UCMA audio conferencing API is widely used by middle tier applications, especially by contact center applications. It allows applications to implement several audio related services (e.g. hold music, call recording, silent monitoring, coaching) in Microsoft Lync Server environment.
Before starting to implement an application based on Lync Server audio conferencing services you need to be aware of the following 2 measures:
- Average conferencing delay: the time typically required to setup an audio conference
- Maximum conference request rate: the maximum number of conferences can be setup in a given timeframe (e.g. in a minute)
These are the 2 most important measures you need to know in order to make an informed decision about whether conferencing is a suitable building block for your application or not. The first measure is quite important because it represents the delay each caller typically experiences. The latter one is also important because it determines whether you can satisfy your application level quality/performance requirements. Simply inferring the maximum conference request rate from the average conferencing delay is impossible since UCMA is asynchronous.
The numerical values you can find below represent average values calculated over an extensive data set collected from several Lync Server 2013 Standard Edition production environment with the cumulative patch “January 2014” applied. Conference requests are generated by a single application thread and requests are randomly/uniformly distributed in the given time interval. The UCMA application performed the following 3 steps for each conference:
- Schedule a conference (UCMA methods: BeginScheduleConference/EndScheduleConference)
- Join the conference (UCMA methods: BeginJoin/EndJoin)
- Dial to the conference (UCMA methods: BeginEstablish/EndEstablish)
This means that the average conferencing delay includes the time required to schedule the conference, join the conference and dial to the conference. The reason why these 3 steps are performed for each conference is very simple: because it has practical relevance. A typical middle tier UCMA application not only schedules conference but also wants to join the conference and dial to the conference.
The following chart shows the average conferencing delay (blue bars) measured at different conference request rates. As the value belonging to conference request rate 1/minute indicates, the average conferencing delay is typically 1.3 seconds. This value is quite acceptable. In case of contact center applications, 1.3 seconds delay is almost imperceptible by a caller who dials customer service.
As the bar chart shows, the average conference setup time remains almost constant until the conference request rate increases to 50 requests per minute. At 50 requests/minute rate, setting up a conference takes 6.9 seconds which represents a borderline case for contact centers. Even in this case, Lync Server performs each conference request issued (red line = 100%). However, if the conference request rate increases further then the average conference setup time becomes more than 1 minute and the Lync Server starts rejecting conference requests. At 60 requests/minutes Lync server performs only 98% of the conference requests issued; UCMA application receives ConferenceFailureException for the remaining 2% of the requests. Increasing the conference request rate beyond 60 requests/minutes is meaningless. It just results in longer and longer average conferencing delay and receiving more and more ConferenceFailureException.
Why does this happen?
According to my observations, the followings might be collectively responsible for the low limit on conferencing rate:
- Using SQL Server Express Edition: Lync Server uses local instances of SQL Server Express Edition as the database for conferences. SQL Server Express Edition has memory and computing restrictions (e.g. maximum 1 GB memory);
- High CPU utilization: High SQL Server (RTCLOCAL instance) CPU utilization if conference request rate is above 50 requests/minute. This CPU utilization issue becomes even worse if monitoring role is also installed on the same Lync Standard Edition Server box with its own SQL Server instance hosting the LcsCDR and QoEMetrics databases (since these databases also hold conference records);
- Database deadlocks: Database deadlock reports ([rtc].[ConfExpireConference] <=> [rtc].[ExppConference]) start appearing in RTCLOCAL SQL Server instance log file if conference request rate increases to 50 requests/minute. This indicates poorly designed concurrency in terms of database locks;
In case of Lync Standard Edition Server, I think the local database server is the real bottleneck and it is responsible for the poor conferencing performance.
What do these values mean for contact center applications?
For inbound contact center applications these values mean the following: assuming 2.5 minutes as the sum of the average talk duration and after-call-work time, the above mentioned 50 requests/minute rate allows up to 50*2.5=125 agents only. If you want your contact center application to support more than 125 agents then you need to leave out conferencing or ask for Microsoft Lync Server Enterprise Edition. In case of Microsoft Lync Server 2013 Enterprise Edition the limit on conferencing rate is beyond the above mentioned 50 requests/minute (more details later …). This is mainly because Lync Server Enterprise Edition uses dedicated, standalone SQL Server Standard or Enterprise Edition as the backend database server. Thus significant SQL workload is moved away from the Front End Servers to the dedicated backend database server. Moreover, remaining SQL workload on local SQL Express instances (e.g. RTCLOCAL) can be decreased by adding more Front End Servers to the Enterprise Edition pool.
Regarding outbound contact centers, - where typically hundreds or thousands of agents are working – you always need to have Microsoft Lync Server 2013 Enterprise Edition. But please perform load tests in advance because you can easily face conferencing performance issues even on Enterprise Edition Pools.
My advice is the following: always ask for Lync Server Enterprise Edition if your application heavily uses audio conferencing and it is used by more than 100 users. In case of contact center application, you also need to ask for Lync Server Enterprise Edition if you expect short average talking time + after-call-work time.
If you face performance issues even on Enterprise Edition pools then there is no any other solution just to redesign your application and stop using conferencing whenever, wherever is possible.