Captain Kerk: Technology Solutions

Improve Sitecore 10 Performance in Azure PaaS (Part 2)

Edwen Kerk's photo
Edwen Kerk
·

4 min read

In previous article, the performance tuning are mostly on Sitecore instance. In this article, the issue are mostly related to Azure Web App Service (PaaS) and especially on optimizing the Disk I/O

Before starting the performance tuning, let’s have a basic understanding of how Azure PaaS scaling works.

download.png

Above diagram show the high level of Azure PaaS scaling. The scaled sites are running on different cloud machines. So, the worker process is separate but file system is shared across all the worker processes/instances. That’s why Sitecore creates a folder for each machine name in App_Data/Logs folder.

Tips here to improve and optimize the Disk I/O

1. Enable Azure "Local Cache" feature

The Local Cache feature is a Microsoft Azure App Service feature. Enable it, will copy the shared wwwroot to the D:\home\wwwroot folder then the site content is read from the local virtual machine instance. This improves the application performance and instead of retrieving it from the shared file system.

The local cache size is limited to 300 MB by default but you can increase it up to 2 GB. If the copied files exceed the size of the local cache, App Service silently ignores local cache and reads from the remote file share. Therefore, reducing the quantity and size of files on the disk for the Sitecore implementation can be a part of this solution.

NOTE: Enable Local Cache ONLY on CD

  1. Before you enable the Local Cache feature, please cleanup the application logs and make sure application file size within 2GB

  2. Login to Azure Portal, go to CD App Service.

  3. Go to Configuration under Settings section.

  4. Add the following Application Settings: image.png WEBSITE_LOCAL_CACHE_OPTION: Set the Value to Always WEBSITE_LOCAL_CACHE_SIZEINMB : Set the Value to 2000

  5. Save the Configuration

  6. Restart the App Service

Important Note: If you using Deployment Slots to do deployment, DO NOT enable Local Cache on Staging Slots, only on Production.

Reference: https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0909298

2. Disable Azure fallback appender.

Once Local Cache is enabled, the log file will never write to the shared file system. It become redundant. The easiest way to do disable this is to open the \App_Config\Sitecore\ExperienceContentManagement.Administration \Sitecore.ExperienceContentManagement.Administration.config file and remove the logDisabled4net node.

Best practice is to write a patch file to remove it :)

3. Implement LogFilter to Reduce the logs

Sitecore does log a lot of INFO that probably should have been marked as debug information. This is fine when logging to a log file. But with Application Insights there are issues with logging too much information which will cause you pay additional when exceed the quota and of cos more important is the performance!

Refer to the following post, I follow the hack to reduce 90% of log does help improve the performance. 

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:useApplicationInsights="http://www.sitecore.net/xmlconfig/useApplicationInsights/" xmlns:role="http://www.sitecore.net/xmlconfig/role/">
  <sitecore>
    <sc.variable name="logFilterRegExPattern" value="^.*(Job started|Job ended|document not found page|HttpModule is being initialized|Cache created).*$" />
    <log4net role:require="ContentDelivery">

      <!--Add new with filter by Performance Patch file-->

      <!--Log4Net appender - Logs in AI-->
      <appender name="LogFileAppender">
        <!-- Added by Performance Patch file -->
        <filter type="log4net.Filter.StringMatchFilter">
          <regexToMatch value="$(logFilterRegExPattern)" />
          <acceptOnMatch value="false" />
        </filter>
      </appender>
    </log4net>
  </sitecore>
</configuration>

Alternatively, you can reduce the total number of operations on the disk by assigning the log level. The log level (Priority) limits the messages that are written to the log. For example, on a production server you might only be interested in errors: <priority value="ERROR" />

References:

  1. https://briancaos.wordpress.com/2019/12/05/sitecore-and-application-insights-how-to-remove-90-of-all-log-lines-without-sacrificing-traceability/
  2. https://www.sitecorefundamentals.com/how-to-reduce-one-or-many-sitecore-log-entries-from-application-insights

4. Enable Auto Scaling (Scale Out)

The most important topic. As a good practice for HA setup, we will always enable auto scaling, depending on CPU/Memory load to increase number of instances. Please make sure the App Service tier is on Premium App Service plan in Azure.

The initial setup of our infra was using Isolated App Service Plan within ASE environment. However, the App Service scale out time in ASE is taking 20-30mins. There is no official documentation from Azure but we have tested in at least on Southeast Asia region. Logged a support issue to MS team to confirm the behaviour as well.

Finally, we have moved our App Service tier to Premium App Service Plan and the scale out take less than a min time (not include Sitecore start up)

NOTE: The Premium service plan is designed to provide enhanced performance for production apps. The upgraded Premium plan, Premium v2, features Dv2-series VMs with faster processors, SSD storage, and double memory-to-core ratio compared to Standard.

5. Enable Health check

As we enabled auto scaling, the App Service plan should be scaled to two or more instances. The Health check increases the application's availability by re-routing requests away from unhealthy instances, and replacing instances if they remain unhealthy.

To Enable Health Check

  1. Go to Monitoring > Health Check under your App Service
  2. Enable
  3. Path: /sitecore/service/keepalive.aspx
  4. Save

Conclusion

With all the performance turning, we managed to perform stress test with 1200 concurrent users with less than 1 second response rate. (on App Service Premium Plan P3V2 x2)

For more information about performance tuning on Sitecore PaaS, please check the official documentation from Sitecore https://doc.sitecore.com/en/developers/100/managed-cloud/recommendations--sitecore-tuning-in-azure.html

Azure.NET