Performance Counters-MultiInstance/MultiCategory

Introduction

Performance counters in Azure have been a talked about topic all over the web. However most of the articles give a very basic sample and do not talk much about the issues that one faces. In the article below, I have tried to highlight some issues that I faced and provided its work around.

So here is how my application looks like:

An azure application is created with Web (1 instance) and Worker (2 instances) role and uses Windows Azure Storage. To check performance , Diagnostics have been enabled in worker roles and custom counters as part of ‘multinstance ‘ category , created using a Powershell Scripts via the startup task running with elevated privilege. These counters are then incremented within the application code, also running with elevated privilege.

The powershell script creates an AverageTimer32 and AverageBase counter pair. These counters (say for example ‘ResponseProcessTime’ and ‘ResponseProcessTimeBase’ are the custom performance counter names respectively ) help us find out the time taken to complete the execution of a method.

Issues Faced

1. The custom performance counters although are seen on windows machine (visible using perfmon and server explorer'machinename'performance counters)  are not being transferred to WADPerformanceCounters Table, neither on local emulator nor on Cloud storage.Only occasionally do we see the counters being saved in WADPerformaceCountersTable. Sometimes the counters only appear in the WADPerformanceCountersTable from a single instance. Without these counters being visible we will not be able to calculate performance. The non-custom counters Committed Bytes and % Processor Time are always visible in the WADPerformanceCountersTable.
2. An error encountered in emulator and seen in WADDiagnosticInfrastructureLogTable.

The Error details:/p>

PdhGetFormattedCounterArray(\MSBAdapter(deployment18(283).
   MyAdapter.MyAdapter.Worker_IN_0)\ResponseProcessTime) failed pdhStatus=c0000bc6; retry 2
PdhGetFormattedCounterArray(\MSBAdapter(deployment18(283).MyAdapter.
   MyAdapter.Worker_IN_0)\ResponseProcessTimeBase) failed pdhStatus=800007d5; retry 2

3. The AverageTimer32 counter doesn't always behave as expected. Sometimes, when the counter values are successfully transferred into the WADPerformanceCounterTable, the  value of the counter is zero.

The remaining of the article will speak about how these issues were solved

Background

Instead of going into details about performance ounter orgion and its type this article will focus on the issues and it resolution. For learning about performance counters ,please refer –

• Windows Azure PerformanceCounters Sample http://code.msdn.microsoft.com/Windows-Azure-PerformanceCo-7d80ebf9
• Performance Data Helper Error Codes http://msdn.microsoft.com/en-us/library/windows/desktop/aa373046(v=vs.85).aspx
• PerformanceCounterType Enumeration http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecountertype.aspx
• Using performance counters in Windows Azure http://www.windowsazure.com/en-us/develop/net/common-tasks/performance-profiling/#step2

Using the code

The above stated issues were found out to be known limitation with Windows regarding the ability for a process to retrieve the instance counters created by another process after the start of the first. This indicates that it is changing the race condition between MonAgent (MA) start and the counter instance creation because we are using multiple worker instances.

"The best workaround is likely to create the instance counters in the startup task as ‘Singleinstance’, however, since instance counter need to be created at arbitrary times during the lifetime of the role it will be necessary to reset the WAD configuration programmatically as below, while creating the new counter instance" , so instead of changing the  Performance counter Category to single instance from Multiisatnce, we continue to create the perf counter category as 'multinsatnce' via the powershell in the startup ,but instead of creating counters at arbitrary times at different levels we will create a class with public properties for each counter that we want to maintain and create them in the constructor. Then access all counters through those properties where actually needed. A single instance can be injected using Ninject.

The code is as follows :

1. The interface/class that creates the counters:
C#

public interface IDiagnosticHelper 
{ 
    PerformanceCounter ResponseProcessTime { get; } 
    PerformanceCounter ResponseProcessTimeBase { get; }     
} 

public class DiagnosticHelper : IDiagnosticHelper 
{ 
    public static string defaultPerformanceCounterCategory = "MyAdapter"; 
    public static string defaultPerformanceCounterInstance = RoleEnvironment.CurrentRoleInstance.Id; 
    
    public PerformanceCounter ResponseProcessTime { get; private set; } 

    public PerformanceCounter ResponseProcessTimeBase { get; private set; } 
   
    public DiagnosticHelper() 
    { 
       ResponseProcessTime = new PerformanceCounter(defaultPerformanceCounterCategory,  
          "ResponseProcessTime ", defaultPerformanceCounterInstance, false); 
       ResponseProcessTimeBase = new PerformanceCounter(defaultPerformanceCounterCategory, 
          "ResponseProcessTimeBase", defaultPerformanceCounterInstance, false); 
                   
        Trace.TraceWarning("DiagnosticHelper Created performance counters."); 
    } 
}

2. We use Ninject for dependecy injection. In our Ninject Module, we specify the singleton scope( in order to avoid race conditions created beacuse of running 2 instances of our worker roles)while binding as follows:
C#

Bind<IDiagnosticHelper>().To<DiagnosticHelper>().InSingletonScope();

3. In the workerrole we ninject this Interface in the Onstart and Run so that these counters are created  as follows:
C#

ninjectKernel.Get<IDiagnosticHelper>();

4. Now in whichever class/handler we want to increment the counters we will not explicitly create the counters again but just call the diagnostic helper:
C#

public class MyHandler 
{ 
    private readonly IDiagnosticHelper diagnosticHelper; 
    public MyHandler(IDiagnosticHelper diagnosticHelper) 
    { 
        this.diagnosticHelper = diagnosticHelper; 
    } 

    void CallMe() 
    { 
        var watch = new Stopwatch(); 
        watch.Start(); 
        int i = 3; 
        int j = 7; 
        int h = i + j; 
        Thread.Sleep(5000); 
        watch.Stop(); 

        if (PerformanceCounterCategory.Exists("MyAdapter")) 
        { 
            if (null != diagnosticHelper.GetFacilityTime) 
            { 
                diagnosticHelper.GetFacilityTime.IncrementBy(watch.ElapsedTicks); 
                diagnosticHelper.GetFacilityTimeBase.Increment(); 
            } 
        } 
    } 
}

This piece of code helped me to solve the pdhStatus=c0000bc6 error (which means The data is not valid) as the now the race condition is not a problem any more. 

Also the error pdhStatus=800007d5(No data to return) was occurring because initially I was adding the base Counter ‘ResponseProcessTimeBase’ to the PerformanceCounters.DataSources ehich was not needed as it’s an average base counter jut used to supplement the AverageTimer32 counter and is itself not required to be added to the datasource. 

Points of Interest

1. Good to choose single instance performance category if you do not explicitly need a multi instance 
2. Cerebretta's tool Azure Diagnotic Manager was useful in viewing the PerfCounters in graphical  format 

History

• Updated the code formatting.