Measuring Performance

With all this talk about how performance affects your applications, I thought it might be worth taking a step back to look at how to measure performance.

Considerations

There are a few considerations to keep in mind when measuring performance. Some are more global, and others are specific to C#.

• System: It may be obvious, but the speed of your computer affects performance. You can't compare your run-times against times run on a different machine. I'd recommend you only compare the numbers relative to each other.
• Frequency Scaling: A lot of modern CPUs, especially laptops, have frequency scaling. If you can, force this to be disabled before testing code. If not, realize that your CPU could start out slower if it hasn't scaled to a higher frequency yet.
• Compilation: Watch if you're compiling in Debug mode or Release. Release will have optimizations and inlining enabled by default. Also keep in mind whether you are using checked or unchecked arithmatic (the latter being faster, since you're saving bound check operations).
• Architecture: If you're set to compile in Win32 mode, and are running on a 64bit system, you can usually count on this being slow. I recommend setting it to AnyCPU.
• Debugger: Is the debugger attached? If so, you'll have a ton of overhead for the debugger. This also prevents the code from optimizing in any way.
• Version: The version of your .NET runtime will have a huge impact on performance. Different JITers perform differently; mono has llvm performance support now, etc. So it's worth noting this with your metrics.
• Other Apps: If you have a game running, or are watching a video, chances are you're already using a significant amount of CPU processing cycles. This will impact your test and you should eliminate as many external applications as you can.

JIT

For any bytecode language, it's important to understand how JITing works when you're testing performance.

In C#, compiled with optimizations enabled, it works with two passes:

1. JIT the bytecode to be machine code
2. Optimize the machine code

This means the first time a section of code is run, it is JIT'd, and the second time, it is optimized.

Code can be optimized in many ways, including inlining, unwinding, branching, etc. If you're interested in what happens, you'll have to look at the IL, but I won't get into that.

Measuring Time

There are three main ways to measure time in C# (Ignoring P/Invoke): DateTime.Now, DateTime.UTCNow, and StopWatch.

DateTime is not terribly useful, since it only has a resolution of 40ms, which, when it comes to measuring the performance of something, is ages. Also, while DateTime.UTCNow has decent performance, DateTime.Now is slow since it will resolve your timezone with each call.

So, for those reasons, I choose to use StopWatch. It has a much higher resolution, using QueryPerformanceCounter on Windows. It's also just handy for measuring time, as it's specifically suited for this case.

To gain accuracy, we'll run a small snippet of code several million times, and then examine the data.

Even with all these samples, I usually like to run the test several times to make sure it's not affected by outside factors, such as other processes running on the system.

Practical Use

Now, let's talk practicality. Let's say I want to examine the overhead of calling a virtual method compared to that of calling a normal method.

Here's the code I wrote:

using System;
using System.Linq;
using System.Diagnostics;

namespace test
{
    public class ClassControl
    {
        public void Test()
        {}
    }

    public class ClassBase
    {
        public virtual void Test()
        {}
    }

    public class ClassOverride : ClassBase
    {
        public override void Test ()
        {

        }
    }

    static class MainClass
    {
        public static void Main (string[] args)
        {
            const int ITERATIONS = 100000000;
            var sw = new Stopwatch ();

            //Test Control
            var control = new ClassControl ();
            sw.Start ();
            for (int i=0; i<ITERATIONS; ++i)
            {
                control.Test ();
            }
            sw.Stop ();
            Console.WriteLine ("Control took {0}ms, {1}ms/each", sw.ElapsedMilliseconds, sw.ElapsedMilliseconds / (double)ITERATIONS);
            sw.Reset ();

            //Test virt
            var virt = new ClassOverride ();
            sw.Start ();
            for (int i=0; i<ITERATIONS; ++i)
            {
                virt.Test ();
            }
            sw.Stop ();
            Console.WriteLine ("Virtual took {0}ms, {1}ms/each", sw.ElapsedMilliseconds, sw.ElapsedMilliseconds / (double)ITERATIONS);
            sw.Reset ();
        }
    }
}

On my system, Linux Mint, running Mono 3.2.8, on 64 bit:

Control took 43ms, 4.3E-07ms/each
Virtual took 213ms, 2.13E-06ms/each

It's important to note that we're performance testing everything between sw.Start and sw.Stop. This means the for-loop itself, and anything else we have in there. That's why I try to simplify it at much as possible. Or, when I can't simplify it, I standardize it across all tests.

We can see from the results that a non-virtual method invokes many more times quicker than a virtual one. You could test static methods too, and find it quicker than them both. Keep in mind, this is over the course of 100,000,000 iterations.

Conclusion

In the scope of performance, even little things can matter. As I spoke about in my previous entry, it's important to know what you're facing. In one of my applications, it may not seem like a big deal, but there's a single method called across multiple threads, on the order of millions of times per second. Knowing not to make it virtual, and optimize entry to the method in this tight loop is very important.