.net, Non-functional Requirements, Performance

Measuring your code’s performance during development with BenchmarkDotNet – Part #1: Getting started

I was inspired to write a couple of posts after recently reading articles by Stephen Toub and Andrey Akinshin about BenchmarkDotNet from the .NET Blog, and I wanted to write about how I could use BenchmarkDotNet to understand my own existing codebase a little bit better.

A common programming challenge is how to manage complexity around code performance – a small change might have a large impact on application performance.

I’ve managed this in the past with page-level performance tests (usually written in JMeter) running on my integration server – and it works well.

However, these page-level performance tests only give me coarse grained results – if the outputs of the JMeter tests start showing a slowdown, I’ll have to do more digging in the code to find the problem. At this point, tools like ANTS or dotTrace are really good for finding the bottlenecks – but even with these, I’m reacting to a problem rather than managing it early.

I’d like to have more immediate feedback – I’d like to be able to perform micro-benchmarks against my code before and after I make small changes, and know right away if I’ve made things better or worse. Fortunately BenchmarkDotNet helps with this.

This isn’t premature optimisation – this is about how I can have a deeper understanding of the quality of code I’ve written. Also, if you don’t know if your code is slow or not, how can you argue that any optimisation is premature?

A simple example

Let’s take a simple example – say that I have a .NET Core website which has a single page that just generates random numbers.

Obviously this application wouldn’t be a lot of use – I’m deliberately choosing something conceptually simple so I can focus on the benchmarking aspects.

I’ve created a simple HomeController, which has an action called Index that returns a random number. This random number is generated from a service called RandomNumberGenerator.

Let’s look at the source for this. I’ve put the code for the controller below – this uses .NET Core’s built in dependency injection feature.

using Microsoft.AspNetCore.Mvc;
using Services;
 
namespace SampleFrameworkWebApp.Controllers
{
    public class HomeController : Controller
    {
        private readonly IRandomNumberGenerator _randomNumberGenerator;
        
        public HomeController(IRandomNumberGenerator randomNumberGenerator)
        {
            _randomNumberGenerator = randomNumberGenerator;
        }
 
        public IActionResult Index()
        {
            ViewData["randomNumber"= _randomNumberGenerator.GetRandomNumber();
 
            return View();
        }
    }
}

The code below shows the RandomNumberGenerator – it uses the Random() class from the System library.

using System;
 
namespace Services
{
    public class RandomNumberGenerator : IRandomNumberGenerator
    {
        private static Random random = new Random();
 
        public int GetRandomNumber()
        {
            return random.Next();
        }
    }
}

A challenge to make it “better”

But after a review, let’s say a colleague tells me that the System.Random class isn’t really random – it’s really only pseudo random, certainly not random enough for any kind of cryptographic purpose. If I want to have a really random number, I need to use the RNGCryptoServiceProvider class.

So I’m keen to make my code “better” – or at least make the output more cryptographically secure – but I’m nervous that this new class is going to make my RandomNumberGenerator class slower for my users. How can I measure the before and after performance without recording a JMeter test?

Using BenchmarkDotNet

With BenchmarkDotNet, I can just decorate the method being examined using the [Benchmark] attribute, and use this to measure the performance of my code as it is at the moment.

To make this attribute available in my Service project, I need to include a nuget package in my project, and you can use the code below at the Package Manager Console:

Install-Package BenchmarkDotNet

The code for the RandomNumberGenerator class now looks like the code below – as you can see, it’s not changed much at all – just an extra library reference at the top, and a single attribute decorating the method I want to test.

using System;
using BenchmarkDotNet.Attributes;
 
namespace Services
{
    public class RandomNumberGenerator : IRandomNumberGenerator
    {
        private static Random random = new Random();
 
        [Benchmark]
        public int GetRandomNumber()
        {
            return random.Next();
        }
    }
}

I like to keep my performance benchmarking code in a separate project (in the same way that I keep my unit tests in a separate project). That project is a simple console application, with a main class that looks like the code below (obviously I need to install the BenchmarkDotNet nuget package in this project as well):

using BenchmarkDotNet.Running;
using Services;
 
namespace PerformanceRunner
{
    class Program
    {
        static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<RandomNumberGenerator>();
        }
    }
}

And now if I run this console application at a command line, BenchmarkDotNet presents me with some experiment results like the ones below.

// * Summary *

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-2640M CPU 2.80GHz (Sandy Bridge), ProcessorCount=4
Frequency=2728183 Hz, Resolution=366.5443 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006127
 [Host] : .NET Core 4.6.25316.03, 64bit RyuJIT
 DefaultJob : .NET Core 4.6.25316.03, 64bit RyuJIT


          Method | Mean     | Error     | StdDev    |
---------------- |---------:|----------:|----------:|
 GetRandomNumber | 10.41 ns | 0.0468 ns | 0.0365 ns |

As you can see above, my machine specifications are listed, and the experiment results suggest that my RandomNumberGenerator class presently takes about 10.41 nanoseconds to generate a random number.

So now I have a baseline – after I change my code to use the more cryptographically secure RNGCryptoServiceProvider, I’ll be able to run this test again and see if I’ve made it faster or slower.

How fast is the service after the code changes?

I’ve changed the service to use the RNGCryptoServiceProvider – the code is below.

using System;
using BenchmarkDotNet.Attributes;
using System.Security.Cryptography;
 
namespace Services
{
    public class RandomNumberGenerator : IRandomNumberGenerator
    {
        private static Random random = new Random();
 
        [Benchmark]
        public int GetRandomNumber()
        {
            using (var randomNumberProvider = new RNGCryptoServiceProvider())
            {
                byte[] randomBytes = new byte[sizeof(Int32)];
 
                randomNumberProvider.GetBytes(randomBytes);
 
                return BitConverter.ToInt32(randomBytes, 0);
            }
        }
    }
}

And now, when I run the same performance test at the console, I get the results below. The code has become slower, and now takes 154.4 nanoseconds instead of 10.41 nanoseconds.

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-2640M CPU 2.80GHz (Sandy Bridge), ProcessorCount=4
Frequency=2728183 Hz, Resolution=366.5443 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006127
 [Host] : .NET Core 4.6.25316.03, 64bit RyuJIT
 DefaultJob : .NET Core 4.6.25316.03, 64bit RyuJIT


          Method | Mean     | Error    | StdDev   |
---------------- |---------:|---------:|---------:|
 GetRandomNumber | 154.4 ns | 2.598 ns | 2.028 ns |

So it’s more functionally correct, and unfortunately it has become a little slower. But I can now go to my technical architect with a proposal to change the code, and present a more complete picture – they’ll be able to not only understand why my proposed code is more cryptographically secure, but also I’ll be able to show some solid metrics around the performance deterioration cost. With this data, they have can make better decisions about what mitigations they might want to put in place.

How should I use these numbers?

A slow down from about 10 to 150 nanoseconds doesn’t mean that the user’s experience deteriorates by a factor of 15 – remember that in this case, a single user’s experience is over the entire lifecycle of the page, so really a single user should only see a slowdown of 140 nanoseconds over the time it takes to refresh the whole page. Obviously a website will have many more users than just one at a time, and this is where our JMeter tests will be able to tell us more accurately how the page performance deteriorates at scales of hundreds or thousands of users.

Wrapping up

BenchmarkDotNet is a great open-source tool (sponsored by the .NET Foundation) that allows us to perform micro-benchmarking experiments on methods in our code. Check out more of the documentation here.

I’ve chosen to demonstrate BenchmarkDotNet with a very small service that has methods which take no parameters. The chances are that your code is more complex than this example, and you can structure your code to so that you can pass parameters to BenchmarkDotNet – I’ll write more about these more complicated scenarios in the next post.

Where I think BenchmarkDotNet is most valuable is that it changes the discussion in development teams around performance. Rather than changing code and hoping for the best – or worse, reacting to an unexpected performance drop affecting users – micro-benchmarking is part of the development process, and helps developers understand and mitigate code problems before they’re even pushed to an integration server.


About me: I regularly post about .NET – if you’re interested, please follow me on Twitter, or have a look at my previous posts here. Thanks!