.net, Non-functional Requirements, Performance

Measuring your code’s performance during development with BenchmarkDotNet – Part #2: Methods with parameters

Last time, I wrote about how to use BenchmarkDotNet (Github here: NuGet: here) to measure code performance for a very simple method with no parameters. This time I’ll write about testing another scenario that I find is more common – methods with parameters.

Let’s start with a simple case – primitive parameters.

Methods with Primitive Parameters

Let’s write a method which takes an integer parameter and calculates the square.

I know that I could use the static System.Math.Pow(int a, int b) for this instead of rolling my own method – but more on this later.

I written a little static method like this.

public class MathFunctions
{
    public static long Square(int number)
    {
        return number * number;
    }
}

Nothing wrong with that – but not that easy to test with BenchmarkDotNet and decorate with a simple [Benchmark] attribute because I need to specify the number parameter.

There are a couple of ways to test this.

Refactor and use the Params attribute

Instead of passing the number as a parameter to the Square method, I can refactor the code so that Number is a property of the class, and the Square method uses this property.

public class MathFunctions
{
    public int Number { getset; }
 
    public long Square()
    {
        return this.Number * this.Number;
    }
}

Now I can decorate Square method with the [Benchmark] attribute, and I can use the ParamsAttribute in BenchmarkDotNet to decorate the property with numbers that I want to test.

public class MathFunctions
{
    [Params(12)]
    public int Number { getset; }
        
    [Benchmark]
    public int Square()
    {
        return this.Number * this.Number;
    }
}

And then it’s very simple to execute a performance runner class like the code below:

using BenchmarkDotNet.Running;
using Services;
 
namespace PerformanceRunner
{
    class Program
    {
        static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<MathFunctions>();
        }
    }
}

Which yields the results:

// * Summary *

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-2640M CPU 2.80GHz (Sandy Bridge), ProcessorCount=4
Frequency=2728178 Hz, Resolution=366.5450 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006127
 [Host] : .NET Core 4.6.25316.03, 64bit RyuJIT
 DefaultJob : .NET Core 4.6.25316.03, 64bit RyuJIT


 Method | Number | Mean      | Error     | StdDev    | Median    |
------- |------- |----------:|----------:|----------:|----------:|
 Square | 1      | 0.0429 ns | 0.0370 ns | 0.0658 ns | 0.0001 ns |
 Square | 2      | 0.0035 ns | 0.0086 ns | 0.0072 ns | 0.0000 ns |

This mechanism has the advantage that you can specify a range of parameters and observe the behaviour for each of the values.

But I think it has a few disadvantages:

  • I’m a bit limited in the type of parameter that I can specify in an attribute. Primitives like integers and strings are easy, but instantiating a more complex data transfer object is harder.
  • I have to refactor my code to measure performance – you could argue that the refactored version is better code, but to me the code below is simple and has a clear intent:
var output = MathFunctions.Square(10);

Whereas I think the code below is more obtuse.

var math = new MathFunctions { Number = 10 };
var output = math.Square();
  • My source code has a tight dependency on the BenchmarkDotNet library, and the attributes add a little litter to the class.

Basically I’m not sure I’ve made my code better by refactoring it to measure performance. Let’s look at other techniques.

Separate performance measurement code into a specific test class

I can avoid some of the disadvantages of the technique above by creating a dedicated class to measure the performance of my method, as shown below.

public class MathFunctions
{
    public static long Square(int number)
    {
        return number * number;
    }
}
 
public class PerformanceTestMathFunctions
{
    [Params(12)]
    public int Number { getset; }
 
    [Benchmark]
    public long Measure_Speed_of_Square_Function()
    {
        return MathFunctions.Square(Number);
    }
}

So now I can run the code below to measure the performance of my method.

using BenchmarkDotNet.Running;
using Services;
 
namespace PerformanceRunner
{
    class Program
    {
        static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<PerformanceTestMathFunctions>();
        }
    }
}

This time I’ve not had to refactor my original code, and I’ve moved the dependency from my source code under test to the dedicated test class. But I’m still a bit limited in what types of parameter I can supply to my test class.

Using GlobalSetup for methods with non-primitive data transfer object parameters

Let’s try benchmarking an example which is a bit more involved – how to measure the performance of some more math functions I’ve written which use Complex Numbers.

Complex numbers are nothing to do with BenchmarkDotNet – I’m just using this as an example of a non-trivial problem space and how to run benchmark tests against it.

At school you might have done some work with Complex Numbers. These numbers have a real and imaginary component – which sounds weird if you’re not used to it, but they can be represented as:

1 + 2i

Where 1 is the real component, and 2 is the size of the ‘imaginary’ component.

If you want to calculate the magnitude of a complex number, you just use Pythagorean maths – namely:

  • Calculate the square of the real component, and the square of the imaginary component.
  • Add these two squares together.
  • The magnitude is the square root of the sum of the two squares.

So I can represent a Complex Number in code in the object class shown below:

public class ComplexNumber
{
    public int Real { getset; }
 
    public int Imaginary { getset; }
}

And I can instantiate a complex number 1 + 2i with the code:

new ComplexNumber { Real = 1, Imaginary = 2 };

If I want to calculate the magnitude of this Complex Number, I can pass the ComplexNumber data transfer object as a parameter to a method shown below.

public class ComplexMathFunctions
{
    public static double Magnitude(ComplexNumber complexNumber)
    {
        return Math.Pow(Math.Pow(complexNumber.Real, 2                        + Math.Pow(complexNumber.Imaginary, 2), 0.5);
    }
}

But how do I benchmark this?

I can’t instantiate a ComplexNumber parameter in the Params attribute supplied by BenchmarkDotNet.

Fortunately there’s a GlobalSetup attribute – this is very similar to the Setup attribute used by some unit test frameworks, were we can arrange our parameters before they are used by a test.

The code below shows how to create a dedicated test class, and instantiate a Complex Number in the GlobalSetup method which is used in the method being benchmarked.

public class PerformanceTestComplexMathFunctions
{
    private ComplexNumber ComplexNumber;
 
    [GlobalSetup]
    public void GlobalSetup()
    {
        this.ComplexNumber = new ComplexNumber { Real = 1, Imaginary = 2 };
    }
 
    [Benchmark]
    public double Measure_Magnitude_of_ComplexNumber_Function()
    {
        return ComplexMathFunctions.Magnitude(ComplexNumber);
    }
}

This yields the results below:

// * Summary *

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-2640M CPU 2.80GHz (Sandy Bridge), ProcessorCount=4
Frequency=2728178 Hz, Resolution=366.5450 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006127
 [Host] : .NET Core 4.6.25316.03, 64bit RyuJIT
 DefaultJob : .NET Core 4.6.25316.03, 64bit RyuJIT


 Method                                      | Mean     | Error    | StdDev    |
-------------------------------------------- |---------:|---------:|----------:|
 Measure_Magnitude_of_ComplexNumber_Function | 110.5 ns | 1.058 ns | 0.9897 ns |

I think this eliminates pretty much all the disadvantages I listed earlier, but does add a restriction that I’m only testing one instantiated value of the data transfer object parameter.

You might wonder why we need the GlobalSetup at all when we could just instantiate a local variable in the method under test – I don’t think we should do that because we’d also be including the time taken to set up the experiment in the method being benchmarked – which reduces the accuracy of the measurement.

Addendum

I was kind of taken aback by how slow my Magnitude function was, so I started playing with some different options – instead of using the built in System.Math.Pow static method, I decide to calculate a square by just multiplying the base by itself. I also decided to use the System.Math.Sqrt function to calculate the square root, rather than the equivalent of raising the base to the power of 0.5. My refactored code is shown in the code below.

public class ComplexMathFunctions
{
    public static double Magnitude(ComplexNumber complexNumber)
    {
        return Math.Sqrt(complexNumber.Real * complexNumber.Real 
                    + complexNumber.Imaginary * complexNumber.Imaginary);
    }
}

Re-running the test yielded the benchmark results below:

// * Summary *

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-2640M CPU 2.80GHz (Sandy Bridge), ProcessorCount=4
Frequency=2728178 Hz, Resolution=366.5450 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006127
 [Host] : .NET Core 4.6.25316.03, 64bit RyuJIT
 DefaultJob : .NET Core 4.6.25316.03, 64bit RyuJIT


 Method                                      | Mean     | Error     | StdDev    |
-------------------------------------------- |---------:|----------:|----------:|
 Measure_Magnitude_of_ComplexNumber_Function | 4.192 ns | 0.0371 ns | 0.0347 ns |

So with a minor code tweak, the time taken to calculate the magnitude dropped from 110.5 nanoseconds to 4.192 nanoseconds. That’s a pretty big performance improvement. If I hadn’t been measuring this, I’d probably never have known that I could have improved my original implementation so much.

Of course, this performance improvement might only work for small integers – it could be that large integers have a different performance profile. But it’s easy to understand how we could set up some other tests to check this.

Wrapping up

This time I’ve written about how to use BenchmarkDotNet to measure the performance of methods which have parameters, even ones that are data transfer objects. The Params attribute can be useful sometimes for methods which have simple primitive parameters, and the GlobalSetup attribute can specify a method which sets up more complicated scenarios. I’ve also shown how we can create classes dedicated to testing individual methods, and keep benchmarking test references isolated in their own classes and projects.

This makes it really simple to benchmark your existing codebase, even code which wasn’t originally designed with performance testing in mind. I think it’s worth doing – even while writing this post, I unexpectedly discovered a simple way to change my example code that made a big performance improvement.

I hope you find this post useful in starting to measure the performance of your codebase. If you want to dig into understanding BenchmarkDotNet more, I highly recommend this post from Andrey Akinshin – it goes into lots more detail.


About me: I regularly post about .NET – if you’re interested, please follow me on Twitter, or have a look at my previous posts here. Thanks!