Benchmarque – Comparative Benchmarking for .NET

Last night, I announced that the first release of my benchmarking library Benchmarque was available on NuGet. This morning, I’d like to share with you what the library is, and how it to use it.

What is Benchmarque?

Benchmarque (pronounced bench-mar-key) allows you to create comparative benchmarks using .NET. An example of a comparative benchmark would be evaluating two or more approaches to performing an operation, such as whether for(), foreach(), or LINQ is faster at enumerating an array of items. While this example often falls into the over-optimization category, there are many related algorithms that may warrant comparison when cycles matter.

How do I use it?

To understand how to use Benchmarque, let’s work through an example. First, start Visual Studio 2010 Service Pack 1 with NuGet 2.0 installed and create a new class library project using the .NET 4.0 runtime. Once created, we’re going to define an interface for our benchmark.

public interface AppendText
{
    string Append(params string[] args);
}

In this benchmark, we are going to compare the performance of the different ways to append text into a single string. Now that we have the interface defining the behavior we want to benchmark, we need to create a few implementations that perform the operation.

First, the good old concatenation operator.

public class ConcatAppendText :
    AppendText
{
    public string Append(params string[] args)
    {
        string result = string.Empty;

        for (int i = 0; i < args.Length; i++)
        {
            result += args[i];
        }

        return result;
    }
}

Next, we’ll use a StringBuilder to handle the work.

public class StringBuilderAppendText :
    AppendText
{
    public string Append(params string[] args)
    {
        var builder = new StringBuilder();

        for (int i = 0; i < args.Length; i++)
        {
            builder.Append(args[i]);
        }
        return builder.ToString();
    }
}

And last, we’ll try to use string.Join with an empty separator.

public class JoinAppendText :
    AppendText
{
    public string Append(params string[] args)
    {
        return string.Join("", args);
    }
}

With our three implementations ready to benchmark, we now need to create an actual benchmark. We’ll take a list of names, and call the interface with those names. Before we can do that, however, it’s time to add Benchmarque to the project. Using the NuGet package manager, install Benchmarque to your class library project.

Installing from Package Manager

Once installed, we can create our benchmark class as shown below.

public class NameAppendBenchmark :
    Benchmark<AppendText>
{
    static readonly string[] Names = new[]
        {
            "Adam", "Betty", "Charles", "David",
            "Edward", "Frodo", "Gandalf", "Henry",
            "Ida", "John", "King", "Larry", "Morpheus",
            "Neo", "Peter", "Quinn", "Ralphie", "Samwise",
            "Trinity", "Umma", "Vincent", "Wanda"
        };

    public void WarmUp(AppendText instance)
    {
        instance.Append(Names);
    }

    public void Shutdown(AppendText instance)
    {
    }

    public void Run(AppendText instance, int iterationCount)
    {
        for (int i = 0; i < iterationCount; i++)
        {
            string result = instance.Append(Names);
        }
    }

    public IEnumerable<int> Iterations
    {
        get { return new[] {1000, 10000}; }
    }
}

A benchmark includes three methods that involve the execution of the benchmark, along with a property that returns the iteration counts for each run. WarmUp is called with the implementation to allow any one-time initialization of the implementation to be established. This allow should include a few runs through the test to allow the runtime to JIT any code to ensure the benchmark only includes actual execution time (versus assembly load and JIT time). The Run method is then called with each of the iteration counts to actually run the benchmark. Once complete, the Shutdown method is called to dispose of any resources used by the implementation.

The benchmark runner (Benchmarque.Console, which is installed in the tools folder by NuGet) will run the benchmark with each implementation and measure the time taken. To run the benchmark, we need to open the NuGet Package Manager Console, change to the assembly folder, and start the benchmark using Start-Benchmark as shown below.

Open the package manager console

Once open, change to the folder for the assembly to benchmark.

Change to the output folder

And now, we’re going to run the actual benchmark and view the results.

Results of benchmark

First, Start-Benchmark is a Powershell function that is added by the init.ps1 that’s included in the NuGet package. It handles the execution of the benchmark using the console runner. Once complete, the output of the benchmark is displayed in the console window.

As shown above, the results of the test execution are ordered with the fastest implementation first, followed by the remaining implementations with the difference and how many times slower it is displayed. The output is pretty basic at this point, without a lot of other calculations displayed. Additional items may be added as some point. For now, it’s enough to give me the answers I need when trying different approaches to the same problem.

The library is open source (I’ll put the Apache 2 documents in place at some point), so feel free to use, abuse, modify, and enhance as needed! 

One request: If anyone is a Powershell megastar and can modify the Benchmarque.psm1 so that if no argument is specified, it looks through the solution for the projects that are referencing Benchmarque, and automatically running the benchmarks in those assemblies so they don’t have to be specified explicitly.


Related Articles:

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Chris Patterson

Chris is a senior architect for RelayHealth, the connectivity business of the nation's leading healthcare services company. There he is responsible for the architecture and development of applications and services that accelerate care delivery by connecting patients, providers, pharmacies, and financial institutions. Previously, he led the development of a new content delivery platform for TV Guide, enabling the launch of a new entertainment network seen on thousands of cable television systems. In his spare time, Chris is an active open-source developer and a primary contributor to MassTransit, a distributed application framework for .NET. In 2009, he was awarded the Most Valuable Professional award by Microsoft for his technical community contributions.
This entry was posted in .net, c#. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Samuel Langlois

    Seems to be a nice and useful tool. Would it be possible to also benchmark approximate memory usage for each function?

    • http://twitter.com/PhatBoyG Chris Patterson

      I’m sure that could be added, right now I’m only capturing time.

  • http://blog.componentoriented.com D. Lambert

    Interesting — looks like it doesn’t like projects with spaces in the path names, and I’m having trouble getting it to see that there’s actual work to be done — does it look for public classes of type “Benchmark”?

    • http://twitter.com/PhatBoyG Chris Patterson

      Public classes that implement Benchmark, yes – then it will look for implementations of the interface T to run against the benchmark.

  • Anonymous

    Nice tool!  It’d be even better if you had a Visual Studio UI instead of having to run it through package manager console – not the first place I’d think of when having to run diagnostics.  But I guess it’s open source, so whoever wants to add it can :)