The problem really exacerbates itself when too much was happening at once, and the need to break things out to improve throughput requires a different protocol for integration.

Consider going to the grocery store. When you place an item in your basket, it’s inherently a synchronous operation. Either the jar of pickles makes it into your cart, or it doesn’t, there is no middle ground there. If you drop the jar of pickles onto the ground, you don’t wait for some async process to come by and let you know by email, “sorry, it seems there was a problem with your request to add the item to your shopping cart.”

Blending async in

Consider another system that requires to you to register with the site, using a local database for “who is registered” and a web service for downstream email communication:

In this system, we block the registration of the user against both the local database and the 3rd party web service. We need to check the local database to make sure the username/email is not already used, but what about that web service? What happens when that web service is slow, or unavailable?

We’ve coupled the availability of our system with a system we don’t own, which is potentially disastrous. In the real-world system that the above diagram is based on, when that 3rd party system goes into scheduled maintenance, the web application is brought down for maintenance too! Not a good user experience by any means. Are we really going to turn users away based on the availability of a downstream service?

But there’s another way – shifting the protocol of how this system is built. Let’s make sure what needs to be synchronous is, what what doesn’t need to be, isn’t. Instead of coupling the web service and local database calls together, let’s separate the two out with messaging:

Instead of performing the web service call in the same thread of the UI request, we instead send an asynchronous command as a message to perform the downstream operation. We’ve now decoupled the operation of calling the web service (what doesn’t need to be synchronous) with what does need (registering the user). The downstream operations of registering the user in the 3rd party email provider doesn’t need to happen at the same time.

To put it another way – does it affect the user’s registration if the email service provider rejects the message? No! Instead, that’s likely an administrative operation to figure out what happened, but the user is still successfully registered.

Defining the boundaries

When it comes time to figure out what should be synchronous and what shouldn’t, the key is to figure out what information, operations and behaviors your system owns versus are owned by someone else. If you own the UI and the database, then synchronous is a good possibility. If you don’t own the system (like our web service above), then async is a good possibility.

Next time: sync, async and CQRS.

In microcontrollers, onboard memory is a huge limiting factor. Modern devices have lots of available memory, but for embedded devices, your ROM pretty much decides how complicated your program can be.

As part of the course, we were graded on how compact our code was, rather than merely how correct it was. And this makes sense, since compactness can allow for more features/decisions etc.

So we all made reasonable efforts to compact our code using common tricks like shift-operators for multiplication and so on. However, what we didn’t know is that our professor had spent 20-30 years optimizing assembly code for compactness, and our efforts were being graded against his. Any deviation from his solution was a deduction in our grade.

After receiving poor marks, and seeing why, we all as a class reviewed the (his) solution. And wouldn’t you know that while compact, code optimized to its maximum is nearly impossible to understand or maintain. No one in the class, viewing the code for the first time, would be able to decipher what it actually did.

Long-term maintainability

Since we change code far more often than we write code, optimizing solely for performance can make it difficult or impossible to change that code in the future. In the case of our college course, we were being held to standards that were nearly impossible to reach, let alone understand. Performance isn’t an accomplishment, it’s a feature.

It’s a feature that needs to be balanced against all other constraints, like the ability to maintain the code in the future. Highly optimized code often becomes more difficult to understand or comprehend, making it difficult to tweak or refactor in the future.

So when looking at performance optimization, which is many times a necessary endeavor, always keep an eye on the true goal of the performance optimization. How much more optimized does it need to be? What is the threshold for success?

Performance optimization without a clear definition of success just leads down the path of obfuscation and unmaintainability. Optimization does have an upper limit, not only in terms of gains, but of losses in maintainability.

But what I leave out of a talk, I can certainly blog about! Suppose we have one of our enumeration classes (available here on NuGet):

public class Color : Enumeration<Color>
{
    private Color(int value, string displayName) 
        : base(value, displayName)
    {
    }

    public static readonly Color Red 
        = new Color(1, "Red");
    public static readonly Color Blue 
        = new Color(2, "Blue");
}

When it comes time to persisting this enumeration class, we want to make sure that the database schema uses the integer value as what gets persisted. When it writes, we want the value to persist, and when it reads, we want the correct enumeration value (Red/Blue) to be hydrated.

To do this in NHibernate, we’ll first need a custom type:

public class EnumerationType<T> : PrimitiveType where T : Enumeration<T>
{
    public EnumerationType()
        : base(new SqlType(DbType.Int32))
    {
    }

    public override object Get(IDataReader rs, int index)
    {
        var o = rs[index];
        var value = Convert.ToInt32(o);
        return Enumeration<T>.FromInt32(value);
    }

    public override object Get(IDataReader rs, string name)
    {
        var ordinal = rs.GetOrdinal(name);
        return Get(rs, ordinal);
    }

    public override Type ReturnedClass
    {
        get { return typeof(T); }
    }

    public override object FromStringValue(string xml)
    {
        return int.Parse(xml);
    }

    public override string Name
    {
        get { return "enumeration"; }
    }

    public override void Set(IDbCommand cmd, object value, int index)
    {
        var parameter = (IDataParameter)cmd.Parameters[index];

        var val = (Enumeration<T>)value;

        parameter.Value = val.Value;
    }

    public override string ObjectToSQLString(object value, Dialect dialect)
    {
        return value.ToString();
    }

    public override Type PrimitiveClass
    {
        get { return typeof(int); }
    }

    public override object DefaultValue
    {
        get { return 0; }
    }
}

This class is the bridge between our ORM (NHibernate) and our enumeration class. NHibernate is fantastic in its ability to provide easy ways to bridge to value objects. Value objects help avoid primitive obsession, but is only useful if you can actually use them when you’re mapping to your persistence layer.

To instruct NHibernate to use our custom types when reading/writing, the easiest way to do so is with a Fluent NHibernate convention:

public class EnumerationTypeConvention : IPropertyConvention, IPropertyConventionAcceptance
{
    private static readonly Type OpenType = typeof(EnumerationType<>);

    public void Apply(IPropertyInstance instance)
    {
        var closedType = OpenType.MakeGenericType(instance.Property.PropertyType);

        instance.CustomType(closedType);
    }

    public void Accept(IAcceptanceCriteria<IPropertyInspector> criteria)
    {
        criteria.Expect(x => IsEnumerationType(x.Property.PropertyType));
    }

    private bool IsEnumerationType(Type type)
    {
        return GetTypeHierarchy(type)
            .Where(t => t.IsGenericType)
            .Select(t => t.GetGenericTypeDefinition())
            .Any(t => t == typeof(Enumeration<>));
    }

    private IEnumerable<Type> GetTypeHierarchy(Type type)
    {
        while (type != null)
        {
            yield return type;
            type = type.BaseType;
        }
    }
}

This convention walks the type hierarchy for each property type given, and checks to see if the property type’s type hierarchy is a generic type that closes the open generic type of our enumeration class. Finally, we just need to hook our convention up to Fluent NHibernate, but that really depends on how we have Fluent NHibernate hooked up. We don’t have to use Fluent NHibernate to hook up our custom user type, but it’s much easier this way.

With NHibernate, we get the benefits of using Java-style enumeration classes, and have it seamlessly plug in to our persistence layer, which is the whole point of ORMs, right?

We first tried the Web.Config setting for custom errors:

<customErrors mode="Off" />

But this didn’t affect anything. Digging a little further, we found that ASP.NET Web API uses a different configuration for error details being passed along. This is for a couple of reasons; first, the custom errors element in the Web.Config is an ASP.NET thing. It’s something that ASP.NET uses to determine if that yellow screen of death with additional detail should be shown to users. However, ASP.NET Web API is designed to be self-hosted, outside of ASP.NET and IIS. While the customErrors element affects requests for ASPX and MVC, it does nothing for Web API.

Instead of relying on a lot of XML configuration, Web API uses a lot of programmatic configuration. This helps self hosting, but for changing policies like error detail, we have to change the code, re-compile and re-deploy. To set the error policy in our application, we need to modify our global Web API configuration:

GlobalConfiguration.Configuration.IncludeErrorDetailPolicy 
    = IncludeErrorDetailPolicy.Always;

With this mode, requests from any source will get us full exception detail. It’s likely not something we want in production, but nice that it is available. We have three familiar options for the IncludeErrorDetailPolicy setting:

• LocalOnly (default)
• Always
• Off

These correspond 1:1 to the customErrror policy in their behavior, but in the reverse. The ASP.NET customErrors setting talks about when to custom errors, where Web API is when to display exception details. Same settings, but approaching from opposite ends.

What if we want to just have Web API use whatever setting that our Web.Config uses? We can just read the ASP.NET setting and apply it to our Web API config:

var config = (CustomErrorsSection)
    ConfigurationManager.GetSection("system.web/customErrors");

IncludeErrorDetailPolicy errorDetailPolicy;

switch (config.Mode)
{
    case CustomErrorsMode.RemoteOnly:
        errorDetailPolicy 
            = IncludeErrorDetailPolicy.LocalOnly;
        break;
    case CustomErrorsMode.On:
        errorDetailPolicy 
            = IncludeErrorDetailPolicy.Never;
        break;
    case CustomErrorsMode.Off:
        errorDetailPolicy 
            = IncludeErrorDetailPolicy.Always;
        break;
    default:
        throw new ArgumentOutOfRangeException();
}

GlobalConfiguration.Configuration.IncludeErrorDetailPolicy 
    = errorDetailPolicy;

With this approach, we just set one policy for custom errors in our Web.Config file, and both our MVC/WebForms and WebAPI requests in the same application use the exact same policy, and we have the ability to change this policy after we deploy.

These days, the de facto standard for building MVC applications looks something like this for GETs:

And for POSTs:

We’re using the same ViewModel for both GET and POST, where in one case the ViewModel is used in the View to build a form, and in the other case, the exact same model is used on the POST side to model bind from HTTP variables into something we can reason with.

The ViewModel usually isn’t the persistence model, as that the concerns of what your persistence layer needs often contradict what your presentation layer needs.

In this model, we have many different pieces at play here:

• Controller – responsible for deciding what to do with a request. Show a view? Redirect?
• ViewModel – Represents the information model for the View in GET, and a deserialization target in a POST
• View – Responsible for translating a ViewModel into instructions for generating HTML for the view engines
• ModelBinder – Responsible for supplying the input model to a controller action

One key piece here I want to highlight is the View. A ViewModel can’t be translated straight to HTML, and shouldn’t. It’s not an object model suited to act as complete instructions on translating the information model to the HTML media type. In fact, you don’t see really any HTML instructions on ViewModels. There are pieces in the ViewModel that can help with HTML generation (like validation attributes), but you don’t see anything like “make this a REALLY HUGE text box”, nor should you. That’s the view’s responsibility.

What’s missing is that View piece in Web API. The piece that takes the model built by a view (that is Media Type-agnostic) and provides instructions on how to then take that into

Bridging to ASP.NET Web API

Coming back to ASP.NET Web API, what objects are in play there? We have some familiar, and not so familiar:

• Controller – Responsible for deciding what to do with a request, at the HTTP level
• Media Type Model – Represents the information needed for a specific media type
• Formatter – Responsible for translating a media type model to the specific media type

What’s bothering me about this model is that I now don’t have that layer to give instructions to the Formatter on how to serialize. Another thing we’ve done here is conflated the responsibilities of the “ViewModel” of this land with not only the information but also instructions.

What do I mean by “instructions”? Putting XML serialization attributes on your model created by your API controller action is mixing the media type in with your model. Having JSON mixed around directly in our controller action. Things like this, that have Json objects created directly in our actions.

I think the above abstractions are the wrong abstractions. We’re coupling the model built inside the controller action with the needs of a generic formatter. We could build a custom formatter, but that’s like building a custom view engine.

Going back to MVC, a View bridges the gap from Resource to View Engine. I believe that a Formatter in Web API is a View Engine. It is a generic translator of instructions into a specific media type (HTML). Formatters attempt to be intelligent on how to build out other media types (JSON, XML etc.), but ultimately, making them too smart means embedding media-specific information into our model. So what’s my thought? I’d like to see something like:

• Controller – Responsible for deciding what to do with a request, at the HTTP level
• Resource Model – Represents informational model of the resource, including links (at a conceptual level, not a technical level)
• Media Type Template – Responsible for translating a Resource Model into instructions for a specific Media Type Formatter
• Formatter – Responsible for using the media type template and resource model to build the response

Why the change from Media Type Model to Resource Model? I don’t want to couple the model in my controller action to a specific media type, that’s why! The conneg piece of Web API is what’s supposed to determine this. Imagine if you will that the resource model generated from the controller action is checked against media type templates to see what is possible to return back to the client.

You could use custom formatters in Web API, but looking at examples, they’re at the wrong abstraction layer, talking to things straight to streams etc.

Do we need generic Media Type Templates? No! When we were building views for different media types in MVC, we built the exact same model from the controller action that fed Telerik reports, PDF reports and HTML, all from the same controller action and model. The concept of views bridged the gap in each case to take the information model from the controller action and provide instructions to the “media type generator” on how to generate that PDF, report, or HTML view.

My ideal world

In my ideal world, my controller action in Web API would care nothing about specific media types. It would know how to build my rich, media-type-ignorant resource model, and that’s it. Web API would then try to line up to figure out what potential formatters are available for my resource model, taking into account accept headers and what media type templates (views) are available for this resource model.

Because wouldn’t it be nice if I could build a single resource model that could actually be used with multiple media types, not just JSON or XML? How about JSON, XML and HTML, and PDF? To do so, the responsibilities of the model generated by the view should only contain the informational model of the resource, and not have media type concerns leaking in.

You could have defaults to make things easier, but what’s really missing here in the Web API landscape is the concept of a view (not the MVC-specific implementation). Something that is responsible for providing (optional) instructions on taking the resource model and telling the formatter how to serialize the model. Looking at Formatters how they are today, you have to take over too much to bridge that gap.

But I could be totally off base here.

After you create your fork, you’ll likely then clone that forked repository locally like:

git clone git@github.com:jbogard/AutoMapper.git

That’ll create a remote called “origin” that points up to your fork (in this case, on github). Check out the Git book for more info on remotes.

But we’re not quite done. We’ve created a fork, which is its own entire repository, separate from the original “central” repository:

In this new distributed reality, we need to be able to work with those upstream repositories effectively.

Making changes

If you want to make changes locally, great! But always make those changes in a branch. What we want to do is still be able to pull in any upstream changes, but isolate any contributions we make to the project. In order to do so, we’ll always create a local AND remote branch for any work we do. Our workflow looks like this:

git checkout -b MyFeature
**work work work & commit changes locally**
git push origin MyFeature

The reason we want to isolate our custom work from the master branch is that it makes it much, much easier to integrate and keep up to date with upstream changes. An example in GitHub is:

I have two branches – the default “master” branch, and one that I’m going to isolate for a pull request. The nice thing about CodePlex and GitHub is that they both allow you to “add” changes in the form of commits to your pull request. By isolating potential pull requests into a separate, dedicated remote branch, we can ensure that we don’t paint ourselves into a corner where we want to pull in upstream changes or work on something else.

Branches – local and remote – isolate work, and both GitHub and CodePlex work well with branches as pull requests.

When you’re done with any upstream changes, and our pull request is either accepted or rejected – kill your local and remote branch:

git branch -D MyFeature
git push origin :MyFeature

Once our changes are integrated (if they are), we want to use the upstream repository’s version of our changes.

Pulling upstream changes

When you create a fork, you’ll also want a way to keep up to date with the upstream repository. The first thing to do will be to add a remote that points to the upstream repository (conventionally named “upstream”):

git remote add upstream git://github.com/AutoMapper/AutoMapper.git

With that remote added, we want to fetch upstream, merge all local branches with that repository, and push that to our upstream:

git checkout master
git fetch upstream
git merge upstream/master (for each upstream branch)
git push origin

This workflow:

1. Fetches all upstream commits
2. Merges our local branches (as fast-forward merges) to the upstream’s repository’s versions
3. Synchronizes our fork with upstream changes by pushing our local repository up.

It’s a little weird that we have to work with our local repository to synchronize our fork with the upstream repository, but that’s just because it would be a little hard to try and do that directly in GitHub.

The *first* thing you should do after forking a project and cloning that fork locally is add that “upstream” remote. It’s just much, much easier to have it set up early and get into the flow of fetch, merge, push for incorporating upstream changes.

To review, working with forks on GitHub and CodePlex is easy if we do two things:

• Create an “upstream” remote to represent the repository our fork originated from
• Always work in named local and remote branches for any commits we do in our fork

For more info, check out GitHub’s help pages.

Prerequisites:

First, get a GitHub account and make sure you’re set up locally. GitHub has a great tutorial here: http://help.github.com/win-set-up-git/

Next, just create a dummy repository and make sure you can push/pull OK.

Forking a CodePlex repository

1. Create an aspnetwebstack repository on GitHub:
   
   You can just call it “aspnetwebstack” to make things easy.
2. Clone the CodePlex repository locally, using the Git bash or Posh-Git

   git clone https://git01.codeplex.com/aspnetwebstack
   cd aspnetwebstack

3. Create a Git remote for your GitHub repository:

   git remote add github git@github.com:jbogard/aspnetwebstack.git

4. Push the aspnetwebstack repository you just cloned up to GitHub:

   git push github master

5. Profit!

Quick side note, to learn more about what “Open Source” means, check out the Open Source Definition from the OSI. “Open Source” does not now nor has ever meant “accepts contributions”.

In ASP.NET MVC 3, the MVC framework was released under an OSI approved license, but Razor was not. But starting today, you can go to CodePlex and not only download the source, but watch as the ASP.NET team commits code on a daily basis.

So how does this affect the products now being developed in the open, and accepting contributions? As a maintainer of AutoMapper, which also went from closed source to open, I’ve felt the challenge of this transition. While I can’t speak for the ASP.NET team, I can offer my perspective on how going OSS affected the development of AutoMapper.

Will the quality of the product be adversely affected because the team accepts contributions?

No. Just because someone has the ability to accept contributions does not mean that anybody’s crappy code will make it in. Code submitted will go through the same review process that all other code commits go through from internal teams.

Will some yahoo be able to sidestep my current feedback channel and take the product in some other (wrong) direction?

Possible, but unlikely. In my experience, large features don’t get submitted as pull requests without intensive discussions with the team. I’ve only received one or two very large features in AutoMapper, and those were after long conversations on GitHub or the mailing list. The discussions are public, and anyone can participate.

Basically, if you submit a large pull request that isn’t on the team’s roadmap, it’s more than likely going to get rejected. I know, because I’ve rejected large pull requests for the same reasons. If a team develops in the open, you don’t go hide in a hole, develop a big feature, then expect it to get rolled in.

Will my pull request get accepted?

I don’t know. I don’t accept every pull request. It’s not even a size thing, sometimes I just don’t like a feature, don’t think it’s needed, or don’t like an implementation. I’ve submitted plenty of patches that got rejected for a wide variety of reasons, but again, OSS is all about the ability to make your own changes, not that your changes will make it back into the fold.

Will someone else’s changes be supported?

Yes. Because the team ultimately makes the call on whether or not code gets into the mainline and released, it’s actually not that important who wrote the code. It really just matters that the team has committed to support whatever code makes it in to the release.

Are we in store for human sacrifice, dogs and cats living together, mass hysteria?

No. But nice Ghostbusters reference.

Can we trust these products now?

I don’t see why not. You’ve never met the team that coded the original products, so it shouldn’t matter who wrote the code, rather that the code is approved.

Is this a Good Thing?

Yes. The biggest benefits I see are that:

1. Small fixes that get lost in Connect issues etc. are much more easily addressed through contributions. The little stuff will get better and more complete.
2. Bigger changes will get feedback more early from the community. Often, by the time the community gets to see a working product, it’s way past the point of being able to provide substantial feedback.

What could go wrong?

It’s not all rainbows and unicorns. The biggest thing is that once you start engaging directly with the community through your code, you’re going to start getting a lot more feedback. Now the feedback is better and more targeted, but it is a different animal.

However, I don’t think it’s going to be a big deal. Right now, the team receives feedback in big batches, after releases, making it quite hard to sift through all those comments at once. By developing in the open, those comments become spread out a lot more throughout the dev cycle. Also, I imagine that much of the feedback received after a release will just go away because things will just be addressed earlier when changes are much easier to make.

Does this mean the Contrib projects are going away?

No. Contrib projects have their place, as a repository of supplemental features that aren’t necessarily a good idea to fold into the main project. The part of the Contrib project that is filling out “missing” features will likely go away, but the “supplemental” parts are still definitely valuable. Also valuable are pieces that build on top of that product and another one, where the core product shouldn’t take that dependency.

As an example, Input Builders would never have made it into the MvcContrib project, as we would have just worked with the MVC team on a common design.

Why CodePlex and not GitHub?

Really? No no no wait. Really? You know, you can mirror the source on GitHub if that’s your style. It’s DISTIRBUTED VCS, ya know. But really? It’s called being a Team Player. And the CodePlex team is (re)committed to make CodePlex a viable place for OSS development. GitHub is still vastly superior, but the level of feedback that the MS teams will start giving the CodePlex team because of this new relationship is also a Good Thing.

In short, this is a Good Thing, and I’m excited that Microsoft is taking this step. It’s a step I never thought Microsoft would take, and just assumed that it would never happen. But it has, and now we can start pushing for other products/tools to be released the same way.

• Sending files
• Receiving files
• Web services

And how each of those ways of integrating can be rife with failures. I walked through a scenario of using an NServiceBus Saga to help manage a process that communicates with multiple web services that each depend on each other. The code and slides can be found on my github here:

https://github.com/jbogard/presentations

Check it out!

However, not everything needs to be up and running in integration tests. I often want to stub out services on the outermost periphery in my application. Places where I call into things I can’t control, like external web services, still need to be stubbed out in my tests.

This poses a challenge – I want to have a configured container in my tests, but provide test-specific behaviors. I want any custom configuration/stubs to go away after my test is done, and have each test start with a blank slate. So you could do something like this:

public class TestClassBaseWithInitialize
{
    private IContainer _container;

    [SetUp]
    public void SetUp()
    {
        _container = Root.BuildContainer();
    }

However, each time a test runs, the container is re-initialized. But in our production apps, we initialize the container just once, so why not just initialize once in our tests? The trick is that we need to reset the container to a known state before each test.

The solution here is to use the concept of nested/child containers. In a nested container, its configuration is cloned but isolated from the parent container. We want our tests to use this nested container instead:

public abstract class TestClassBase
{
    private readonly IContainer _container 
        = Root.Container.GetNestedContainer();

    protected T GetInstance<T>()
    {
        return _container.GetInstance<T>();
    }

    protected void Inject<T>(T instance)
    {
        _container.Configure(cfg => cfg.For<T>().Use(instance));
    }
}

Folks using the container need to go through this base GetInstance method, and not go straight to the container. For injecting dummy/stub instances, the Inject method configures just that nested container, and leaves the root container untouched.

If you’re doing a lot of integration/full-system-tests, you can get quite a drop in the time it takes your tests to run if you make sure those things that are expensive to initialize (container/ORM) are initialized only once per entire test suite. Once you have that in place, it’s just a matter of integrating and isolating changes from one test to the other. With nested containers, it’s a cinch.