Pearls: EventStore transaction log

I thought for a while about presenting a few projects which are in my opinion real pearls. Let’s start with the EventStore and one in one of its aspects: the transaction log.
If you’re not familiar with this project, EventStore is a stream database providing complex event processing. It’s oriented around streams of events, which can be easily aggregated or repartitioned with projections. Based on ever appended streams and projections chasing the streams one can build a truly powerful logic around processing events.
One of the interesting aspects of EventStore is its storage engine. You can find a bit of description in here. ES does not abstract a storage away, the storage is a built-in part of the database itself. Let’s take a look at its parts before discussing its further:

Appending to the log
One the building blocks of ES is SEDA architecture – the communication within db is based on publishing and consuming messages, which one can notice reviewing StorageWriterService. The service subscribes to multiple messages, mentioned in implementations of the IHandle interface. The arising question is how often does the service flushed it’s messages to disk. One can notice, that method EnqueueMessage beside enqueuing incoming messages counts ones marked by interface IFlushableMessage. What is it for?
Each Handle method call Flush at its very end. Additionally, as the EnqueueMessage increases the counter of messages requiring flush, each Handle method decreases the counter when it handles a flushable message. This brings us to the conclusion that the mentioned counter is equal 0 iff there are no more flushable messages in the queue.

Flushing the log
Once the Flush is called a condition is checked whether:

  • the call was made with force=true (this never happens) or
  • there are no more flush messages in the queue or
  • the given time from the last time has passed

This provides a very powerful batching behavior. Under stress, the flush-to-be counter will be constantly greater than 0, providing flushing every given period of time. Under less stress, with no more flushables in the queue, ES will flush every message which needs to flush the log file.

Acking the client
The final part of the processing is the acknowledgement part. The client should be informed about persisting a transaction to disk. I spent a bit of time (with help of Greg Young and James Nugent) of chasing the place where the ack is generated. It does not happen in the StorageWriterService. What’s responsible for considering the message written then? Here comes the second part of the solution, the StorageChaser. In a dedicated thread, in an infinite loop, a method ChaserIteration is called. The method tries to read a next record from a chunk of unmanaged memory, that was ensured to be flushed by the StorageWriterService. Once the chaser finds CommitRecord, written when a transaction is commited, it acks the client by publishing the StorageMessage.CommitAck in ProcessCommitRecord method. The message will be translated to a client message, confirming the commit and sent back to the client.

Sum up
One cannot deny the beauty and simplicity of this solution. One component tries to flush as fast as possible, or batches a few messages if it cannot endure the pressure. Another one waits for the position to which a file is flushed to be increased. Once it changes, it reads the record (from the in-memory chunk matched with the file on disk) processes it and sends acks. Simple and powerful.

Composition vs derivation

Assume you’re writing some reports in your application. You’ve just created the third controller covering some kind of reporting and it seems to be, that all the three controllers have a very similar code, modulo type passed as the entity type to your NH ISession.QueryOver() or another data source. It seems that, if the method creating report base was generic, it could be used in all the cases. You want to extract it, make it clearer and to stop repeating yourself. What would you do?

Derivation
The very first thing is to extract method in each of the controllers. Now they seem almost the same. A place is needed where the method can be easily moved. How about a super type? Let’s create a controller, call it in a fashionable way, for instance: ReportControllerBase, and move the method in there. Now you can easily remove the methods in the deriving controllers. Yeah! It’s reusable, everyone writing his/her report can derive from the ReportControllerBase and use its methods to speed up his/her task.

Composition
The very first step is exactly the same: the extract method must be done, to see the common code. Once it’s done, you notice that the whole method has only a few dependencies which can be easily pushed to parameters, for instance: isession, the entity type passed to query, the value used in some complex where clause, etc. You change all the field and properties usages to parameters which allows this method to be static. You create a static method and turn it into an extension method of a session. The refactorization is done, you can easily call this method in all the controllers, by simply calling an extension onto a session.

What’s the difference and why composition should be preferred
If you use C# or Java you should always be aware of one limitation: you can derive from only one type. Spending this ‘once in a lifetime’ for a simple functionality extraction, for me – that’s the wrong way. Using composition, and deriving only when you truly see that one type is another type, that’s the right way to go. In the next post I’ll write about ASP MVC Detached Actions, a simple mechanism you can use, to derive your controllers only, when it is needed and delegating common actions, without it.

ASP MVC Model binders

Recently I had to create a model for ASP MVC, which would be bound by the model binder and had its properties set according to the request. It’s the most common case you can imagine, but there was one ‘but’. The model, because of its nature had to have non-default constructor. It was only a few parameters but it broke the DefaultModelBinder, which simply couldn’t find all the needed values. My response was quite fast, specially having the controller factory implemented in the following way:

/// <summary>
/// The controller factory building up the controllers with the unity container.
/// </summary>
public class ControllerFactory : DefaultControllerFactory
{
    private readonly IUnityContainer _container;

    public ControllerFactory(IUnityContainer container)
    {
        _container = container;
    }

    protected override IController GetControllerInstance(RequestContext requestContext, 
        Type controllerType)
    {
        return (IController)_container.Resolve(controllerType, null);
    }
}

The very next step was to replace the default binder stored in ModelBinders.Binders.DefaultBinder with another, controller-based implementation:

/// <summary>
/// The model binder building up the model objects with the unity container.
/// </summary>
public class ModelBinder : DefaultModelBinder
{
    private readonly IUnityContainer _container;

    public ModelBinder(IUnityContainer container)
    {
        _container = container;
    }

    protected override object CreateModel(ControllerContext controllerContext, 
                                          ModelBindingContext bindingContext,
                                          Type modelType)
    {
        var type = modelType;
        if (modelType.IsGenericType)
        {
            var genericTypeDefinition = modelType.GetGenericTypeDefinition();
            if (genericTypeDefinition == typeof (IDictionary<,>))
            {
                type = typeof (Dictionary<,>).MakeGenericType(modelType.GetGenericArguments());
            }
            else if (((genericTypeDefinition == typeof (IEnumerable<>)) ||
                      (genericTypeDefinition == typeof (ICollection<>))) ||
                     (genericTypeDefinition == typeof (IList<>)))
            {
                type = typeof (List<>).MakeGenericType(modelType.GetGenericArguments());
            }
        }

        // in the base method: return Activator.CreateInstance(type);
        return _container.Resolve(type, null); 
    }
}

As you can compare, the method simply delegates creating model to the container. Having this binder set allowed passing models with non-default constructors, so the problem was solved:] After a while, I considered following case: if the type is constructed with a container, the controller can also be passed an object implementing an interface registered in container. Hence, if a method needs a NHibernate ISession, this can be expressed not only as a parameter in Controller constructor, but also as a method parameter. This paradigm creates controllers with no parameters needed in constructor (or only common parameters, used in all actions) and all the others passed as action parameters. The result is a method, which can be perfectly tested in an environment where the only smallest, needed set of dependencies is passed (no more constructors with not needed in the current method parameters). What do you think about it?

An example of a controller:

public class CarController : Controller
{
    private readonly ISession _session;
    private const int PageSize = 10;

    public HomeController(ISession session)
    {
        _session = session;
    }

    public ActionResult Rent(IUserService userService, Guid carId)
    {
        using (var tx = _session.BeginTransaction())
        {
            var car = _session.Load<Car>(carId);
            var user = userService.GetCurrentUser();

            car.RentBy(user);
            
            tx.Commit();
        }

        return View();
    }

    public ActionResult Index( int? pageNo)
    {
        var page = pageNo ?? 0;

        var cars = _session.CreateQuery("from Car")
            .SetFirstResult(page*PageSize)
            .SetMaxResults(PageSize)
            .Future<Car>();

        return View(cars);
    }
}