Snapshot time!

It’s snapshot time! There’s been a lot of event sourcing content so far. Let’s do a recap!

Below you will find a short summary of event sourcing related articles that I have published here so far. Treat it as a table of content or a lookup or a pattern collection. It’s ordered by date, the later – the older. Enjoy!

  1. Why did it happen – how to make your event sourced system even easier to reason about
  2. Event sourcing and interim stream – how to embrace new modelling techniques with short living streams
  3. Multitenant Event Sourcing with Azure – how to design a multitenant event sourced system using Azure Storage Services
  4. Rediscover your domain with Event Sourcing – how to use your events and astonish your business with meaningful insights
  5. Event Sourcing for DBAs – a short introduction for any relational person into the amazing world of event sourcing. Can be used as an argument during a conversation.
  6. Enriching your events – what are events metadata and why should we care? how to select the most important ones
  7. Aggregate, an idempotent receiver – how to receive a command or dispatch an event exactly once?
  8. Process managers – what is a process manager, how can you simplify it?
  9. Optimizing queries – how to make queries efficient, especially when dealing with multiple version of the same application running in parallel
  10. Event sourcing and failure handling – an exception is thrown. Is it an event or not? How to deal and model it?
  11. Embracing domain leads towards event oriented design – how event oriented design emerges from understanding of a domain

Why did it happen?

TL;DR

Do you know that feeling of being powerless? Of being not able to tell why your system acted in a specific way? Of not being able to recognize whether it’s a hacker or your system malfunction? Event sourcing, by storing all the events that happened in your system helps a lot. Still, you can improve it and provide much better answers for ‘why did it happen?’.

Why?

The question ‘why did it happen?’ reminds me of one case from my career. It was a few years back, when I was a great fan of AutoMapper. It’s a good tool, but as with every tool one should use it wisely (if you have only a hammer …). I think I stretched it a little to much and landed in a point where nobody was able to tell what the mapping come from. It took only 3 days to provide an extension method Why that was showing which mapping will be applied for a specific object.

I’d say, that being able to answer these ‘whys’ within a reasonable time frame is vital for any project. And I don’t mean only failures. When introducing a junior to your team, being able to show how and why things work is important as well.

Event sourcing

Event sourcing helps a lot. It just stores every business delta, every single change of your domain objects. You can query these changes in any time. You can improve your queries even more by enhancing events with some metadata (which I covered recently in here). There’s a case though that you should consider to tell if the reasoning is always that easy.

Dispatching an event

Some of actions on your aggregates are results of dispatching an event. Something happened and another part of your systems turns that into an action. For instance consider the following

When OrderFinished then AddBonusPoints

Some bonus points are added to an account whenever an order is finished. When looking at an account history, you’ll see a lot of events BonusPointsAdded. Yes, you could introduce a lot of events like BonusPointsAddedBecauseOfOrderFinished but this just leaks the process into your account aggregate. If you don’t do it, can you answer the following question

Why BonusPointsAdded were appended?

Because somebody added points? Yes, but WHY? It looks that the reason is disconnected.

Point back

What if following metadata were added to every event that is a result of another event. In this case, what if the following metadata were added to this specific BonusPointsAdded

enhance

Now, when somebody asked Why did it happen you can easily point to the original event. If that was a reason of some process, of dispatching the event you can follow the link again and again to find the original event that was created because of an user action.

Summary

Links are a powerful tool. If you use it with event sourcing you can get a history of your system that’s easy to navigate, follow and reason about.

Event Sourcing and interim streams

TL;DR

When modelling with event sourcing, people often tend to create long living streams/aggregates. I encourage you to improve your modelling with interim streams.

Long live the king

A user, an account, a company. Frequently this kinds of aggregates are distinguished during first modelling attempts. They are long lasting, never ending streams of events. Sometimes they combine events from different contexts, which is a smell on its own. The extended longevity can be a problem though and even when events are correlated and an aggregate is dense, in the business meaning, rethinking it’s lifetime can help you design a better system.

Interim streams

One of the frequent questions raised when applying event sourcing is: how do I ensure that a user is registered with a unique email? For sure in majority of cases handling duplicates by a person would be just enough (how many people would actually try to register twice…), but let’s try to make this a real business requirement.

Because of the disability to span a transaction across different streams, we can model this requirement in a different way. Let’s start with creation of a stream which name is derived from email, like:

useraggregate_email@domain.com

This would ensure, that for a specific email, only one aggregate can be created. Next step is to put an event holding all the needed data to add a user, like:

UserRegistered:

{name: “Test”, surname: “Test”, phone: “111111111111”, userid: “some_guid”}

The final step is to have a projection that consumes this event and appends it to a new stream named:

useraggregate_some_guid

This ensures that we don’t use a natural key and still preserve the uniqueness requirement. At the end we have an aggregate that can be identified with its GUID.

Time to die

The interim stream example with the unique email wasn’t the best one, but was good enough that a stream can make interim streams or that can be based on an interim stream. It’s vital to liberate from thinking that every stream needs to live forever. I’d say that in majority of cases there’s a moment when a stream needs to ends its life.

stocksnap_6qj9xjwicj

The longevity isn’t the most important virtue of a stream. It’s created either to model a behavior in a right way or be just a passage for other streams. Next time when you model a stream ask yourself, how long will it live?

Multitenant Event Sourcing with Azure Table Storage

TL;DR

Designing a multitenant system puts a hard requirement on a designer to do not leak data between tenants. Is there anyone who would like to show a list of employees’ emails from one company to another?

Azure Table Storage

Azure Table Storage is a part of Azure Storage Services. It’s mentioned in the original Windows Azure Storage whitepaper and provides a stable foundation with known limitations, quotas and API that hasn’t changes for ages (ok, years). The most important aspect of it is the throughput which limited in two dimensions:

  • partitions – a partition is defined by a partition key value and can CRUD at most 500 entities (rows) per second
  • storage account – an account can process at most 10k operations per second

These two numbers can impact the performance of an app and should be taken into consideration when designing storage.

StreamStone

There’s a library which provides an event sourcing store on top of the Azure Table Storage. It’s called StreamStone. It provides a lot of capabilities but not a from-all projection (see this PR for more info, including my notes). This can be added (not easily), which I’ve done introducing some overhead on the write side.

Having a storage problem solved, how would you define and design a multitenant system?

One to rule them all

The initial attempt could be to add the company identifier to the partition key. Just use it as a prefix. That could work. Until one of the following happens:

  • a scan query without a condition is issued – just like SELECT * without where, yeah, that would be scary
  • a company uses our app in a way that impacts others – it’s easy, you need to saturate 10k operations per second

It looks that this could work, but it could fail as well. So it’s not an option.

Separated accounts

Fortunately, Azure provides management API for storage accounts. This means that from an application, one can instantiate storage accounts under the same subscription but totally separated from each other. Like in a container or something. This boxes performance limitations for a company into its own account. The problem of a potential leakage is also addressed by storing data of a company in a totally separated account.

As mentioned by Adrian in the comments, there’s a limit of 200 storage accounts per single subscription which is a high number to reach. Once you do it, additional layer of subscription management should be applied.

stocksnap_g9pflyymu1

Who knows them all?

Of course there’s a need of a governor. A module that will know all the accounts and that will manage them. This, is a limited surface of possible leakage, leaving a good separation for the rest of the application/system.

Concurrent conditional deletes

TL;DR

Sometimes System.Collections.Concurrent provide not enough methods to squeeze the top performance out of them. Below you can find a small extension method that will make your life easier in some cases

Concurrent dictionary

The concurrent dictionary provides a lot of useful methods. You can TryAdd, AddOrUpdate. Additionally, you can use TryUpdate which provides a very nice method for ensuring optimistic concurrency. Let’s take a look at its signature:

public bool TryUpdate(TKey key, TValue newValue, TValue comparisonValue)

It enables to replace a value under a specific key with the newValue only if the previous value is equal to comparisonValue. It’s an extremely powerful API. If you create a new object for a specific key to update it, it enables to replace that object without locks with a single method call. Of course, if the method fails, it returns false and it’s up to you to retry.

What about deletes? What if I wanted to remove an entry only if nobody changed it’s value in the meantime? What if I wanted to have an optimistic concurrency for deletes as well. Is there a method for it? Unfortunately no. The only method for removal is

public bool TryRemove(TKey key, out TValue value)

which removes the value unconditionally returning it. This breaks the optimistic concurrency as we can’t ensure that the removed entry wasn’t modified. What can be done to make it conditional?

Explicit interfaces

The ConcurrentDictionary class implements a lot of interfaces, one of them is

ICollection<KeyValuePair<TKey, TValue>>

This interface has one particular method, enabling to remove a pair of values.

ICollection<KeyValuePair<TKey, TValue>>.Remove(KeyValuePair<TKey, TValue> kvp

If you take a look into implementation, it uses a private method of the dictionary to remove the key only if the value is equal to the value of the pair. Now we can write a simple extension method to provide a conditional, optimistically concurrent removal of a key

static class ConcurrentDictionaryExtensions
{
    public static bool TryRemoveConditionally<TKey, TValue>(
        this ConcurrentDictionary<TKey, TValue> dictionary, TKey key, TValue previousValueToCompare)
    {
        var collection = (ICollection<KeyValuePair<TKey, TValue>>)dictionary;
        var toRemove = new KeyValuePair<TKey, TValue>(key, previousValueToCompare);
        return collection.Remove(toRemove);
    }
}

which closes the gap and makes the API of the concurrent dictionary support all the operations under optimistic concurrency.

Summary

With this simple tweak you can use a concurrent dictionary as a collection that supports fully an optimistic concurrency.

Rediscover your domain with Event Sourcing

TL;DR

Beside common advantages of event sourcing like the auditing, projections and sticking closely to the domain, you can use events to discover the domain again and provide meaningful insights to your business.

Include.Metadata

I’ve already described the idea of enriching your events. This is the main enabler for analyzing your events in various way. The basic metadata one could are:

  • date
  • action performer
  • on behalf of who action is taken

You could add a screen of your app and IP address and many many more.

Reason

Having these additional data, it’s quite easy to aggregate all the events of a specific user. This, with time attached, could provide various information:

  • how the work is distributed during a work day
  • how big is the area of business handled by a single user
  • is the user behavior pattern the same all the time or maybe somebody has overtaken this account?

The same with projection by event type:

  • is it a frequent business event
  • are these event clustered in time – maybe two events are only one event

Or looking at a mixed projection finding sequences of events for a user that might indicate:

  • an opportunity for remodeling your implementation
  • finding hot spots in the application.

Rediscover

stocksnap_crlx1ud87t

All of the above may be treated as simple aggregations/projections. On the other hand, they may provide important trends for a system and might be used to get an event based insight to the business domain. Can you imagine the business being informed about a high probability of a successful cross selling of two or three products? That’s where a competitive advantage can be born.

 

 

Event sourcing for DBAs

TL;DR

Are you in a team trying to convince a database administrator  to use event sourcing? Are you a DBA that is being convinced? Or maybe it’s you and a person that does not want to change the relational point of view. In any case, read forward.

Drop. Every. Single. Table

That’s the argument you should start with. You can drop every single table and your database will be just right. Why is that? There’s a recovery option. If you don’t run your database with some silly settings, you can just recover all the tables with all the data. Alright, but where do I recover from? What’s the source of truth if I don’t have tables.

Log

The tables are just a cache. All the data are stored in the transaction file log. Yes, the file that is always to big. If all the data are in there, can it be queried somehow?

SELECT * FROM fn_dblog(null, null)

That’s the query using an undocumented SQL Server function. It simply reads a database log file and displays it as a nicely formatted table like this:

dblog

Take a look at the operation column now. What can you see?

  • LOP_INSERT_ROWS
  • LOP_MODIFY_ROW
  • LOP_MODIFY_COLUMNS
  • LOP_COMMIT_XACT
  • LOP_BEGIN_XACT
  • LOP_COMMIT_XACT

What are these? That’s right. That’s every single change that has been applied to this database so far. If you spent enough of time with these data, you’d be able to decode  payloads or changes applied to the database in every single event. Having this said, it looks like the database itself is built with an append only store that saves all the changes done to the db.

Event Sourcing

Event sourcing does exactly the same. Instead of using database terms and names of operations, it incorporates the business language, so that you won’t find LOP_MODIFY_COLUMNS with a payload that needs to be parsed, but rather an event with an explicit business related name appended to a store. Or to a log, if you want it call it this way. Of course there’s a cost of making tables out of it once again, but it pays back by pushing the business understanding and meaning much deeper into the system and bringing the business, closer to the system.

At the end these tables will be needed as well to query something. They won’t be treated as a source of truth, they are just a cache anyway, right? The truth is in the log.