The tests that come to your mind, when all you have are messages

The majority of the developers do the enterprise related development. Probably, you’re one of them. Even if pet projects or OSS contributions are important to you, enterprise related development is the thing you spend with majority of your professional time. Enterprise applications are quite dependent. You don’t have just a database and an app sitting on top of it. There are many services, applications, databases that your solution need to talk to. Additionally, the communication may be very different, from REST services using http to TCP, queues, or even shared databases… The question which arises is are the true unit tests the tests that you should use to cover your whole application? How do you test at all an application which is in the worst case just a mediator?

How would you describe the API provided by your service, the database, queues etc? Follow my way of thinking and consider them as output-input. In the center there’s your application configured with a fancy DI container, properly configured. Consider now calling an API to get a current account state. Your application access the database to get cached glossaries to provide properly named properties, gets some data from external transactional service, possibly sends a notification with some queue. Now, let turn these into named entities:

  • GetAccountsData (call to your API)
    • FetchGlossaries (call to your DB)
    • GetAccounts (call to transactional system)
    • NotifyCompliance (push to queue)

How would you test this flow? Using strict mocks, some substitutions? I’d say, instead of interfaces, services repositories go with a simple approach of messages and handlers. The whole flow is a result of initial message GetAccountsData. Having provided responses to FetchGlossaries, GetAccounts, NotifyCompliance in a simple form: just take the request object and compare it serialized version with the one registered as fake. If that fits, return the preregistered response. At the end you should assert the result of GetAccountsData.

To minimize the set of faked responses, fail the test if the response was not used at all. Fail if it was used to many times if you want. You may go into world of mocking libraries, but it’s not needed when every call is a simple single object passed to a mediator, dispatching it to a handler. You know the thing about objects. You easily deep compare them with a simple serialization output.

Is it a test? Yes. Is it a unit test? Not at all. You’re asserting the whole configured system. Is it worth? In my opinion with that approach you can have more business oriented tests, describing some scenarios rather than tests handling external mock/assert libraries. If I would this approach everywhere? Probably not. With a highly collaborative and dependent applications it seems to me know as a time and sanity saver. Define input-output and assert the system as a whole. Nothing more, nothing less.

Nautral identifiers as subresources in RESTful services

There’s a project I’m working on, which provides a great study of legacy. Beside the code, there’s a database which frequently uses complex natural keys, consisting of various values. As modelling using natural complex keys may be natural, when it comes to put a layer of REST services on top of that structure, a question may be raised: how to model the identifiers in the API, should this natural keys be leaked into the API?

REST provides various ways of modelling API. One of them are REST subresources, which are represented as URIs with additional identifiers at the end. The subresource is nothing more than an identified subpart of the resource itself. Having that said, and taking as an example a simple row with complex natural key consisting of two values <Country, City> how could one model accessing cities (for sake of this example I assume that, there are cities all around the world having the same name but being in different countries and all the cities in the given country have distinct names). How one could provide a URI for that? Is the following the right one?


The API shows Warsaw as the Polish city. That’s true. This API has that nice notion of being easy to consume, navigate. Consider following example:


Now it’s a big uglier, both the country and the city name are at the end. This is a bit different for sure and tells nothing about country accessible under /api/country/Poland. The question is which is better?

Let me abuse a bit the DDD Aggregate term and follow its definition. Are there any operations that can be performed against the city resource/subresource that does not change the state of the country? If yes, then in my opinion modelling your API with resources shows something totally different, saying: hey, this is a city, a part of this country; it’s a subresource and should be treated as a part of the country ALWAYS. Consider the second take. This one, presents a city as a standalone resource. Yes, it is identified by a complex natural key consisting of two dimentions, but this is a mere implementation detail. Once a usual identifiers like int, Guid are introduced the API won’t change that much, or even better, API could accept both of them, using the older combined id for consumers that don’t want to change their usage (easier versioninig).

To sum up: do not leak your internal design either it’s a database design or an application design. Present your user a consistent view grouping resources under wings of transactional consistency.

Kramer vs Kramer

The reality of project management is simple. A project manager has a project he/she’s responsible for. The project can span across multiple teams, areas, resources, whatever you want to call them (just use the name managish enough). The responsibility of the manager is to deliver the project, in majority of cases within the given budget (manhours). The mythical man-month rule can be applied to deliver it faster by simply throwing more resources…

On the other hand, one may take a look at the teams and products. There’s one thing about team and product – they are bound and it’s quite interesting to see the cross-functional team delivering the whole product experience and simply being happy because of the created product. It’s a natural thing to be happy because of your own creation. We all know it: build the house, etc…

And here goes Kramers! Pulling the team from one to another, arguing about behavior only from his/her project point of view, trying to match the high-level goals and estimates and discussing who paid for the ice cream, I meant, who is responsible for providing time to learn a new technology or fixing this or that.

Kramers may think that they just fulfill some orders, or that they’ve been given some kind of a power over others, but the question which everyone should answer is how is Billy.

Enriching your events with important metadata

When considering the application of event sourcing it’s quite common to allow a common part for all the events, the metadata. Various stores handle it in separate but common ways. EventStore lets you append the metadata with events. The same you can do with NEventStore using headers. But what information can be useful to store in the metadata, which info is worth to store despite the fact that it was not captured in the creation of the model?

Audit data

The most common case considered by various lists and blog posts are audit data. This set of data can be described as:

  1. who? – simply store the user id of the action invoker
  2. when? – the timestamp of the action and the event(s)
  3. why? – the serialized intent/action of the actor

That’s an obvious choice and you can easily find examples filled with repartitioning by the who or gather event in a time frame or window as it’s done in the complex event processing. But is there something more one could store? Is it there any particular set of additional dimensions that are worth to remember?

Important metadata

The event sourcing deals with the effect of the actions. An action executed on a state results in an action according to the current implementation. Wait. The current implementation? Yes, the implementation of your aggregate can change and it will either because of bug fixing or introducing new features. Wouldn’t it be nice if the version, like a commit id (SHA1 for gitters) or a semantic version could be stored with the event as well? Imagine that you published a broken version and your business sold 100 tickets before fixing a bug. It’d be nice to be able which events were created on the basis of the broken implementation. Having this knowledge you can easily compensate transactions performed by the broken implementation.

It’s quite common to introduce canary releases, feature toggling and A/B tests for users. With automated deployment and small code enhancement all of the mentioned approaches are feasible to have on a project board. If you consider the toggles or different implementation coexisting in the very same moment, storing the version only may be not enough. How about adding information which features were applied for the action? Just create a simple set of features enabled, or map feature-status and add it to the event as well. Having this and the command, it’s easy to repeat the process. Additionally, it’s easy to result in your A/B experiments. Just run the scan for events with A enabled and another for the B ones.

Optimization (when needed)

If you think that this is too much, create a lookup for sets of versions x features. It’s not that big and is repeatable across many users, hence you can easily optimize storing the set elsewhere, under a reference key. You can serialize this map and calculate SHA1, put the values in a map (a table will do as well) and use identifiers to put them in the event. There’s plenty of options to shift the load either to the query (lookups) or to the storage (store everything as named metadata).

Summing up

If you create an event sourced architecture, consider adding the temporal dimension (version) and a bit of configuration to the metadata. Once you have it, it’s much easier to reason about the sources of your events and introduce tooling like compensation. There’s no such thing like too much data, is there?

Feature oriented design got wrong

The fourth link in my google search for ‘feature toggle’ is a link to this Building Real Software post. It’s about not about feature toggles described by Martin Folwer. It’s about feature toggles got wrong.

If you consider toggling features with flags and apply it literally, what you get is a lot of branching. That’s all. Some tests should be written twice to handle a positive and a negative scenario for the branch. The reason for this is a design not prepared to handle toggling properly. In the majority of cases, it’s a design which is not feature-based on its own.

The featured based design is created on the basis of closed components, which handle the given domain aspect. Some of them may be big like ‘basket’, some may be much smaller, like ‘notifications’ reacting to various changes and displaying needed information. The important thing is to design the features as closed components. Once you have it done this way, it’s easier to think about the page without notifications or ads. Again, disabling the feature is not a mere flag thrown in different pieces of code. It’s disabling or replacing the whole feature.

One of my favorite architecture styles, event driven architecture helps in a great manner to build this kind of toggles. It’s quite easy to simply… not handle the event at all. If you consider the notifications, if they are disabled, they simply do not react to various events like ‘order-processed’, etc. The separate story is to not create cycles of dependencies, but still, if you consider the reactive nature of connections between features, that’s a great enabler for introducing toggling with all of advantages one can derive from it with A/B tests, canary releases in mind.

I’m not a fan boy of feature toggling, I consider it as an important tool in architects arsenal though.


The cost of scan queries in Azure Table Storage

There are multiple articles describing the performance of Azure Table Storage. You probably read the entry of Troy Hunt, Working with 154 million records on Azure Table Storage…. You may have invested your time in reading How to get most out of Windows Azure Tables as well. My question is have you really considered the limitations of the queries, specifically scan queries and how they can consume the major part of Azure Performance Targets.

The PartitionKey and RowKey create the primary and the only index in ATS (Azure Table Storage). Depending on the query the following kinds can be distinguished:

  1. Point Queries, which are queries to retrieve a single entity by specifying a single PartitionKey and RowKey using equality as predicate
  2. Row Range Queries, which  are queries to get a set of entities defined with the same PartitionKey and a range of RowKeys
  3. Partition Range Queries, which are run with a range of ParitionKeys
  4. Full table scans, which have no predicate for ParitionKey

What are the costs and limitations of the following queries? Unfortunately, every row that is accessed by the query to perform scan over will be counted as the table operation, Tthere ain’t no such thing as a free lunch. This means, that if you scan your entire table (4th scenario), you’ll be able to process no more than 20,000 entities per second. This limits the usage of large data sets’ scans. If you have to model queries across different keys, then you may consider storing the same value twice: once under the natural Parition/RowKey pair and the second time to match the other index, to create an inverted index. If any case, you’ll have to scan through the entire data set, then using ATS is not the way to go, and you should consider some other ways of modelling your data, like asynchronous copy data to blob, etc.

Lokad.CQRS Retrospective

In the recent post Rinat Abdullin provides a retrospective for Lokad.CQRS framework which was/is a starting point for many CQRS journeys. It’s worth to mention that Rinat is the author of this library. The whole article may sound a bit harsh, it provides a great retrospection from the author’s and user’s point of view though.

I agree with the majority points of this post. The library provided abstractions allowing to change the storage engine, but the directions taken were very limiting. The tooling for messages, ddd console, was the thing at the beginning, but after spending a few days with it, I didn’t use it anyway. The library encouraged to use one-way messaging all the way down, to separate every piece. Today, when CQRS mailing lists are filled with messages like ‘you don’t have to use queues all the time’ and CQRS people are much more aware of the ability to handle the requests synchronously it’d be easier to give some directions.

The author finishes with

So, Lokad.CQRS was a big mistake of mine. I’m really sorry if you were affected by it in a bad way.

Hopefully, this recollection of my mistakes either provided you with some insights or simply entertained.

which I totally disagree with! Lokad.CQRS was the tool that shaped thinking of many people, when nothing like that was available on the market. Personally, it helped me to build a event-driven project (you can see the presentation about this here) based on somehow on Lokad.CQRS but with other abstractions and targeted at very good performance, not to mention living documentation built with Mono.Cecil.


Lokad.CQRS was a ground breaking library providing a bit too much tooling and abstracting too many things. I’m really glad if it helped you to learn about CQRS as it helped me.  Without this, I wouldn’t ask all the questions and wouldn’t learn so much.

The provided retrospective is invaluable and brings a lot of insights. I’m wishing you all to make that kind of ground breaking mistakes someday.