Podsumowanie roku 2016

Koniec roku to dobry czas na podsumowanie.

Particular Software

Rok 2016 to rok spędzony z firmą Particular Software. To pierwsza firma, dla której pracowałem całkowicie zdalnie. Rozróżnijmy od razu dwa rodzaje pracy zdalnej: remote i distributed. Praca zdalna się zdarza w różnych firmach. Rozproszona natomiast, gdzie zdalny pracownik nie musi borykać się z synchronizacją i prosić o rozmowy przy kamerze a nie przy kawie, to coś zupełnie innego. Ten rok minął pozytywnie właśnie w pracy rozproszonej.

Ramp Up

Rok 2016 to także wielkie otwarcie jak i zamknięcie projektu “drogi” RampUp. Nauczyłem się przy nim bardzo dużo. O ile wcześniej myślałem, że rozumiałem jak działa pamięć oraz procesor to teraz już to wiem. Podobnie to, co mogę im zrobić z poziomu .NET. Było dużo pracy ze wskaźnikami i usuwaniem chociażby najmniejszych alokacji. Był IL Emit. To był ciekawy projekt, który rozwinął i ugruntował mechaniczną sympatię, którą pałałem od dawna.

Community

To mój najaktywniejszy rok jeżeli chodzi o community. Średnio raz w miesiącu występowałem ze swoją prezentacją. Czy na grupie, czy na konferencji, działo się dużo. Ciekawym przeżyciem było wzięcie udziału po raz pierwszy w konferencji wirtualnej DotNetConfPL. Brak kontaktu wzrokowego ze słuchaczem, to coś czego doświadczyłem po raz pierwszy. Było warto, oceny pokazują, że było dobrze (mediana 5/5).

Open Source

Poza własnym RampUpem, zaktywizowałem się w Open Sourcie. Nie chodziło mi o dużą liczbę małych kontrybucji, a raczej skupienie się na zaaplikowaniu wyżej wspomaniej mechanicznej sympatii. Tak też stałem się drugim kontrybutorem Wire, nowego serializatora AKKA.NET. Wiem, to tylko 8 commitów, ale zachęcam do porównania performance’u przed i po. Tony alokacji, boxowania, branchy i metod wirtualnych zostały wyeliminowane.

Dodatkowo sama Akka.NET (szkice kolejek i duże dyskusje na ten temat), wtyczka do Fody i PR do EventStore’a usuwający alokacje z podstawowego kompoenentu tej bazy – QueuedHandler i przyśpieszający go o 50%.

Social media

Koniec roku to większa aktywność w social media i zmiany w formacie bloga. Jest obrazek nawiązujący do treści, lżejszy layout i mniejsze akapity. Pojawił się też TL;DR dla tych, co lubią przeczytać o czym będą czytać 😉

Slack devspl

Ostatnim medium, w którym zacząłem się udzielać jest slackowy kanał założony przez Maćka Aniserowicza devspl. Słyszałem o nim wcześniej, wszedłem przy DotNetConfPL i jakoś zostałem. Można mnie spotkać co najmniej kilka razy w tygodniu. Maciek, szacun za to.

Podsumowanie

To był świetny rok. Kolejny zapowiada się już równie ciekawie, co oznacza, że będzie jeszcze lepiej. Udanego wypoczynku i do zobaczenia w kolejnym!

Year 2016 in .NET ecosystem

TL;DR

This was a truly amazing year for .NET ecosystem. Let’s go through some of the important events in our ecosystem.

Core & Standard

.NET Core and Standard moved closer to the mainstream development. There are applications running Core in production already. The Standard addresses the fragile point of the cross platform solutions – APIs which you can use. It had some versioning problems, but they seem to be solved for now.

Kestrel

The ASP team with Ben A Adams are working really hard! ASP .NET Core, on Linux, is 10th in the ranking of the plain text speed. Let me say it again: Linux, .NET Core, 10th place. And it will be faster for sure.

Slices, memory pools and more

.NET Core and .NET at all are getting more and more attention in terms of performance. Slices (previously spans) which can be seen as a better alternative to ArraySegment abstracting access to an allocated collection of values (an array, an unsafe pointer-based memory segment). Memory pools and UTF8 strings are being brought to the mainstream. This won’t be used in every project, but having these foundations is much easier to think about .NET as a platform for writing well performing systems, databases etc.

Benchmarking

BenchmarkingDotNet isn’t a library used only occasionally. Many projects embed tests based on it as a part of their tests suites. You can see the benchmarking output table in many PRs which is really good. I’ve heard Knuth’s phrase about performance much to often. It’s good to see that performance is getting attention. You don’t have to benchmark everything, but as a software engineer you should be aware of the tooling and apply it when needed.

Asyncing

Async await is globally accepted. The understanding of it seems to be better. Hopefully, we’ll see less of the sync/async battles and new APIs will follow only one of these.

Summary

It was a really good year. Let’s push it even further in the next one!

Service kata with Business Rules

TL;DR

In the previous post we started working on a code kata and discovered that instead of creating a new monolithic giant we could tackle the complexity of a process by modelling it right in its natural boundaries: contexts. It this post we continue this journey.

Requesting payment

Let’s spend some time on modelling a process of ordering a membership. It’s been said that it requires a payment and as soon as the payment is done, the membership is activated. We introduced the PaymentReceived event as an asynchronous response to the payment request.

Consider a membership order with the following identifier

11112222-3333-4444-5555-666677778888

When accepting the request for a membership, Membership sends a request to the Payments with the following information

payment_request

It is important to see, that the caller generates identifier, which has following properties:

  • In this case it reuses it’s own identifier for a different context to use snowy identifiers to create snowflake entities
  • As the caller generates id and stores it, in case of the failure when requesting a payment, it can be POSTed again as it’s idempotent (any http status indicating that it already exists means that the previous call was accepted and processed).

Using approach in a service oriented architecture enables idempotence (everyone knows the id upfront).

Events strike back

The result of the payment, after receiving money is an event PaymentReceived which is published to all the interested parties. One of them will be Membership, which would simply take the paymentId and check if there’s an order for a membership with the same identifier. It there is, it can be checked as paid. Simple and easy. The same will apply to other rules in other contexts.

There’s really no point of making this ONE BIG APP TO RULE THEM ALL. You can separate services according to business units and design towards integrating them.

Again, depending on the used tools, you can have events delivered by a bus to all the subscribers all use ATOM feed to publish events in there, and consume them by polling from other services.

Summary

These two posts show that raising modelling questions is important and that it can help to reuse existing structures and applications in creating new robust systems. They do not cover transactions, retries and more. You can use tools that solve it for you like a messaging bus or you’ll need to handle it on your own. Whatever path you choose, the modelling techniques will be generally the same and you can use them to bring real value into the existing ecosystem instead of creating the single new shiny application that will rule them all.

 

Code kata with Business Rules

TL;DR

How many times you were given an implicit requirements that you’d create one application or two services? How many times the architecture and design were predetermined before any modelling with business stake holders? Let’s take a dive into a code kata, that will reveal much more than code.

Kata

The kata we’ll be working on is presented here. It covers writing a tool for a set of business rules gathered across the whole company. The business rules depends on the payment (the fact that it is done) and some other conditions, for example:

  • If the payment is for a physical product, generate a packing slip for shipping.
  • If the payment is for a book, create a duplicate packing slip for the royalty department.
  • If the payment is for a membership, activate that membership.

The starting point for every rule is a payment that is accepted. Another observation is that these rules are scattered across the whole company (as the author mentions Carol on the second floor handles that kind of order). Do you think that having a new single application that gathers all the rules is the way to go?

Contexts

If a mythical Carol is responsible for some part of the rules, maybe another department/team is responsible for membership? What about the payments? Is the bookstore part of your organization really interested if a payment was done with a credit card or a transfer? Is a membership rule really valid outside of the membership context? Is it needed to anyone without membership specific knowledge should be able to say when the membership is activated?

I hope you see the way through these questions. There are multiple contexts that are somehow dependent but that are not a truly one:

  • payments – a part responsible for accepting money and making them transferred to the company’s account
  • membership – taking care of (possibly) accounts, monitoring activity, activating/disactivating accounts
  • bookstore/videostore or simply store – the sales part
  • shipping – for physical products

Are these areas connected? Of course they are!

PaymentReceived

The first visible connector is the payment. To be precise, the fact of receiving it, which can be described in a passive tense PaymentReceived. You can imagine, when requesting a membership, a payment is required. This can be perceived as a whole process, but could be split into following phases:

  • gathering membership data
  • requesting a payment
  • receiving a payment
  • completing the membership order

This is the Membership point of view. As you can see it requests a payment but does not handle it. We will see in the next post how it can be solved.

 

Why persistent memory will change your world

TL;DR

If you haven’t heard, the non volatile RAM memory is coming to town and for sure will change the persistence patterns of databases, queues, loggers. Want to know more about this new wave of hardware? Read along.

API

The first and the most important aspect is that persistent memory on Windows, reuses already existing APIs. First, if you want to use the drive just as a block storage, you can do it. You’ll be able to create files, write them etc. There’s another way of using it, which is much faster, called DAX.

The direct access enables to use the non volatile memory directly. What do I mean by directly? I mean accessing the memory with a raw pointer. How do you obtain a pointer? The old fashioned memory mapped file API is used. First, create a file, then map it and here it is! No FlushFileBuffers, no fsync. Just a raw pointer to the memory. Can you imagine writing to a mapped file and just having it persisted?

Speed

The non volatile memory is really fast. You can write 4GB per second. Yes, it’s 4GB per second of a persistent memory. The latency is extremely low. It’s so low, that using any form of asynchronous programming (raw completion ports, async-await) brings more havoc than just waiting for having this memory written. Yes, this means that your methods hitting files mem mapped with DAX will not need async signatures. Of course you’ll be able to preserve them just for the compatibility.

Ordering matters

It looks like tech heaven. No more data loss during power outages, right? It’s not entirely true. The persistent memory acts as a memory. There is an order of execution in which data are transported there. Imagine now writing the following string:

BLAH

If power went down after copying first three letters you’d be left with

BLA

Which although shows the same attitude, is not what we wish for when thinking about persistence. This example shows, that good old fashioned IO access patterns will still be important, like write-ahead logging or copy-on-write. But let me remind you again, they will be free with no synchronization, no flushing required.

Adopters

The persistent memory will change the world. SQL Server 2016 has already adopted it, as you can see here. Some databases are already there, like LMDB using memory mapped files (same API for the win!) with an ability to run as non-durable. Guess what. Now it’s durable. More databases will follow.

Summary

The persistent memory is here. You won’t probably rewrite or rethink your application as majority of apps do not deal with IO directly, but just by applying it to your database or other IO bound systems will be a real game changer.

Snowy identifiers

TL;DR

When using the snowflake entities pattern, it’s quite easy to forget about using external identifiers that we need to communicate with external systems. This post provides an easy way to address this concern.

Identity revisited

The identifier of a snowflake entity was presented as a guid. We use an artificial non-colliding client-generated identifier to ensure, that any part of the system can generate one without validating that a specific value hasn’t been used before. This enables storing different pieces of data, belonging to different contexts in different services of our system. No system leaves in vacuum though, and sometimes it requires communication with the rest of the world.

Gate away!

A common aspect that is handled by an external system are payments. When you consider credit cards, native bank applications, PayPay, BitCoin and all the rest, providing that kind of a service on your own is not a reasonable option. That’s why external services are used – the price of using one is much cheaper than delivering one. Let’s stick to the payments example. How would you approach this? Would you call the external payment service from each of your services? I hope you’d not. A better approach is to create a gateway, that will act as a translator between your system and the external one.

How many ids do I need?

Using a gateway provides a really interesting property. As the payment gateway is a part of your system, it can use the snowflake identifier. In other words, if there’s an order, it’s ok (under given circumstances) to use its identifier as identifier of the payment as well. Of course if you want to model these two as a part of a snowflake entity spanning across services. It’d be the payment gateway responsibility to correlate the system snowflake identifier with the external system id (integer, some string, whatever). This would create a coherent view of an entity within your system boundaries, closing the mapping in a small dedicated area of the payment gateway.

An integration with an external system closed in a small component leaving your system agnostic to this? Do we need more?

Summary

As you can see, closing the external dependency as a gateway provides value not only by separating the interface of the external provider from your system components, but also preserves a coherent (but distributed) view of your entities.

Snowflake entities

TL;DR

Much to often we incorporate the fallacy of grasping it all the next time. We say, that now, after absorbing some knowledge we will KNOW it all and design the system THE RIGHT way. Unfortunately, even using good methods like the strangler pattern, it’s simply impossible to design in a right way that covers the WHOLE domain of a company. Are there any design or architectural patterns that can be helpful? It there a way to make the uncertainty play by our rules?

Identity

The most important part of any design is an identity. I don’t mean the identity from the identity management point of view. I mean the identity that makes the entities, aggregate roots different, the good old-fashioned Id. It’s quite to common to find entities’ properties pointing to another context/domain. A User will have an employee’s identifier, a car in an insurance company will be referenced in accounting and other contexts. Basically, it will spread its carId across the whole company. Have you ever encountered ThisExternalId in your system? I bet you have. It’s time to end this.

One thing, multiple views

It’s not a car that is referenced in the accounting. It’s not a user that is referencing an employee. It’s the same entity spread across different contexts. Let me give you an example.

The same entity, depending on the context, can have a different meaning. A car in an insurance company will be seen in different dimensions. In some contexts it won’t occur at all (mortgage insurance), in others, it will be present. What’s the MAIN context? Can one tell what is it? Again, I bet noone can.

It’s time to end this referential wars. A car is not referenced by this or that context. None of them requires to have a car identifier as a foreign key. Everyone requires to have an id which can have the SAME VALUE in different context. Why? Because the same thing will be understood differently in different contexts! When signing an insurance, it will be just a thing. When paying for an accident it will be another, but after all, it’s the same car.

The same thing can be seen in many contexts. In each of them, it will have a separate set of properties that is unique, but will share one and only one property – the identity. The identity will be probably artificial and uniquely generated (you don’t want to have duplicates, do you?). A perfect match for that would be GUID or UUID.

Snowflakes

When asking what a car is, you could imagine asking all the contexts for the same identifier. Getting an insurance info

GET /insurances/38e5c55b-1b44-4bdc-bd9e-632580736f22

Getting the mailing info

GET /mailing/38e5c55b-1b44-4bdc-bd9e-632580736f22

And so on and so forth. Getting an empty response from a context means nothing but the fact, that an entity does not exist in a specific context. You can see this as a snowflake, which consists of the center which is an identifier, and arms which are responses from different contexts. None of the contexts creates the entity on its own, but every single one contains a meaningful information about the snowflake.

Further modelling

It’s quite easy to follow this rule and imagine adding another context to a system like this. You don’t have to add foreign keys or make changes to other components. You just map a new context with the same bluntly simple rule: share just the identity. After all, if another context wants to handle this entity in some way, it will just use this single meaningful property of an entity, the identity of the snowflake.

I hope that this article shed some light on this modelling technique and showed itself as a solid and an extensible approach towards modelling your domain.