Reconcilation between systems

From time to time a system is replaced with another system being capable of doing more, or doing the thing better. It’s quite to common to ask whether no data is lost or does the system preserve needed behaviors of the old one. Sometimes it’s human-application comparison, when a procedure followed by people is replaced with an application, sometimes it’s a question of an old system vs a new system

Cassandra integrity check
Let’s presume one migrates some old-fashioned SQL system to a Cassandra based solution given following:

  1. total payload of daily data is quite high
  2. data are written to the Cassandra cluster (more than one node) with ConsistencyLevel Local Quorum
  3. there should be a possibility to check whether all the data stored in the previous systems are written to the new one

After a bit of consideration one can propose that as the data are written with LocalQuorum, they should be queried with the same level and match in the old solution. This would ensure that data which has been written are being read (famous R + W > N). This could cost a lot as querying hits [N+1]/2 nodes of your cluster, streaming a daily payload through network twice: once to the coordinator, second – to the client. Can we do this better?

Possibly faster integrity check
How about using Consistency Level of One? How can this be done to ensure that the given node consists of all the needed data? By running repair in your local data center on each node, one can ensure that each node consist of all the data it’s responsible for. Then, querying with One is ok. What’s important about nodetool repair is that it does not stream data if it’s not needed. The information sent to match if the given node contains all the data is a Merkle tree, a tree made by hash of hashes of hashes of… Sending this structure is cheap and doesn’t your network so much.
If you consider (know that) running repairs daily is a heavy task for your cluster, you’ll be happy to read about Cassandra 2.1 repair improvements, including incremental repairs.

So stop complaining about your good old fashioned RMDB and get yourself a new shiny cluster of Cassandra nodes :)

Multidatacenter Cassandra cluster with slow cross DC connection

I’d like to discuss a particular failure scenario for a multi datacenter Cassandra cluster.
The setup to reproduce is following:

  • Two Cassandra data centers
    • DC1: n nodes
    • DC2: m nodes
  • TestKeyspace
  • NetworkTopologyStrategy with replication factors:
    • DC1: n (each key on each node)
    • DC2: m (each key on each node)
  • Tables in TestKeyspace are created with default settings
  • hinted hand-off enabled
  • read repair enabled

The writes and reads goes to the DC1. What can go wrong when whole DC2 goes down (or you get a network split)?
It occurs that read_repair is defined not by one but two probabilities:

What’s the difference between them? The first one shows probability of read repair across whole cluster, the second – rr across the same DC. If you have an occasionally failing connection, or a slow one using the first can bring you some troubles. If you plan for multi DC cluster and you can live with periodical runs nodetool repair instead of failing some of your LOCAL_QUORUM reads from time to time, switch to dc read repair and disable the global one.

For curious readers the class responsible for performing reads with read-repairs as well is AbstractReadExecutor

Simple Cassandra backups

Cassandra is one of the most interesting NoSQL databases which resolves plenty of complex problems with extremely simple solutions. They are not the easiest options, but can be deduced from this db foundations.
Cassandra uses Sorted String Tables as its store for rows values. When queried, it simply finds the value offset with the index file and searched the data file for this offset. New files are flushed once in a while to disc and a new memory representation of SST is started again. The files, once stored on disc are no longer modified (the compactation is another scenario). How would you backup them? Here comes the simplicity and elegance of this solution. Cassandra stores hard links to each SST flushed from memory in a special directory. Hard links preserves removing of a file system inodes, allowing to backup your data to another media. Once once backup them, they can be removed and it’d be the file system responsibility to count whether it was the last hard link and all the inodes can be set free. Having your data written once into not modified files gives you this power and provides great simplicity. That’s one of the reasons I like Cassandra’s design so much.

The docs for Cassandra backups are available here.

Deiphobus, no more SELECT n + 1

The previous post contained an information about lazy loading of group of properties, let’s call them families as it is called in the Cassandra. What about the following code. How many db hits you’d like to get by default?

using (var s = sessionFactory.Open())
	var user = s.Load<IUser>(5);
	foreach(var post in user.Posts)

I’ll tell you how many you’ll get. The answer is two: first hit will occur, when a collection of posts is accessed in the foreach loop, the second – when a title is printed on the console. During the second hit all the posts loaded in the session will have their titles loaded. In some cases it may drive to a small overhead, but it simplifies batching and working with your entities in the majority of cases. Would anyone like to set FetchMode, like it was done in the NHibernate? ;)

Deiphobus, mapping properties groups to column families

As i it is stated in the official documentation, the column family can be compared to a table in the relational database and as with table, the main target of creating one is to hold the same type of objects together to create a better model, allow querying, etc. Speaking about Cassandra it has one more advantage: the whole column family is stored on one server (the data are consistently hashed by key), so you may consider a column family as a collection of items frequently used together, for instance: the surname and the name, or the user name and the password.
In NHibernate you can embed those values within component to make it look like a whole, but they’re still stored in the same table. Speaking about Deiphobus, it can be easily configured to match the needs of grouping properties (simple properties, and referencing other entities in many-to-one or one-to-one way) by implementing a convention interface:

public interface IPropertyFamilyConvention
    /// <summary>
    /// Gets the Cassandra family name.
    /// </summary>
    /// <param name="mappedType">The mapped type.</param>
    /// <param name="propertyInfo">The info of a mapped property.</param>
    /// <returns>The family name.</returns>
    FamilyName GetFamilyName(Type mappedType, PropertyInfo propertyInfo);

It’s worth to mention, that by default all the properties are mapped to one column family called ‘Entity’. Ok, you know how it influences the storage from the Cassandra point of view, but what about Deiphobus? As it was implemented, every time you access previously not loaded property of an entity mapped with Deiphobus, the whole column family, containing the specified property is loaded from the database (actually a bit more data is retrieved, but for the sake of simplicity it can be omitted in here). It means, that once you started using a property which is strongly connected with others, all the needed properties will be loaded in one db hit. Simple and powerful, isn’t it?

Deiphobus, lazy load

Last time, when an identity map of the Deiphobus was described, one user entity of was asked about several times. We already know, that the same object was returned, but what about hitting the Cassandra DB? Consider the following code:

var user = session.Load<IUser>(5);
// some other loads and operations
var model = user.Login; // here goes db hit!

As you can see, the database is not hit till one of the properties is queried. It’s default and only mode of loading entities with Deiphobus. Is allows a great reduction of db calls, if your code is structured in a right way (query for data first, then operate). The question is, what if a user holds a few massive, in terms of bytes transported, properties. Will of them will be loaded at once, even if they’re unneeded? The answer will be revealed in a very next post.

Deiphobus, first level caching

Another feature of my mapper for Cassandra database, is the first level cache, commonly known as the identity map. Consider loading an entity of a user type only for displaying its login in the page header. Next, you’d like to get the same property to display it in several places across the page (for instance an author of a posts). I assume, that you use DI and your app creates only one session object per web request, injecting it in all the needed places. Consider the following code:

var user1 = session.Load<IUser>(5);
var user2 = session.Load<IUser>(5);
var user3 = session.Load<IUser>(5);

Now try to ReferenceEquals all the returned instances. Yes, that’s the same object! It means that there will be no problems with dirty tracking (what instance should I persist?). Also once an user is loaded with a session object, it can be re-retrieved without db hits. Isn’t it nice?