Azure Functions: processing 2 billions items per day (1)

In this series I’ll describe a few patterns that enabled me to process 2 billions items per day using Azure Functions. Yes 2 billions items per day. The aim of this trial was not to check whether you can do it with Azure Functions. You can do it easily. The goal was to do it in a cost-aware and cost-wise manner, enabling fast processing with a small amount of money spent on this.

Initial phase

The start point was simple. To have a single queue, in my case Azure Storage Queue, and simply enqueue items to it, and run processing on a Consumption Plan. This looked pretty nicely. If you ever try Azure Functions you’ll see the ability to scale up instances when needed, just to make your workload processed in a timely manner.

I must admit that I skipped that part. When you calculate the number of operations that a single queue can handle, it won’t be enough to cope with 2 billions item per day. Yes, you can scale to multiple queues or use a different kind of queue. This was not the case for my experiment though.

It comes in batches

The important part that I intentionally didn’t mention, was the fact, that the numbers of items’ producers was limited. Also, they were able to batch items and flush them once in a while. With this assumption I was able to use a dense serialization protocol (big no no for JSON) and fill every single message that is being sent with hundreds, sometimes, thousands of items to get them processed.

In my case this lowered the number of messages greatly, by a factor of 1000, leaving the whole thing working as it was supposed to. Yes, the receiving part become a bit different as it was required to deserialize the densely packed payload properly.

You may ask why not Event Hubs? Being able to pack data on my own, being given the possibility of a delayed write and comparing prices for the scale I talk about, Azure Storage Queues with a properly selected serializer still won in my calculations.

Cheating Seeing opportunities

This was the first opportunity that I used to make the processing faster and cheaper. We saw that using a batch (smart-batching in this case) greatly lowered the number of moving pieces, still delivering the same value. In the following entry, we’ll move a bit deeper into the solution I built.

Unboxing yourself with books

20 books. That’s the number of books I challenged myself to read during 2017. One week ago I ended my challenge with a positive score 20 out of 20. 20 books per year? It’s not a small number. It wasn’t that easy to achieve either.

There was a tricky part in this challenge. Every book that I read was followed by a short review. Also, my goal was to read non-technical books. Not only non-technical, but also no novels, no poetry. This leaves things like economy, psychology, philosophy, presentation skills. Things that developers’ brotherhood is often afraid or not willing to invest their time in.

I must admit that this year provided me an experience that I could name as unboxing. Reading 20 books from areas much different than my profession or, for example. science fiction novels, really changed my perspective. Lets not dive into details now, but rather focus on the outcome of this experiment. I truly perceive some things in a different way now.

It’s your turn. If I can encourage you to anything, please settle a goal for 2018 and reach out for some non-technical, non-fiction, non-prose books.

Appendix

For sake of reference, these are books that I read in 2017.

Yes, I put one technical book in there as it was really good.

 

Pricing SaaS in the clouds

Why is it so pricey? This is a question that might have popped in your head too many times. Especially, when looking at the pricing pages of SaaS applications. The second that might have followed up, is Why? What’s behind this pricing model? How did they come up with it? Of course one answer could be I don’t care. They did it to get rich. With my money!, but this isn’t very constructive, is it? What would be the minimum price you’d charge for a single user, or a single account of your app? These are much better questions to ask.

Recently, I’ve been playing with Azure Functions. They provide this beautiful FaaS (Function as a Service) environment, where you pay for what you use. In a Consumption Plan, you don’t even pay for you app lying in there as long as nobody uses. Not paying for having no users is a good thing. Having users and paying something is a much better situation though. Imagine now, that you have your first account registered. Let’s put aside the cost of staff/work/development. How much money do you need to handle this single account. How would you estimate costs?

I think that using word estimation in this case, would be a really underestimation. It’s so easy to put a single Function App, with a single storage account and just run a synthetic workload. A single account for one month. Then, using Azure cost management, just to take a look at your bill. See the numbers. No guessing, no estimation, but real costs, real money. Now, with these numbers, you can go back to the pricing model and put something on top of it, just to make it work for you. And for clouds’ sake, remember to make it rain!

A missing image of a manager

You must have seen this meme. A group of people is pulling a stone. The first one, that is in the front of the group is labeled as a leader. There’s also another picture, showing a person sitting on the stone and doing nothing. This person is labeled as a boss. If you are a software engineer, this image probably resonates with you. In my opinion, this resonance, is a result of the confirmation bias kicking in, just proving that technical leadership is the only one that’s needed. In my opinion, there’s one more image that should have been added, but was omitted.

The missing picture came to my mind, when I was on my book reading quest, consuming First, Break All the Rules. The  book is based on a Gallup’s institute research, testing how people are being managed and how this changes the way they work. At the beginning, authors are dissecting the poll that they used for their tests. Also, they show a very important difference between the outward and the inward thinking. They describe a leader as a person that looks outward, pulling the line in a new direction, helping other to conquer new territories. It’s interesting that they don’t discuss the boss figure. They discuss the manager, looking inward.

The third picture that is missing is one showing a manager role. It’s not for looking outward, it’s for looking inward, at the people, at the team. Asking them about their goals, their needs and motivating them. I wrote manager role, as this is just a role. Maybe in your organization you’ll find people having two, or even three roles (startups, anyone?). Maybe, unfortunately for you, you’ll find none (complex organizations, gov related companies ruled by policies).

At the end, I’d like to ask you for one thing. Next time, when you see this extremely fitting or soothing presentation slide or meme, think again, why does it suit you? Maybe it’s just a confirmation kicking in? It’s popular to question authorities. Unfortunately for us, it’s still not popular to question self.

Software as Cheap Cr*p

‘You mean the cheap cr*p?’. This is what I heard, when I ask a colleague in London about a local souvenir shop. Unfortunately for me, fortunately for the cheap cr*p shop, I bought a not so beautiful Big Ben replica and brought it back home. My wife was not so happy. The only good thing was that I was able to buy 2 for 3 or 3 for 4. You know, this kind of deal.

When I bought The Cheap Cr*p, I felt that I signed a simple contract containing two points. The first part was easy, it was about being cheap. That’s it. The second was not that obvious. It was about being replaceable or ready to be thrown away.

The cheap cr*p situation recently reminded me of all these not so good code pieces that we all meet and endure. They probably were introduced as a quick fix, something, that is cheap in terms of time spent on the fix. They do not fulfill the other part of the cheap cr*p contract though. I could easily throw away my not so beautiful Big Ben. I did not use it to build foundations of my house. I do not use it as a oven knob. It’s just a piece on a small shelf, that sooner or later will be cleaned up.

Imagine that you’re under pressure, delivering your product on the deadline. You want to make this small change, ensure that your work fulfills the cheap cr*p contract. Remember it. It’s not only about being cheap. The first and foremost is about being easily replaceable and having a short expiration date. Otherwise, this cr*p won’t be so cheap at all.