Today is about handling events real time. I’ve been talking to different people about events handling and the outcome of these conversations is that you get a different opinion based on who you’re talking to. It’s hard to get some consensus on what to choose. Imho it’s too easy to say that A is better than B without considering all aspects of the solution.
Questions that you should ask yourself before jumping on one technology:
Remember to consider both internal/external man-hours, cost and time line. Be honest with yourself. Listen to people who have done this before and check different sources. Never trust just one blog post.
Twitter is handling millions of events every second. It would be interesting to know who many servers and ops people it takes to run their platform.
Handling five billion sessions a day – in real time
// blog.twitter.com - Thanks to @joawan for the tip.
LinkedId is also handling a couple of events every second. The say they have 8 million incoming messages a second and 32 million outgoing messages a second.
Kafka at LinkedIn: Current and Future
// engineering.linkedin.com - Thanks to @eide for the tip.
Both Twitter and LinkedIn are using more or less self managed services for their real time data pipeline. Is this the best solution?
Recently Ebay entered the game with their own solution called Pulsar. They’ve launched it in production and it is currently handling all user behavior events.
Pulsar can be used to collect and process user and business events in real time, providing key insights and enabling systems to react to user activities within seconds.
Announcing Pulsar: Real-time Analytics at Scale
Amazon has been in the game since December 2013 with their managed service Kinesis.
Amazon Kinesis is a fully managed, cloud-based service for real-time processing of large, distributed data streams.
When you make a mistake remember that it’s not the end of the world. There is always someone who is in deeper shit than you.
Knightmare: A DevOps Cautionary Tale
And when everything seems a bit dark you should always take care of yourself and drink more coffee.
This week ends with a couple of fun links.
Language Perl (bottled by Acme::EyeDrops)
Have a nice weekend!