Apache Kafka[1], the open source distributed messaging system, has steadily carved a foothold as the de facto real-time standard for brokering messages in scale out environments. Confluent[2], the company whose founders created Kafka, has just released their third annual report[3] on implementation. The report reached a much bigger sample, hinting at growth, while showing some modest changes in how Kafka is being used.
Gartner analyst Merv Adrian's[4] point that in Silicon Valley, if an idea is on a whiteboard, it must be commonplace could apply to Kafka. There are alternatives: MapR Streams[5] allows you to broker messages without requiring a separate Kafka cluster, while streaming services such as Amazon Kinesis Firehose[6] offer similar capabilities. Nonetheless, Kafka has become the de facto standard for highly distributed, high volume, real-time message queuing with wide vendor support. But when we reviewed Kafka a year ago[7], we found that the tooling was still primitive.
So it shouldn't be surprising that take-up is still largely the domain of early adopters. The survey sample, which doubled this year to 600 respondents, clearly skewed toward organizations that are ahead of the curve. Case in point? 78% of them are already using microservices architectures, and 63% of them are using Kafka to manage state with those microservices. In the general population, you won't find a majority of enterprises redesigning their application stacks to expose functionality as microservices.
So it shouldn't be surprising that the most represented sectors in the sample were the usual suspects for early adopters: computer systems, financial services, and media and entertainment.
Nonetheless, the data provides a useful glimpse on where first-generation Kafka implementation is headed. While 30% of the sample was in