Recently there has been some interesting discussion on the Mechanical Sympathy mailing list about Martin Thompson’s new extremely high performance, extremely low latency, open source messaging system called Aeron. For those of you that don’t know, Martin Thompson is an expert on wringing as much throughput and as little latency as possible out of hardware for financial firms so his background made me very interested in this.
The high level gist is that Aeron has publishers that write messages to a partitioned, replicated, immutable log, and subscribers that are delivered messages from the immutable log using message offsets. Sound familiar? There are a few differences (https://groups.google.com/d/msg/mechanical-sympathy/fr7moLcsuiI/ZfUMk4QUbN8J), the key ones being that there are no brokers in this system, and that Aeron is a pure OSI layer 4 implementation. At first it seems like Aeron is a competing product to Kafka, and indeed for some cases it could be competing. However because it is implemented as a layer 4 transport, after thinking about it for a while, I think it could be implemented into Kafka as an alternative to the TCP transport that Kafka uses. The semantics would need to be thought through, but I think you could implement the brokered model on top of this, as well as supporting point to point subscriptions bypassing the brokers. I wanted to bring this to the groups attention and see what others thought. Aeron is still pretty new, and there’s a lot to digest in this so I wouldn’t recommend trying to land it in 0.8.2 :), but I thought it’s worth considering it for future development. The Youtube video where he talks about it is at https://www.youtube.com/watch?v=tM4YskS94b0 and the GitHub repo is at https://github.com/real-logic/Aeron. Discussion is spread across a few threads but starts here: https://groups.google.com/d/msg/mechanical-sympathy/fr7moLcsuiI/04unxUfIJ8sJ and https://groups.google.com/forum/#!topic/mechanical-sympathy/GrEce0gP7RU. Daniel.
