I wish to use either the ConsumerConnector or the SimpleConsumer to read messages from all partitions across multiple brokers using a fixed number of threads (hopefully leveraging asynchronous IO for high performance).
I know that the ConsumerConnector sounds like this, but the documentation was not clear about a few things. Would somebody who has used it (or knows the code) be willing to answer these questions about it: - Does the ConsumerConnector manage connections to multiple brokers, or just a single broker? - Does the ConsumerConnector require a thread for each partition on each broker? (If not, how many threads does it require?) - Does the ConsumerConnector use actual asynchronous IO, or does it mimic it by using a dedicated behind-the-scenes thread (and the traditional java socket API)? If the ConsumerConnector won't serve my purpose I think I could use the SimpleConsumer to implement a version of this, but I worry that it's not thread safe; That is, can I use the same SimpleConsumer instance on Thread-A, then on Thread-B, and then again on Thread-A (though never at the same time)? Thanks in advance! --Tom
