I was just fighting this same situation. I never expected the new producer send() method to block as it returns a Future and accepts a Callback. However, when I tried my unit test, just replacing the old producer with the new, I immediately started getting timeouts waiting for metadata. I struggled with this until I went into the source code and found the wait() that waits for the metadata.
At that point I realized that this new "async" producer would have to be executed on its own thread, unlike the old producer, which complicates my code unnecessarily. I totally agree with Paul that the contract of send() is being completely violated with internal code that can block. I did try fetching the metadata first, but that only worked for a few calls before the producer decided it was time to update the metadata again. Again, I agree with Paul that this API should be fixed so that it is truly asynchronous in all cases. Otherwise, it cannot be used on the main thread of an application as it will block and fail.
