Also, make On Wed, Jul 19, 2017 at 5:10 AM, Ismael Juma <ism...@juma.me.uk> wrote:
> Hi Anish, > > What should happen in that case is that the broker down converts the > messages to a format supported by the older clients. We have tests for this > scenario so it would be good to understand what is different in your > environment. Please file a JIRA and provide as much information as possible. > > Ismael > > On Wed, Jul 19, 2017 at 12:39 AM, Anish Mashankar < > an...@systeminsights.com> wrote: > >> After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol >> to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting >> Invalid message error logs. >> >> On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma <ism...@juma.me.uk> wrote: >> >> > Hi all, >> > >> > 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have >> > system tests for all the relevant combinations. One thing to be careful >> > about is that Scala consumers and Java consumers store offsets and group >> > management information differently and the Java consumer was only >> > introduced in 0.9.0.0. >> > >> > Ismael >> > >> > On Tue, Jul 18, 2017 at 5:18 AM, John Yost <hokiege...@gmail.com> >> wrote: >> > >> > > Hi Everyone, >> > > >> > > I personally found that the 0.8.x clients do not work with 0.10.0. We >> > > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and >> > then >> > > Kafka produce/consume worked fine. >> > > >> > > --John >> > > >> > > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sjmit...@gmail.com> >> > wrote: >> > > >> > > > OK. >> > > > >> > > > Just a doubt I have is that my broker is 0.10.2 and producer is >> also of >> > > the >> > > > same version and writes to a topic. >> > > > Then I have a client from version 0.8.2 trying to fetch these >> messages >> > > and >> > > > what I see is that all messages are getting dropped. >> > > > >> > > > However since this is an older client I tried to fetch its offsets >> > using: >> > > > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper >> > > > localhost:2181 --group test >> > > > >> > > > I saw that lag is 0. >> > > > >> > > > So probably better question would be can 0.8.2 read a message >> written >> > by >> > > > 0.10.2 >> > > > >> > > > Thanks >> > > > Sachin >> > > > >> > > > >> > > > >> > > > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tcrayf...@heroku.com >> > >> > > > wrote: >> > > > >> > > > > All broker versions support all older client versions >> > > > > >> > > > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal < >> sjmit...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > Hi, >> > > > > > This gives me some information but still not the complete >> picture. >> > > > > > >> > > > > > It says: >> > > > > > 0.10.2, Java clients have acquired the ability to communicate >> with >> > > > older >> > > > > > brokers. >> > > > > > >> > > > > > It also says >> > > > > > Version 0.11.0 brokers support 0.8.x and newer clients >> > > > > > >> > > > > > Question is does 0.10.2 broker support 0.8.x clients? >> > > > > > >> > > > > > This may solve answer my temporary doubt, but a compatibility >> > matrix >> > > > will >> > > > > > be much helpful to have a yes/no answer. >> > > > > > >> > > > > > Thanks >> > > > > > Sachin >> > > > > > >> > > > > > >> > > > > > >> > > > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki < >> > > > > > michal.borowie...@openbet.com> wrote: >> > > > > > >> > > > > >> Have you seen this: http://kafka.apache.org/ >> > > > documentation.html#upgrade >> > > > > >> >> > > > > >> Starting with version 0.10.2, Java clients (producer and >> consumer) >> > > > have >> > > > > >> acquired the ability to communicate with older brokers. Version >> > > 0.11.0 >> > > > > >> clients can talk to version 0.10.0 or newer brokers. However, >> if >> > > your >> > > > > >> brokers are older than 0.10.0, you must upgrade all the >> brokers in >> > > the >> > > > > >> Kafka cluster before upgrading your clients. Version 0.11.0 >> > brokers >> > > > > support >> > > > > >> 0.8.x and newer clients. >> > > > > >> >> > > > > >> Hope that helps. >> > > > > >> >> > > > > >> Cheers, >> > > > > >> >> > > > > >> Michał >> > > > > >> >> > > > > >> On 18/07/17 08:17, Sachin Mittal wrote: >> > > > > >> >> > > > > >> Hi, >> > > > > >> I would like some help/information on what client versions are >> > > > > compatible >> > > > > >> with what broker versions in kafka. >> > > > > >> >> > > > > >> Some table like this would be good >> > > > > >> >> > > > > >> server >> > > > > >> client 0.8 0.9 0.10 0.11 >> > > > > >> 0.8 yes ? ? ? >> > > > > >> 0.9 ? yes ? ? >> > > > > >> 0.10 ? ? yes ? >> > > > > >> 0.11 ? ? ? yes >> > > > > >> >> > > > > >> So if question marks are filled it would be of great help. >> > > > > >> >> > > > > >> Reason I am asking is many times we need to use other >> > > > > libraries/frameworks >> > > > > >> to pull/push data from/into kafka and sometimes these support >> > only a >> > > > > >> particular version of clients. >> > > > > >> >> > > > > >> Like right now I am trying to pull data from kafka via >> > > > druid/tranquility >> > > > > >> and they have clients of version 0.8.x implemented but my >> broker >> > is >> > > > > running >> > > > > >> 0.10.x. >> > > > > >> >> > > > > >> Also if such a table can be posted on kafka documentation page >> or >> > > > github >> > > > > >> page that would be great. >> > > > > >> >> > > > > >> Thanks >> > > > > >> Sachin >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> -- >> > > > > >> <http://www.openbet.com/> Michal Borowiecki >> > > > > >> Senior Software Engineer L4 >> > > > > >> T: +44 208 742 1600 <+44%2020%208742%201600> >> > <+44%2020%208742%201600> >> > > > > >> >> > > > > >> >> > > > > >> +44 203 249 8448 <+44%2020%203249%208448> >> > <+44%2020%203249%208448> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> E: michal.borowie...@openbet.com >> > > > > >> W: www.openbet.com >> > > > > >> OpenBet Ltd >> > > > > >> >> > > > > >> Chiswick Park Building 9 >> > > > > >> >> > > > > >> 566 Chiswick High Rd >> > > > > >> >> > > > > >> London >> > > > > >> >> > > > > >> W4 5XT >> > > > > >> >> > > > > >> UK >> > > > > >> <https://www.openbet.com/email_promo> >> > > > > >> This message is confidential and intended only for the >> addressee. >> > If >> > > > you >> > > > > >> have received this message in error, please immediately notify >> the >> > > > > >> postmas...@openbet.com and delete it from your system as well >> as >> > > any >> > > > > >> copies. The content of e-mails as well as traffic data may be >> > > > monitored >> > > > > by >> > > > > >> OpenBet for employment and security purposes. To protect the >> > > > environment >> > > > > >> please do not print this e-mail unless necessary. OpenBet Ltd. >> > > > > Registered >> > > > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, >> London, >> > W4 >> > > > > 5XT, >> > > > > >> United Kingdom. A company registered in England and Wales. >> > > Registered >> > > > > no. >> > > > > >> 3134634. VAT no. GB927523612 >> > > > > >> >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> -- >> >> Regards, >> Anish Samir Mashankar >> R&D Engineer >> System Insights >> +91-9789870733 >> > >
