Hi Anish, What should happen in that case is that the broker down converts the messages to a format supported by the older clients. We have tests for this scenario so it would be good to understand what is different in your environment. Please file a JIRA and provide as much information as possible.
Ismael On Wed, Jul 19, 2017 at 12:39 AM, Anish Mashankar <an...@systeminsights.com> wrote: > After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol > to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting > Invalid message error logs. > > On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi all, > > > > 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x brokers. We have > > system tests for all the relevant combinations. One thing to be careful > > about is that Scala consumers and Java consumers store offsets and group > > management information differently and the Java consumer was only > > introduced in 0.9.0.0. > > > > Ismael > > > > On Tue, Jul 18, 2017 at 5:18 AM, John Yost <hokiege...@gmail.com> wrote: > > > > > Hi Everyone, > > > > > > I personally found that the 0.8.x clients do not work with 0.10.0. We > > > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and > > then > > > Kafka produce/consume worked fine. > > > > > > --John > > > > > > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal <sjmit...@gmail.com> > > wrote: > > > > > > > OK. > > > > > > > > Just a doubt I have is that my broker is 0.10.2 and producer is also > of > > > the > > > > same version and writes to a topic. > > > > Then I have a client from version 0.8.2 trying to fetch these > messages > > > and > > > > what I see is that all messages are getting dropped. > > > > > > > > However since this is an older client I tried to fetch its offsets > > using: > > > > kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper > > > > localhost:2181 --group test > > > > > > > > I saw that lag is 0. > > > > > > > > So probably better question would be can 0.8.2 read a message written > > by > > > > 0.10.2 > > > > > > > > Thanks > > > > Sachin > > > > > > > > > > > > > > > > On Tue, Jul 18, 2017 at 3:26 PM, Tom Crayford <tcrayf...@heroku.com> > > > > wrote: > > > > > > > > > All broker versions support all older client versions > > > > > > > > > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal < > sjmit...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > This gives me some information but still not the complete > picture. > > > > > > > > > > > > It says: > > > > > > 0.10.2, Java clients have acquired the ability to communicate > with > > > > older > > > > > > brokers. > > > > > > > > > > > > It also says > > > > > > Version 0.11.0 brokers support 0.8.x and newer clients > > > > > > > > > > > > Question is does 0.10.2 broker support 0.8.x clients? > > > > > > > > > > > > This may solve answer my temporary doubt, but a compatibility > > matrix > > > > will > > > > > > be much helpful to have a yes/no answer. > > > > > > > > > > > > Thanks > > > > > > Sachin > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 18, 2017 at 2:38 PM, Michal Borowiecki < > > > > > > michal.borowie...@openbet.com> wrote: > > > > > > > > > > > >> Have you seen this: http://kafka.apache.org/ > > > > documentation.html#upgrade > > > > > >> > > > > > >> Starting with version 0.10.2, Java clients (producer and > consumer) > > > > have > > > > > >> acquired the ability to communicate with older brokers. Version > > > 0.11.0 > > > > > >> clients can talk to version 0.10.0 or newer brokers. However, if > > > your > > > > > >> brokers are older than 0.10.0, you must upgrade all the brokers > in > > > the > > > > > >> Kafka cluster before upgrading your clients. Version 0.11.0 > > brokers > > > > > support > > > > > >> 0.8.x and newer clients. > > > > > >> > > > > > >> Hope that helps. > > > > > >> > > > > > >> Cheers, > > > > > >> > > > > > >> Michał > > > > > >> > > > > > >> On 18/07/17 08:17, Sachin Mittal wrote: > > > > > >> > > > > > >> Hi, > > > > > >> I would like some help/information on what client versions are > > > > > compatible > > > > > >> with what broker versions in kafka. > > > > > >> > > > > > >> Some table like this would be good > > > > > >> > > > > > >> server > > > > > >> client 0.8 0.9 0.10 0.11 > > > > > >> 0.8 yes ? ? ? > > > > > >> 0.9 ? yes ? ? > > > > > >> 0.10 ? ? yes ? > > > > > >> 0.11 ? ? ? yes > > > > > >> > > > > > >> So if question marks are filled it would be of great help. > > > > > >> > > > > > >> Reason I am asking is many times we need to use other > > > > > libraries/frameworks > > > > > >> to pull/push data from/into kafka and sometimes these support > > only a > > > > > >> particular version of clients. > > > > > >> > > > > > >> Like right now I am trying to pull data from kafka via > > > > druid/tranquility > > > > > >> and they have clients of version 0.8.x implemented but my broker > > is > > > > > running > > > > > >> 0.10.x. > > > > > >> > > > > > >> Also if such a table can be posted on kafka documentation page > or > > > > github > > > > > >> page that would be great. > > > > > >> > > > > > >> Thanks > > > > > >> Sachin > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> <http://www.openbet.com/> Michal Borowiecki > > > > > >> Senior Software Engineer L4 > > > > > >> T: +44 208 742 1600 <+44%2020%208742%201600> > > <+44%2020%208742%201600> > > > > > >> > > > > > >> > > > > > >> +44 203 249 8448 <+44%2020%203249%208448> > > <+44%2020%203249%208448> > > > > > >> > > > > > >> > > > > > >> > > > > > >> E: michal.borowie...@openbet.com > > > > > >> W: www.openbet.com > > > > > >> OpenBet Ltd > > > > > >> > > > > > >> Chiswick Park Building 9 > > > > > >> > > > > > >> 566 Chiswick High Rd > > > > > >> > > > > > >> London > > > > > >> > > > > > >> W4 5XT > > > > > >> > > > > > >> UK > > > > > >> <https://www.openbet.com/email_promo> > > > > > >> This message is confidential and intended only for the > addressee. > > If > > > > you > > > > > >> have received this message in error, please immediately notify > the > > > > > >> postmas...@openbet.com and delete it from your system as well > as > > > any > > > > > >> copies. The content of e-mails as well as traffic data may be > > > > monitored > > > > > by > > > > > >> OpenBet for employment and security purposes. To protect the > > > > environment > > > > > >> please do not print this e-mail unless necessary. OpenBet Ltd. > > > > > Registered > > > > > >> Office: Chiswick Park Building 9, 566 Chiswick High Road, > London, > > W4 > > > > > 5XT, > > > > > >> United Kingdom. A company registered in England and Wales. > > > Registered > > > > > no. > > > > > >> 3134634. VAT no. GB927523612 > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Regards, > Anish Samir Mashankar > R&D Engineer > System Insights > +91-9789870733 >
