Hi Gwen,
the protocol version in an InfoResponse message would be the maximum
supported mainline protocol version, each change to the
effective protocol specification bumps the version by one.
The current per-request-type version ID is unfortunately useless to the
client, it only allows the broker to act differently
on different versions of a known request type.
Since the broker currently does not handle unknown request types gracefully
- it simply disconnects the client, which from the client's perspective
could mean anything and nothing - the addition of an InfoRequest/Respones
would allow the client to know if a desired broker-feature is available or
not and thus allows the client library to provide the application (and in
the end the user) with usable error message (compare "Broker disconnected"
to "offsetCommit not supported by 0.8.0 broker").
Having the broker return a fabricated response with the header.err field
set to UNKNOWN_REQUEST_TYPE would be very useful as well of course.
Regards,
Magnus
2014-11-14 18:17 GMT+01:00 Gwen Shapira <gshap...@cloudera.com>:
> I'm not sure there's a single "protocol version". Each
> request/response has its own version ID embedded in the request
> itself.
>
> Broker version, broker ID and (as Joel suggested) git hash are all
> reasonable. And +1 for adding this to API to support non-JVM clients.
>
> Gwen
>
> On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill <mag...@edenhill.se>
> wrote:
> > Please let's not add dependencies on more third party protocols/software,
> > the move away from Zookeeper dependence on the clients is very much
> welcomed
> > and it'd be a pity to see a new foreign dependency added for such a
> trivial
> > thing
> > as propagating the version of a broker to its client.
> >
> > My suggestion is to add a new protocol request which returns:
> > - broker version
> > - protocol version
> > - broker id
> >
> > A generic future-proof solution would be the use of tags (or named TLVs):
> > InfoResponse: [InfoTag]
> > InfoTag:
> > intX tag ( KAFKA_TAG_BROKER_VERSION, KAFKA_TAG_PROTO_VERSION,
> > KAFKA_TAG_SSL, ... )
> > intX type ( KAFKA_TYPE_STR, KAFKA_TYPE_INT32, KAFKA_TYPE_INT64,
> ...)
> > intX len
> > bytes payload
> >
> > Local site/vendor tags could be defined in the configuration file,
> >
> >
> > This would also allow clients to enable/disable features based on
> protocol
> > version,
> > e.g., there is no point in trying offsetCommit on a 0.8.0 broker and the
> > client library
> > can inform the application about this early, rather than having its
> > offsetCommit requests
> > fail by connection teardown (which is not much of an error propagation).
> >
> >
> > My two cents,
> > Magnus
> >
> >
> > 2014-11-12 20:11 GMT+01:00 Mark Roberts <wiz...@gmail.com>:
> >
> >> I haven't worked much with JMX before, but some quick googling (10-20
> >> minutes) is very inconclusive as to how I would go about getting the
> server
> >> version I'm connecting to from a Python client. Can someone please
> >> reassure me that it's relatively trivial for non Java clients to query
> JMX
> >> for the server version of every server in the cluster? Is there a reason
> >> not to include this in the API itself?
> >>
> >> -Mark
> >>
> >> On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy <jjkosh...@gmail.com>
> wrote:
> >>
> >> > +1 on the JMX + gradle properties. Is there any (seamless) way of
> >> > including the exact git hash? That would be extremely useful if users
> >> > need help debugging and happen to be on an unreleased build (say, off
> >> > trunk)
> >> >
> >> > On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira wrote:
> >> > > Actually, Jun suggested exposing this via JMX.
> >> > >
> >> > > On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira <
> gshap...@cloudera.com>
> >> > wrote:
> >> > >
> >> > > > Good question.
> >> > > >
> >> > > > The server will need to expose this in the protocol, so Kafka
> clients
> >> > will
> >> > > > know what they are talking to.
> >> > > >
> >> > > > We may also want to expose this in the producer and consumer, so
> >> people
> >> > > > who use Kafka's built-in clients will know which version they
> have in
> >> > the
> >> > > > environment.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts <wiz...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > >> Just to be clear: this is going to be exposed via some Api the
> >> clients
> >> > > >> can call at startup?
> >> > > >>
> >> > > >>
> >> > > >> > On Nov 12, 2014, at 08:59, Guozhang Wang <wangg...@gmail.com>
> >> > wrote:
> >> > > >> >
> >> > > >> > Sounds great, +1 on this.
> >> > > >> >
> >> > > >> >> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira <
> >> > gshap...@cloudera.com>
> >> > > >> wrote:
> >> > > >> >>
> >> > > >> >> So it looks like we can use Gradle to add properties to
> manifest
> >> > file
> >> > > >> and
> >> > > >> >> then use getResourceAsStream to read the file and parse it.
> >> > > >> >>
> >> > > >> >> The Gradle part would be something like:
> >> > > >> >> jar.manifest {
> >> > > >> >> attributes('Implementation-Title': project.name,
> >> > > >> >> 'Implementation-Version': project.version,
> >> > > >> >> 'Built-By': System.getProperty('user.name'),
> >> > > >> >> 'Built-JDK': System.getProperty('java.version'),
> >> > > >> >> 'Built-Host': getHostname(),
> >> > > >> >> 'Source-Compatibility':
> project.sourceCompatibility,
> >> > > >> >> 'Target-Compatibility': project.targetCompatibility
> >> > > >> >> )
> >> > > >> >> }
> >> > > >> >>
> >> > > >> >> The code part would be:
> >> > > >> >>
> >> > > >> >>
> >> > > >>
> >> >
> >>
> this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF")
> >> > > >> >>
> >> > > >> >> Does that look like the right approach?
> >> > > >> >>
> >> > > >> >> Gwen
> >> > > >> >>
> >> > > >> >> On Tue, Nov 11, 2014 at 10:43 AM, Bhavesh Mistry <
> >> > > >> >> mistry.p.bhav...@gmail.com
> >> > > >> >>> wrote:
> >> > > >> >>
> >> > > >> >>> If is maven artifact then you will get following pre-build
> >> > property
> >> > > >> file
> >> > > >> >>> from maven build called pom.properties under
> >> > > >> >>> /META-INF/maven/groupid/artifactId/pom.properties folder.
> >> > > >> >>>
> >> > > >> >>> Here is sample:
> >> > > >> >>> #Generated by Maven
> >> > > >> >>> #Mon Oct 10 10:44:31 EDT 2011
> >> > > >> >>> version=10.0.1
> >> > > >> >>> groupId=com.google.guava
> >> > > >> >>> artifactId=guava
> >> > > >> >>>
> >> > > >> >>> Thanks,
> >> > > >> >>>
> >> > > >> >>> Bhavesh
> >> > > >> >>>
> >> > > >> >>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen Shapira <
> >> > gshap...@cloudera.com
> >> > > >> >
> >> > > >> >>> wrote:
> >> > > >> >>>
> >> > > >> >>>> In Sqoop we do the following:
> >> > > >> >>>>
> >> > > >> >>>> Maven runs a shell script, passing the version as a
> parameter.
> >> > > >> >>>> The shell-script generates a small java class, which is then
> >> > built
> >> > > >> >> with a
> >> > > >> >>>> Maven plugin.
> >> > > >> >>>> Our code references this generated class when we expose
> >> > > >> "getVersion()".
> >> > > >> >>>>
> >> > > >> >>>> Its complex and ugly, so I'm kind of hoping that there's a
> >> > better way
> >> > > >> >> to
> >> > > >> >>> do
> >> > > >> >>>> it :)
> >> > > >> >>>>
> >> > > >> >>>> Gwen
> >> > > >> >>>>
> >> > > >> >>>>> On Tue, Nov 11, 2014 at 9:42 AM, Jun Rao <jun...@gmail.com
> >
> >> > wrote:
> >> > > >> >>>>>
> >> > > >> >>>>> Currently, the version number is only stored in our build
> >> config
> >> > > >> >> file,
> >> > > >> >>>>> gradle.properties. Not sure how we can automatically
> extract
> >> it
> >> > and
> >> > > >> >>>> expose
> >> > > >> >>>>> it in an mbean. How do other projects do this?
> >> > > >> >>>>>
> >> > > >> >>>>> Thanks,
> >> > > >> >>>>>
> >> > > >> >>>>> Jun
> >> > > >> >>>>>
> >> > > >> >>>>> On Tue, Nov 11, 2014 at 7:05 AM, Otis Gospodnetic <
> >> > > >> >>>>> otis.gospodne...@gmail.com> wrote:
> >> > > >> >>>>>
> >> > > >> >>>>>> Hi Jun,
> >> > > >> >>>>>>
> >> > > >> >>>>>> Sounds good. But is the version number stored anywhere
> from
> >> > where
> >> > > >> >> it
> >> > > >> >>>>> could
> >> > > >> >>>>>> be gotten?
> >> > > >> >>>>>>
> >> > > >> >>>>>> Thanks,
> >> > > >> >>>>>> Otis
> >> > > >> >>>>>> --
> >> > > >> >>>>>> Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> >> > > >> >>> Management
> >> > > >> >>>>>> Solr & Elasticsearch Support * http://sematext.com/
> >> > > >> >>>>>>
> >> > > >> >>>>>>
> >> > > >> >>>>>> On Tue, Nov 11, 2014 at 12:45 AM, Jun Rao <
> jun...@gmail.com>
> >> > > >> >> wrote:
> >> > > >> >>>>>>
> >> > > >> >>>>>>> Otis,
> >> > > >> >>>>>>>
> >> > > >> >>>>>>> We don't have an api for that now. We can probably expose
> >> this
> >> > > >> >> as a
> >> > > >> >>>> JMX
> >> > > >> >>>>>> as
> >> > > >> >>>>>>> part of kafka-1481.
> >> > > >> >>>>>>>
> >> > > >> >>>>>>> Thanks,
> >> > > >> >>>>>>>
> >> > > >> >>>>>>> Jun
> >> > > >> >>>>>>>
> >> > > >> >>>>>>> On Mon, Nov 10, 2014 at 7:17 PM, Otis Gospodnetic <
> >> > > >> >>>>>>> otis.gospodne...@gmail.com> wrote:
> >> > > >> >>>>>>>
> >> > > >> >>>>>>>> Hi,
> >> > > >> >>>>>>>>
> >> > > >> >>>>>>>> Is there a way to detect which version of Kafka one is
> >> > running?
> >> > > >> >>>>>>>> Is there an API for that, or a constant with this
> value, or
> >> > > >> >> maybe
> >> > > >> >>>> an
> >> > > >> >>>>>>> MBean
> >> > > >> >>>>>>>> or some other way to get to this info?
> >> > > >> >>>>>>>>
> >> > > >> >>>>>>>> Thanks,
> >> > > >> >>>>>>>> Otis
> >> > > >> >>>>>>>> --
> >> > > >> >>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized
> Log
> >> > > >> >>>>> Management
> >> > > >> >>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > --
> >> > > >> > -- Guozhang
> >> > > >>
> >> > > >
> >> > > >
> >> >
> >> >
> >>
>
