Agreed. We can have the response be the list of all brokers and each the current version. Also an option to single broker just for that brokers version. On start up the broker can register with meta data store (zookeeper for now) their current version using one of the mentioned gathered info.
Something like that, yeah. I will create the JIRA when I get back to computer desk later today. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ********************************************/ On Nov 15, 2014 9:42 AM, "Jeff Holoman" <jholo...@cloudera.com> wrote: > Seems like this aspect of Daniel's request could (would be) included here > pretty easily: > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements > > Jeff > > On Sat, Nov 15, 2014 at 3:29 AM, Daniel Compton <d...@danielcompton.net> > wrote: > > > This has been covered in passing in the preceding threads but I'd like to > > make it explicit, can we have a command line option/script for getting > the > > Kafka version (probably with Scala version too)? > > > > I ran into this recently, where we wanted to verify that the right > version > > of Kafka had been installed on a number of machines. I ended up getting > the > > version number from the bundled jars but it wasn't ideal. Being able to > run > > kafka --version would have been great. > > > > --- > > Daniel > > > > > On 15/11/2014, at 8:33 am, Magnus Edenhill <mag...@edenhill.se> wrote: > > > > > > Hi Gwen, > > > > > > the protocol version in an InfoResponse message would be the maximum > > > supported mainline protocol version, each change to the > > > effective protocol specification bumps the version by one. > > > > > > > > > The current per-request-type version ID is unfortunately useless to the > > > client, it only allows the broker to act differently > > > on different versions of a known request type. > > > > > > Since the broker currently does not handle unknown request types > > gracefully > > > - it simply disconnects the client, which from the client's perspective > > > could mean anything and nothing - the addition of an > InfoRequest/Respones > > > would allow the client to know if a desired broker-feature is available > > or > > > not and thus allows the client library to provide the application (and > in > > > the end the user) with usable error message (compare "Broker > > disconnected" > > > to "offsetCommit not supported by 0.8.0 broker"). > > > Having the broker return a fabricated response with the header.err > field > > > set to UNKNOWN_REQUEST_TYPE would be very useful as well of course. > > > > > > > > > Regards, > > > Magnus > > > > > > > > > > > > 2014-11-14 18:17 GMT+01:00 Gwen Shapira <gshap...@cloudera.com>: > > > > > >> I'm not sure there's a single "protocol version". Each > > >> request/response has its own version ID embedded in the request > > >> itself. > > >> > > >> Broker version, broker ID and (as Joel suggested) git hash are all > > >> reasonable. And +1 for adding this to API to support non-JVM clients. > > >> > > >> Gwen > > >> > > >> On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill <mag...@edenhill.se> > > >> wrote: > > >>> Please let's not add dependencies on more third party > > protocols/software, > > >>> the move away from Zookeeper dependence on the clients is very much > > >> welcomed > > >>> and it'd be a pity to see a new foreign dependency added for such a > > >> trivial > > >>> thing > > >>> as propagating the version of a broker to its client. > > >>> > > >>> My suggestion is to add a new protocol request which returns: > > >>> - broker version > > >>> - protocol version > > >>> - broker id > > >>> > > >>> A generic future-proof solution would be the use of tags (or named > > TLVs): > > >>> InfoResponse: [InfoTag] > > >>> InfoTag: > > >>> intX tag ( KAFKA_TAG_BROKER_VERSION, KAFKA_TAG_PROTO_VERSION, > > >>> KAFKA_TAG_SSL, ... ) > > >>> intX type ( KAFKA_TYPE_STR, KAFKA_TYPE_INT32, KAFKA_TYPE_INT64, > > >> ...) > > >>> intX len > > >>> bytes payload > > >>> > > >>> Local site/vendor tags could be defined in the configuration file, > > >>> > > >>> > > >>> This would also allow clients to enable/disable features based on > > >> protocol > > >>> version, > > >>> e.g., there is no point in trying offsetCommit on a 0.8.0 broker and > > the > > >>> client library > > >>> can inform the application about this early, rather than having its > > >>> offsetCommit requests > > >>> fail by connection teardown (which is not much of an error > > propagation). > > >>> > > >>> > > >>> My two cents, > > >>> Magnus > > >>> > > >>> > > >>> 2014-11-12 20:11 GMT+01:00 Mark Roberts <wiz...@gmail.com>: > > >>> > > >>>> I haven't worked much with JMX before, but some quick googling > (10-20 > > >>>> minutes) is very inconclusive as to how I would go about getting the > > >> server > > >>>> version I'm connecting to from a Python client. Can someone please > > >>>> reassure me that it's relatively trivial for non Java clients to > query > > >> JMX > > >>>> for the server version of every server in the cluster? Is there a > > reason > > >>>> not to include this in the API itself? > > >>>> > > >>>> -Mark > > >>>> > > >>>> On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy <jjkosh...@gmail.com> > > >> wrote: > > >>>> > > >>>>> +1 on the JMX + gradle properties. Is there any (seamless) way of > > >>>>> including the exact git hash? That would be extremely useful if > users > > >>>>> need help debugging and happen to be on an unreleased build (say, > off > > >>>>> trunk) > > >>>>> > > >>>>>> On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira wrote: > > >>>>>> Actually, Jun suggested exposing this via JMX. > > >>>>>> > > >>>>>> On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira < > > >> gshap...@cloudera.com> > > >>>>> wrote: > > >>>>>> > > >>>>>>> Good question. > > >>>>>>> > > >>>>>>> The server will need to expose this in the protocol, so Kafka > > >> clients > > >>>>> will > > >>>>>>> know what they are talking to. > > >>>>>>> > > >>>>>>> We may also want to expose this in the producer and consumer, so > > >>>> people > > >>>>>>> who use Kafka's built-in clients will know which version they > > >> have in > > >>>>> the > > >>>>>>> environment. > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts <wiz...@gmail.com> > > >>>>> wrote: > > >>>>>>> > > >>>>>>>> Just to be clear: this is going to be exposed via some Api the > > >>>> clients > > >>>>>>>> can call at startup? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> On Nov 12, 2014, at 08:59, Guozhang Wang <wangg...@gmail.com> > > >>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> Sounds great, +1 on this. > > >>>>>>>>> > > >>>>>>>>>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira < > > >>>>> gshap...@cloudera.com> > > >>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> So it looks like we can use Gradle to add properties to > > >> manifest > > >>>>> file > > >>>>>>>> and > > >>>>>>>>>> then use getResourceAsStream to read the file and parse it. > > >>>>>>>>>> > > >>>>>>>>>> The Gradle part would be something like: > > >>>>>>>>>> jar.manifest { > > >>>>>>>>>> attributes('Implementation-Title': project.name, > > >>>>>>>>>> 'Implementation-Version': project.version, > > >>>>>>>>>> 'Built-By': System.getProperty('user.name'), > > >>>>>>>>>> 'Built-JDK': System.getProperty('java.version'), > > >>>>>>>>>> 'Built-Host': getHostname(), > > >>>>>>>>>> 'Source-Compatibility': > > >> project.sourceCompatibility, > > >>>>>>>>>> 'Target-Compatibility': project.targetCompatibility > > >>>>>>>>>> ) > > >>>>>>>>>> } > > >>>>>>>>>> > > >>>>>>>>>> The code part would be: > > >> > > > this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF") > > >>>>>>>>>> > > >>>>>>>>>> Does that look like the right approach? > > >>>>>>>>>> > > >>>>>>>>>> Gwen > > >>>>>>>>>> > > >>>>>>>>>> On Tue, Nov 11, 2014 at 10:43 AM, Bhavesh Mistry < > > >>>>>>>>>> mistry.p.bhav...@gmail.com > > >>>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> If is maven artifact then you will get following pre-build > > >>>>> property > > >>>>>>>> file > > >>>>>>>>>>> from maven build called pom.properties under > > >>>>>>>>>>> /META-INF/maven/groupid/artifactId/pom.properties folder. > > >>>>>>>>>>> > > >>>>>>>>>>> Here is sample: > > >>>>>>>>>>> #Generated by Maven > > >>>>>>>>>>> #Mon Oct 10 10:44:31 EDT 2011 > > >>>>>>>>>>> version=10.0.1 > > >>>>>>>>>>> groupId=com.google.guava > > >>>>>>>>>>> artifactId=guava > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> > > >>>>>>>>>>> Bhavesh > > >>>>>>>>>>> > > >>>>>>>>>>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen Shapira < > > >>>>> gshap...@cloudera.com > > >>>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> In Sqoop we do the following: > > >>>>>>>>>>>> > > >>>>>>>>>>>> Maven runs a shell script, passing the version as a > > >> parameter. > > >>>>>>>>>>>> The shell-script generates a small java class, which is then > > >>>>> built > > >>>>>>>>>> with a > > >>>>>>>>>>>> Maven plugin. > > >>>>>>>>>>>> Our code references this generated class when we expose > > >>>>>>>> "getVersion()". > > >>>>>>>>>>>> > > >>>>>>>>>>>> Its complex and ugly, so I'm kind of hoping that there's a > > >>>>> better way > > >>>>>>>>>> to > > >>>>>>>>>>> do > > >>>>>>>>>>>> it :) > > >>>>>>>>>>>> > > >>>>>>>>>>>> Gwen > > >>>>>>>>>>>> > > >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 9:42 AM, Jun Rao <jun...@gmail.com > > >>> > > >>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Currently, the version number is only stored in our build > > >>>> config > > >>>>>>>>>> file, > > >>>>>>>>>>>>> gradle.properties. Not sure how we can automatically > > >> extract > > >>>> it > > >>>>> and > > >>>>>>>>>>>> expose > > >>>>>>>>>>>>> it in an mbean. How do other projects do this? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Jun > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 7:05 AM, Otis Gospodnetic < > > >>>>>>>>>>>>> otis.gospodne...@gmail.com> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Hi Jun, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Sounds good. But is the version number stored anywhere > > >> from > > >>>>> where > > >>>>>>>>>> it > > >>>>>>>>>>>>> could > > >>>>>>>>>>>>>> be gotten? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>> Otis > > >>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized > > >> Log > > >>>>>>>>>>> Management > > >>>>>>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 12:45 AM, Jun Rao < > > >> jun...@gmail.com> > > >>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Otis, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> We don't have an api for that now. We can probably expose > > >>>> this > > >>>>>>>>>> as a > > >>>>>>>>>>>> JMX > > >>>>>>>>>>>>>> as > > >>>>>>>>>>>>>>> part of kafka-1481. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Jun > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Mon, Nov 10, 2014 at 7:17 PM, Otis Gospodnetic < > > >>>>>>>>>>>>>>> otis.gospodne...@gmail.com> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Hi, > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Is there a way to detect which version of Kafka one is > > >>>>> running? > > >>>>>>>>>>>>>>>> Is there an API for that, or a constant with this > > >> value, or > > >>>>>>>>>> maybe > > >>>>>>>>>>>> an > > >>>>>>>>>>>>>>> MBean > > >>>>>>>>>>>>>>>> or some other way to get to this info? > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>> Otis > > >>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized > > >> Log > > >>>>>>>>>>>>> Management > > >>>>>>>>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- > > >>>>>>>>> -- Guozhang > > >> > > > > > > -- > Jeff Holoman > Systems Engineer > 678-612-9519 >
