FYI there is already https://issues.apache.org/jira/browse/KAFKA-1768
Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Sat, Nov 15, 2014 at 10:45 AM, Joe Stein <joe.st...@stealth.ly> wrote: > Agreed. We can have the response be the list of all brokers and each the > current version. Also an option to single broker just for that brokers > version. On start up the broker can register with meta data store > (zookeeper for now) their current version using one of the mentioned > gathered info. > > Something like that, yeah. > > I will create the JIRA when I get back to computer desk later today. > > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop > ********************************************/ > On Nov 15, 2014 9:42 AM, "Jeff Holoman" <jholo...@cloudera.com> wrote: > > > Seems like this aspect of Daniel's request could (would be) included here > > pretty easily: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements > > > > Jeff > > > > On Sat, Nov 15, 2014 at 3:29 AM, Daniel Compton <d...@danielcompton.net> > > wrote: > > > > > This has been covered in passing in the preceding threads but I'd like > to > > > make it explicit, can we have a command line option/script for getting > > the > > > Kafka version (probably with Scala version too)? > > > > > > I ran into this recently, where we wanted to verify that the right > > version > > > of Kafka had been installed on a number of machines. I ended up getting > > the > > > version number from the bundled jars but it wasn't ideal. Being able to > > run > > > kafka --version would have been great. > > > > > > --- > > > Daniel > > > > > > > On 15/11/2014, at 8:33 am, Magnus Edenhill <mag...@edenhill.se> > wrote: > > > > > > > > Hi Gwen, > > > > > > > > the protocol version in an InfoResponse message would be the maximum > > > > supported mainline protocol version, each change to the > > > > effective protocol specification bumps the version by one. > > > > > > > > > > > > The current per-request-type version ID is unfortunately useless to > the > > > > client, it only allows the broker to act differently > > > > on different versions of a known request type. > > > > > > > > Since the broker currently does not handle unknown request types > > > gracefully > > > > - it simply disconnects the client, which from the client's > perspective > > > > could mean anything and nothing - the addition of an > > InfoRequest/Respones > > > > would allow the client to know if a desired broker-feature is > available > > > or > > > > not and thus allows the client library to provide the application > (and > > in > > > > the end the user) with usable error message (compare "Broker > > > disconnected" > > > > to "offsetCommit not supported by 0.8.0 broker"). > > > > Having the broker return a fabricated response with the header.err > > field > > > > set to UNKNOWN_REQUEST_TYPE would be very useful as well of course. > > > > > > > > > > > > Regards, > > > > Magnus > > > > > > > > > > > > > > > > 2014-11-14 18:17 GMT+01:00 Gwen Shapira <gshap...@cloudera.com>: > > > > > > > >> I'm not sure there's a single "protocol version". Each > > > >> request/response has its own version ID embedded in the request > > > >> itself. > > > >> > > > >> Broker version, broker ID and (as Joel suggested) git hash are all > > > >> reasonable. And +1 for adding this to API to support non-JVM > clients. > > > >> > > > >> Gwen > > > >> > > > >> On Fri, Nov 14, 2014 at 8:46 AM, Magnus Edenhill < > mag...@edenhill.se> > > > >> wrote: > > > >>> Please let's not add dependencies on more third party > > > protocols/software, > > > >>> the move away from Zookeeper dependence on the clients is very much > > > >> welcomed > > > >>> and it'd be a pity to see a new foreign dependency added for such a > > > >> trivial > > > >>> thing > > > >>> as propagating the version of a broker to its client. > > > >>> > > > >>> My suggestion is to add a new protocol request which returns: > > > >>> - broker version > > > >>> - protocol version > > > >>> - broker id > > > >>> > > > >>> A generic future-proof solution would be the use of tags (or named > > > TLVs): > > > >>> InfoResponse: [InfoTag] > > > >>> InfoTag: > > > >>> intX tag ( KAFKA_TAG_BROKER_VERSION, > KAFKA_TAG_PROTO_VERSION, > > > >>> KAFKA_TAG_SSL, ... ) > > > >>> intX type ( KAFKA_TYPE_STR, KAFKA_TYPE_INT32, > KAFKA_TYPE_INT64, > > > >> ...) > > > >>> intX len > > > >>> bytes payload > > > >>> > > > >>> Local site/vendor tags could be defined in the configuration file, > > > >>> > > > >>> > > > >>> This would also allow clients to enable/disable features based on > > > >> protocol > > > >>> version, > > > >>> e.g., there is no point in trying offsetCommit on a 0.8.0 broker > and > > > the > > > >>> client library > > > >>> can inform the application about this early, rather than having its > > > >>> offsetCommit requests > > > >>> fail by connection teardown (which is not much of an error > > > propagation). > > > >>> > > > >>> > > > >>> My two cents, > > > >>> Magnus > > > >>> > > > >>> > > > >>> 2014-11-12 20:11 GMT+01:00 Mark Roberts <wiz...@gmail.com>: > > > >>> > > > >>>> I haven't worked much with JMX before, but some quick googling > > (10-20 > > > >>>> minutes) is very inconclusive as to how I would go about getting > the > > > >> server > > > >>>> version I'm connecting to from a Python client. Can someone > please > > > >>>> reassure me that it's relatively trivial for non Java clients to > > query > > > >> JMX > > > >>>> for the server version of every server in the cluster? Is there a > > > reason > > > >>>> not to include this in the API itself? > > > >>>> > > > >>>> -Mark > > > >>>> > > > >>>> On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy <jjkosh...@gmail.com> > > > >> wrote: > > > >>>> > > > >>>>> +1 on the JMX + gradle properties. Is there any (seamless) way of > > > >>>>> including the exact git hash? That would be extremely useful if > > users > > > >>>>> need help debugging and happen to be on an unreleased build (say, > > off > > > >>>>> trunk) > > > >>>>> > > > >>>>>> On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira wrote: > > > >>>>>> Actually, Jun suggested exposing this via JMX. > > > >>>>>> > > > >>>>>> On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira < > > > >> gshap...@cloudera.com> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>>> Good question. > > > >>>>>>> > > > >>>>>>> The server will need to expose this in the protocol, so Kafka > > > >> clients > > > >>>>> will > > > >>>>>>> know what they are talking to. > > > >>>>>>> > > > >>>>>>> We may also want to expose this in the producer and consumer, > so > > > >>>> people > > > >>>>>>> who use Kafka's built-in clients will know which version they > > > >> have in > > > >>>>> the > > > >>>>>>> environment. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On Wed, Nov 12, 2014 at 9:09 AM, Mark Roberts < > wiz...@gmail.com> > > > >>>>> wrote: > > > >>>>>>> > > > >>>>>>>> Just to be clear: this is going to be exposed via some Api the > > > >>>> clients > > > >>>>>>>> can call at startup? > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>> On Nov 12, 2014, at 08:59, Guozhang Wang <wangg...@gmail.com > > > > > >>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> Sounds great, +1 on this. > > > >>>>>>>>> > > > >>>>>>>>>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira < > > > >>>>> gshap...@cloudera.com> > > > >>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> So it looks like we can use Gradle to add properties to > > > >> manifest > > > >>>>> file > > > >>>>>>>> and > > > >>>>>>>>>> then use getResourceAsStream to read the file and parse it. > > > >>>>>>>>>> > > > >>>>>>>>>> The Gradle part would be something like: > > > >>>>>>>>>> jar.manifest { > > > >>>>>>>>>> attributes('Implementation-Title': project.name, > > > >>>>>>>>>> 'Implementation-Version': project.version, > > > >>>>>>>>>> 'Built-By': System.getProperty('user.name'), > > > >>>>>>>>>> 'Built-JDK': System.getProperty('java.version'), > > > >>>>>>>>>> 'Built-Host': getHostname(), > > > >>>>>>>>>> 'Source-Compatibility': > > > >> project.sourceCompatibility, > > > >>>>>>>>>> 'Target-Compatibility': > project.targetCompatibility > > > >>>>>>>>>> ) > > > >>>>>>>>>> } > > > >>>>>>>>>> > > > >>>>>>>>>> The code part would be: > > > >> > > > > > > this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF") > > > >>>>>>>>>> > > > >>>>>>>>>> Does that look like the right approach? > > > >>>>>>>>>> > > > >>>>>>>>>> Gwen > > > >>>>>>>>>> > > > >>>>>>>>>> On Tue, Nov 11, 2014 at 10:43 AM, Bhavesh Mistry < > > > >>>>>>>>>> mistry.p.bhav...@gmail.com > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> If is maven artifact then you will get following pre-build > > > >>>>> property > > > >>>>>>>> file > > > >>>>>>>>>>> from maven build called pom.properties under > > > >>>>>>>>>>> /META-INF/maven/groupid/artifactId/pom.properties folder. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Here is sample: > > > >>>>>>>>>>> #Generated by Maven > > > >>>>>>>>>>> #Mon Oct 10 10:44:31 EDT 2011 > > > >>>>>>>>>>> version=10.0.1 > > > >>>>>>>>>>> groupId=com.google.guava > > > >>>>>>>>>>> artifactId=guava > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Bhavesh > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Tue, Nov 11, 2014 at 10:34 AM, Gwen Shapira < > > > >>>>> gshap...@cloudera.com > > > >>>>>>>>> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> In Sqoop we do the following: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Maven runs a shell script, passing the version as a > > > >> parameter. > > > >>>>>>>>>>>> The shell-script generates a small java class, which is > then > > > >>>>> built > > > >>>>>>>>>> with a > > > >>>>>>>>>>>> Maven plugin. > > > >>>>>>>>>>>> Our code references this generated class when we expose > > > >>>>>>>> "getVersion()". > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Its complex and ugly, so I'm kind of hoping that there's a > > > >>>>> better way > > > >>>>>>>>>> to > > > >>>>>>>>>>> do > > > >>>>>>>>>>>> it :) > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Gwen > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 9:42 AM, Jun Rao < > jun...@gmail.com > > > >>> > > > >>>>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Currently, the version number is only stored in our build > > > >>>> config > > > >>>>>>>>>> file, > > > >>>>>>>>>>>>> gradle.properties. Not sure how we can automatically > > > >> extract > > > >>>> it > > > >>>>> and > > > >>>>>>>>>>>> expose > > > >>>>>>>>>>>>> it in an mbean. How do other projects do this? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Tue, Nov 11, 2014 at 7:05 AM, Otis Gospodnetic < > > > >>>>>>>>>>>>> otis.gospodne...@gmail.com> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi Jun, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Sounds good. But is the version number stored anywhere > > > >> from > > > >>>>> where > > > >>>>>>>>>> it > > > >>>>>>>>>>>>> could > > > >>>>>>>>>>>>>> be gotten? > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>> Otis > > > >>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized > > > >> Log > > > >>>>>>>>>>> Management > > > >>>>>>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 12:45 AM, Jun Rao < > > > >> jun...@gmail.com> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Otis, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> We don't have an api for that now. We can probably > expose > > > >>>> this > > > >>>>>>>>>> as a > > > >>>>>>>>>>>> JMX > > > >>>>>>>>>>>>>> as > > > >>>>>>>>>>>>>>> part of kafka-1481. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Jun > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> On Mon, Nov 10, 2014 at 7:17 PM, Otis Gospodnetic < > > > >>>>>>>>>>>>>>> otis.gospodne...@gmail.com> wrote: > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Hi, > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Is there a way to detect which version of Kafka one is > > > >>>>> running? > > > >>>>>>>>>>>>>>>> Is there an API for that, or a constant with this > > > >> value, or > > > >>>>>>>>>> maybe > > > >>>>>>>>>>>> an > > > >>>>>>>>>>>>>>> MBean > > > >>>>>>>>>>>>>>>> or some other way to get to this info? > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>> Otis > > > >>>>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>>>> Monitoring * Alerting * Anomaly Detection * > Centralized > > > >> Log > > > >>>>>>>>>>>>> Management > > > >>>>>>>>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> -- > > > >>>>>>>>> -- Guozhang > > > >> > > > > > > > > > > > -- > > Jeff Holoman > > Systems Engineer > > 678-612-9519 > > >
