Yeah I agree there isn't a good way to auto-detect the presence of a given
API.
I think #1 may be tricky in practice. The response format is always
dictated by the request format so how do I know if the bytes I got back are
a valid response to the given request or are the UnknownRequestResponse?
#2 would be a good fix for the problem I think. This might be a good
replacement for the echo api and would probably serve the same purpose
(checking if the server is alive).
#3 is a little dangerous because we actually want clients to only pay
attention to the protocol versions which are per-api, not the server
version. I.e. we actually don't want the client to do something like check
serverVersion.equals("0.8.2") because we want to be able to release the
server at will and have it keep answering protocols in a backwards
compatible way. I.e. a client that uses just metadata request and produce
request should only care about the version of these protocols it implements
being supported not about the version of the server or the version of any
other protocol it doesn't use. This is the rationale behind versioning the
apis independently rather than having a single protocol version that we
would have to bump every time an internal broker-broker protocol changed.
-Jay
On Mon, Jan 12, 2015 at 1:32 PM, Gwen Shapira <gshap...@cloudera.com> wrote:
> We ran into similar difficulties, both when trying to get Kafka to use
> new APIs when available and when testing the wire protocol.
>
> +1 for all three suggestions.
>
> #1 sounds like the bare minimum, but I'm not sure how much it will
> complicate the clients (now we expect either a response or an Unknown
> message and need to be able to distinguish between them from the byte
> array).
>
> #2 and #3 both makes lots of sense.
>
> Gwen
>
>
> On Mon, Jan 12, 2015 at 1:15 PM, Dana Powers <dana.pow...@rd.io> wrote:
> > Hi all -- continuing on the compatibility discussion:
> >
> > I've found that it is very difficult to identify when a server does not
> > recognize an api
> > (I'm using kafka-python to submit wire-protocol requests). For example,
> > when I
> > send a ConsumerMetadataRequest to an 0.8.1.1 server, I get a closed
> socket
> > *[stacktrace below]. The server raises an error internally, but does not
> > send any
> > meaningful response. I'm not sure whether this is the intended behavior,
> > but
> > maintaining clients in an ecosystem of multiple server versions with
> > different API
> > support it would be great to have a way to determine what the server
> > supports
> > and what it does not.
> >
> > Some suggestions:
> >
> > (1) An UnknownAPIResponse that is returned for any API or API Version
> > request
> > that is unsupported.
> >
> > (2) A server metadata API to get the list of supported APIs and/or API
> > versions supported.
> >
> > (3) A server metadata API to get the published version of the server
> (0.8.2
> > v. 0.8.1.1, etc).
> >
> >
> > Thoughts?
> >
> >
> > Dana Powers
> > Rdio, Inc.
> > dana.pow...@rd.io
> > rdio.com/people/dpkp/
> >
> > *stacktrace:
> > ```
> > [2015-01-12 13:03:55,719] ERROR Closing socket for /127.0.0.1 because of
> > error (kafka.network.Processor)
> > kafka.common.KafkaException: Wrong request type 10
> > at kafka.api.RequestKeys$.deserializerForKey(RequestKeys.scala:57)
> > at
> kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:53)
> > at kafka.network.Processor.read(SocketServer.scala:353)
> > at kafka.network.Processor.run(SocketServer.scala:245)
> > at java.lang.Thread.run(Thread.java:722)
> > ```
>
