Hi Lincong Li, I agree that server-side instrumentation is helpful. However, I don't think this is the right approach.
The problem is that RequestChannel.Request and AbstractResponse are internal classes that should not be exposed. These are implementation details that we may change in the future. Freezing these into a public API would really hold back the project. For example, for really large responses, we might eventually want to avoid materializing the whole response all at once. It would make more sense to return it in a streaming fashion. But if we need to support this API forever, we can't do that. I think it's fair to say that this is, at best, half a solution to the problem of tracing requests. Users still need to write the plugin code and arrange for it to be on their classpath to make this work. I think the alternative here is not client-side instrumentation, but simply making the change to the broker without using a plugin interface. If a public interface is absolutely necessary here we should expose only things like the API key, client ID, time, etc. that don't constrain the implementation a lot in the future. I think we should also use java here to avoid the compatibility issues we have had with Scala APIs in the past. best, Colin On Thu, Nov 8, 2018, at 11:34, radai wrote: > another downside to client instrumentation (beyond the number of > client codebases one would need to cover) is that in a large > environments you'll have a very long tail of applications using older > clients to upgrade - it would be a long and disruptive process (as > opposed to updating broker-side instrumentation) > On Thu, Nov 8, 2018 at 11:04 AM Peter M. Elias <petermel...@gmail.com> wrote: > > > > I know we have a lot of use cases for this type of functionality at my > > enterprise deployment. I think it's helpful for maintaining reliability of > > the cluster especially and identifying clients that are not properly tuned > > and therefore applying excessive load to the brokers. Additionally, there > > is a bit of a dark spot without something like as currently. For example, > > if a client is not using a consumer group, there is no direct way to query > > the state of the consumer without looking at raw network connections to > > determine the extent of the traffic generated by that particular consumer. > > > > While client instrumentation can certainly help with this currently, given > > that Kafka is intended to be a shared service across a potentially very > > large surface area of clients, central observation of client activity is in > > my opinion an essential feature. > > > > Peter > > > > On Thu, Nov 8, 2018 at 12:13 PM radai <radai.rosenbl...@gmail.com> wrote: > > > > > bump. > > > > > > I think the proposed API (Observer) is useful for any sort of > > > multi-tenant environment for chargeback and reporting purposes. > > > > > > if no one wants to comment, can we initiate a vote? > > > On Mon, Nov 5, 2018 at 6:31 PM Lincong Li <andrewlinc...@gmail.com> wrote: > > > > > > > > Hi everyone. Here > > > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response > > > > > > > > is > > > > my KIP. Any feedback is appreciated. > > > > > > > > Thanks, > > > > Lincong Li > > >
