Hello Apache Kafka community, Couldn't broker return a special error code in FetchResponse for a given partition(s) where it detects that there was something to return/read from partition but actual FetchResponse contains no messages for that partition since fetch size in FetchRequest for that partition is too small?
I don't see it already in existing Kafka wire protocol docs <https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol>. It doesn't seem to be supported, since docs for broker and consumer configuration mention that the maximums should be in sync, otherwise too large message could block consumer. If it was available this feature would be useful for simple consumer API users, in the cases when range of possible message sizes on same topic is relatively large (e.g. most very small messages, but some small fraction of relatively large messages). Would help with being able to soft and hard maximums, so most of the time one could read with soft max, and only in case of this error code is on, attempt a read using hard max limit. Assuming this feature is not there, I guess what would need to be done at minimum is: - minimal change in protocol and docs, just one new error code - accompanying detection of this condition on broker side, so extend handling of FetchRequest - maybe in high level consumer make use of this error code in a special way (e.g. just log a warning/error that there is potential misconfiguration). Kind regards, Stevo Slavic.
