By the way, I don't think non-Java clients have to lag on this feature even if we don't do anything because the SASL mechanism is defined by a client property. So, if a non-GSSAPI mechanism is chosen, the clients would optimistically use the new way (older brokers would close the connection). This is not great from a validation perspective (for the case where users choose a mechanism that is not supported by the broker), but it's better than not supporting the feature at all.
Having said that, I agree with you that it would be good to think about good ways of supporting such cases. Finally, as Magnus said and Jun implied, it would be nice to use the Kafka protocol for the handshake phase as well, but that's not straightforward. Ismael