sujitn commented on PR #3038: URL: https://github.com/apache/iggy/pull/3038#issuecomment-4144899997
Agreed on the shard 0 bottleneck that's the real architectural issue. Fault isolation is also a fair point given the size of the Kafka protocol surface. the security/correctness gaps (auto-auth, no CRC validation, dropped record keys) are implementation maturity issues, not in-process vs gateway issues. The gateway will have the same Phase 1 limitations until they're addressed. Questions on the gateway approach before I open a Discussion: - send_messages() returning base_offset is a hard blocker without it a Kafka ProduceResponse is invalid regardless of architecture. Is there an existing issue for this, or should I propose the binary protocol change? Happy to contribute that work. - For ListOffsets I need earliest/latest offset per partition. Is that exposed through the SDK today, or would it require a new SDK method? - The connector runtime comparison makes sense for lifecycle isolation, but the interaction pattern is different connectors are async pipelines, a Kafka gateway is synchronous request-response on the same TCP connection. I'll look at the runtime structure but the internal design will diverge. The protocol codec from this spike (varint, RecordBatch v2, request/response types, flexible header negotiation) transfers directly to the gateway. I'll close this PR and open a Discussion with the gateway design. Point taken on discussions-before-code will do that going forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
