Hi Kafka Community, I’d like to start a discussion on KIP-1153: Extending Support for Microsecond Precision for Kafka Connect <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect> .
The primary motivation behind this KIP is to enhance the precision of timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to millisecond-level precision for timestamps. However, many modern data formats and platforms have moved beyond this limitation: - Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and even nanosecond) precision. For example, Avro specifies support for timestamp-micros (spec link <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). - Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* offer *microsecond and nanosecond precision* for time-based fields, making millisecond precision inadequate for accurate data replication and analytics in some use cases. This gap can lead to *loss of precision* when transferring data through Kafka Connect, especially when interacting with systems that depend on high-resolution timestamps for change tracking, event ordering, or deduplication. The goal of this KIP is to: - Introduce microsecond-level timestamp handling in Kafka Connect schema and data representation. - Ensure connectors (both source and sink) can leverage this precision when supported by the underlying data systems. - Maintain backward compatibility with existing millisecond-based configurations and data. We welcome community feedback on: - Potential implementation concerns or edge cases we should address - Suggestions for schema enhancements or conversion strategies - Impacts on connector compatibility and testing Looking forward to your thoughts and input on this proposal! Thanks. Link to the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API>
