Hi Pritam ! Thanks for the KIP ! Let’s extend the KIP to also add support for Nanosecond precision!
Thanks Sudesh On 2025/04/05 01:30:49 pritam kumar wrote: > Hi Kafka Community, > Sorry due to some changes I had to change the link to the kip. > Here is the updated KIP link: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect > > On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com> > wrote: > > > Hi Kafka Community, > > > > I’d like to start a discussion on KIP-1153: Extending Support for > > Microsecond Precision for Kafka Connect > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect> > > . > > > > The primary motivation behind this KIP is to enhance the precision of > > timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to > > millisecond-level precision for timestamps. However, many modern data > > formats and platforms have moved beyond this limitation: > > > > - > > > > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and > > even nanosecond) precision. For example, Avro specifies support for > > timestamp-micros (spec link > > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). > > - > > > > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* > > offer *microsecond and nanosecond precision* for time-based fields, > > making millisecond precision inadequate for accurate data replication and > > analytics in some use cases. > > > > This gap can lead to *loss of precision* when transferring data through > > Kafka Connect, especially when interacting with systems that depend on > > high-resolution timestamps for change tracking, event ordering, or > > deduplication. > > > > The goal of this KIP is to: > > > > - > > > > Introduce microsecond-level timestamp handling in Kafka Connect schema > > and data representation. > > - > > > > Ensure connectors (both source and sink) can leverage this precision > > when supported by the underlying data systems. > > - > > > > Maintain backward compatibility with existing millisecond-based > > configurations and data. > > > > We welcome community feedback on: > > > > - > > > > Potential implementation concerns or edge cases we should address > > - > > > > Suggestions for schema enhancements or conversion strategies > > - > > > > Impacts on connector compatibility and testing > > > > Looking forward to your thoughts and input on this proposal! > > > > Thanks. > > Link to the KIP. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API> > > > > Sent with a Spark
