Thanks Sudesh, for taking a look at this. I am already working on extending this for nanosecond precision as most sinks like iceberg have started giving nanosecond precision options.
On Wed, Apr 16, 2025 at 4:41 PM Sudesh Wasnik <wasnik...@gmail.com> wrote: > Hi Pritam ! Thanks for the KIP ! > Let’s extend the KIP to also add support for Nanosecond precision! > > Thanks > Sudesh > > On 2025/04/05 01:30:49 pritam kumar wrote: > > Hi Kafka Community, > > Sorry due to some changes I had to change the link to the kip. > > Here is the updated KIP link: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect > > > > On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com> > > wrote: > > > > > Hi Kafka Community, > > > > > > I’d like to start a discussion on KIP-1153: Extending Support for > > > Microsecond Precision for Kafka Connect > > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect > > > > > . > > > > > > The primary motivation behind this KIP is to enhance the precision of > > > timestamp handling in Kafka Connect. Currently, Kafka Connect is > limited to > > > millisecond-level precision for timestamps. However, many modern data > > > formats and platforms have moved beyond this limitation: > > > > > > - > > > > > > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and > > > even nanosecond) precision. For example, Avro specifies support for > > > timestamp-micros (spec link > > > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). > > > - > > > > > > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* > > > offer *microsecond and nanosecond precision* for time-based fields, > > > making millisecond precision inadequate for accurate data replication > and > > > analytics in some use cases. > > > > > > This gap can lead to *loss of precision* when transferring data through > > > Kafka Connect, especially when interacting with systems that depend on > > > high-resolution timestamps for change tracking, event ordering, or > > > deduplication. > > > > > > The goal of this KIP is to: > > > > > > - > > > > > > Introduce microsecond-level timestamp handling in Kafka Connect schema > > > and data representation. > > > - > > > > > > Ensure connectors (both source and sink) can leverage this precision > > > when supported by the underlying data systems. > > > - > > > > > > Maintain backward compatibility with existing millisecond-based > > > configurations and data. > > > > > > We welcome community feedback on: > > > > > > - > > > > > > Potential implementation concerns or edge cases we should address > > > - > > > > > > Suggestions for schema enhancements or conversion strategies > > > - > > > > > > Impacts on connector compatibility and testing > > > > > > Looking forward to your thoughts and input on this proposal! > > > > > > Thanks. > > > Link to the KIP. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect > > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API > > > > > > > > > > Sent with a Spark >
