Hi Chris, Can you please review this one too? On Sat, Apr 5, 2025 at 7:00 AM pritam kumar <kumarpritamm...@gmail.com> wrote:
> Hi Kafka Community, > Sorry due to some changes I had to change the link to the kip. > Here is the updated KIP link: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect > > On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <kumarpritamm...@gmail.com> > wrote: > >> Hi Kafka Community, >> >> I’d like to start a discussion on KIP-1153: Extending Support for >> Microsecond Precision for Kafka Connect >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect> >> . >> >> The primary motivation behind this KIP is to enhance the precision of >> timestamp handling in Kafka Connect. Currently, Kafka Connect is limited to >> millisecond-level precision for timestamps. However, many modern data >> formats and platforms have moved beyond this limitation: >> >> - >> >> Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and >> even nanosecond) precision. For example, Avro specifies support for >> timestamp-micros (spec link >> <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). >> - >> >> Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* >> offer *microsecond and nanosecond precision* for time-based fields, >> making millisecond precision inadequate for accurate data replication and >> analytics in some use cases. >> >> This gap can lead to *loss of precision* when transferring data through >> Kafka Connect, especially when interacting with systems that depend on >> high-resolution timestamps for change tracking, event ordering, or >> deduplication. >> >> The goal of this KIP is to: >> >> - >> >> Introduce microsecond-level timestamp handling in Kafka Connect >> schema and data representation. >> - >> >> Ensure connectors (both source and sink) can leverage this precision >> when supported by the underlying data systems. >> - >> >> Maintain backward compatibility with existing millisecond-based >> configurations and data. >> >> We welcome community feedback on: >> >> - >> >> Potential implementation concerns or edge cases we should address >> - >> >> Suggestions for schema enhancements or conversion strategies >> - >> >> Impacts on connector compatibility and testing >> >> Looking forward to your thoughts and input on this proposal! >> >> Thanks. >> Link to the KIP. >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API> >> >>
