I have made Nanosecond related changes in the KIP. Please have a look. Thanks Pritam.
On Thu, Apr 17, 2025 at 4:25 PM pritam kumar <kumarpritamm...@gmail.com> wrote: > Hi Sudesh, > Sorry for the earlier comment, I just checked that avro in 1.12 has > timestamp-nanos support. I will update the KIP to have nanosecond support > also and correspondingly I will make changes for nanosecond. > > Thanks > Pritam. > > On Wed, Apr 16, 2025 at 7:13 PM pritam kumar <kumarpritamm...@gmail.com> > wrote: > >> Also just to add I did not add this in the first place as I think Avro >> itself does not have the Nanosecond logical type. >> >> On Wed, Apr 16, 2025 at 7:01 PM pritam kumar <kumarpritamm...@gmail.com> >> wrote: >> >>> Thanks Sudesh, for taking a look at this. I am already working on >>> extending this for nanosecond precision as most sinks like iceberg have >>> started giving nanosecond precision options. >>> >>> On Wed, Apr 16, 2025 at 4:41 PM Sudesh Wasnik <wasnik...@gmail.com> >>> wrote: >>> >>>> Hi Pritam ! Thanks for the KIP ! >>>> Let’s extend the KIP to also add support for Nanosecond precision! >>>> >>>> Thanks >>>> Sudesh >>>> >>>> On 2025/04/05 01:30:49 pritam kumar wrote: >>>> > Hi Kafka Community, >>>> > Sorry due to some changes I had to change the link to the kip. >>>> > Here is the updated KIP link: >>>> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect >>>> > >>>> > On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com> >>>> > wrote: >>>> > >>>> > > Hi Kafka Community, >>>> > > >>>> > > I’d like to start a discussion on KIP-1153: Extending Support for >>>> > > Microsecond Precision for Kafka Connect >>>> > > < >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >>>> > >>>> > > . >>>> > > >>>> > > The primary motivation behind this KIP is to enhance the precision >>>> of >>>> > > timestamp handling in Kafka Connect. Currently, Kafka Connect is >>>> limited to >>>> > > millisecond-level precision for timestamps. However, many modern >>>> data >>>> > > formats and platforms have moved beyond this limitation: >>>> > > >>>> > > - >>>> > > >>>> > > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond >>>> (and >>>> > > even nanosecond) precision. For example, Avro specifies support for >>>> > > timestamp-micros (spec link >>>> > > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). >>>> > > - >>>> > > >>>> > > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* >>>> > > offer *microsecond and nanosecond precision* for time-based fields, >>>> > > making millisecond precision inadequate for accurate data >>>> replication and >>>> > > analytics in some use cases. >>>> > > >>>> > > This gap can lead to *loss of precision* when transferring data >>>> through >>>> > > Kafka Connect, especially when interacting with systems that depend >>>> on >>>> > > high-resolution timestamps for change tracking, event ordering, or >>>> > > deduplication. >>>> > > >>>> > > The goal of this KIP is to: >>>> > > >>>> > > - >>>> > > >>>> > > Introduce microsecond-level timestamp handling in Kafka Connect >>>> schema >>>> > > and data representation. >>>> > > - >>>> > > >>>> > > Ensure connectors (both source and sink) can leverage this precision >>>> > > when supported by the underlying data systems. >>>> > > - >>>> > > >>>> > > Maintain backward compatibility with existing millisecond-based >>>> > > configurations and data. >>>> > > >>>> > > We welcome community feedback on: >>>> > > >>>> > > - >>>> > > >>>> > > Potential implementation concerns or edge cases we should address >>>> > > - >>>> > > >>>> > > Suggestions for schema enhancements or conversion strategies >>>> > > - >>>> > > >>>> > > Impacts on connector compatibility and testing >>>> > > >>>> > > Looking forward to your thoughts and input on this proposal! >>>> > > >>>> > > Thanks. >>>> > > Link to the KIP. >>>> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >>>> > > < >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API >>>> > >>>> > > >>>> > > >>>> >>>> Sent with a Spark >>>> >>>
