Hi Sudesh, Sorry for the earlier comment, I just checked that avro in 1.12 has timestamp-nanos support. I will update the KIP to have nanosecond support also and correspondingly I will make changes for nanosecond.
Thanks Pritam. On Wed, Apr 16, 2025 at 7:13 PM pritam kumar <kumarpritamm...@gmail.com> wrote: > Also just to add I did not add this in the first place as I think Avro > itself does not have the Nanosecond logical type. > > On Wed, Apr 16, 2025 at 7:01 PM pritam kumar <kumarpritamm...@gmail.com> > wrote: > >> Thanks Sudesh, for taking a look at this. I am already working on >> extending this for nanosecond precision as most sinks like iceberg have >> started giving nanosecond precision options. >> >> On Wed, Apr 16, 2025 at 4:41 PM Sudesh Wasnik <wasnik...@gmail.com> >> wrote: >> >>> Hi Pritam ! Thanks for the KIP ! >>> Let’s extend the KIP to also add support for Nanosecond precision! >>> >>> Thanks >>> Sudesh >>> >>> On 2025/04/05 01:30:49 pritam kumar wrote: >>> > Hi Kafka Community, >>> > Sorry due to some changes I had to change the link to the kip. >>> > Here is the updated KIP link: >>> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1154%3A+Extending+support+for+Microsecond+Precision+for+Kafka+Connect >>> > >>> > On Sat, Apr 5, 2025 at 12:14 AM pritam kumar <ku...@gmail.com> >>> > wrote: >>> > >>> > > Hi Kafka Community, >>> > > >>> > > I’d like to start a discussion on KIP-1153: Extending Support for >>> > > Microsecond Precision for Kafka Connect >>> > > < >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >>> > >>> > > . >>> > > >>> > > The primary motivation behind this KIP is to enhance the precision of >>> > > timestamp handling in Kafka Connect. Currently, Kafka Connect is >>> limited to >>> > > millisecond-level precision for timestamps. However, many modern data >>> > > formats and platforms have moved beyond this limitation: >>> > > >>> > > - >>> > > >>> > > Formats such as *Avro*, *Parquet*, and *ORC* support microsecond (and >>> > > even nanosecond) precision. For example, Avro specifies support for >>> > > timestamp-micros (spec link >>> > > <https://avro.apache.org/docs/1.11.0/spec.html#timestamp-micros>). >>> > > - >>> > > >>> > > Sink systems like *Apache Iceberg*, *Delta Lake*, and *Apache Hudi* >>> > > offer *microsecond and nanosecond precision* for time-based fields, >>> > > making millisecond precision inadequate for accurate data >>> replication and >>> > > analytics in some use cases. >>> > > >>> > > This gap can lead to *loss of precision* when transferring data >>> through >>> > > Kafka Connect, especially when interacting with systems that depend >>> on >>> > > high-resolution timestamps for change tracking, event ordering, or >>> > > deduplication. >>> > > >>> > > The goal of this KIP is to: >>> > > >>> > > - >>> > > >>> > > Introduce microsecond-level timestamp handling in Kafka Connect >>> schema >>> > > and data representation. >>> > > - >>> > > >>> > > Ensure connectors (both source and sink) can leverage this precision >>> > > when supported by the underlying data systems. >>> > > - >>> > > >>> > > Maintain backward compatibility with existing millisecond-based >>> > > configurations and data. >>> > > >>> > > We welcome community feedback on: >>> > > >>> > > - >>> > > >>> > > Potential implementation concerns or edge cases we should address >>> > > - >>> > > >>> > > Suggestions for schema enhancements or conversion strategies >>> > > - >>> > > >>> > > Impacts on connector compatibility and testing >>> > > >>> > > Looking forward to your thoughts and input on this proposal! >>> > > >>> > > Thanks. >>> > > Link to the KIP. >>> > > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Extending+Support+for+Microsecond+Precision+for+Kafka+Connect >>> > > < >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+prefix+filter+to+ListTransactions+API >>> > >>> > > >>> > > >>> >>> Sent with a Spark >>> >>
