Hi all, Closing the loop here, the below PR incorporates the outcome of this discussion regarding the persistence of request ID and OTel context. The former is persisted as before (option 2) and the latter is being stored in the additional_properties column, as suggested by Yufei.
https://github.com/apache/polaris/pull/2998 Thanks, Alex On Wed, Nov 5, 2025 at 8:39 AM Jean-Baptiste Onofré <[email protected]> wrote: > > Hi Alex > > I think option 2 is slightly better. We can imagine adding a prefix > for context (not sure it's needed now). > > Regards > JB > > On Fri, Oct 31, 2025 at 1:22 PM Alexandre Dutra <[email protected]> wrote: > > > > Hi all, > > > > As a follow-up to [1], I'm starting this thread to discuss how we can > > persist client-generated request IDs and OTel context in our database. > > > > Quoting myself [2], the requirements I think we want to fulfil are: > > > > 1. Only one correlation ID is enough > > 2. The correlation ID is an opaque string > > 3. The main use case is to find events with matching correlation IDs > > 4. The only query pattern is by exact match > > > > A few options were suggested: > > > > 1) Two separate columns: request_id and otel_context, both of type > > TEXT (nullable). > > - Pros: Easy to implement and offers good read performance, > > especially with indexes. > > - Cons: Could be overkill, as often one context is sufficient for > > correlating records. > > > > 2) A single column: (e.g., correlation_id, final name TBD) of type TEXT. > > - Pros: Same as option 1. > > - Cons: If both a request ID and OTel context are available, we > > can only store one. There's no straightforward way to identify the > > type of context stored (unless we use a prefix). > > > > 3) Two columns: correlation_id and correlation_id_type. > > - Pros: Same as option 1, and it addresses the issue of > > identifying the ID type. > > - Cons: Might be over-engineered. Is the ID type truly essential? > > Isn't it opaque? > > > > 4) Leverage the existing additional_properties column: (JSONB in > > Postgres, TEXT in H2). > > - Pros: Simple and flexible due to its schemaless nature, allowing > > us to add anything we need. > > - Cons: Query performance might not be optimal, though indexes could > > help. > > > > What are your thoughts? > > > > Thanks, > > Alex > > > > [1] https://lists.apache.org/thread/bb1qyxjt827t3tomv2xp0s1kovxjsp94 > > [2] https://lists.apache.org/thread/fqjjmxc6v8bbynwd5xfz83ngmp6gqqxj
