Hi Yufei, Using one datasource for now sounds reasonable to me.
Cheers, Dmitri. On Wed, Feb 25, 2026 at 3:37 PM Yufei Gu <[email protected]> wrote: > Thanks a lot Anand and Dmitri for working on this! > > Supporting multiple data sources introduces significant complexity as I > recorded in this issue, https://github.com/apache/polaris/issues/3890. It > requires a well-thought-out design before we commit to it. I'd propose to > leave it now and move forward with the single data source as the initial > step. WDYT? > > Yufei > > > On Tue, Feb 24, 2026 at 12:17 PM Anand Kumar Sankaran via dev < > [email protected]> wrote: > > > Hi Dmitri > > > > Thanks for all your support again. > > > > I talked to Yufei and I have addressed his review feedback in this > commit. > > > https://github.com/apache/polaris/pull/3385/changes/9e3d4d0f409720ab4005db176190ebde29a9d2b6 > > > > Yufei and Prashant - can you please review these changes? Dmitri has > > already reviewed the PR and approved. > > > > - > > Anand > > > > From: Dmitri Bourlatchkov <[email protected]> > > Date: Tuesday, February 24, 2026 at 10:38 AM > > To: [email protected] <[email protected]> > > Cc: Anand Kumar Sankaran <[email protected]> > > Subject: Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1xO1e2JFuKSh-PdIU2YhFBQOvhapv4xXZ0ucXUR7TPwnqpAVpb05t786Kda3YPcEe-qOyYlMkJM8hj02m4Yv48r-c47jSruOAIe3mbeFZ5dBQBrfOUk$ > > > > > > > Hi Yufei, > > > > I see that you propose a different approach to schema evolution for > > metrics in GH [1] > > > > Please share your view of this in light of the previous discussion > > regarding combining the metrics persistence with different metastore > > persistence options. It would be good to sync up on this email thread > since > > GH comments tend to be hidden on long PRs and not all readers may be able > > to easily find them. > > > > [1] https://github.com/apache/polaris/pull/3385#issuecomment-3953906091< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385*issuecomment-3953906091__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKT8_s2eg$ > > > > > > > Thanks, > > Dmitri. > > > > On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev < > > [email protected]<mailto:[email protected]>> wrote: > > Hi Dmitri > > > > The SPI is designed to be pluggable via > > MetaStoreManagerFactory.getOrCreateMetricsPersistence()< > > > https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51 > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLKXRU26ZA$ > >>. > > The default returns MetricsPersistence.NOOP, and only JDBC with schema > v4+ > > returns a functional implementation. > > > > Your proposal makes sense to me. To allow NoSQL Entity Persistence + > JDBC > > Metrics Persistence would require: > > > > 1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that > > can be bootstrapped independently > > 2. Separate datasource configuration for metrics (e.g., > > polaris.metrics.jdbc.*) > > 3. A dedicated `MetricsPersistenceFactory` interface (separate from > > MetaStoreManagerFactory) > > 4. Updates to the Admin Tool bootstrap workflow to handle both > schemas > > > > Does that sound reasonable? If we do this, all this work should go into > > https://github.com/apache/polaris/pull/3523< > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!-KOwW_42EOhdoNvlXd2GvBD6Z7u_HteBuHkQAow0Z11eX9-Q5LAWMr02rP-8OE8VlcwlG4hSnLIg5ApDFA$ > > > > as one atomic commit, correct? > > > > — > > Anand > > > > > > > > From: Dmitri Bourlatchkov <[email protected]<mailto:[email protected]>> > > Date: Thursday, February 5, 2026 at 8:28 AM > > To: Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]>> > > Cc: [email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > Subject: Re: Polaris Telemetry and Audit Trail > > > > This Message Is From an External Sender > > This message came from outside your organization. > > Report Suspicious< > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$ > > > > > > > > > Hi Anand, > > > > Thanks for driving the SPI definition for scan/commits metrics. > > > > I did not think from this angle before, but now I suppose it might be > worth > > introducing a separate SQL schema for metrics, detached from the main > JDBC > > Persistence schema. > > > > My rationale is that the SPI may be exercised independently of the JDBC > > Entity Persistence. I can imagine people using NoSQL Entity Persistence > > plus JDBC Scan Metrics Persistence in the same deployment. > > > > Using a separate schema will add further work on the Bootstrap workflow > > (Admin Tool), but I hope the end result is going to be more convenient > and > > versatile for end users. > > > > WDYT? > > > > Thanks, > > Dmitri. > > > > On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran < > > [email protected]<mailto:[email protected]>> wrote: > > > > > Hi Dmitri, > > > > > > Thank you for taking the time to review. I had pulled out the schema in > > > its own PR (based on comments) here: > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$ > > > > > > I will update the document with the SPI and carve out another PR that > > > focuses on the SPI. > > > > > > — > > > Anand > > > > > > *From: *Dmitri Bourlatchkov <[email protected]<mailto: > > [email protected]>> > > > *Date: *Thursday, January 29, 2026 at 10:43 AM > > > *To: *[email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > > *Cc: *Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]>> > > > *Subject: *Re: Polaris Telemetry and Audit Trail > > > > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Report Suspicious > > > < > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ > > > > > > > > > Hi Anand, > > > > > > Thanks for making a proposal doc and starting this thread. Apologies > for > > > late comments. > > > > > > As you know I already reviewed a couple of related PRs, which had a > > > smaller impact on the codebase, but now, I believe, we're coming to the > > > core of this feature and it might need a deeper discussion. > > > > > > For the sake of awareness of other project members, I'd like to > > > highlight some aspects by email. > > > > > > * Storing scan metrics in RDBMS in general looks like a > > > reasonable approach to me. However, Polaris Persistence is an > extensible > > > mechanism. Other backends are possible, including privately developed > > > backends. > > > > > > From this POV, I believe it might be preferable to start not with an > > RDBMS > > > schema, but with a java SPI outlining expected read/write operations > and > > > the data model. This will make it easier to assess impact on other > > > Persistence implementations. Naturally, Polaris service code will have > to > > > be able to work seamlessly regardless of the backend impl. (including > the > > > do-nothing impl.). > > > > > > I see that some connection between the SQL schema and java code is made > > in > > > PR [3385], but it does not appear to offer a clean SPI that could be > > > implemented by different Persistence backends. Would it be ok from your > > > perspective to make another PR with just SPI code? > > > > > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for > > > custom code to query the Polaris database directly. However, those > ad-hoc > > > queries may require certain indexes, which may not have a relevant use > > case > > > in OSS code. In that case, I believe it would be preferable to manage > the > > > extra indexes in custom code (not in Polaris). Whether this is the case > > or > > > not currently, I cannot say with certainty, so apologies if this is a > > false > > > alarm... Having the java SPI available would be a huge aid to reasoning > > > about the proposed SQL schema, I think. > > > > > > That said, I do not mean to complicate the development of this feature, > > > just trying to structure it in a way that is hopefully easy to > understand > > > (including myself), maintain and expand later. If I missed something, > > > please feel free to point it out. > > > > > > [3385] > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ > > > > > > > > >Thanks, > > > Dmitri. > > > > > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev < > > > [email protected]<mailto:[email protected]>> wrote: > > > > > > Hi Yufei, > > > > > > Gave commenting privileges to all. > > > > > > Thanks. > > > > > > — > > > Anand > > > > > > From: Yufei Gu <[email protected]<mailto:[email protected]>> > > > Date: Wednesday, January 14, 2026 at 6:13 PM > > > To: [email protected]<mailto:[email protected]> < > > [email protected]<mailto:[email protected]>> > > > Cc: Anand Kumar Sankaran <[email protected]<mailto: > > [email protected]>> > > > Subject: Re: Polaris Telemetry and Audit Trail > > > > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Report Suspicious< > > > > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$ > > > > > > > > > > Hi Anand, thanks for the design doc and PR. Can you allow comments in > the > > > doc so that people can chime in? Thanks! > > > > > > Yufei > > > > > > > > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev < > > > [email protected]<mailto:[email protected]><mailto: > > [email protected]<mailto:[email protected]>>> wrote: > > > Hi all > > > > > > My first PR for adding AWS STS Session Tags support for credential > > vending > > > was merged. > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$ > > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$ > > > > > >< > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$ > > >> > > > > > > I have been working with Prashant on an end-to-end telemetry and audit > > > trail tracking for Polaris. It is documented here. > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$ > > > < > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$ > > > > > >< > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$ > > >> > > > > > > Based on Prashant’s initial feedback (parity with Apache Gravitino for > > > metrics reports), I have an initial PR here. > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ > > > < > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ > > > > > >< > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$ > > >> > > > > > > I will be reaching out to various folks for review. I am open to > breaking > > > down the PR into smaller logical groups of PR if it helps. > > > > > > I request you to look at the Google doc and let me know what you think. > > > > > > — > > > Anand > > > > > > > > > > > > -- > > > Dmitri Bourlatchkov > > > Senior Staff Software Engineer, Dremio > > > Dremio.com > > > < > > > https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$ > > >/ > > > Follow Us on LinkedIn > > > < > > > https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$ > > >/ > > > Get Started > > > < > > > https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$ > > > > > > > > > > > >The Agentic Lakehouse > > > *The only lakehouse built for agents, managed by agents* > > > > > > > > > > > > > >
