Hi All, The SPI PR [3616] has been merged.
I propose to merge the Schema PR [3523] on Mon if there are no objections or fresh comments. [3616] https://github.com/apache/polaris/pull/3616 [3523] https://github.com/apache/polaris/pull/3523 Cheers, Dmitri. On Thu, Feb 12, 2026 at 6:00 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > The SPI PR [3616] for Scan/Commit metrics has been in review for 2 weeks > including 3 days in an approved state. > > Unless there are any last minute comments, I propose to merge it tomorrow > (Feb 13). > > [3616] https://github.com/apache/polaris/pull/3616 > > Cheers, > Dmitri. > > On Mon, Feb 9, 2026 at 2:08 PM Anand Kumar Sankaran via dev < > [email protected]> wrote: > >> Dmitri and others, >> >> Thanks for all the review feedback and being patient while we iterated. >> My initial proposal was to persist the table metrics as events (least >> amount of work 😊), we have come far from that proposal. >> >> To detach the SPI fully, there are a few things to do >> (loadSchemaVersion() and shared DatasourceOperations). We can add a >> separate datasource config to specify a different connection pool, perhaps. >> Can be done as a follow up PR. >> >> — >> Anand >> >> From: Dmitri Bourlatchkov <[email protected]> >> Date: Monday, February 9, 2026 at 10:52 AM >> To: [email protected] <[email protected]> >> Cc: Anand Kumar Sankaran <[email protected]> >> Subject: Re: Polaris Telemetry and Audit Trail >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Report Suspicious< >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumYduJ20ONMhMxljYPF8uZOe6R8BpEWyJ7tSG5GUUf8our05nYHWov0t1Vi7Rx_O8JuRNBBQQ57aBoctrR0wH7VSR9VxRuqYJPpz0q6I0JboPRfdLIwqal$ >> > >> >> Hi Anand and all, >> >> Thanks for pushing this feature forward! From my POV the SPI [3616] and >> the schema [3523] PRs are good to merge as the initial implementation of >> this feature (annotated as "beta"). The way is open to the JDBC impl. PR >> [3385] (which will probably need a rebase after merging the SPI and schema >> code). >> >> I'm sending this email to highlight the direction taken with respect to >> the SQL schema in this feature, as it might have been visible enough to the >> broader community during the PR reviews. >> >> * The scan / commit metrics schema is managed / versioned separately from >> the JDBC MetaStore schema >> >> * The Metrics Persistence SPI is detached (mostly at this time, but >> supposedly fully detached later) from the JDBC MetaStore >> >> * The JDBC DataSource for metrics may be the same or different from the >> MetaStore datasource. >> >> * Metrics Persistence may have non-JDBC implementation in the future (not >> implemented ATM) >> >> * JDBC Metrics Persistence may co-exist with other MetaStore >> implementations. >> >> I believe this design allows great flexibility for end users to try this >> feature out and keeps options open for future enhancements in Scan / Commit >> Metrics in Polaris. >> >> [3616] https://github.com/apache/polaris/pull/3616< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3616__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XEpN6Okdg$ >> > >> >> [3523] https://github.com/apache/polaris/pull/3523< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$ >> > >> >> [3385] https://github.com/apache/polaris/pull/3385< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFlxLMhvQ$ >> > >> >> Thoughts / comments? >> >> Cheers, >> Dmitri. >> >> >> On Thu, Feb 5, 2026 at 6:50 PM Anand Kumar Sankaran via dev < >> [email protected]<mailto:[email protected]>> wrote: >> Hi Dmitri, >> >> Following up on your feedback regarding the metrics schema separation >> approach. I've implemented the simplified design you suggested in >> https://github.com/apache/polaris/pull/3523< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$ >> >: >> >> >> The metrics schema can now evolve independently from the entity schema. >> Let me know if you'd like any changes to this approach. >> >> Thanks, >> Anand >> >> >> >> From: Dmitri Bourlatchkov <[email protected]<mailto:[email protected]>> >> Date: Thursday, February 5, 2026 at 2:57 PM >> To: [email protected]<mailto:[email protected]> < >> [email protected]<mailto:[email protected]>> >> Cc: Anand Kumar Sankaran <[email protected]<mailto: >> [email protected]>> >> Subject: Re: Polaris Telemetry and Audit Trail >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Report Suspicious< >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1iAus5GNbkIxSXoQaI552sXgYyv7Z9Wrvb7qeTskSy5WkXTVDJ_Bel2AbNh_fQVn-HS_UtOSdOtpsNfzhXDkK4fYWdTeoeBe70A6oQVGpAYT3SrbEvs$ >> > >> >> Hi Anand, >> >> Re: MetricsPersistence Factory - TBH I do not think it is required. If we >> go with an MetricsPersistence impl. independent of the JDBC MetaStore, it >> could be injected into PersistingMetricsReporter directly by CDI. >> >> Re: DataSource - for a start we could probably reuse the same DataSource >> as the JDBS MetaStore uses. In principle, we could introduce a separate >> DataSource, but I do not think we need to complicate the initial >> implementation with that. It can be added later, if required. >> >> Re: schema / bootstrap - I'm thinking, indeed, of having a separate *.sql >> file for metrics DDL. Then we could add an option to the bootstrap command >> in the Admin Tool, or even a new command for creating / updating the >> metrics schema. This might require some refactoring, but I believe the >> bootstrap code needs some cleanup anyway. >> >> The general idea is to allow the metrics schema to evolve independently. >> It's a new and actively developed feature, so having flexibility is going >> to be good for making progress / improvements, I hope. >> >> Does that sound reasonable? >> >> Thanks, >> Dmitri. >> >> On Thu, Feb 5, 2026 at 3:47 PM Anand Kumar Sankaran via dev < >> [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>> wrote: >> Hi Dmitri >> >> The SPI is designed to be pluggable via >> MetaStoreManagerFactory.getOrCreateMetricsPersistence()< >> https://github.com/apache/polaris/pull/3385/changes#diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51 >> < >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XFxRr4qRw$ >> >< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385/changes*diff-0e34667d083d30cace067936b87c736201e5a8cb001aa0b12e203ee52cddae99R51__;Iw!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh65taUhcw$>>. >> The default returns MetricsPersistence.NOOP, and only JDBC with schema v4+ >> returns a functional implementation. >> >> Your proposal makes sense to me. To allow NoSQL Entity Persistence + >> JDBC Metrics Persistence would require: >> >> 1. Separate metrics schema files (e.g., metrics-schema-v1.sql) that >> can be bootstrapped independently >> 2. Separate datasource configuration for metrics (e.g., >> polaris.metrics.jdbc.*) >> 3. A dedicated `MetricsPersistenceFactory` interface (separate from >> MetaStoreManagerFactory) >> 4. Updates to the Admin Tool bootstrap workflow to handle both >> schemas >> >> Does that sound reasonable? If we do this, all this work should go into >> https://github.com/apache/polaris/pull/3523< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!7IwwfPXDN0ThhNZgol-n9Ww3FFLKcpZbSQ17lWEtcQnTHtJM9aVXnqGW99_6vmVkWACy7rYC8XGOCHKhSg$ >> >< >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!_K1mcH0_2x0nbU-mZlbBi3tk9QAGjB6VInPiqkQU21mpIP_ft2bXXyIiADNoAcotNr9AKqlphh4ZISZcDQ$> >> as one atomic commit, correct? >> >> — >> Anand >> >> >> >> From: Dmitri Bourlatchkov <[email protected]<mailto:[email protected] >> ><mailto:[email protected]<mailto:[email protected]>>> >> Date: Thursday, February 5, 2026 at 8:28 AM >> To: Anand Kumar Sankaran <[email protected]<mailto: >> [email protected]><mailto:[email protected]<mailto: >> [email protected]>>> >> Cc: [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>> < >> [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>> >> Subject: Re: Polaris Telemetry and Audit Trail >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Report Suspicious< >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumbRcPlIOttKwvGr4f38TF-J2-LmOw8H1KsFpWaM3W5g28Oq5qW2gnw2-WHz0rsWhm7F7hSGykA9l01LKYGuCcCkexkHp1sUssM_IegCtjk_id9s891orv$ >> > >> >> >> Hi Anand, >> >> Thanks for driving the SPI definition for scan/commits metrics. >> >> I did not think from this angle before, but now I suppose it might be >> worth >> introducing a separate SQL schema for metrics, detached from the main JDBC >> Persistence schema. >> >> My rationale is that the SPI may be exercised independently of the JDBC >> Entity Persistence. I can imagine people using NoSQL Entity Persistence >> plus JDBC Scan Metrics Persistence in the same deployment. >> >> Using a separate schema will add further work on the Bootstrap workflow >> (Admin Tool), but I hope the end result is going to be more convenient and >> versatile for end users. >> >> WDYT? >> >> Thanks, >> Dmitri. >> >> On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran < >> [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>> wrote: >> >> > Hi Dmitri, >> > >> > Thank you for taking the time to review. I had pulled out the schema in >> > its own PR (based on comments) here: >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3523__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt3GRsh6AA$ >> > >> > I will update the document with the SPI and carve out another PR that >> > focuses on the SPI. >> > >> > — >> > Anand >> > >> > *From: *Dmitri Bourlatchkov <[email protected]<mailto: >> [email protected]><mailto:[email protected] >> <mailto:[email protected]>>> >> > *Date: *Thursday, January 29, 2026 at 10:43 AM >> > *To: *[email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>> < >> [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>> >> > *Cc: *Anand Kumar Sankaran <[email protected]<mailto: >> [email protected]><mailto:[email protected]<mailto: >> [email protected]>>> >> > *Subject: *Re: Polaris Telemetry and Audit Trail >> > >> > This Message Is From an External Sender >> > This message came from outside your organization. >> > Report Suspicious >> > < >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$ >> > >> > >> > Hi Anand, >> > >> > Thanks for making a proposal doc and starting this thread. Apologies for >> > late comments. >> > >> > As you know I already reviewed a couple of related PRs, which had a >> > smaller impact on the codebase, but now, I believe, we're coming to the >> > core of this feature and it might need a deeper discussion. >> > >> > For the sake of awareness of other project members, I'd like to >> > highlight some aspects by email. >> > >> > * Storing scan metrics in RDBMS in general looks like a >> > reasonable approach to me. However, Polaris Persistence is an extensible >> > mechanism. Other backends are possible, including privately developed >> > backends. >> > >> > From this POV, I believe it might be preferable to start not with an >> RDBMS >> > schema, but with a java SPI outlining expected read/write operations and >> > the data model. This will make it easier to assess impact on other >> > Persistence implementations. Naturally, Polaris service code will have >> to >> > be able to work seamlessly regardless of the backend impl. (including >> the >> > do-nothing impl.). >> > >> > I see that some connection between the SQL schema and java code is made >> in >> > PR [3385], but it does not appear to offer a clean SPI that could be >> > implemented by different Persistence backends. Would it be ok from your >> > perspective to make another PR with just SPI code? >> > >> > * Regarding ad-hoc SQL queries - it is certainly a valid use case for >> > custom code to query the Polaris database directly. However, those >> ad-hoc >> > queries may require certain indexes, which may not have a relevant use >> case >> > in OSS code. In that case, I believe it would be preferable to manage >> the >> > extra indexes in custom code (not in Polaris). Whether this is the case >> or >> > not currently, I cannot say with certainty, so apologies if this is a >> false >> > alarm... Having the java SPI available would be a huge aid to reasoning >> > about the proposed SQL schema, I think. >> > >> > That said, I do not mean to complicate the development of this feature, >> > just trying to structure it in a way that is hopefully easy to >> understand >> > (including myself), maintain and expand later. If I missed something, >> > please feel free to point it out. >> > >> > [3385] >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ >> > < >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ >> > >> > >> >Thanks, >> > Dmitri. >> > >> > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev < >> > [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>> wrote: >> > >> > Hi Yufei, >> > >> > Gave commenting privileges to all. >> > >> > Thanks. >> > >> > — >> > Anand >> > >> > From: Yufei Gu <[email protected]<mailto:[email protected] >> ><mailto:[email protected]<mailto:[email protected]>>> >> > Date: Wednesday, January 14, 2026 at 6:13 PM >> > To: [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>> < >> [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>> >> > Cc: Anand Kumar Sankaran <[email protected]<mailto: >> [email protected]><mailto:[email protected]<mailto: >> [email protected]>>> >> > Subject: Re: Polaris Telemetry and Audit Trail >> > >> > This Message Is From an External Sender >> > This message came from outside your organization. >> > Report Suspicious< >> > >> https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$ >> > > >> > >> > Hi Anand, thanks for the design doc and PR. Can you allow comments in >> the >> > doc so that people can chime in? Thanks! >> > >> > Yufei >> > >> > >> > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev < >> > [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>><mailto: >> [email protected]<mailto:[email protected]><mailto: >> [email protected]<mailto:[email protected]>>>> wrote: >> > Hi all >> > >> > My first PR for adding AWS STS Session Tags support for credential >> vending >> > was merged. >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt0YHY9sdA$ >> > < >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$ >> > >> >< >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$ >> >> >> > >> > I have been working with Prashant on an end-to-end telemetry and audit >> > trail tracking for Polaris. It is documented here. >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt11Y1rbNQ$ >> > < >> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$ >> > >> >< >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$ >> >> >> > >> > Based on Prashant’s initial feedback (parity with Apache Gravitino for >> > metrics reports), I have an initial PR here. >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!65BVFQDe_GtKbDW8LkNdzVS-WYewzWsSns5zFHKclbMU3vUy2UaTXLB5G-wql1yyqoTFyDc9Bt2W-eKl7A$ >> > < >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$ >> > >> >< >> > >> https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$ >> >> >> > >> > I will be reaching out to various folks for review. I am open to >> breaking >> > down the PR into smaller logical groups of PR if it helps. >> > >> > I request you to look at the Google doc and let me know what you think. >> > >> > — >> > Anand >> > >> > >> > >> > -- >> > Dmitri Bourlatchkov >> > Senior Staff Software Engineer, Dremio >> > Dremio.com >> > < >> https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$ >> >/ >> > Follow Us on LinkedIn >> > < >> https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$ >> >/ >> > Get Started >> > < >> https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$ >> > >> > >> > >> >The Agentic Lakehouse >> > *The only lakehouse built for agents, managed by agents* >> > >> > >> > >> >>
