yuqi1129 opened a new pull request, #13620: URL: https://github.com/apache/gravitino/pull/13620
### What changes were proposed in this pull request? Make JDBC idle-connection retention configurable for the entity-store pool and for JDBC catalog pools. - Entity store: add `gravitino.entity.store.relational.maxIdleConnections` (default `32`, capped by `maxConnections`) in place of the hard-coded `maxIdle = 10` in `SqlSessionFactoryHelper`. `minIdle` is capped by the effective `maxIdle`. The Helm chart exposes it as `entity.maxIdleConnections`. - JDBC catalogs: add the catalog property `jdbc.pool.max-idle` (default `8`, DBCP's previous default, capped by `jdbc.pool.max-size`). When it is not set, a `maxIdle` supplied through `gravitino.bypass.` is still honored; when both are set, `jdbc.pool.max-idle` wins. - Document both settings, including that entity-store idle connections are released slowly (the evictor checks at most three idle connections every ten minutes) and that catalog pools do not evict idle connections by default. - Add `dev/performance/jdbc_idle_retention.py`, a script that measures MySQL connection churn and HTTP tail latency under concurrent reads. ### Why are the changes needed? With `maxConnections = 100` but `maxIdle = 10`, a 64-client metalake-list workload repeatedly destroyed and recreated database connections. The measurements in #13582 compare `maxIdle = 10` with `maxIdle = 100`: about 3,400 physical connections per measurement window versus 4–36, and a median p99 of 214 ms versus 90 ms. The idle limit was not configurable. The default of `32` was not benchmarked. It is a compromise between those two measured points and the database connection budget of multi-node deployments: with N servers, the entity store can keep up to 32 × N idle connections (previously 10 × N), in addition to JDBC catalog pools. The peak per server is unchanged (still bounded by `maxConnections`); only the number of connections kept after a burst grows. High-concurrency deployments can raise it; deployments with a tight connection budget can set it back to `10`. Out of scope, left for follow-ups: pool created/destroyed metrics for diagnosing churn (MySQL's `Connections` counter and the script cover it for now), and validating catalog pool properties before datasource creation so an invalid value is reported by name rather than as "Error creating datasource" (an existing behavior shared with `jdbc.pool.max-size`). Both new settings accept only positive values on purpose; a catalog that needs DBCP's `0` or `-1` can still use `gravitino.bypass.maxIdle`. Fix: #13582 ### Does this PR introduce _any_ user-facing change? Yes. - New server configuration `gravitino.entity.store.relational.maxIdleConnections`. The entity-store idle limit changes from 10 to 32 per server, and idle connections are released slowly, so budget for the full value on every server. - New JDBC catalog property `jdbc.pool.max-idle`. Its default equals the previous behavior, so existing catalogs are unchanged. - New Helm value `entity.maxIdleConnections` (unset by default). ### How was this patch tested? - Unit tests for the new settings, their caps and defaults, and the `gravitino.bypass.maxIdle` precedence (tested at the `DataSourceUtils` level; the prefix stripping in `JdbcCatalogOperations.initialize` is unchanged): `TestSqlSession`, `TestConfig`, `TestJdbcConfig`, `TestJdbcCatalogOperations` (including a burst test that checks physical connections are reused), and `TestJdbcCatalogPropertiesMetadata`. - Every test class that stubs the entity-store config, plus the full `catalog-jdbc-common` and `idp-basic` suites, pass locally. Reverting either production change makes five of the new tests fail. - `:core:compileJmhJava`, javadoc for `core` and `catalog-jdbc-common`, and `spotlessApply`. The Helm chart tests are covered by CI (`chart-test`); Helm is not installed locally. - The benchmark script's self-test: `cd dev/performance && python3 -m unittest test_jdbc_idle_retention`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
