[
https://issues.apache.org/jira/browse/HDDS-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129368#comment-18129368
]
Siyao Meng commented on HDDS-16668:
-----------------------------------
I dig a bit on using sqlite (again), regarding some previous issues:
h3. Native driver (HDDS-3411)
The reason given in HDDS-3411 no longer applies to ppc64le. sqlite-jdbc 3.25.2
(used at the time) already shipped a Linux ppc64le library, but did not map
{{os.arch=ppc64le}} to it, so loading failed. Releases since about 3.39 have
the mapping, including 3.53.4.0 used today. s390x is still not bundled (RocksDB
JNI does ship it), so making SQLite mandatory, especially for the build time
jOOQ codegen, would drop s390x unless users build {{libsqlitejdbc}} themselves.
h3. Concurrency
SQLite allows many connections and concurrent readers, but only one writer at a
time. Recon's remaining SQL writes are small and infrequent: per chunk replace
of {{UNHEALTHY_CONTAINERS}}, {{RECON_TASK_STATUS}} upserts from task threads,
{{CONTAINER_COUNT_BY_SIZE}}, and the container count stat. Serializing them is
cheap. The real problem is the default rollback journal, where a long read (the
unhealthy containers export cursor) blocks every writer.
Synthetic test with sqlite-jdbc 3.53.4.0 (8 task status writers, 50k row
replace transactions, a 12s streaming export, new connection per operation as
in {{SqliteDataSourceProvider}}):
||Config||OK||SQLITE_BUSY||Max write latency||
|default journal, busy_timeout 3s|2109|23|12.1 s|
|default journal, no export|5423|0|256 ms|
|WAL, busy_timeout 3s|5608|0|155 ms|
So SQLite is viable for Recon if {{SqliteDataSourceProvider}} enables WAL
({{?journal_mode=WAL}} in the URL works) and sets the busy timeout with
{{setBusyTimeout}} (the URL parameter is ignored by {{SQLiteDataSource}}). WAL
needs a local filesystem. The Derby specific code listed in HDDS-14451
(codegen, {{SYS.SYSCOLUMNS}} checks, hard coded URLs) still has to be ported.
h3. Other issues hit while Recon used SQLite
* HDDS-3561: the failing write was a one row {{RECON_TASK_STATUS}} update
({{SQLITE_BUSY}}), which aborted the OM delta sync and made Recon fall back to
a full snapshot. That path still exists: a single failed status write during OM
sync triggers a full OM DB snapshot and reprocess. Filed HDDS-17193 to make it
non fatal, which should land before any switch, together with a generous busy
timeout.
* HDDS-4497: {{SQLITE_CONSTRAINT}} in the file size count task. Root cause was
a {{truncate}} without {{execute()}}, not SQLite, and that table is in RocksDB
now.
* HDDS-2086: SQLite file path error, closed as Invalid (config issue).
* HDDS-8678: CVE-2023-32697 in sqlite-jdbc (code execution through a crafted
JDBC URL), fixed by a version bump. The URL is operator configuration, and
sqlite-jdbc is already a dependency that we keep current.
> Migrate from Derby
> ------------------
>
> Key: HDDS-16668
> URL: https://issues.apache.org/jira/browse/HDDS-16668
> Project: Apache Ozone
> Issue Type: Wish
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> Derby is retired. https://db.apache.org/derby/
> bq. On 2025-10-10, the Derby developers voted to retire the project into a
> read-only state. Derby development and bug-fixing have ended. No further
> releases will be published.
> Plus, there are known security vulnerabilities associated with Derby. We're
> not affected but from a supportability perspective we should move away from
> Derby as soon as feasibly possible.
> Possible solutions:
> 1. SQLite (least effort)
> 2. PostgreSQL, MariaDB, MySqL (if fullweight DB is needed, HA.. etc)
> 3. DuckDB (if we plan to have more analytical capabilities in Recon)
> 4. RocksDB (???)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
