peterxcli commented on PR #10743:
URL: https://github.com/apache/ozone/pull/10743#issuecomment-5793511193
@amaliujia I ran TPC-H SF1, 10 and 100 with:
* Spark querying HBase on Ozone
* store: Spark generates HFiles from the TPC-H data (one HBase table per
TPC-H table, row key = primary key, one column per TPC-H column), and HBase
bulk-loads them into its root dir on Ozone (RATIS/THREE).
* query: Spark SQL reads each table through HBase scans
(`MultiTableInputFormat`, 64 key-range scans, block cache off), and does all
filters, joins and aggregation itself. The Ozone reads are done by the HBase
RegionServers, as small 64 KB positional reads of HFile blocks.
* Spark querying Ozone
* store: the TPC-H Parquet files are copied unchanged with `ozone fs
-put` into a RATIS/THREE bucket, one directory per table.
* query: Spark SQL reads each table with `spark.read.parquet("ofs://…")`.
The Ozone reads are done by Spark itself, as large sequential range reads of
Parquet column chunks, with column pruning and row-group skipping.
In both cases only the Ozone client read setting changes between gRPC and
Ratis streaming read. For HBase it's set on the RegionServers, for the direct
case on Spark. Both use the same data, the same 22 queries and the same
cluster: one host, 3 datanodes, Spark `local[64]`.
Hardware:
- CPU: 2× AMD EPYC 9654 (192 cores / 384 threads)
- Memory: 377 GB
- Disk: one 2 TB Kingston KC3000 NVMe;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
