maomaodev commented on PR #7436: URL: https://github.com/apache/kyuubi/pull/7436#issuecomment-4406992290
> How can KSHC be faster than Vanilla Spark? Thanks for the review! After investigation, KSHC being faster than vanilla Spark on TEXT-format Hive tables mainly comes from two factors: 1、Different file-splitting strategies * Vanilla Spark reads via HadoopRDD + `mapred.FileInputFormat.getSplits(JobConf, numSplits)`: ``` splitSize = max(minSize, min(goalSize, blockSize)), goalSize = totalSize / numSplits ``` With the defaults (minSize=1B, blockSize=128MB, numSplits=2), splitSize is only ~2MB, so each small file (1–4MB) is split into ~2 tasks, and scheduling overhead dominates. * KSHC uses DataSource V2 FileScan + FilePartition: ``` maxSplitBytes = min(maxPartitionBytes, max(openCostInBytes, bytesPerCore)) ``` With the defaults (maxPartitionBytes=128MB, openCostInBytes=4MB), each small file becomes at most one task. * Validation: on q88, setting mapreduce.input.fileinputformat.split.minsize=128M on vanilla Spark to align splitting with KSHC drops the query from 249.8s → 153.9s (~52% of the gap). 2、KSHC reuses FileStatus via FileStatusCache * Vanilla Spark goes through HiveMetastoreCatalog → HadoopFsRelation on every scan and re-listStatus per partition every time. KSHC's HiveCatalogFileIndex reuses FileStatus across scans within a session. * Validation: on q88, replacing KSHC's fileStatusCache with NoopCache increases the query from 65.9s → 105.2s (~22% of the gap). **Note on ORC** We also tested 10GB TPC-DS in ORC format, where vanilla Spark is faster than KSHC, this issue does not appear there. **On the Spark 3.3 CI failure** SupportsRuntimeV2Filtering was introduced in Spark 3.4, so cross-version compilation against 3.3 fails. Any suggestions on the preferred direction? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
