morningman commented on code in PR #68848:
URL: https://github.com/apache/doris/pull/68848#discussion_r4235999678
##########
fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java:
##########
@@ -462,6 +468,124 @@ public void missingReusePropertyDoesNotReuse() {
"an older planning FE's missing property must not enable reuse
in a newer connector");
}
+ @Test
+ public void statementReuseKeepsScansWithDifferentInputsApart() {
+ // Separately built but equal handles share one plan; a scan that
differs in any input the planned ranges
+ // depend on plans on its own (the ranges carry the partition files
and the BE file format).
+ // MUTATION: dropping a fact from HiveScanReuseKey (or the catalog id
from the memo key) makes that scan
+ // return an earlier plan -> red.
+ HiveScanPlanProvider provider = provider(new FakeHmsClient(), new
CountingLister());
+ TestStatementScope scope = new TestStatementScope();
+ ConnectorSession session = new ScopeSession(7L, "same-statement",
scope);
+ HiveTableHandle text = textHandle("t", LAZY_SIMPLE_SERDE,
part("year=2024/month=01"));
+
+ List<ConnectorScanRange> plain = provider.planScan(session,
scanOf(text));
+ Assertions.assertEquals(1, plain.size());
+ Assertions.assertSame(plain, provider.planScan(session,
+ scanOf(textHandle("t", LAZY_SIMPLE_SERDE,
part("year=2024/month=01")))),
+ "separately built but equal handles must share one plan");
+ List<ConnectorScanRange> otherPartition = provider.planScan(session,
+ scanOf(textHandle("t", LAZY_SIMPLE_SERDE,
part("year=2024/month=02"))));
+ List<ConnectorScanRange> otherTable = provider.planScan(session,
+ scanOf(textHandle("t2", LAZY_SIMPLE_SERDE,
part("year=2024/month=01"))));
+ List<ConnectorScanRange> csv = provider.planScan(session,
+ scanOf(textHandle("t", OPEN_CSV_SERDE,
part("year=2024/month=01"))));
+ List<ConnectorScanRange> stringFirstColumn = provider.planScan(session,
+ scanOf(text.toBuilder().firstColumnIsString(true).build()));
+ List<ConnectorScanRange> parquet = provider.planScan(session,
scanOf(text.toBuilder()
+ .inputFormat(PARQUET_INPUT_FORMAT)
+ .serializationLib(PARQUET_SERDE)
+ .build()));
Review Comment:
Fixed in 6ebef6bf755. The test now also plans the same partition with
`DeprecatedLzoTextInputFormat` and the plain text serde (`LazySimpleSerDe`), so
only the input format differs from the plain text scan. That difference matters
on its own: an LZO text table reads only `*.lzo` data files, so this scan's
plan is empty (the fixture lists only `000000_0`) while the plain text scan
gets one range. Removing `inputFormat` from `HiveScanReuseKey#equals` and
`#hashCode` now fails the test, because the LZO scan gets the plain text scan's
range. I kept the existing listing rather than adding a `.lzo` file, so the two
plans differ in what they read and not only in object identity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
