This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a change to branch
xiangfu0/codex/codec-stack/07-integration-docs
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard 03550e6b0e0 Add codec pipeline integration tests and design doc
discard 6e90858fafe Report uncompressed value size for V7 codec-pipeline
columns
discard 7fb73752349 Harden codec reload and compression stats
discard a028cb0d2d7 Support codecSpec on segment reload and enable codecSpec
end to end
discard 01973633547 Resolve configured codec specs once per reload and add a
V7 write benchmark
discard e34f73a464a Validate V7 frames lazily and harden the review follow-ups
discard 96e77eca5b4 Add V7 raw forward index format for codec pipelines
add 554f9cda26b Add null-aware star-tree index (#19367)
add 8e814b0dbc6 Reuse UTF-8 scratch for DataTable dictionary decoding
(#19535)
add ff2eee7fafb Support PostgreSQL hex bytea literals (#19263)
add 222b41877c1 Bump software.amazon.awssdk:bom from 2.55.0 to 2.55.1
(#19630)
add 5c5a7940755 Hash the full value in fixed-byte BYTES and BIG_DECIMAL
dictionary batch murmur3 reads (#19634)
add fe8cd0261ec Change SegmentMetadata.getCrc and getDataCrc to return
long (#19629)
add 0306987954e Keep orphaned multipart temp files out of java.io.tmpdir
(#19627)
add bcb0d466e9d [MSE] Build ORDER BY on SortOperator instead of
SortedMailboxReceiveOperator (#19412)
add d30d683ba7f Bump com.dynatrace.hash4j:hash4j from 0.30.0 to 0.31.0
(#19641)
add c6efbfb4177 Bump joda-time:joda-time from 2.14.3 to 2.14.4 (#19640)
add 5725f884739 Bump com.github.luben:zstd-jni from 1.5.7-17 to 1.5.7-18
(#19639)
add cfd14629af2 Fix re-reading a huge value with the same
VarByteChunkForwardIndexReaderV4 context (#19638)
add 06c8f7e2ac2 Upgrade Apache parent to 40 and migrate to native Surefire
3.6 (#19633)
add 33d88e480ba Remove unused ValueReader.getPaddedString API (#19635)
add 8d67b0a3489 Add V7 raw forward index format for codec pipelines
add 5aa087054db Validate V7 frames lazily and harden the review follow-ups
add 2607ae372bd Resolve configured codec specs once per reload and add a
V7 write benchmark
add 1502224e6cf Support codecSpec on segment reload and enable codecSpec
end to end
add 10c5aa67bbb Harden codec reload and compression stats
add f2b14ebef2c Report uncompressed value size for V7 codec-pipeline
columns
add d57fdbdc0a1 Add codec pipeline integration tests and design doc
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (03550e6b0e0)
\
N -- N -- N
refs/heads/xiangfu0/codex/codec-stack/07-integration-docs (d57fdbdc0a1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.mvn/jvm.config | 3 +-
.../LiteralOnlyBrokerRequestTest.java | 50 +-
pinot-common/pom.xml | 26 +-
pinot-common/src/main/codegen/config.fmpp | 2 +
.../src/main/codegen/includes/parserImpls.ftl | 25 +
.../pinot/common/datablock/BaseDataBlock.java | 35 -
.../common/datablock/ZeroCopyDataBlockSerde.java | 22 +-
.../pinot/common/datatable/DataTableImplV4.java | 7 +-
.../pinot/common/datatable/DataTableUtils.java | 29 +
.../metadata/segment/SegmentZKMetadataUtils.java | 6 +-
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 3 +
.../pinot/sql/parsers/PostgreSqlCastRewriter.java | 201 ++++
.../datablock/ZeroCopyDataBlockSerdeTest.java | 27 +
.../DataTableUtilsDecodeStringArrayTest.java | 174 ++++
.../pinot/common/request/LiteralSerDeTest.java | 2 +
.../pinot/common/utils/NativeTestNGSuiteTest.java | 230 +++++
.../pinot/common/utils/OrderedTestNGSuite.java | 49 +
.../common/utils/request/RequestUtilsTest.java | 6 +
.../pinot/sql/parsers/CalciteSqlParserTest.java | 233 +++++
pinot-controller/pom.xml | 85 +-
.../api/ControllerAdminApiApplication.java | 58 ++
.../api/resources/ControllerFilePathProvider.java | 12 +
.../PinotSegmentUploadDownloadRestletResource.java | 19 +-
.../pinot/controller/api/upload/ZKOperator.java | 2 +-
.../realtime/provisioning/MemoryEstimator.java | 4 +-
.../pinot/controller/ControllerTestSetup.java | 2 +-
.../api/ControllerAdminApiApplicationTest.java | 216 ++++
.../api/ControllerFilePathProviderTest.java | 48 +
.../api/PinotSegmentRestletResourceTest.java | 4 +-
...otSegmentUploadDownloadRestletResourceTest.java | 39 +
.../controller/api/upload/ZKOperatorTest.java | 20 +-
.../LineageDeleteInterleavingIntegrationTest.java | 2 +-
.../PinotLLCRealtimeSegmentManagerTest.java | 14 +-
.../controller/suites/ControllerStatefulSuite.java | 28 +-
.../suites/ControllerStatelessSuite.java | 28 +-
.../controller/utils/SegmentMetadataMockUtils.java | 32 +-
.../validation/ValidationManagerTest.java | 2 +-
pinot-controller/testng-statefull.xml | 39 -
pinot-controller/testng-stateless.xml | 41 -
.../core/data/manager/BaseTableDataManager.java | 17 +-
.../core/operator/query/AggregationOperator.java | 11 +-
.../query/FilteredAggregationOperator.java | 6 +-
.../operator/query/FilteredGroupByOperator.java | 6 +-
.../pinot/core/operator/query/GroupByOperator.java | 13 +-
.../function/AggregationFunctionUtils.java | 72 +-
.../function/AvgAggregationFunction.java | 32 +-
.../apache/pinot/core/startree/StarTreeUtils.java | 204 ++--
.../executor/StarTreeAggregationExecutor.java | 17 +-
.../startree/executor/StarTreeGroupByExecutor.java | 26 +-
.../datatable/DataTableDictionarySerDeTest.java | 95 ++
...ableDataManagerEnqueueSegmentToReplaceTest.java | 2 +-
.../data/manager/BaseTableDataManagerTest.java | 24 +-
.../offline/DimensionTableDataManagerTest.java | 2 +-
.../realtime/RealtimeSegmentDataManagerTest.java | 2 +-
...centileSmartTDigestAggregationFunctionTest.java | 49 -
...igestHighThresholdAggregationFunctionTest.java} | 8 +-
...gestSmallThresholdAggregationFunctionTest.java} | 31 +-
.../pinot/core/startree/StarTreeUtilsTest.java | 15 +-
.../queries/NullAwareStarTreeQueriesTest.java | 387 +++++++
pinot-integration-tests/pom.xml | 65 +-
.../GroupByEnableTrimOptionIntegrationTest.java | 4 +-
.../tests/OfflineClusterIntegrationTest.java | 2 +-
.../integration/tests/custom/BytesMvTypeTest.java | 56 +-
.../integration/tests/custom/BytesTypeTest.java | 21 +
.../tests/custom/GroupByOptionsTest.java | 18 +-
.../tests/custom/GroupByTrimmingTest.java | 10 +-
.../tests/suites/CustomClusterSuite.java | 27 +-
.../integration/tests/suites/KinesisSuite.java | 41 +
.../tests/suites/MultiNodesOfflineSuite.java | 38 +
.../tests/suites/SharedHybridSuite.java | 43 +
.../tests/suites/SharedKafkaRealtimeSuite.java | 45 +
.../custom-cluster-integration-test-suite.xml | 28 -
.../resources/kinesis-integration-test-suite.xml | 29 -
.../multi-nodes-offline-integration-test-suite.xml | 36 -
.../shared-hybrid-integration-test-suite.xml | 31 -
...hared-kafka-realtime-integration-test-suite.xml | 33 -
.../perf/BenchmarkMseSortImplementations.java | 182 ++++
.../tasks/BaseSingleSegmentConversionExecutor.java | 2 +-
.../pinot/plugin/minion/tasks/MinionTaskUtils.java | 8 +-
.../UpsertCompactionTaskExecutor.java | 6 +-
.../UpsertCompactMergeTaskExecutor.java | 11 +-
.../BaseSingleSegmentConversionExecutorTest.java | 4 +-
.../plugin/minion/tasks/MinionTaskUtilsTest.java | 46 +-
.../UpsertCompactMergeTaskExecutorTest.java | 44 +-
.../PinotAggregateExchangeNodeInsertRule.java | 11 +-
.../rel/rules/PinotSortExchangeNodeInsertRule.java | 9 +-
.../rules/PinotWindowExchangeNodeInsertRule.java | 49 +-
.../apache/pinot/query/QueryCompilationTest.java | 26 +
.../pinot/query/QueryPlannerRuleOptionsTest.java | 12 +-
.../src/test/resources/queries/GroupByPlans.json | 8 +-
.../src/test/resources/queries/JoinPlans.json | 4 +-
.../src/test/resources/queries/OrderByPlans.json | 22 +-
.../test/resources/queries/PinotHintablePlans.json | 8 +-
.../resources/queries/WindowFunctionPlans.json | 1066 +++++++++++---------
.../query/runtime/operator/FullSortOperator.java | 106 ++
.../query/runtime/operator/LimitSortOperator.java | 107 ++
.../pinot/query/runtime/operator/SortOperator.java | 255 +++--
.../operator/SortedMailboxReceiveOperator.java | 13 +
.../query/runtime/operator/TopNSortOperator.java | 109 ++
.../query/runtime/plan/PlanNodeToOpChain.java | 2 +-
.../operator/OperatorBufferReleaseTest.java | 44 +-
.../query/runtime/operator/SortOperatorTest.java | 221 +++-
.../src/test/resources/queries/BinaryTypes.json | 27 +-
.../local/aggregator/CountValueAggregator.java | 11 +
.../DistinctCountBitmapValueAggregator.java | 4 +-
.../DistinctCountThetaSketchValueAggregator.java | 4 +-
.../segment/local/aggregator/ValueAggregator.java | 22 +
.../local/data/manager/SegmentDataManager.java | 2 +-
.../immutable/ImmutableSegmentLoader.java | 2 +-
.../local/io/util/FixedByteValueReaderWriter.java | 6 +-
.../pinot/segment/local/io/util/ValueReader.java | 25 +-
.../local/io/util/VarLengthValueReader.java | 26 +-
.../index/readers/BaseImmutableDictionary.java | 20 +-
.../index/readers/BigDecimalDictionary.java | 7 +-
.../segment/index/readers/BytesDictionary.java | 7 +-
.../segment/index/readers/StringDictionary.java | 7 +-
.../forward/VarByteChunkForwardIndexReaderV4.java | 3 +-
.../local/segment/store/StarTreeIndexReader.java | 21 +
.../SegmentCrcVirtualColumnProvider.java | 23 +-
.../local/startree/StarTreeBuilderUtils.java | 4 +
.../startree/v2/builder/BaseSingleTreeBuilder.java | 236 ++++-
.../v2/builder/OffHeapSingleTreeBuilder.java | 73 +-
.../startree/v2/builder/StarTreeIndexCombiner.java | 16 +
.../v2/builder/StarTreeIndexSeparator.java | 5 +
.../v2/builder/StarTreeV2BuilderConfig.java | 39 +-
.../startree/v2/store/StarTreeDataSource.java | 9 +-
.../startree/v2/store/StarTreeIndexMapUtils.java | 14 +-
.../startree/v2/store/StarTreeLoaderUtils.java | 24 +-
.../segment/local/utils/TableConfigUtils.java | 7 +-
.../converter/RealtimeSegmentConverterTest.java | 2 +-
.../segment/index/SegmentMetadataImplTest.java | 2 +-
.../SegmentV1V2ToV3FormatConverterTest.java | 2 +-
.../segment/index/creator/VarByteChunkV4Test.java | 66 ++
.../local/segment/index/loader/LoaderTest.java | 2 +-
.../index/readers/ImmutableDictionaryTest.java | 97 ++
.../SegmentMetadataVirtualColumnProviderTest.java | 2 +-
.../v2/builder/NullAwareStarTreeBuilderTest.java | 230 +++++
.../v2/builder/StarTreeBuilderUtilsTest.java | 2 +-
.../apache/pinot/segment/spi/SegmentMetadata.java | 7 +-
.../spi/index/metadata/SegmentMetadataImpl.java | 8 +-
.../startree/AggregationFunctionColumnPair.java | 49 +-
.../spi/index/startree/StarTreeV2Constants.java | 2 +
.../spi/index/startree/StarTreeV2Metadata.java | 21 +-
.../spi/loader/SegmentDirectoryLoaderContext.java | 11 +-
.../spi/index/startree/StarTreeV2MetadataTest.java | 3 +-
.../api/resources/SegmentMetadataFetcher.java | 4 +
.../pinot/server/api/resources/TablesResource.java | 16 +-
.../server/predownload/PredownloadSegmentInfo.java | 2 +-
.../pinot/server/api/TablesResourceTest.java | 441 ++++----
.../spi/config/table/StarTreeIndexConfig.java | 25 +-
.../compile/DdlCompilerMaterializedViewTest.java | 22 +
pom.xml | 49 +-
152 files changed, 5498 insertions(+), 1957 deletions(-)
create mode 100644
pinot-common/src/main/java/org/apache/pinot/sql/parsers/PostgreSqlCastRewriter.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/datatable/DataTableUtilsDecodeStringArrayTest.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/utils/NativeTestNGSuiteTest.java
create mode 100644
pinot-common/src/test/java/org/apache/pinot/common/utils/OrderedTestNGSuite.java
create mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/api/ControllerAdminApiApplicationTest.java
copy
pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPhysicalExplain.java
=>
pinot-controller/src/test/java/org/apache/pinot/controller/suites/ControllerStatefulSuite.java
(55%)
copy
pinot-common/src/main/java/org/apache/pinot/sql/parsers/parser/SqlPhysicalExplain.java
=>
pinot-controller/src/test/java/org/apache/pinot/controller/suites/ControllerStatelessSuite.java
(55%)
delete mode 100644 pinot-controller/testng-statefull.xml
delete mode 100644 pinot-controller/testng-stateless.xml
create mode 100644
pinot-core/src/test/java/org/apache/pinot/core/common/datatable/DataTableDictionarySerDeTest.java
copy
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/{PercentileAggregationFunctionTest.java
=> PercentileSmartTDigestHighThresholdAggregationFunctionTest.java} (71%)
copy
pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/{PercentileKLLAggregationFunctionTest.java
=> PercentileSmartTDigestSmallThresholdAggregationFunctionTest.java} (62%)
create mode 100644
pinot-core/src/test/java/org/apache/pinot/queries/NullAwareStarTreeQueriesTest.java
copy
pinot-broker/src/main/java/org/apache/pinot/broker/cursors/JsonResponseSerde.java
=>
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/suites/CustomClusterSuite.java
(59%)
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/suites/KinesisSuite.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/suites/MultiNodesOfflineSuite.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/suites/SharedHybridSuite.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/suites/SharedKafkaRealtimeSuite.java
delete mode 100644
pinot-integration-tests/src/test/resources/custom-cluster-integration-test-suite.xml
delete mode 100644
pinot-integration-tests/src/test/resources/kinesis-integration-test-suite.xml
delete mode 100644
pinot-integration-tests/src/test/resources/multi-nodes-offline-integration-test-suite.xml
delete mode 100644
pinot-integration-tests/src/test/resources/shared-hybrid-integration-test-suite.xml
delete mode 100644
pinot-integration-tests/src/test/resources/shared-kafka-realtime-integration-test-suite.xml
create mode 100644
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkMseSortImplementations.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/FullSortOperator.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LimitSortOperator.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/TopNSortOperator.java
create mode 100644
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/startree/v2/builder/NullAwareStarTreeBuilderTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
