This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch
Add-queue-based-throttling-to-ThrottleOnCriticalHeapUsageExecutor
in repository https://gitbox.apache.org/repos/asf/pinot.git
discard 3e45be20407 Address PR review comments for
ThrottleOnCriticalHeapUsageExecutor
discard 691d48a3663 Add queue-based throttling to
ThrottleOnCriticalHeapUsageExecutor
add 90346d5f379 Allow Reset of ThreadResourceUsageAccountant in
Tracing.java (#16360)
add c1c89160276 Cleanup logs (#16410)
add c7dd57c6562 Enhance LookupTable implementations and add comprehensive
test coverage for join/window operators (#16408)
add ccd55239c5a Bump commons-codec:commons-codec from 1.18.0 to 1.19.0
(#16413)
add da510929298 Bump software.amazon.awssdk:bom from 2.32.5 to 2.32.6
(#16412)
add 0189f92523c Add a protected method for deleting snapshots to be
extended (#16414)
add 1a3f2bd3278 Emit segment download metrics for local (NFS/GPFS) file
access (#16379)
add 8316aa39a2e udf test framework (#16258)
add 0415a2bc750 Make ForwardIndexReader pluggable by removing static
methods from ForwardIndexType and ForwardIndexReaderFactory (#16363)
add 30119129f83 Support executing backfill jobs on clusters that use
self-signed certificates (#16411)
add 9a18becd159 Use correlation ID instead of request id in
PerQueryCpuMemAccountant (#16040)
add 92a16d4ecbc Bump software.amazon.awssdk:bom from 2.32.6 to 2.32.7
(#16417)
add e8b1c52641f Bump curator.version from 5.8.0 to 5.9.0 (#16418)
add 4935e417f3a [timeseries] Adding error banner for timeseries error
handling in Controller UI (#16406)
add bd333b4ff73 Cleanup stream message related APIs (#16388)
add c1fceb23986 Queries now self terminate if in panic mode. (#16380)
add 4f1c40c6f1d feat: Enhance PurgeTask to automatically delete empty
segments (#16368)
add 8887c884b13 Add queue-based throttling to
ThrottleOnCriticalHeapUsageExecutor
add 0513ab07f00 Address PR review comments for
ThrottleOnCriticalHeapUsageExecutor
add b9b8be46cde Address PR review comments for
ThrottleOnCriticalHeapUsageExecutor
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3e45be20407)
\
N -- N -- N
refs/heads/Add-queue-based-throttling-to-ThrottleOnCriticalHeapUsageExecutor
(b9b8be46cde)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.gitignore | 3 +
.../broker/broker/helix/BaseBrokerStarter.java | 34 +-
.../requesthandler/BaseBrokerRequestHandler.java | 7 +-
.../BaseSingleStageBrokerRequestHandler.java | 9 +-
.../requesthandler/GrpcBrokerRequestHandler.java | 5 +-
.../MultiStageBrokerRequestHandler.java | 10 +-
.../SingleConnectionBrokerRequestHandler.java | 8 +-
.../requesthandler/TimeSeriesRequestHandler.java | 5 +-
.../BaseSingleStageBrokerRequestHandlerTest.java | 4 +-
.../LiteralOnlyBrokerRequestTest.java | 10 +-
.../pinot/common/function/FunctionRegistry.java | 54 +-
.../pinot/common/function/PinotScalarFunction.java | 48 +
.../segment/generation/SegmentGenerationUtils.java | 68 +-
.../PinotSegmentUploadDownloadRestletResource.java | 20 +-
.../controller/api/resources/ResourceUtils.java | 4 +-
.../controller/helix/ControllerRequestClient.java | 21 +
.../app/components/Query/TimeseriesQueryPage.tsx | 21 +-
.../api/resources/ResourceUtilsTest.java | 136 ++
.../pinot/controller/helix/ControllerTest.java | 10 +
.../CPUMemThreadLevelAccountingObjects.java | 1 +
.../PerQueryCPUMemAccountantFactory.java | 35 +-
.../pinot/core/accounting/QueryMonitorConfig.java | 21 +
.../accounting/ResourceUsageAccountantFactory.java | 11 +-
.../manager/realtime/IngestionDelayTracker.java | 8 +-
.../realtime/RealtimeSegmentDataManager.java | 21 +-
.../manager/realtime/RealtimeTableDataManager.java | 8 +-
.../function/TransformFunctionFactory.java | 5 +
.../core/query/reduce/BrokerReduceService.java | 12 +-
.../core/query/reduce/GroupByDataTableReducer.java | 7 +-
.../core/query/reduce/ResultReducerFactory.java | 5 +-
.../pinot/core/query/scheduler/QueryScheduler.java | 2 +-
.../main/java/org/apache/pinot/core/udf/Udf.java | 169 ++
.../java/org/apache/pinot/core/udf/UdfExample.java | 201 ++
.../apache/pinot/core/udf/UdfExampleBuilder.java | 296 +++
.../org/apache/pinot/core/udf/UdfParameter.java | 178 ++
.../org/apache/pinot/core/udf/UdfSignature.java | 73 +
.../core/accounting/QueryMonitorConfigTest.java | 17 +
.../accounting/ResourceManagerAccountingTest.java | 61 +-
.../tests/BaseClusterIntegrationTest.java | 16 +-
pinot-integration-tests/pom.xml | 5 +
.../tests/MultiStageEngineIntegrationTest.java | 2 +-
.../tests/OOMProtectionEnabledIntegrationTest.java | 54 +
.../tests/OfflineGRPCServerIntegrationTest.java | 4 +-
.../tests/PurgeMinionClusterIntegrationTest.java | 178 +-
.../tests/WindowResourceAccountingTest.java | 1 +
.../pinot/integration/tests/udf/AvroSink.java | 191 ++
.../tests/udf/IntegrationUdfTestCluster.java | 227 +++
.../pinot/integration/tests/udf/UdfTest.java | 444 +++++
.../src/test/resources/udf-test-results/README.md | 35 +
.../src/test/resources/udf-test-results/abs.yaml | 870 +++++++++
.../src/test/resources/udf-test-results/acos.yaml | 268 +++
.../test/resources/udf-test-results/adler32.yaml | 198 ++
.../resources/udf-test-results/all-functions.yaml | 1952 ++++++++++++++++++++
.../src/test/resources/udf-test-results/and.yaml | 303 +++
.../src/test/resources/udf-test-results/array.yaml | 27 +-
.../udf-test-results/arrayConcatDouble.yaml | 339 ++++
.../udf-test-results/arrayElementAtInt.yaml | 233 +++
.../test/resources/udf-test-results/arrayMax.yaml | 158 ++
.../resources/udf-test-results/arrayaverage.yaml | 138 ++
.../test/resources/udf-test-results/degrees.yaml | 233 +++
.../test/resources/udf-test-results/isFalse.yaml | 144 ++
.../src/test/resources/udf-test-results/mult.yaml | 520 ++++++
.../src/test/resources/udf-test-results/not.yaml | 163 ++
.../src/test/resources/udf-test-results/or.yaml | 373 ++++
.../src/test/resources/udf-test-results/plus.yaml | 520 ++++++
.../resources/udf-test-results/toDateTime.yaml | 128 ++
.../src/test/resources/udf-test-results/year.yaml | 163 ++
.../spark/SparkSegmentGenerationJobRunner.java | 6 +-
.../spark3/SparkSegmentGenerationJobRunner.java | 6 +-
.../pinot/plugin/inputformat/avro/AvroUtils.java | 26 +-
.../minion/tasks/purge/PurgeTaskGenerator.java | 48 +-
.../kafka20/KafkaStreamMetadataProvider.java | 4 +-
.../kafka20/KafkaPartitionLevelConsumerTest.java | 5 -
.../kafka30/KafkaStreamMetadataProvider.java | 4 +-
.../kafka30/KafkaPartitionLevelConsumerTest.java | 5 -
.../stream/kafka/KafkaStreamMessageMetadata.java | 15 +-
.../plugin/stream/kinesis/KinesisConsumer.java | 4 +-
.../kinesis/KinesisStreamMessageMetadata.java | 21 +-
.../kinesis/KinesisStreamMetadataProvider.java | 4 +-
.../stream/kinesis/KinesisMessageBatchTest.java | 1 -
.../pulsar/PulsarPartitionLevelConsumer.java | 6 +-
.../stream/pulsar/PulsarStreamMessageMetadata.java | 21 +-
.../plugin/stream/pulsar/PulsarConsumerTest.java | 4 -
.../apache/pinot/tsdb/m3ql/parser/Tokenizer.java | 29 +
.../pinot/query/runtime/function/AbsUdf.java | 63 +
.../pinot/query/runtime/function/AcosUdf.java | 73 +
.../pinot/query/runtime/function/Adler32Udf.java | 66 +
.../pinot/query/runtime/function/AndUdf.java | 82 +
.../query/runtime/function/ArrayAverageUdf.java | 76 +
.../runtime/function/ArrayConcatDoubleUdf.java | 71 +
.../runtime/function/ArrayElementAtIntUdf.java | 65 +
.../pinot/query/runtime/function/ArrayMaxUdf.java | 77 +
.../pinot/query/runtime/function/ArrayUdf.java | 32 +-
.../pinot/query/runtime/function/DegreesUdf.java | 72 +
.../pinot/query/runtime/function/IsFalseUdf.java | 75 +
.../pinot/query/runtime/function/MultUdf.java | 81 +
.../pinot/query/runtime/function/NotUdf.java | 82 +
.../apache/pinot/query/runtime/function/OrUdf.java | 82 +
.../pinot/query/runtime/function/PlusUdf.java | 82 +
.../query/runtime/function/ToDateTimeUdf.java | 67 +
.../pinot/query/runtime/function/YearUdf.java | 71 +
.../query/runtime/operator/AsofJoinOperator.java | 5 +
.../query/runtime/operator/HashJoinOperator.java | 14 +-
.../runtime/operator/NonEquiJoinOperator.java | 4 +
.../runtime/operator/WindowAggregateOperator.java | 4 +
.../runtime/operator/join/DoubleLookupTable.java | 5 +
.../runtime/operator/join/FloatLookupTable.java | 5 +
.../runtime/operator/join/IntLookupTable.java | 5 +
.../runtime/operator/join/LongLookupTable.java | 5 +
.../query/runtime/operator/join/LookupTable.java | 5 +
.../runtime/operator/join/ObjectLookupTable.java | 5 +
.../pinot/query/service/server/QueryServer.java | 4 +-
.../runtime/operator/MultiStageAccountingTest.java | 1 +
.../MultistageResourceUsageAccountingTest.java | 1 +
.../runtime/operator/join/LookupTableTest.java | 484 +++++
.../indexsegment/mutable/MutableSegmentImpl.java | 8 +-
.../writer/StatelessRealtimeSegmentWriter.java | 7 +-
.../index/forward/ForwardIndexReaderFactory.java | 6 +-
.../segment/index/forward/ForwardIndexType.java | 35 -
.../segment/index/loader/ForwardIndexHandler.java | 33 +-
.../loader/bloomfilter/BloomFilterHandler.java | 6 +-
.../ColumnMinMaxValueGenerator.java | 3 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 12 +-
.../index/loader/invertedindex/H3IndexHandler.java | 6 +-
.../loader/invertedindex/InvertedIndexHandler.java | 6 +-
.../loader/invertedindex/JsonIndexHandler.java | 10 +-
.../invertedindex/MultiColumnTextIndexHandler.java | 9 +-
.../loader/invertedindex/RangeIndexHandler.java | 10 +-
.../loader/invertedindex/TextIndexHandler.java | 7 +-
.../loader/invertedindex/VectorIndexHandler.java | 6 +-
.../startree/v2/store/StarTreeLoaderUtils.java | 4 +-
.../upsert/BasePartitionUpsertMetadataManager.java | 6 +-
.../indexsegment/mutable/IndexingFailureTest.java | 14 +-
.../MutableSegmentEntriesAboveThresholdTest.java | 12 +-
.../MutableSegmentImplAggregateMetricsTest.java | 6 +-
...MutableSegmentImplIngestionAggregationTest.java | 19 +-
.../mutable/MutableSegmentImplRawMVTest.java | 13 +-
.../mutable/MutableSegmentImplTest.java | 17 +-
.../MultiValueFixedByteRawIndexCreatorTest.java | 3 +-
.../MultiValueVarByteRawIndexCreatorTest.java | 6 +-
.../segment/index/creator/RawIndexCreatorTest.java | 11 +-
.../index/loader/ForwardIndexHandlerTest.java | 48 +-
.../index/loader/SegmentPreProcessorTest.java | 12 +-
.../apache/pinot/segment/spi/MutableSegment.java | 6 +-
.../server/starter/helix/BaseServerStarter.java | 30 +-
.../accounting/ThreadResourceUsageAccountant.java | 16 +
.../ThrottleOnCriticalHeapUsageExecutor.java | 24 +-
.../pinot/spi/ingestion/batch/spec/TlsSpec.java | 18 +
.../pinot/spi/stream/BytesStreamMessage.java | 8 +-
.../pinot/spi/stream/ConsumerPartitionState.java | 7 +-
.../org/apache/pinot/spi/stream/MessageBatch.java | 49 +-
.../org/apache/pinot/spi/stream/RowMetadata.java | 102 -
.../pinot/spi/stream/StreamDataDecoderImpl.java | 18 +-
.../org/apache/pinot/spi/stream/StreamMessage.java | 23 +-
.../pinot/spi/stream/StreamMessageMetadata.java | 51 +-
.../java/org/apache/pinot/spi/trace/Tracing.java | 54 +-
.../apache/pinot/spi/utils/CommonConstants.java | 4 +
.../utils/builder/ControllerRequestURLBuilder.java | 8 +
.../ThrottleOnCriticalHeapUsageExecutorTest.java | 23 +-
.../spi/stream/StreamDataDecoderImplTest.java | 29 +-
{pinot-clients => pinot-udf-test}/pom.xml | 13 +-
.../pinot/udf/test/PinotFunctionEnvGenerator.java | 369 ++++
.../org/apache/pinot/udf/test/ResultByExample.java | 233 +++
.../apache/pinot/udf/test/UdfExampleResult.java | 97 +
.../org/apache/pinot/udf/test/UdfReporter.java | 358 ++++
.../org/apache/pinot/udf/test/UdfTestCluster.java | 70 +
.../apache/pinot/udf/test/UdfTestFramework.java | 288 +++
.../org/apache/pinot/udf/test/UdfTestResult.java | 220 +++
.../org/apache/pinot/udf/test/UdfTestScenario.java | 63 +
.../test/scenarios/AbstractUdfTestScenario.java | 125 ++
.../ExpressionTransformerTestScenario.java | 78 +
.../scenarios/IntermediateUdfTestScenario.java | 130 ++
.../test/scenarios/PredicateUdfTestScenario.java | 117 ++
.../scenarios/TransformationUdfTestScenario.java | 85 +
pom.xml | 12 +-
175 files changed, 13851 insertions(+), 680 deletions(-)
create mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/ResourceUtilsTest.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/udf/Udf.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/udf/UdfExample.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/udf/UdfExampleBuilder.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/udf/UdfParameter.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/udf/UdfSignature.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/udf/AvroSink.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/udf/IntegrationUdfTestCluster.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/udf/UdfTest.java
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/README.md
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/abs.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/acos.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/adler32.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/all-functions.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/and.yaml
copy helm/helm-rbac.yaml =>
pinot-integration-tests/src/test/resources/udf-test-results/array.yaml (62%)
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/arrayConcatDouble.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/arrayElementAtInt.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/arrayMax.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/arrayaverage.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/degrees.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/isFalse.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/mult.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/not.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/or.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/plus.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/toDateTime.yaml
create mode 100644
pinot-integration-tests/src/test/resources/udf-test-results/year.yaml
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/AbsUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/AcosUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/Adler32Udf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/AndUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ArrayAverageUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ArrayConcatDoubleUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ArrayElementAtIntUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ArrayMaxUdf.java
copy
pinot-common/src/main/java/org/apache/pinot/common/auth/NullAuthProvider.java
=>
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ArrayUdf.java
(53%)
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/DegreesUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/IsFalseUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/MultUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/NotUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/OrUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/PlusUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/ToDateTimeUdf.java
create mode 100644
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/function/YearUdf.java
create mode 100644
pinot-query-runtime/src/test/java/org/apache/pinot/query/runtime/operator/join/LookupTableTest.java
delete mode 100644
pinot-spi/src/main/java/org/apache/pinot/spi/stream/RowMetadata.java
copy {pinot-clients => pinot-udf-test}/pom.xml (86%)
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/PinotFunctionEnvGenerator.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/ResultByExample.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/UdfExampleResult.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/UdfReporter.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/UdfTestCluster.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/UdfTestFramework.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/UdfTestResult.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/UdfTestScenario.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/scenarios/AbstractUdfTestScenario.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/scenarios/ExpressionTransformerTestScenario.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/scenarios/IntermediateUdfTestScenario.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/scenarios/PredicateUdfTestScenario.java
create mode 100644
pinot-udf-test/src/main/java/org/apache/pinot/udf/test/scenarios/TransformationUdfTestScenario.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
