This is an automated email from the ASF dual-hosted git repository.
vhs pushed a change to branch variant-intro-shredded-avro-support
in repository https://gitbox.apache.org/repos/asf/hudi.git
discard 291b967512c7 Fix checkstyle
discard c5745d1a93ce Add log + compaction test for shredded + unshredded
discard df6644d4c035 Fix malformed sql state exception
discard f80cb5a3f4e9 feat(schema): Add read + write support for shredded for
AVRO
discard b702b7d75d83 feat(schema): Config path implemented for spark record
type
discard 78109aa88b4a feat(schema): Add support to write shredded variants
add 236dc2233a70 test(lance): Add test of bloomFilter support to
TestLanceDataSource (#18388)
add 1ff0506ec340 fix: Use target schema for non-FileBased/SchemaRegistry
providers in SourceFormatAdapter (#17946)
add e270e256bac6 perf: Improve Serialization Performance of BufferedRecord
(#18418)
add 2b56eae4b8c0 feat(utilities): add option to make all schema columns
nullable for backwards compatibility (#17777)
add 75918eabcb33 feat(blob): Create blobs in Spark SQL (#18347)
add dd8fe9953f68 refactor: remove
HoodieWriteConfig.getOrcCompressionCodec() function (#18422)
add bef0c54ac5b4 [HUDI-3055] Fix hardcoded GZIP compression codec in
HFileUtils (#18263)
add f6a33fd828d5 feat(lance): Implement canWrite() in
HoodieSparkLanceWriter with configurable max file size for Lance (#18341)
add 35e2bbf9813f refactor: add import (#18428)
add 6079b6a7ed65 feat: support limit push down in Hudi Flink Source V2
(#18406)
add ea59d60a86d1 fix(spark): validate and normalize incremental start/end
instants (#18426)
add 3e1d3008ec14 feat(vector): Add guard for user creating nested VECTOR
(#18431)
add 447af5a8dbc7 fix(spark): Ignore duplicate fields when merging schema
in IncrementalRelation (#17776)
add 180592a0ad71 feat(spark): implement column pruning for incremental
queries (#17514)
add 9af7f29ceef7 perf(table-services): Only attempt scheduling log
compaction if number of deltacommits is at least LogCompactionBlocksThreshold
(#18306)
add c9d0ffb6ccca fix(common): close parquet reader iterator on EOF (#18407)
add 149d84f89705 feat(metrics): Add table-specific metrics registry
support for multi-tenant scenarios (#18179)
add 76b4fa404c72 feat(table-services): Support
hoodie.clustering.enable.expirations to allow cleanup of failed clustering
plans (intended for PreferWriterConflictResolutionStrategy) (#18302)
add 3e662f9828d3 feat (flink): improvement bucket assignment for MOR with
bucket index (#18444)
add d69826cc5dc2 fix(flink): Trigger a failover after pending instants
recommitted to b… (#18434)
add 57c3f632a203 refactor: consolidate common utility classes for Flink
CDC read (#18436)
add 0baabba65f36 feat(common): Add API to fetch log files created on or
before given instant time (#18142)
add d60be1448d5b fix: do not shutdown distruptor thread in snapshotState
in flink connnector (#18446)
add 3c53b91d9633 perf(metadata): avoid recursive calls for partition
listing using catalog (#18265)
add fd20018e05f1 feat(vector): Add Spark VECTOR Search TVF with intial KNN
algorithm (#18432)
add f2dea5263f1f feat(schema): Add support to write shredded variants
(#18036)
add 8862585a1b4e feat(schema): Config path implemented for spark record
type
add e4fcb807db52 feat(schema): Add read + write support for shredded for
AVRO
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (291b967512c7)
\
N -- N -- N refs/heads/variant-intro-shredded-avro-support
(e4fcb807db52)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../org/apache/hudi/client/BaseHoodieClient.java | 5 -
.../hudi/client/BaseHoodieTableServiceClient.java | 69 +-
.../apache/hudi/config/HoodieClusteringConfig.java | 37 +-
.../org/apache/hudi/config/HoodieWriteConfig.java | 15 +-
.../compact/ScheduleCompactionActionExecutor.java | 65 +-
.../apache/hudi/util/DistributedRegistryUtil.java | 56 +
.../client/TestBaseHoodieTableServiceClient.java | 245 ++++
.../apache/hudi/config/TestHoodieWriteConfig.java | 83 ++
.../apache/hudi/util/HoodieSchemaConverter.java | 2 +-
.../apache/hudi/client/SparkRDDWriteClient.java | 30 +-
.../client/common/HoodieSparkEngineContext.java | 54 +-
.../io/storage/HoodieSparkFileWriterFactory.java | 12 +-
.../hudi/io/storage/HoodieSparkLanceWriter.java | 104 +-
.../row/HoodieInternalRowFileWriterFactory.java | 19 +-
.../storage/row/HoodieRowParquetWriteSupport.java | 45 +-
.../apache/hudi/metrics/DistributedRegistry.java | 5 +
.../apache/spark/sql/HoodieInternalRowUtils.scala | 37 -
.../sql/avro/HoodieSparkSchemaConverters.scala | 30 +-
.../parquet/HoodieParquetFileFormatHelper.scala | 3 +-
.../parquet/HoodieParquetReadSupport.scala | 17 +-
.../org/apache/spark/sql/hudi/SparkAdapter.scala | 26 -
.../hudi/common/table/log/TestLogReaderUtils.java | 245 ++++
.../hudi/metrics/TestDistributedRegistry.java | 231 ++++
.../parquet/TestHoodieParquetReadSupport.scala | 85 --
.../org/apache/hudi/BaseHoodieTableFileIndex.java | 22 +-
.../hudi/common/config/HoodieStorageConfig.java | 11 +
.../hudi/common/engine/HoodieEngineContext.java | 14 +
.../apache/hudi/common/schema/HoodieSchema.java | 50 +-
.../hudi/common/table/log/LogReaderUtils.java | 79 ++
.../common/table/timeline/BaseHoodieTimeline.java | 11 +
.../hudi/common/table/timeline/HoodieTimeline.java | 5 +
.../apache/hudi/common/util/CompactionUtils.java | 65 +
.../org/apache/hudi/common/util/HFileUtils.java | 3 +-
.../common/util/HoodieCommonKryoRegistrar.java | 4 +-
.../common/util/queue/DisruptorMessageQueue.java | 27 +
.../metadata/FileSystemBackedTableMetadata.java | 12 +-
.../hudi/metadata/HoodieBackedTableMetadata.java | 13 +-
.../apache/hudi/metadata/HoodieTableMetadata.java | 7 +
.../apache/hudi/util/PartitionPathFilterUtil.java | 38 +
.../hudi/common/schema/TestHoodieSchema.java | 69 +-
.../hudi/util/TestPartitionPathFilterUtil.java | 56 +
.../apache/hudi/configuration/FlinkOptions.java | 8 -
.../hudi/sink/StreamWriteOperatorCoordinator.java | 48 +-
...AppendWriteFunctionWithDisruptorBufferSort.java | 30 +-
.../hudi/sink/bootstrap/RLIBootstrapOperator.java | 89 +-
.../org/apache/hudi/sink/event/Correspondent.java | 38 -
.../org/apache/hudi/sink/utils/EventBuffers.java | 15 +-
.../java/org/apache/hudi/sink/utils/Pipelines.java | 7 +-
.../org/apache/hudi/source/HoodieScanContext.java | 1 +
.../java/org/apache/hudi/source/HoodieSource.java | 2 +-
.../hudi/source/reader/HoodieSourceReader.java | 9 +-
.../source/reader/HoodieSourceSplitReader.java | 32 +-
.../apache/hudi/source/reader/RecordLimiter.java | 83 ++
.../function/AbstractSplitReaderFunction.java | 67 +
.../function/HoodieCdcSplitReaderFunction.java | 815 +-----------
.../reader/function/HoodieSplitReaderFunction.java | 32 +-
.../source/split/assign/HoodieSplitAssigners.java | 2 +-
.../split/assign/HoodieSplitBucketAssigner.java | 23 +-
.../org/apache/hudi/table/HoodieTableSource.java | 7 +-
.../hudi/table/format/cdc/CdcImageManager.java | 197 +++
.../hudi/table/format/cdc/CdcInputFormat.java | 788 +-----------
.../apache/hudi/table/format/cdc/CdcIterators.java | 667 ++++++++++
.../sink/TestStreamWriteOperatorCoordinator.java | 28 -
.../org/apache/hudi/sink/TestWriteCopyOnWrite.java | 2 +-
.../org/apache/hudi/sink/TestWriteMergeOnRead.java | 51 +
.../TestAppendWriteFunctionWithBufferSort.java | 163 +++
...lueState.java => MockStateSnapshotContext.java} | 28 +-
.../sink/utils/StreamWriteFunctionWrapper.java | 48 +-
.../apache/hudi/sink/utils/TestEventBuffers.java | 42 -
.../org/apache/hudi/sink/utils/TestWriteBase.java | 13 +
.../source/reader/TestHoodieSourceSplitReader.java | 198 ++-
.../hudi/source/reader/TestRecordLimiter.java | 372 ++++++
.../function/TestAbstractSplitReaderFunction.java | 246 ++++
.../function/TestHoodieCdcSplitReaderFunction.java | 63 +-
.../function/TestHoodieSplitReaderFunction.java | 58 +
.../split/TestDefaultHoodieSplitProvider.java | 3 +-
.../assign/TestHoodieSplitBucketAssigner.java | 401 +++---
.../apache/hudi/table/ITTestHoodieDataSource.java | 55 +
.../apache/hudi/avro/HoodieAvroWriteSupport.java | 43 +-
.../hudi/common/util/ParquetReaderIterator.java | 16 +-
.../hudi/hadoop/fs/HoodieWrapperFileSystem.java | 10 +
.../hudi/io/lance/HoodieBaseLanceWriter.java | 22 +
.../avro/AvroSchemaConverterWithTimestampNTZ.java | 1 +
...TestHoodieAvroWriteSupportVariantShredding.java | 508 --------
.../hudi/common/table/TestHoodieTableConfig.java | 1 +
.../hudi/common/util/TestCompactionUtils.java | 315 +++++
.../common/util/TestParquetReaderIterator.java | 2 +-
...oodieAvroFileWriterFactoryVariantShredding.java | 275 ----
.../apache/hudi/common/metrics/LocalRegistry.java | 5 +
.../org/apache/hudi/common/metrics/Registry.java | 107 +-
hudi-spark-datasource/hudi-spark-common/pom.xml | 5 +
.../scala/org/apache/hudi/DataSourceOptions.scala | 26 +-
.../scala/org/apache/hudi/HoodieFileIndex.scala | 7 +
.../hudi/HoodieHadoopFsRelationFactory.scala | 5 +-
.../org/apache/hudi/IncrementalRelationV1.scala | 38 +-
.../org/apache/hudi/IncrementalRelationV2.scala | 38 +-
.../apache/hudi/SparkHoodieTableFileIndex.scala | 32 +-
.../hudi/metadata/CatalogBackedTableMetadata.scala | 123 ++
.../apache/hudi/util/IncrementalRelationUtil.scala | 110 ++
.../plans/logical/HoodieTableChanges.scala | 7 +-
.../HoodieVectorSearchTableValuedFunction.scala | 203 +++
.../spark/sql/hudi/HoodieSqlCommonUtils.scala | 71 +-
.../hudi/analysis/HoodieSparkBaseAnalysis.scala | 81 +-
.../analysis/HoodieVectorSearchPlanBuilder.scala | 329 +++++
.../sql/hudi/analysis/TableValuedFunctions.scala | 12 +-
.../sql/hudi/analysis/VectorDistanceUtils.scala | 136 ++
.../sql/hudi/streaming/HoodieStreamSourceV1.scala | 4 +-
.../sql/hudi/streaming/HoodieStreamSourceV2.scala | 4 +-
.../org/apache/spark/sql/types/BlobType.scala | 24 +-
hudi-spark-datasource/hudi-spark/pom.xml | 5 +
.../io/storage/TestHoodieSparkLanceReader.java | 15 +-
.../io/storage/TestHoodieSparkLanceWriter.java | 153 ++-
.../org/apache/hudi/blob/BlobTestHelpers.scala | 129 ++
.../org/apache/hudi/blob/TestBlobSupport.scala | 142 ++
.../apache/hudi/functional/TestCOWDataSource.scala | 71 +
.../TestHoodieVectorSearchFunction.scala | 1359 ++++++++++++++++++++
.../TestIncrementalQueryColumnPruning.scala | 242 ++++
.../hudi/functional/TestLanceDataSource.scala | 69 +-
.../apache/hudi/functional/TestMORDataSource.scala | 88 ++
.../hudi/functional/TestVectorDataSource.scala | 26 +
.../spark/sql/avro/TestSchemaConverters.scala | 50 +
.../common/TestCatalogBackedTableMetadata.scala | 574 +++++++++
.../hudi/common/TestInstantTimeValidation.scala | 278 ++++
.../spark/sql/hudi/ddl/TestCreateTable.scala | 124 ++
.../sql/hudi/dml/schema/TestVariantDataType.scala | 473 +------
.../spark/sql/adapter/BaseSpark3Adapter.scala | 8 -
.../HoodieSpark3_3ExtendedSqlAstBuilder.scala | 29 +-
.../parser/HoodieSpark3_3ExtendedSqlParser.scala | 3 +-
.../HoodieSpark3_4ExtendedSqlAstBuilder.scala | 29 +-
.../parser/HoodieSpark3_4ExtendedSqlParser.scala | 3 +-
.../HoodieSpark3_5ExtendedSqlAstBuilder.scala | 29 +-
.../parser/HoodieSpark3_5ExtendedSqlParser.scala | 3 +-
.../spark/sql/adapter/BaseSpark4Adapter.scala | 27 +-
.../TestSpark4VariantShreddingProvider.java | 279 ----
.../apache/spark/sql/avro/AvroDeserializer.scala | 23 +-
.../HoodieSpark4_0ExtendedSqlAstBuilder.scala | 29 +-
.../parser/HoodieSpark4_0ExtendedSqlParser.scala | 3 +-
.../TestHoodieRowParquetWriteSupportVariant.java | 2 +-
.../apache/hudi/utilities/HoodieClusteringJob.java | 68 +-
.../org/apache/hudi/utilities/UtilHelpers.java | 14 +
.../utilities/config/HoodieStreamerConfig.java | 12 +
.../apache/hudi/utilities/sources/RowSource.java | 5 +-
.../utilities/streamer/SourceFormatAdapter.java | 12 +-
.../apache/hudi/utilities/streamer/StreamSync.java | 8 +-
.../org/apache/hudi/utilities/TestUtilHelpers.java | 141 ++
.../deltastreamer/TestSourceFormatAdapter.java | 94 ++
.../offlinejob/TestHoodieClusteringJob.java | 151 +++
pom.xml | 6 +
148 files changed, 9914 insertions(+), 4159 deletions(-)
create mode 100644
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/util/DistributedRegistryUtil.java
create mode 100644
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/common/table/log/TestLogReaderUtils.java
create mode 100644
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java
create mode 100644
hudi-common/src/main/java/org/apache/hudi/util/PartitionPathFilterUtil.java
create mode 100644
hudi-common/src/test/java/org/apache/hudi/util/TestPartitionPathFilterUtil.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/RecordLimiter.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/reader/function/AbstractSplitReaderFunction.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/cdc/CdcImageManager.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/cdc/CdcIterators.java
copy
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/{MockValueState.java
=> MockStateSnapshotContext.java} (50%)
create mode 100644
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/reader/TestRecordLimiter.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/reader/function/TestAbstractSplitReaderFunction.java
delete mode 100644
hudi-hadoop-common/src/test/java/org/apache/hudi/avro/TestHoodieAvroWriteSupportVariantShredding.java
delete mode 100644
hudi-hadoop-common/src/test/java/org/apache/hudi/io/storage/hadoop/TestHoodieAvroFileWriterFactoryVariantShredding.java
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/metadata/CatalogBackedTableMetadata.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/util/IncrementalRelationUtil.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/HoodieVectorSearchTableValuedFunction.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieVectorSearchPlanBuilder.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/analysis/VectorDistanceUtils.scala
copy
hudi-common/src/main/java/org/apache/hudi/common/data/HoodieAccumulator.java =>
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/types/BlobType.scala
(55%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/blob/BlobTestHelpers.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/blob/TestBlobSupport.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestHoodieVectorSearchFunction.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalQueryColumnPruning.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestCatalogBackedTableMetadata.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestInstantTimeValidation.scala
delete mode 100644
hudi-spark-datasource/hudi-spark4-common/src/test/java/org/apache/hudi/variant/TestSpark4VariantShreddingProvider.java
