[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1035952410 ## CI report: * bc24e231f7f7e86f0eeac7716f0b1e9293374169 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5903) * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1035961795 ## CI report: * bc24e231f7f7e86f0eeac7716f0b1e9293374169 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5903) * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope commented on issue #3868: [SUPPORT] hive syncing with `--spark-datasource` (first title was: Querying hudi datasets from standalone metastore)
codope commented on issue #3868: URL: https://github.com/apache/hudi/issues/3868#issuecomment-1035963670 @matthiasdg Could you remove the partition extraction config (by default it is slash encoded day partition) and try again? I have updated the gist with both failed and successful runs in the gist: https://gist.github.com/codope/c4487d35beb60e322316d9a18773103a only difference in the cases of successful run is that i'm using the default partition extractor class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
danny0405 commented on a change in pull request #4753:
URL: https://github.com/apache/hudi/pull/4753#discussion_r804430279
##
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkCompactor.java
##
@@ -43,19 +48,24 @@ public HoodieSparkCompactor(BaseHoodieWriteClient>, J
}
@Override
- public void compact(HoodieInstant instant) throws IOException {
+ public void compact(HoodieInstant instant) {
LOG.info("Compactor executing compaction " + instant);
SparkRDDWriteClient writeClient = (SparkRDDWriteClient)
compactionClient;
-JavaRDD res = writeClient.compact(instant.getTimestamp());
-this.context.setJobStatus(this.getClass().getSimpleName(), "Collect
compaction write status");
-long numWriteErrors =
res.collect().stream().filter(WriteStatus::hasErrors).count();
+HoodieWriteMetadata> compactionMetadata =
writeClient.compact(instant.getTimestamp());
+List writeStats =
compactionMetadata.getCommitMetadata().get()
+.getPartitionToWriteStats()
+.values()
Review comment:
Is there any possibility we add a util method named
`HoodieCommitMetadata#getWriteStats` to wrap these stream lines ? I saw the
similar code invoked in several places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
danny0405 commented on a change in pull request #4753:
URL: https://github.com/apache/hudi/pull/4753#discussion_r804432099
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java
##
@@ -293,4 +297,17 @@ private String getSchemaFromLatestInstant() throws
Exception {
Schema schema = schemaUtil.getTableAvroSchema(false);
return schema.toString();
}
+
+ private int handleErrors(HoodieCommitMetadata metadata, String instantTime) {
+List writeStats =
metadata.getPartitionToWriteStats().entrySet().stream().flatMap(e ->
Review comment:
Can we also put it into the `UtilHelpers` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4791: [HUDI-3412] TypedProperties no need to create new set when check key …
danny0405 commented on a change in pull request #4791:
URL: https://github.com/apache/hudi/pull/4791#discussion_r804433456
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/config/TypedProperties.java
##
@@ -44,27 +43,22 @@ public TypedProperties(Properties defaults) {
}
private void checkKey(String property) {
-if (!keyExists(property)) {
+if (!containsKey(property)) {
throw new IllegalArgumentException("Property " + property + " not
found");
Review comment:
The old code requires that the key should be a string, what about the
new code then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1035971918 ## CI report: * bc24e231f7f7e86f0eeac7716f0b1e9293374169 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5903) * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1035961795 ## CI report: * bc24e231f7f7e86f0eeac7716f0b1e9293374169 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5903) * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yihua opened a new pull request #4792: [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests
yihua opened a new pull request #4792: URL: https://github.com/apache/hudi/pull/4792 ## What is the purpose of the pull request This PR removes hardcoded logic of disabling metadata table in tests and fixes test that are broken with metadata table. ## Verify this pull request This pull request is already covered by existing tests. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3366) Remove unnecessary hardcoded logic of disabling metadata table in tests
[ https://issues.apache.org/jira/browse/HUDI-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3366: - Labels: pull-request-available (was: ) > Remove unnecessary hardcoded logic of disabling metadata table in tests > --- > > Key: HUDI-3366 > URL: https://issues.apache.org/jira/browse/HUDI-3366 > Project: Apache Hudi > Issue Type: Task >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Blocker > Labels: pull-request-available > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4792: [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests
hudi-bot commented on pull request #4792: URL: https://github.com/apache/hudi/pull/4792#issuecomment-1035979794 ## CI report: * 1b2bdd81dc3f7edea2934b1cd461ee50dff029b7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4792: [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests
hudi-bot removed a comment on pull request #4792: URL: https://github.com/apache/hudi/pull/4792#issuecomment-1035979794 ## CI report: * 1b2bdd81dc3f7edea2934b1cd461ee50dff029b7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4792: [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests
hudi-bot commented on pull request #4792: URL: https://github.com/apache/hudi/pull/4792#issuecomment-1035981836 ## CI report: * 1b2bdd81dc3f7edea2934b1cd461ee50dff029b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5907) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3648: [HUDI-2413] fix Sql source's checkpoint issue
codope commented on a change in pull request #3648:
URL: https://github.com/apache/hudi/pull/3648#discussion_r804443369
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##
@@ -378,6 +378,10 @@ private boolean onDeltaSyncShutdown(boolean error) {
@Parameter(names = {"--max-retry-count"}, description = "the max retry
count if --retry-on-source-failures is enabled")
public Integer maxRetryCount = 3;
+@Parameter(names = {"--allow-commit-on-no-checkpoint-change"}, description
= "allow commits even if checkpoint has not changed before and after fetch data"
++ "from souce. This might be useful in sources like SqlSource where
there is not checkpoint. ")
Review comment:
Should we also add `Not recommended to enable in continuous mode`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] pan3793 opened a new issue #4793: [SUPPORT] Integration test broken after upgrade from 0.10.0 to 0.10.1
pan3793 opened a new issue #4793: URL: https://github.com/apache/hudi/issues/4793 **Describe the problem you faced** The Apache Kyuubi (Incubating) Hudi Integration test broken after upgrade from 0.10.0 to 0.10.1. https://github.com/apache/incubator-kyuubi/runs/5152924363 **To Reproduce** https://github.com/apache/incubator-kyuubi/pull/1897 **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.10.1 * Spark version : 3.1.3 * Hive version : 2.3.7 * Hadoop version : 3.3.1 * Storage (HDFS/S3/GCS..) : Local File System * Running on Docker? (yes/no) : Not sure, failed in GitHub Action. **Additional context** Add any other context about the problem here. **Stacktrace** ``` - get tables *** FAILED *** java.sql.SQLException: Error operating EXECUTE_STATEMENT: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.CatalogTable.copy(Lorg/apache/spark/sql/catalyst/TableIdentifier;Lorg/apache/spark/sql/catalyst/catalog/CatalogTableType;Lorg/apache/spark/sql/catalyst/catalog/CatalogStorageFormat;Lorg/apache/spark/sql/types/StructType;Lscala/Option;Lscala/collection/Seq;Lscala/Option;Ljava/lang/String;JJLjava/lang/String;Lscala/collection/immutable/Map;Lscala/Option;Lscala/Option;Lscala/Option;Lscala/collection/Seq;ZZLscala/collection/immutable/Map;)Lorg/apache/spark/sql/catalyst/catalog/CatalogTable; at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand$.createTableInCatalog(CreateHoodieTableCommand.scala:136) at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:71) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79) at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228) at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685) at org.apache.spark.sql.Dataset.(Dataset.scala:228) at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.$anonfun$executeStatement$1(ExecuteStatement.scala:79) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.kyuubi.engine.spark.operation.SparkOperation.withLocalProperties(SparkOperation.scala:88) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement.org$apache$kyuubi$engine$spark$operation$ExecuteStatement$$executeStatement(ExecuteStatement.scala:73) at org.apache.kyuubi.engine.spark.operation.ExecuteStatement$$anon$1.run(ExecuteStatement.scala:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:405) at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeWithConfOverlay(KyuubiStatement.java:255) at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:249) at org.apache.kyuubi.operation.HudiMetadataTests.$anonfun$$init$$10(HudiMetadataTests.scala:74) at org.apache.kyuubi.operation.HudiMetadataTests.$anonfun$$init$$10$adapted(HudiMetadataTests.scala:66) at org.apache.kyuubi.operation.JDBCTestHelper.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestHelper.scala:60) a
[GitHub] [hudi] boneanxs commented on a change in pull request #4791: [HUDI-3412] TypedProperties no need to create new set when check key …
boneanxs commented on a change in pull request #4791:
URL: https://github.com/apache/hudi/pull/4791#discussion_r804452202
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/config/TypedProperties.java
##
@@ -44,27 +43,22 @@ public TypedProperties(Properties defaults) {
}
private void checkKey(String property) {
-if (!keyExists(property)) {
+if (!containsKey(property)) {
throw new IllegalArgumentException("Property " + property + " not
found");
Review comment:
You mean we can support key with different type? After this,
TypedProperties also can only accept keys which is strings, because all public
methods can only accept key as String
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4772: [HUDI-3362] Fix restore to rollback pending clustering operations followed by other rolling back other commits
hudi-bot removed a comment on pull request #4772: URL: https://github.com/apache/hudi/pull/4772#issuecomment-1035954025 ## CI report: * d92b221a24d5b04a21e7320e43d04ed699adadc4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5902) * 9c26457de8ff1bca99f7098a676e17aa1f62b4a8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5904) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4791: [HUDI-3412] TypedProperties no need to create new set when check key …
hudi-bot removed a comment on pull request #4791: URL: https://github.com/apache/hudi/pull/4791#issuecomment-1035958612 ## CI report: * 3c061f10be730b7edb473cdb90be1aa5236a28a7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5905) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4772: [HUDI-3362] Fix restore to rollback pending clustering operations followed by other rolling back other commits
hudi-bot commented on pull request #4772: URL: https://github.com/apache/hudi/pull/4772#issuecomment-1035990281 ## CI report: * 9c26457de8ff1bca99f7098a676e17aa1f62b4a8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5904) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4791: [HUDI-3412] TypedProperties no need to create new set when check key …
hudi-bot commented on pull request #4791: URL: https://github.com/apache/hudi/pull/4791#issuecomment-1035990377 ## CI report: * 3c061f10be730b7edb473cdb90be1aa5236a28a7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5905) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] cuibo01 commented on pull request #4699: [HUDI-3336][HUDI-FLINK] Support custom hadoop config options for flink
cuibo01 commented on pull request #4699: URL: https://github.com/apache/hudi/pull/4699#issuecomment-1035995109 > Yes, the `toMap` has similar functionality, but the `getXXX` and `setXXX` methods always prefix the option with prefix, which is not what we want, i still think there is no need to introduce this too class. got it, thx @danny0405 , i will update the PR :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1035971918 ## CI report: * bc24e231f7f7e86f0eeac7716f0b1e9293374169 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5903) * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036004814 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4775: [WIP] [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs
hudi-bot commented on pull request #4775: URL: https://github.com/apache/hudi/pull/4775#issuecomment-1036010826 ## CI report: * 1143c99f4169aa458928a89cb4f6f6dce98d93ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5837) * 93db7c550338eb4368a69d3a5ee0c33c68a5b5cd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4775: [WIP] [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs
hudi-bot removed a comment on pull request #4775: URL: https://github.com/apache/hudi/pull/4775#issuecomment-1033699581 ## CI report: * 1143c99f4169aa458928a89cb4f6f6dce98d93ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5837) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4775: [WIP] [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs
hudi-bot commented on pull request #4775: URL: https://github.com/apache/hudi/pull/4775#issuecomment-1036013092 ## CI report: * 1143c99f4169aa458928a89cb4f6f6dce98d93ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5837) * 93db7c550338eb4368a69d3a5ee0c33c68a5b5cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5908) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4775: [WIP] [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs
hudi-bot removed a comment on pull request #4775: URL: https://github.com/apache/hudi/pull/4775#issuecomment-1036010826 ## CI report: * 1143c99f4169aa458928a89cb4f6f6dce98d93ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5837) * 93db7c550338eb4368a69d3a5ee0c33c68a5b5cd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4792: [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests
hudi-bot removed a comment on pull request #4792: URL: https://github.com/apache/hudi/pull/4792#issuecomment-1035981836 ## CI report: * 1b2bdd81dc3f7edea2934b1cd461ee50dff029b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5907) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4792: [HUDI-3366] Remove hardcoded logic of disabling metadata table in tests
hudi-bot commented on pull request #4792: URL: https://github.com/apache/hudi/pull/4792#issuecomment-1036035557 ## CI report: * 1b2bdd81dc3f7edea2934b1cd461ee50dff029b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5907) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-3413) fix jackson parse error when empty message from JsonKafkaSource Using HoodieDeltaStreamer
xiang zhang created HUDI-3413: - Summary: fix jackson parse error when empty message from JsonKafkaSource Using HoodieDeltaStreamer Key: HUDI-3413 URL: https://issues.apache.org/jira/browse/HUDI-3413 Project: Apache Hudi Issue Type: Bug Reporter: xiang zhang when there is an empty kafka empty message, HoodieDeltaStreamer will occur _'No content to map due to end-of-input_ ' error, to solve that, we should filter empty message from JsonKafkaSource -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3413) fix jackson parse error when empty message from JsonKafkaSource Using HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiang zhang updated HUDI-3413: -- Priority: Minor (was: Major) > fix jackson parse error when empty message from JsonKafkaSource Using > HoodieDeltaStreamer > --- > > Key: HUDI-3413 > URL: https://issues.apache.org/jira/browse/HUDI-3413 > Project: Apache Hudi > Issue Type: Bug >Reporter: xiang zhang >Priority: Minor > > when there is an empty kafka empty message, HoodieDeltaStreamer will occur > _'No content to map due to end-of-input_ ' error, to solve that, we should > filter empty message from JsonKafkaSource -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] zhangxiang17 opened a new pull request #4794: [HUDI-3413]fix jackson parse error when empty message from JsonKafkaS…
zhangxiang17 opened a new pull request #4794: URL: https://github.com/apache/hudi/pull/4794 …ource Using HoodieDeltaStreamer ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3413) fix jackson parse error when empty message from JsonKafkaSource Using HoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-3413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3413: - Labels: pull-request-available (was: ) > fix jackson parse error when empty message from JsonKafkaSource Using > HoodieDeltaStreamer > --- > > Key: HUDI-3413 > URL: https://issues.apache.org/jira/browse/HUDI-3413 > Project: Apache Hudi > Issue Type: Bug >Reporter: xiang zhang >Priority: Minor > Labels: pull-request-available > > when there is an empty kafka empty message, HoodieDeltaStreamer will occur > _'No content to map due to end-of-input_ ' error, to solve that, we should > filter empty message from JsonKafkaSource -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4794: [HUDI-3413]fix jackson parse error when empty message from JsonKafkaS…
hudi-bot commented on pull request #4794: URL: https://github.com/apache/hudi/pull/4794#issuecomment-1036046786 ## CI report: * d313daba428b2b0007ee688219a6273ddaa7aea0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4794: [HUDI-3413] Fix empty message jackson parse error in JsonKafkaSource
hudi-bot removed a comment on pull request #4794: URL: https://github.com/apache/hudi/pull/4794#issuecomment-1036046786 ## CI report: * d313daba428b2b0007ee688219a6273ddaa7aea0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4794: [HUDI-3413] Fix empty message jackson parse error in JsonKafkaSource
hudi-bot commented on pull request #4794: URL: https://github.com/apache/hudi/pull/4794#issuecomment-1036048922 ## CI report: * d313daba428b2b0007ee688219a6273ddaa7aea0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5909) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #4794: [HUDI-3413] Fix empty message jackson parse error in JsonKafkaSource
wangxianghu commented on pull request #4794: URL: https://github.com/apache/hudi/pull/4794#issuecomment-1036048925 @zhangxiang17 Thanks for this fix :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
hudi-bot removed a comment on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1035874974 ## CI report: * ea61391876e0f18cd1cde28dcbc1de4623ab66de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5890) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5895) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4775: [WIP] [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs
hudi-bot removed a comment on pull request #4775: URL: https://github.com/apache/hudi/pull/4775#issuecomment-1036013092 ## CI report: * 1143c99f4169aa458928a89cb4f6f6dce98d93ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5837) * 93db7c550338eb4368a69d3a5ee0c33c68a5b5cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5908) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
hudi-bot commented on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1036055497 ## CI report: * ea61391876e0f18cd1cde28dcbc1de4623ab66de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5890) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5895) * 5834378fe9ef4ee4ad4e44df6e0a61362e522fe5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4775: [WIP] [HUDI-2883] Refactor hive sync tool / config to use reflection and standardize configs
hudi-bot commented on pull request #4775: URL: https://github.com/apache/hudi/pull/4775#issuecomment-1036055589 ## CI report: * 93db7c550338eb4368a69d3a5ee0c33c68a5b5cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5908) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
hudi-bot removed a comment on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1036055497 ## CI report: * ea61391876e0f18cd1cde28dcbc1de4623ab66de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5890) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5895) * 5834378fe9ef4ee4ad4e44df6e0a61362e522fe5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
hudi-bot commented on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1036057820 ## CI report: * ea61391876e0f18cd1cde28dcbc1de4623ab66de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5890) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5895) * 5834378fe9ef4ee4ad4e44df6e0a61362e522fe5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5910) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] geoffroyatkwiff commented on issue #4778: [SUPPORT] Row with _hoodie_is_deleted=True stored into target table
geoffroyatkwiff commented on issue #4778: URL: https://github.com/apache/hudi/issues/4778#issuecomment-1036073470 Hey @nsivabalan , I'm using `upsert`. I'm using a timestamp for `hoodie.datasource.write.precombine.field`. So, in the case of a dataframe being processed that would contain an Insert and a Delete for the same record (so same key value), only the Delete is taken into account, and since there is nothing to be deleted in the target table, it looks as if it is inserting this Delete.. Any advice on the best way to handle this case? Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4794: [HUDI-3413] Fix empty message jackson parse error in JsonKafkaSource
hudi-bot removed a comment on pull request #4794: URL: https://github.com/apache/hudi/pull/4794#issuecomment-1036048922 ## CI report: * d313daba428b2b0007ee688219a6273ddaa7aea0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5909) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4794: [HUDI-3413] Fix empty message jackson parse error in JsonKafkaSource
hudi-bot commented on pull request #4794: URL: https://github.com/apache/hudi/pull/4794#issuecomment-1036095788 ## CI report: * d313daba428b2b0007ee688219a6273ddaa7aea0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5909) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] andykrk commented on issue #4604: [SUPPORT] Archive functionality fails
andykrk commented on issue #4604: URL: https://github.com/apache/hudi/issues/4604#issuecomment-1036096714 Hello, I will try to describe the procedure with more details. It's very close to what you have described though. Please hold on with reproduction attempt and let me try again now since we are on 0.9.0 in production environment but without archival (1000 commits needed). I see that change from 0.7.0 to 0.9.0 without archiving is not causing any problems since it took place. I will try with enabled "hoodie.metadata.enable" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036108147 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036004814 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
hudi-bot removed a comment on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1036057820 ## CI report: * ea61391876e0f18cd1cde28dcbc1de4623ab66de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5890) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5895) * 5834378fe9ef4ee4ad4e44df6e0a61362e522fe5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5910) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
hudi-bot commented on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1036112386 ## CI report: * 5834378fe9ef4ee4ad4e44df6e0a61362e522fe5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5910) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
zhangyue19921010 commented on pull request #4753: URL: https://github.com/apache/hudi/pull/4753#issuecomment-1036129050 > Thanks for the contribution @zhangyue19921010 , overall looks good, i left some minor comments. Thanks @danny0405 All comments are addressed. Also CI passed. PTAL :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
zhangyue19921010 commented on a change in pull request #4753:
URL: https://github.com/apache/hudi/pull/4753#discussion_r804578059
##
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkCompactor.java
##
@@ -43,19 +48,24 @@ public HoodieSparkCompactor(BaseHoodieWriteClient>, J
}
@Override
- public void compact(HoodieInstant instant) throws IOException {
+ public void compact(HoodieInstant instant) {
LOG.info("Compactor executing compaction " + instant);
SparkRDDWriteClient writeClient = (SparkRDDWriteClient)
compactionClient;
-JavaRDD res = writeClient.compact(instant.getTimestamp());
-this.context.setJobStatus(this.getClass().getSimpleName(), "Collect
compaction write status");
-long numWriteErrors =
res.collect().stream().filter(WriteStatus::hasErrors).count();
+HoodieWriteMetadata> compactionMetadata =
writeClient.compact(instant.getTimestamp());
+List writeStats =
compactionMetadata.getCommitMetadata().get()
+.getPartitionToWriteStats()
+.values()
Review comment:
Sure, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #4753: [HUDI-3370] The files recorded in the commit may not match the actual ones for MOR Compaction
zhangyue19921010 commented on a change in pull request #4753:
URL: https://github.com/apache/hudi/pull/4753#discussion_r804578214
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java
##
@@ -293,4 +297,17 @@ private String getSchemaFromLatestInstant() throws
Exception {
Schema schema = schemaUtil.getTableAvroSchema(false);
return schema.toString();
}
+
+ private int handleErrors(HoodieCommitMetadata metadata, String instantTime) {
+List writeStats =
metadata.getPartitionToWriteStats().entrySet().stream().flatMap(e ->
Review comment:
Sure, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
XuQianJin-Stars commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036154342 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036108147 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036155469 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5911) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
XuQianJin-Stars removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036154342 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036155469 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5911) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036164056 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5911) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036164056 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5911) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036187779 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5911) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot removed a comment on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036187779 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 5dd7309bb81f9b4b5b22de04659d37418a408ba4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5906) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5911) * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4752: [WIP][HUDI-3088] Use Spark 3.2 as default Spark version
hudi-bot commented on pull request #4752: URL: https://github.com/apache/hudi/pull/4752#issuecomment-1036198959 ## CI report: * d5f1fbad92cd451d5ac7cf81f5f8612ff18d85ed UNKNOWN * 42cbe5434d4b24acd24d3dca5f1ff027f5e95fcb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot commented on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036203793 ## CI report: * 974c5ecb0234bcf34b0f0ef86057decb9812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5776) * 6d40a75e6d159f9c961f2e960760022af915beee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot removed a comment on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1033224445 ## CI report: * 974c5ecb0234bcf34b0f0ef86057decb9812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5776) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot removed a comment on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036203793 ## CI report: * 974c5ecb0234bcf34b0f0ef86057decb9812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5776) * 6d40a75e6d159f9c961f2e960760022af915beee UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot commented on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036206376 ## CI report: * 974c5ecb0234bcf34b0f0ef86057decb9812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5776) * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036211540 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1576) Add ability to perform archival synchronously
[ https://issues.apache.org/jira/browse/HUDI-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1576: - Labels: pull-request-available (was: ) > Add ability to perform archival synchronously > - > > Key: HUDI-1576 > URL: https://issues.apache.org/jira/browse/HUDI-1576 > Project: Apache Hudi > Issue Type: Improvement > Components: archiving >Reporter: Nishith Agarwal >Assignee: Raymond Xu >Priority: Blocker > Labels: pull-request-available > Fix For: 0.11.0 > > > Currently, archival runs inline. We want to move archival to a table service > like cleaning, compaction etc.. > and treat it like that. of course, no new action will be introduced. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036211540 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036214116 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036214116 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036219570 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036219570 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036222166 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036222166 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036249771 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot removed a comment on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036206376 ## CI report: * 974c5ecb0234bcf34b0f0ef86057decb9812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5776) * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot commented on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036260406 ## CI report: * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036249771 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036268815 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036268815 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036271576 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) * 9e22cac5f5bf269aee3f41a72ea0d4f8067954a4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036271576 ## CI report: * 361f176bbe028130f1e5a23d7673c22f169ec7b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5914) * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) * 9e22cac5f5bf269aee3f41a72ea0d4f8067954a4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036274270 ## CI report: * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) * 9e22cac5f5bf269aee3f41a72ea0d4f8067954a4 UNKNOWN * b4d9ab2f772fed882bd58d03fa74c793110c7f0e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-3391) presto and hive beeline fails to read MOR table w/ 2 or more array fields
[
https://issues.apache.org/jira/browse/HUDI-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490960#comment-17490960
]
Harsha Gudladona commented on HUDI-3391:
Adding some more context on the dependency fixes.
Original issue on the parquet lib was reported here:
https://issues.apache.org/jira/browse/PARQUET-1441 and fixed in
[https://github.com/apache/parquet-mr/pull/560] - which is available in 1.11.x
On the hudi mvn deps, it appears that the parquet version prop is set to
[1.10.1|https://github.com/apache/hudi/blob/master/pom.xml#L100]. however its
overridden for the spark context build
[here][[https://github.com/apache/hudi/blob/master/pom.xml#L1589]]
I think, updating the version in the parent pom global prop should fix it for
presto as well?
> presto and hive beeline fails to read MOR table w/ 2 or more array fields
> -
>
> Key: HUDI-3391
> URL: https://issues.apache.org/jira/browse/HUDI-3391
> Project: Apache Hudi
> Issue Type: Task
> Components: reader-core
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Critical
> Fix For: 0.11.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> We have an issue reported by user
> [here|[https://github.com/apache/hudi/issues/2657].] Looks like w/ 0.10.0 or
> later, spark datasource read works, but hive beeline does not work. Even
> spark.sql (hive table) querying works as well.
> Another related ticket:
> [https://github.com/apache/hudi/issues/3834#issuecomment-997307677]
>
> Steps that I tried:
> [https://gist.github.com/nsivabalan/fdb8794104181f93b9268380c7f7f079]
> From beeline, you will encounter below exception
> {code:java}
> Failed with exception
> java.io.IOException:org.apache.hudi.org.apache.avro.SchemaParseException:
> Can't redefine: array {code}
> All linked ticket states that upgrading parquet to 1.11.0 or greater should
> work. We need to try it out w/ latest master and go from there.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HUDI-3391) presto and hive beeline fails to read MOR table w/ 2 or more array fields
[
https://issues.apache.org/jira/browse/HUDI-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490960#comment-17490960
]
Harsha Gudladona edited comment on HUDI-3391 at 2/11/22, 2:47 PM:
--
Adding some more context on the dependency fixes.
Original issue on the parquet lib was reported here:
https://issues.apache.org/jira/browse/PARQUET-1441 and fixed in
[https://github.com/apache/parquet-mr/pull/560] - which is available in 1.11.x
On the hudi mvn deps, it appears that the parquet version prop is set to
[1.10.1|https://github.com/apache/hudi/blob/master/pom.xml#L100]. however its
overridden for the spark context build
https://github.com/apache/hudi/blob/master/pom.xml#L1589
I think, updating the version in the parent pom global prop should fix it for
presto as well?
was (Author: gudladona):
Adding some more context on the dependency fixes.
Original issue on the parquet lib was reported here:
https://issues.apache.org/jira/browse/PARQUET-1441 and fixed in
[https://github.com/apache/parquet-mr/pull/560] - which is available in 1.11.x
On the hudi mvn deps, it appears that the parquet version prop is set to
[1.10.1|https://github.com/apache/hudi/blob/master/pom.xml#L100]. however its
overridden for the spark context build
[here][[https://github.com/apache/hudi/blob/master/pom.xml#L1589]]
I think, updating the version in the parent pom global prop should fix it for
presto as well?
> presto and hive beeline fails to read MOR table w/ 2 or more array fields
> -
>
> Key: HUDI-3391
> URL: https://issues.apache.org/jira/browse/HUDI-3391
> Project: Apache Hudi
> Issue Type: Task
> Components: reader-core
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Critical
> Fix For: 0.11.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> We have an issue reported by user
> [here|[https://github.com/apache/hudi/issues/2657].] Looks like w/ 0.10.0 or
> later, spark datasource read works, but hive beeline does not work. Even
> spark.sql (hive table) querying works as well.
> Another related ticket:
> [https://github.com/apache/hudi/issues/3834#issuecomment-997307677]
>
> Steps that I tried:
> [https://gist.github.com/nsivabalan/fdb8794104181f93b9268380c7f7f079]
> From beeline, you will encounter below exception
> {code:java}
> Failed with exception
> java.io.IOException:org.apache.hudi.org.apache.avro.SchemaParseException:
> Can't redefine: array {code}
> All linked ticket states that upgrading parquet to 1.11.0 or greater should
> work. We need to try it out w/ latest master and go from there.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot commented on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036289601 ## CI report: * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) * 9e22cac5f5bf269aee3f41a72ea0d4f8067954a4 UNKNOWN * b4d9ab2f772fed882bd58d03fa74c793110c7f0e UNKNOWN * 758c072dd67f8869645435eb2dcc97fd13bf13b6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4795: [HUDI-1576] Make archiving an async service
hudi-bot removed a comment on pull request #4795: URL: https://github.com/apache/hudi/pull/4795#issuecomment-1036274270 ## CI report: * 068629b671abbf6fa2f634657d82cf6363eb5f19 UNKNOWN * 9fbfabc1b190d493e54fda5edced1baa18175c02 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5915) * 9e22cac5f5bf269aee3f41a72ea0d4f8067954a4 UNKNOWN * b4d9ab2f772fed882bd58d03fa74c793110c7f0e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot removed a comment on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036260406 ## CI report: * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot commented on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036293797 ## CI report: * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) * b0350aa89d478eedddf483214ba42024880efb27 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot removed a comment on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036293797 ## CI report: * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) * b0350aa89d478eedddf483214ba42024880efb27 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4761: [HUDI-3356][HUDI-3142][HUDI-1492] Metadata column stats index - handling delta writes
hudi-bot commented on pull request #4761: URL: https://github.com/apache/hudi/pull/4761#issuecomment-1036297912 ## CI report: * 6d40a75e6d159f9c961f2e960760022af915beee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5913) * b0350aa89d478eedddf483214ba42024880efb27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5916) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4788: [SUPPORT] run_sync_tool throws java.lang.NoSuchFieldError: TIMESTAMP_MICROS
nsivabalan commented on issue #4788: URL: https://github.com/apache/hudi/issues/4788#issuecomment-1036301189 likely its a jar version mismatch. class of interest if from parquet-avro. Do you happened to have parquet-avro in your class path. if yes, can you remove it and give it a try. CC @xushiyan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4778: [SUPPORT] Row with _hoodie_is_deleted=True stored into target table
nsivabalan commented on issue #4778: URL: https://github.com/apache/hudi/issues/4778#issuecomment-1036303695 sorry. is your requirement, if we have insert and delete for the same record within one batch thats being ingested, you prefer final snapshot in hudi to show the insert record and not delete record? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4778: [SUPPORT] Row with _hoodie_is_deleted=True stored into target table
nsivabalan commented on issue #4778: URL: https://github.com/apache/hudi/issues/4778#issuecomment-1036305248 guess during dedup, hudi goes by preCombine field value. whichever record has higher preCombine value will win. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] geoffroyatkwiff commented on issue #4778: [SUPPORT] Row with _hoodie_is_deleted=True stored into target table
geoffroyatkwiff commented on issue #4778: URL: https://github.com/apache/hudi/issues/4778#issuecomment-1036325464 > sorry. is your requirement, if we have insert and delete for the same record within one batch thats being ingested, you prefer final snapshot in hudi to show the insert record and not delete record? Hi @nsivabalan , no, I would expect the two to cancel each other out. For example: say someone creates a new user in a table, but realises they already had them under another name. They delete this user that was just created 1 minute before, and wouldn't expect the target table to show a row related to this duplicate user. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4784: [SUPPORT] Partition column not appearing in spark dataframe
nsivabalan commented on issue #4784:
URL: https://github.com/apache/hudi/issues/4784#issuecomment-1036330505
I could not reproduce the partitioning issue you are facing.
local spark shell
```
import java.sql.Timestamp
import spark.implicits._
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
val df1 = Seq(
("row1", 1, "part1" ,1578283932000L ),
("row2", 1, "part1", 1578283942000L)
).toDF("row", "ppath", "preComb","eventTime")
df1.write.format("hudi").
options(getQuickstartWriteConfigs).
option(PRECOMBINE_FIELD_OPT_KEY, "preComb").
option(RECORDKEY_FIELD_OPT_KEY, "row").
option(PARTITIONPATH_FIELD_OPT_KEY,
"preComb:simple,ppath:timestamp").
option("hoodie.datasource.write.keygenerator.class","org.apache.hudi.keygen.CustomKeyGenerator").
option("hoodie.deltastreamer.keygen.timebased.timestamp.type","EPOCHMILLISECONDS").
option("hoodie.deltastreamer.keygen.timebased.output.dateformat","-MM-dd").
option("hoodie.deltastreamer.keygen.timebased.timezone","GMT+8:00").
option(TABLE_NAME, "timestamp_tbl4").
mode(Overwrite).
save("/tmp/hudi_timestamp_tbl4")
val hudiDF4 = spark.read.format("hudi").load("/tmp/hudi_timestamp_tbl4")
hudiDF4.registerTempTable("tbl4")
spark.sql("describe tbl4").show()
spark.sql("select * from tbl4 limit 3").show()
```
Output
```
spark.sql("select * from tbl4 limit 3").show()
+---++--+--+++-+---+-+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|
_hoodie_file_name| row|eventTime|preComb|ppath|
+---++--+--+++-+---+-+
| 20220211102107283|20220211102107283...| row1|
part1/1970-01-01|dfc23d4b-8177-4fa...|row1|1578283932000| part1|0|
| 20220211102107283|20220211102107283...| row2|
part1/1970-01-01|dfc23d4b-8177-4fa...|row2|1578283942000| part1|0|
+---++--+--+++-+---+-+
```
specifically values for _hoodie_partition_path are
part1/1970-01-01
2: if you disable hive style partitioning, you may not see the "fieldname=".
But if you want to enable it, don't think hudi allows changing the fieldname
for partition paths.
3: I am not sure on how to leverage partition pruning for custom key gen
based tables. @xushiyan @YannByron @bhasudha : do you folks have any pointers
here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #4784: [SUPPORT] Partition column not appearing in spark dataframe
nsivabalan edited a comment on issue #4784:
URL: https://github.com/apache/hudi/issues/4784#issuecomment-1036330505
I could not reproduce the partitioning issue you are facing. I could see my
partition is well formed and I could see the two original columns which i used
to generate the partition col as well.
local spark shell
```
import java.sql.Timestamp
import spark.implicits._
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
val df1 = Seq(
("row1", 1, "part1" ,1578283932000L ),
("row2", 1, "part1", 1578283942000L)
).toDF("row", "ppath", "preComb","eventTime")
df1.write.format("hudi").
options(getQuickstartWriteConfigs).
option(PRECOMBINE_FIELD_OPT_KEY, "preComb").
option(RECORDKEY_FIELD_OPT_KEY, "row").
option(PARTITIONPATH_FIELD_OPT_KEY,
"preComb:simple,ppath:timestamp").
option("hoodie.datasource.write.keygenerator.class","org.apache.hudi.keygen.CustomKeyGenerator").
option("hoodie.deltastreamer.keygen.timebased.timestamp.type","EPOCHMILLISECONDS").
option("hoodie.deltastreamer.keygen.timebased.output.dateformat","-MM-dd").
option("hoodie.deltastreamer.keygen.timebased.timezone","GMT+8:00").
option(TABLE_NAME, "timestamp_tbl4").
mode(Overwrite).
save("/tmp/hudi_timestamp_tbl4")
val hudiDF4 = spark.read.format("hudi").load("/tmp/hudi_timestamp_tbl4")
hudiDF4.registerTempTable("tbl4")
spark.sql("describe tbl4").show()
spark.sql("select * from tbl4 limit 3").show()
```
Output
```
spark.sql("select * from tbl4 limit 3").show()
+---++--+--+++-+---+-+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|
_hoodie_file_name| row|eventTime|preComb|ppath|
+---++--+--+++-+---+-+
| 20220211102107283|20220211102107283...| row1|
part1/1970-01-01|dfc23d4b-8177-4fa...|row1|1578283932000| part1|0|
| 20220211102107283|20220211102107283...| row2|
part1/1970-01-01|dfc23d4b-8177-4fa...|row2|1578283942000| part1|0|
+---++--+--+++-+---+-+
```
specifically values for _hoodie_partition_path are
part1/1970-01-01
2: if you disable hive style partitioning, you may not see the "fieldname=".
But if you want to enable it, don't think hudi allows changing the fieldname
for partition paths.
3: I am not sure on how to leverage partition pruning for custom key gen
based tables. @xushiyan @YannByron @bhasudha : do you folks have any pointers
here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
