[GitHub] [inlong] featzhang opened a new pull request, #7684: [INLONG-7672][Dashboard] Support kudu node management
featzhang opened a new pull request, #7684: URL: https://github.com/apache/inlong/pull/7684 ### Prepare a Pull Request *([INLONG-7672][Dashboard] Support kudu node management* - Fixes #7672   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang closed pull request #7684: [INLONG-7672][Dashboard] Support kudu node management
featzhang closed pull request #7684: [INLONG-7672][Dashboard] Support kudu node management URL: https://github.com/apache/inlong/pull/7684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] e-mhui opened a new pull request, #7685: [INLONG-7683][Sort] Fix errors in Oracle UT
e-mhui opened a new pull request, #7685: URL: https://github.com/apache/inlong/pull/7685 ### Prepare a Pull Request [INLONG-7683][Sort] Fix errors in Oracle UT - Fixes #7683 ### Motivation In Flink SQL, the Oracle table should be in the format of , not just . For more details, please refer to the documentation at https://inlong.apache.org/zh-CN/docs/next/data_node/extract_node/oracle-cdc. ### Modifications Modify to -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] yunqingmoswu merged pull request #7654: [INLONG-7653][Sort] Support archiving dirty data and metrics for Iceberg connector
yunqingmoswu merged PR #7654: URL: https://github.com/apache/inlong/pull/7654 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[inlong] branch master updated: [INLONG-7653][Sort] Support archiving dirty data and metrics for Iceberg connector (#7654)
This is an automated email from the ASF dual-hosted git repository.
yunqing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong.git
The following commit(s) were added to refs/heads/master by this push:
new 6d4aca310 [INLONG-7653][Sort] Support archiving dirty data and metrics
for Iceberg connector (#7654)
6d4aca310 is described below
commit 6d4aca310330578bc591ca90c1b25366425f8832
Author: Liao Rui
AuthorDate: Fri Mar 24 18:01:41 2023 +0800
[INLONG-7653][Sort] Support archiving dirty data and metrics for Iceberg
connector (#7654)
---
.../sort/iceberg/FlinkDynamicTableFactory.java | 23 +++
.../inlong/sort/iceberg/IcebergTableSink.java | 9 +
.../apache/inlong/sort/iceberg/sink/FlinkSink.java | 34 +++-
.../sink/multiple/DynamicSchemaHandleOperator.java | 207 +
.../multiple/IcebergMultipleFilesCommiter.java | 16 +-
.../sink/multiple/IcebergSingleFileCommiter.java | 16 +-
.../sink/multiple/IcebergSingleStreamWriter.java | 1 -
.../iceberg/sink/multiple/RecordWithSchema.java| 4 +
8 files changed, 214 insertions(+), 96 deletions(-)
diff --git
a/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/FlinkDynamicTableFactory.java
b/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/FlinkDynamicTableFactory.java
index c6d3ed874..32d3a21bd 100644
---
a/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/FlinkDynamicTableFactory.java
+++
b/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/FlinkDynamicTableFactory.java
@@ -34,6 +34,7 @@ import
org.apache.flink.table.factories.DynamicTableSinkFactory;
import org.apache.flink.table.factories.DynamicTableSourceFactory;
import org.apache.flink.table.utils.TableSchemaUtils;
import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.TableProperties;
import org.apache.iceberg.actions.ActionsProvider;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.common.DynMethods;
@@ -112,6 +113,25 @@ public class FlinkDynamicTableFactory implements
DynamicTableSinkFactory, Dynami
.orNoop()
.build();
+public static final ConfigOption WRITE_COMPACT_ENABLE =
+ConfigOptions.key("write.compact.enable")
+.booleanType()
+.defaultValue(false)
+.withDescription("Whether to enable compact small file.");
+
+public static final ConfigOption WRITE_COMPACT_INTERVAL =
+ConfigOptions.key("write.compact.snapshot.interval")
+.intType()
+.defaultValue(20)
+.withDescription("Compact snapshot interval.");
+
+public static final ConfigOption WRITE_DISTRIBUTION_MODE =
+ConfigOptions.key(TableProperties.WRITE_DISTRIBUTION_MODE)
+.stringType()
+.defaultValue(TableProperties.WRITE_DISTRIBUTION_MODE_NONE)
+.withDescription("Distribute the records from input data
stream based "
++ "on the write.distribution-mode.");
+
private final FlinkCatalog catalog;
public FlinkDynamicTableFactory() {
@@ -274,6 +294,9 @@ public class FlinkDynamicTableFactory implements
DynamicTableSinkFactory, Dynami
options.add(SINK_MULTIPLE_SCHEMA_UPDATE_POLICY);
options.add(SINK_MULTIPLE_PK_AUTO_GENERATED);
options.add(SINK_MULTIPLE_TYPE_MAP_COMPATIBLE_WITH_SPARK);
+options.add(WRITE_COMPACT_ENABLE);
+options.add(WRITE_COMPACT_INTERVAL);
+options.add(WRITE_DISTRIBUTION_MODE);
return options;
}
diff --git
a/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/IcebergTableSink.java
b/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/IcebergTableSink.java
index ef93e1f6f..cd239633e 100644
---
a/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/IcebergTableSink.java
+++
b/inlong-sort/sort-connectors/iceberg/src/main/java/org/apache/inlong/sort/iceberg/IcebergTableSink.java
@@ -29,6 +29,7 @@ import
org.apache.flink.table.connector.sink.abilities.SupportsOverwrite;
import org.apache.flink.table.connector.sink.abilities.SupportsPartitioning;
import org.apache.flink.types.RowKind;
import org.apache.flink.util.Preconditions;
+import org.apache.iceberg.DistributionMode;
import org.apache.iceberg.actions.ActionsProvider;
import org.apache.iceberg.flink.CatalogLoader;
import org.apache.iceberg.flink.TableLoader;
@@ -54,6 +55,7 @@ import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_PK_AUTO_GENERA
import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_SCHEMA_UPDATE_POLICY;
import static
org.apache.inlong.sort.base.Constants.SINK_MULTIPLE_TABLE_PATTERN;
import static
org.apache.inlong.sort.
[GitHub] [inlong] featzhang opened a new pull request, #7687: [INLONG-7686][Manager] Support kudu node management
featzhang opened a new pull request, #7687: URL: https://github.com/apache/inlong/pull/7687 ### Prepare a Pull Request *[INLONG-7686][Manager] Support kudu node management* - Fixes #7686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang opened a new pull request, #7689: [INLONG-7688][Manager] Creating schema of StreamSource by SQL
featzhang opened a new pull request, #7689: URL: https://github.com/apache/inlong/pull/7689 ### Prepare a Pull Request *[INLONG-7688][Manager] Creating schema of StreamSource by SQL* - Fixes #7688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang commented on issue #7352: [Bug]Create hudi sink fail
featzhang commented on issue #7352: URL: https://github.com/apache/inlong/issues/7352#issuecomment-1483099176 > @featzhang PTAL OK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [inlong] featzhang opened a new pull request, #7691: [INLONG-7690][Manager] Creating schema of StreamSource by CSV
featzhang opened a new pull request, #7691: URL: https://github.com/apache/inlong/pull/7691 ### Prepare a Pull Request *[INLONG-7690][Manager] Creating schema of StreamSource by CSV* - Fixes #7690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
