[jira] [Created] (FLINK-30257) SqlClientITCase#testMatchRecognize failed
Martijn Visser created FLINK-30257:
--
Summary: SqlClientITCase#testMatchRecognize failed
Key: FLINK-30257
URL: https://issues.apache.org/jira/browse/FLINK-30257
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.17.0
Reporter: Martijn Visser
{code:java}
Nov 30 21:54:41 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 224.683 s <<< FAILURE! - in SqlClientITCase
Nov 30 21:54:41 [ERROR] SqlClientITCase.testMatchRecognize Time elapsed:
50.164 s <<< FAILURE!
Nov 30 21:54:41 org.opentest4j.AssertionFailedError:
Nov 30 21:54:41
Nov 30 21:54:41 expected: 1
Nov 30 21:54:41 but was: 0
Nov 30 21:54:41 at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Nov 30 21:54:41 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Nov 30 21:54:41 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Nov 30 21:54:41 at
SqlClientITCase.verifyNumberOfResultRecords(SqlClientITCase.java:297)
Nov 30 21:54:41 at
SqlClientITCase.testMatchRecognize(SqlClientITCase.java:255)
Nov 30 21:54:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Nov 30 21:54:41 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Nov 30 21:54:41 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 30 21:54:41 at java.lang.reflect.Method.invoke(Method.java:498)
Nov 30 21:54:41 at
org.junit.platform.commons.util.ReflectionUtils.invokeMetho
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43635&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=14817
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30258) PyFlink supports closing loop back server
Xuannan Su created FLINK-30258: -- Summary: PyFlink supports closing loop back server Key: FLINK-30258 URL: https://issues.apache.org/jira/browse/FLINK-30258 Project: Flink Issue Type: Improvement Components: API / Python Affects Versions: 1.16.0 Reporter: Xuannan Su Currently, a loopback server will be started whenever a StreamExecutionEnvironment or StreamTableEnvironment is created. The loopback server can only be closed after the process exit. This might not be a problem for regular uses where only one environment object is used. However, when running tests, such as the unit tests for PyFlink itself, as the environment objects are created, the process starts more and more loopback servers and takes more and more resources. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30259) Using flink Preconditions Util instead of uncertain Assert keyword to do checking
Ran Tao created FLINK-30259:
---
Summary: Using flink Preconditions Util instead of uncertain
Assert keyword to do checking
Key: FLINK-30259
URL: https://issues.apache.org/jira/browse/FLINK-30259
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
SQL / API
Affects Versions: 1.16.0
Reporter: Ran Tao
The code of some modules of the current Flink project uses the 'assert' keyword
of java to do checking, which actually depends on the enablement of the
-enableassertions (-ea) option (default is false), otherwise it may lead to
unexpected behavior. In fact, flink already has a mature Preconditions tool, we
can use it to replace 'assert' keyword. it is more clean and consistent with
flink.
The following is an example of some patch. (by using idea, we can fix it
easily.)
RowDataPrintFunction
{code:java}
@Override
public void invoke(RowData value, Context context) {
Object data = converter.toExternal(value);
assert data != null;
writer.write(data.toString());
}
{code}
KubernetesUtils
{code:java}
public static KubernetesConfigMap checkConfigMaps(
List configMaps, String expectedConfigMapName)
{
assert (configMaps.size() == 1);
assert (configMaps.get(0).getName().equals(expectedConfigMapName));
return configMaps.get(0);
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30260) FLIP-271: Autoscaling
Maximilian Michels created FLINK-30260: -- Summary: FLIP-271: Autoscaling Key: FLINK-30260 URL: https://issues.apache.org/jira/browse/FLINK-30260 Project: Flink Issue Type: New Feature Components: Kubernetes Operator Reporter: Maximilian Michels Assignee: Maximilian Michels https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling -- This message was sent by Atlassian Jira (v8.20.10#820010)
FW: Why streaming over agg not supporting ranged upper boundary?
From: Renjie Liu Date: Tuesday, November 1, 2022 at 15:13 To: u...@flink.apache.org Subject: Why streaming over agg not supporting ranged upper boundary? Hi, flink community: Per the doc here https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/over-agg/#range-definitions Flink streaming over agg only supports current row as upper boundary. I can understand it’s impossible to support unlimited upper boundary, but I don’t know why we can’t support ranges upper boundary? For example I believe following sql is reasonable: SELECT order_id, order_time, amount, SUM(amount) OVER ( PARTITION BY product ORDER BY order_time RANGE BETWEEN INTERVAL '1' HOUR PRECEDING AND INTERVAL '1' HOUR AFTER ) AS one_hour_prod_amount_sum FROM Orders
Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store
Hi @Martijn Thanks for your comments, and I'd like to reply to them 1. It sounds good to me, I'll update the content structure in FLIP later and give the problems first. 2. "Each ETL job creates snapshots with checkpoint info on sink tables in Table Store" -> That reads like you're proposing that snapshots need to be written to Table Store? Yes. To support the data consistency in the FLIP, we need to get through checkpoints in Flink and snapshots in store, this requires a close combination of Flink and store implementation. In the first stage we plan to implement it based on Flink and Table Store only, snapshots written to external storage don't support consistency. 3. If you introduce a MetaService, it becomes the single point of failure because it coordinates everything. But I can't find anything in the FLIP on making the MetaService high available or how to deal with failovers there. I think you raise a very important problem and I missed it in FLIP. The MetaService is a single point and should support failover, we will do it in future in the first stage we only support standalone mode, THX 4. The FLIP states under Rejected Alternatives "Currently watermark in Flink cannot align data." which is not true, given that there is FLIP-182 https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources Watermark alignment in FLIP-182 is different from requirements "watermark align data" in our FLIP. FLIP-182 aims to fix watermark generation in different sources for "slight imbalance or data skew", which means in some cases the source must generate watermark even if they should not. When the operator collects watermarks, the data processing is as described in our FLIP, and the data cannot be aligned through the barrier like Checkpoint. 5. Given the MetaService role, it feels like this is introducing a tight dependency between Flink and the Table Store. How pluggable is this solution, given the changes that need to be made to Flink in order to support this? This is a good question, and I will try to expand it. Most of the work will be completed in the Table Store, such as the new SplitEnumerator and Source implementation. The changes in Flink are as followed: 1) Flink job should put its job id in context when creating source/sink to help MetaService to create relationship between source and sink tables, it's tiny 2) Notify a listener when job is terminated in Flink, and the listener implementation in Table Store will send "delete event" to MetaService. 3) The changes are related to Flink Checkpoint includes a) Support triggering checkpoint with checkpoint id by SplitEnumerator b) Create the SplitEnumerator in Table Store with a strategy to perform the specific checkpoint when all "SplitEnumerator"s in the job manager trigger it. Best, Shammon On Thu, Dec 1, 2022 at 3:43 PM Martijn Visser wrote: > Hi all, > > A couple of first comments on this: > 1. I'm missing the problem statement in the overall introduction. It > immediately goes into proposal mode, I would like to first read what is the > actual problem, before diving into solutions. > 2. "Each ETL job creates snapshots with checkpoint info on sink tables in > Table Store" -> That reads like you're proposing that snapshots need to be > written to Table Store? > 3. If you introduce a MetaService, it becomes the single point of failure > because it coordinates everything. But I can't find anything in the FLIP on > making the MetaService high available or how to deal with failovers there. > 4. The FLIP states under Rejected Alternatives "Currently watermark in > Flink cannot align data." which is not true, given that there is FLIP-182 > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources > > 5. Given the MetaService role, it feels like this is introducing a tight > dependency between Flink and the Table Store. How pluggable is this > solution, given the changes that need to be made to Flink in order to > support this? > > Best regards, > > Martijn > > > On Thu, Dec 1, 2022 at 4:49 AM Shammon FY wrote: > > > Hi devs: > > > > I'd like to start a discussion about FLIP-276: Data Consistency of > > Streaming and Batch ETL in Flink and Table Store[1]. In the whole data > > stream processing, there are consistency problems such as how to manage > the > > dependencies of multiple jobs and tables, how to define and handle E2E > > delays, and how to ensure the data consistency of queries on flowing > data? > > This FLIP aims to support data consistency and answer these questions. > > > > I'v discussed the details of this FLIP with @Jingsong Lee and @libenchao > > offline several times. We hope to support data consistency of queries on > > tables, managing relationships between Flink jobs and tables and revising > > tables on streaming in Flink and Table Store to improve the whole data > > stream processing. > > > > Looking forward to your feedback
[jira] [Created] (FLINK-30261) PartiallyFinishedSourcesITCase.test timed out while waiting for tasks to finish
Matthias Pohl created FLINK-30261:
-
Summary: PartiallyFinishedSourcesITCase.test timed out while
waiting for tasks to finish
Key: FLINK-30261
URL: https://issues.apache.org/jira/browse/FLINK-30261
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.15.3
Reporter: Matthias Pohl
{{PartiallyFinishedSourcesITCase.test}} timed out while waiting for tasks to
finish:
{code}
"main" #1 prio=5 os_prio=0 tid=0x7fe78800b800 nid=0x3a11f waiting on
condition [0x7fe78e6fb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:138)
at
org.apache.flink.runtime.testutils.CommonTestUtils.waitForSubtasksToFinish(CommonTestUtils.java:291)
at
org.apache.flink.runtime.operators.lifecycle.TestJobExecutor.waitForSubtasksToFinish(TestJobExecutor.java:226)
at
org.apache.flink.runtime.operators.lifecycle.PartiallyFinishedSourcesITCase.test(PartiallyFinishedSourcesITCase.java:138)
[...]
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43637&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=39302
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30262) UpsertKafkaTableITCase failed when starting the container because waiting for a port timed out
Matthias Pohl created FLINK-30262:
-
Summary: UpsertKafkaTableITCase failed when starting the container
because waiting for a port timed out
Key: FLINK-30262
URL: https://issues.apache.org/jira/browse/FLINK-30262
Project: Flink
Issue Type: Bug
Components: Build System, Connectors / Kafka, Test Infrastructure
Affects Versions: 1.16.0
Reporter: Matthias Pohl
{code:java}
Dec 01 08:35:00 Caused by:
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for
container port to open (172.17.0.1 ports: [60109, 60110] should be listening)
Dec 01 08:35:00 at
org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:90)
Dec 01 08:35:00 at
org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
Dec 01 08:35:00 at
org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:926)
Dec 01 08:35:00 at
org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:480)
Dec 01 08:35:00 ... 33 more
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43643&view=logs&j=aa18c3f6-13b8-5f58-86bb-c1cffb239496&t=502fb6c0-30a2-5e49-c5c2-a00fa3acb203&l=37366
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[RESULT][VOTE] Release flink-connector-cassandra, release candidate #1
I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Etienne * Dawid (binding) * Martijjn (binding) * Danny (binding) * Ryan There are no disapproving votes. Thanks everyone! On 25/11/2022 10:31, Chesnay Schepler wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 3.0.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], which are signed with the key with fingerprint C2EED7B111D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag v3.0.0-1 [5], * website pull request listing the new release [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. This is the first externalized released of the Cassandra connector and functionally identical to 1.16.0. Thanks, Chesnay [1] https://issues.apache.org/jira/projects/FLINK/versions/12352593 [2] https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.0.0-rc1 [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1549/ [5] https://github.com/apache/flink-connector-cassandra/releases/tag/v3.0.0-1 [6] https://github.com/apache/flink-web/pull/587
[jira] [Created] (FLINK-30263) Introduce schemas table to table store
Shammon created FLINK-30263: --- Summary: Introduce schemas table to table store Key: FLINK-30263 URL: https://issues.apache.org/jira/browse/FLINK-30263 Project: Flink Issue Type: New Feature Components: Table Store Affects Versions: table-store-0.3.0 Reporter: Shammon You can query the historical schemas of the table through SQL, for example, query the historical schemas of table "T" through the following SQL: SELECT * FROM T$schemas; -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
Yes, the StreamingFileSink is not affected. Best, Fabian
[jira] [Created] (FLINK-30264) Set session job status to FAILED if already have retried max attempts
Xin Hao created FLINK-30264: --- Summary: Set session job status to FAILED if already have retried max attempts Key: FLINK-30264 URL: https://issues.apache.org/jira/browse/FLINK-30264 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Xin Hao Sometimes, the session job deployment fails because of the user code. There is nothing the Flink operator can do to fix the failure. So can we add a new reconciliation state *FAILED* and set the status to this if the failure still exists after we have retried for the max attempts? The reconciliation status will continuously be *UPGRADING* currently. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax
+1 (binding) Thanks for the FLIP! On Thu, Dec 1, 2022 at 12:20 PM Shengkai Fang wrote: > Hi All, > > Thanks for all the feedback so far. Based on the discussion[1] we seem > to have a consensus, so I would like to start a vote on FLIP-273. > > The vote will last for at least 72 hours (Dec 5th at 13:00 GMT, > excluding weekend days) unless there is an objection or insufficient votes. > > Best, > Shengkai > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-273%3A+Improve+the+Catalog+API+to+Support+ALTER+TABLE+syntax > [2] https://lists.apache.org/thread/2v4kh2bpzvk049zdxb687q7o1pcmnnnw >
Re: [DISCUSS]Introduce a time-segment based restart strategy
Hi Gen, Thanks for the suggestions! Regarding how to implement the per-region RestartBackoffTimeStrategy as proposed previously, I think your approach works well. Here are more details: - Keep the RestartBackoffTimeStrategy interface API unchanged and only change its semantics, such that all strategies (e.g. failure rate, fixed delay, exponential delay) are applied per region. - Update ExecutionFailureHandler to create one RestartBackoffTimeStrategy instance for each region. ExecutionFailureHandler can get the region information from its SchedulingTopology. - ExecutionFailureHandler::getFailureHandlingResult() will use the strategy instance for the given failedTask's region to make the failover decision. Please see the other comment inline. Regards, Dong On Fri, Nov 25, 2022 at 7:42 PM Gen Luo wrote: > Hi all, > > Sorry for the late jumping in. > > To meet Weihua's need, Dong's proposal seems pretty fine, but the > modification it requires, I'm afraid, is not really easy. > RestartBackoffTimeStrategy is quite a simple interface. The strategy even > doesn't know which task is failing, not to mention the division of pipeline > regions. > To distinguish the failure count of each regions, it lacks too much > information, which is not easy to acquire for the strategy. > One approch I can figure out is to create different strategy instances to > different regions. In this way we do not need to modify the strategy but do > need to modify the schedulers or the ExecutionFailureHandler. > > On the other hand, I realize another case that the restart strategy may > need to be aware of the types and occurrence rate of the exceptions. That > is to avoid failing over but directly fail the job when some errors happen. > I know that there's an annotation > `@ThrowableAnnotation(ThrowableType.NonRecoverableError)` that can fail the > job, but I'm afraid there can be some scenarios that can not annotate the > exceptions, or catch and warp with an annotated exception. > While this is possible, do you have a concrete use-case that can not use and catch the annotated exception? It is probably safer to only add a new strategy (which is a public API) when we are sure we need it :) In such cases, handling in the restart strategy can be a good choice. > Such a strategy can even combines with other existing strategies which > handle the failure rate rather than the cause type. > > Besides, given that new strategies may be necessary, and existing > strategies may also need to enhance, maybe we should make the > RestartBackoffTimeStrategy a plugin rather than the enumerations, or > introduce a new custom type strategy which can load customized > implementations. > This can not solve the problem immediately, but makes the choice of restart > strategy more flexiable. > What do you think about this? > > Thanks. > > Paul Lam 于 2022年11月21日周一 17:46写道： > > > Dong’s proposal LGTM. > > > > Best, > > Paul Lam > > > > > 2022年11月19日 10:50，Dong Lin 写道： > > > > > > Hey Weihua, > > > > > > Thanks for proposing the new strategy! > > > > > > If I understand correctly, the main issue is that different failover > > > regions can be restarted independently, but they share the same counter > > > when counting the number of failures in an interval. So the number of > > > failures for a given region is less than what users expect. > > > > > > Given that regions can be restarted independently, it might be more > > usable > > > and intuitive to count the number of failures for each region when > > > executing the failover strategy. Thus, instead of adding a new failover > > > strategy, how about we update the existing failure-rate strategy, and > > > probably other existing strategies as well, to use the following > > semantics: > > > > > > - For any given region in the job, its number of failures in > > > failure-rate-interval should not exceed max-failures-per-interval. > > > Otherwise, the job will fail without being restarted. > > > > > > By using this updated semantics, the keyby-connected job will have the > > same > > > behavior as the existing Flink when we use failure-rate strategy. For > > > the rescale-connected > > > job, in the case you described above, after the TM fails, each of the 3 > > > regions will increment its failure count from 0 to 1, which is still > less > > > than max-failures-per-interval. Thus the rescale-connected job can > > continue > > > to work. > > > > > > This alternative approach can solve the problem without increasing the > > > complexity of the failover strategy choice. And this approach does not > > > require us to check whether two exceptions belong to the same root > cause. > > > Do you think it can work? > > > > > > Thanks, > > > Dong > > > > > > > > > On Fri, Nov 4, 2022 at 4:46 PM Weihua Hu > wrote: > > > > > >> Hi, everyone > > >> > > >> I'd like to bring up a discussion about restart strategy. Flink > > supports 3 > > >> kinds of restart strategy. These work very well for jobs with specific > > >> c
[jira] [Created] (FLINK-30265) Turn on debug logs for k8s operator e2e tests in CI when tests failing
Gabor Somogyi created FLINK-30265: - Summary: Turn on debug logs for k8s operator e2e tests in CI when tests failing Key: FLINK-30265 URL: https://issues.apache.org/jira/browse/FLINK-30265 Project: Flink Issue Type: Bug Components: Kubernetes Operator Reporter: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30266) Recovery reconciliation loop fails if no checkpoint has been created yet
Maximilian Michels created FLINK-30266:
--
Summary: Recovery reconciliation loop fails if no checkpoint has
been created yet
Key: FLINK-30266
URL: https://issues.apache.org/jira/browse/FLINK-30266
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.3.0
Reporter: Maximilian Michels
Assignee: Gyula Fora
When the upgradeMode is LAST-STATE, the operator fails to reconcile a failed
application unless at least one checkpoint has already been created. The
expected behavior would be that the job starts with empty state.
{noformat}
2022-12-01 10:58:35,596 o.a.f.k.o.l.AuditUtils [INFO ] [app] >>> Status
| Error | UPGRADING |
{"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"HA
metadata not available to restore from last state. It is possible that the job
has finished or terminally failed, or the configmaps have been deleted. Manual
restore
required.","additionalMetadata":{"reason":"RestoreFailed"},"throwableList":[]}
{noformat}
{noformat}
2022-12-01 10:44:49,480 i.j.o.p.e.ReconciliationDispatcher [ERROR] [app] Error
during event processing ExecutionScope{ resource id: ResourceID{name='app',
namespace='namespace'}, version: 216933301} failed.
org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.RuntimeException: This indicates a bug...
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:133)
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:54)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
at
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
at
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
at
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: This indicates a bug...
at
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:180)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:61)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.restoreJob(AbstractJobReconciler.java:212)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.reconcileSpecChange(AbstractJobReconciler.java:144)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:167)
at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:64)
at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:123)
... 13 more {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[DISCUSS] Retroactively externalize some connectors for 1.16
Hello, let me clarify the title first. In the original proposal for the connector externalization we said that an externalized connector has to exist in parallel with the version shipped in the main Flink release for 1 cycle. For example, 1.16.0 shipped with the elasticsearch connector, but at the same time there's the externalized variant as a drop-in replacement, and the 1.17.0 release will not include a ES connector. The rational was to give users some window to update their projects. We are now about to externalize a few more connectors (cassandra, pulsar, jdbc), targeting 1.16 within the next week. The 1.16.0 release has now been about a month ago; so it hasn't been a lot of time since then. I'm now wondering if we could/should treat these connectors as externalized for 1.16, meaning that we would remove them from the master branch now, not ship them in 1.17 and move all further development into the connector repos. The main benefit is that we won't have to bother with syncing changes across repos all the time. We would of course need some sort-of cutoff date for this (December 9th?), to ensure there's still some reasonably large gap left for users to migrate. Let me know what you think. Regards, Chesnay
[jira] [Created] (FLINK-30267) PyFlink YARN per-job on Docker test failed because it could not build hadoop image
Martijn Visser created FLINK-30267:
--
Summary: PyFlink YARN per-job on Docker test failed because it
could not build hadoop image
Key: FLINK-30267
URL: https://issues.apache.org/jira/browse/FLINK-30267
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.17.0
Reporter: Martijn Visser
{code:java}
Dec 01 15:10:48 E: Unable to locate package net-tools
Dec 01 15:10:48 E: Unable to locate package build-essential
The command '/bin/sh -c set -x && apt-get update &&
DEBIAN_FRONTEND=noninteractive apt-get install -y curl tar sudo
openssh-server openssh-client rsync unzip krb5-user net-tools build-essential'
returned a non-zero code: 100
Service 'master' failed to build : Build failed
Dec 01 15:10:48 Command: build_image failed. Retrying...
Dec 01 15:10:50 Command: build_image failed 5 times.
Dec 01 15:10:50 ERROR: Could not build hadoop image. Aborting...
Dec 01 15:10:50 Debugging failed YARN Docker test:
Dec 01 15:10:50
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43650&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=8414
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax
+1 (non-binding). Looking forward to it! On Thu, Dec 1, 2022 at 5:06 AM Dong Lin wrote: > +1 (binding) > > Thanks for the FLIP! > > On Thu, Dec 1, 2022 at 12:20 PM Shengkai Fang wrote: > > > Hi All, > > > > Thanks for all the feedback so far. Based on the discussion[1] we seem > > to have a consensus, so I would like to start a vote on FLIP-273. > > > > The vote will last for at least 72 hours (Dec 5th at 13:00 GMT, > > excluding weekend days) unless there is an objection or insufficient > votes. > > > > Best, > > Shengkai > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-273%3A+Improve+the+Catalog+API+to+Support+ALTER+TABLE+syntax > > [2] https://lists.apache.org/thread/2v4kh2bpzvk049zdxb687q7o1pcmnnnw > > >
Re: [DISCUSS] Retroactively externalize some connectors for 1.16
Hello, +1 I was thinking the same. With regard to the cut off date I would be inclined to be more aggressive and say feature freeze for 1.17. Users do not *need* to migrate for 1.16. Thanks On Thu, 1 Dec 2022, 15:01 Chesnay Schepler, wrote: > Hello, > > let me clarify the title first. > > In the original proposal for the connector externalization we said that > an externalized connector has to exist in parallel with the version > shipped in the main Flink release for 1 cycle. > > For example, 1.16.0 shipped with the elasticsearch connector, but at the > same time there's the externalized variant as a drop-in replacement, and > the 1.17.0 release will not include a ES connector. > > The rational was to give users some window to update their projects. > > > We are now about to externalize a few more connectors (cassandra, > pulsar, jdbc), targeting 1.16 within the next week. > The 1.16.0 release has now been about a month ago; so it hasn't been a > lot of time since then. > I'm now wondering if we could/should treat these connectors as > externalized for 1.16, meaning that we would remove them from the master > branch now, not ship them in 1.17 and move all further development into > the connector repos. > > The main benefit is that we won't have to bother with syncing changes > across repos all the time. > > We would of course need some sort-of cutoff date for this (December > 9th?), to ensure there's still some reasonably large gap left for users > to migrate. > > Let me know what you think. > > Regards, > Chesnay > >
Re: [DISCUSS] Retroactively externalize some connectors for 1.16
Hi! I think this would be a good idea. I was wondering that could we include the hbase connector to this group as well? The externalization PR [1] should be in a good shape now and Dec 9th as a release date sounds doable. WDYT? [1] https://github.com/apache/flink-connector-hbase/pull/2 Best, F --- Original Message --- On Thursday, December 1st, 2022 at 16:01, Chesnay Schepler wrote: > > > Hello, > > let me clarify the title first. > > In the original proposal for the connector externalization we said that > an externalized connector has to exist in parallel with the version > shipped in the main Flink release for 1 cycle. > > For example, 1.16.0 shipped with the elasticsearch connector, but at the > same time there's the externalized variant as a drop-in replacement, and > the 1.17.0 release will not include a ES connector. > > The rational was to give users some window to update their projects. > > > We are now about to externalize a few more connectors (cassandra, > pulsar, jdbc), targeting 1.16 within the next week. > The 1.16.0 release has now been about a month ago; so it hasn't been a > lot of time since then. > I'm now wondering if we could/should treat these connectors as > externalized for 1.16, meaning that we would remove them from the master > branch now, not ship them in 1.17 and move all further development into > the connector repos. > > The main benefit is that we won't have to bother with syncing changes > across repos all the time. > > We would of course need some sort-of cutoff date for this (December > 9th?), to ensure there's still some reasonably large gap left for users > to migrate. > > Let me know what you think. > > Regards, > Chesnay
[VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0
Hi Danny, +1 (non-binding) - Checked release notes - Verified hashes & signature - Build the source with Maven - Verified tag is present - Verified web PR - Verified streaming sink writes to DynamoDB Best regards, Yuri Gusev On 2022/11/28 16:42:42 Danny Cranmer wrote: > Hi everyone, > Please review and vote on the release candidate #0 for the version 3.0.0 as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org [2], > which are signed with the key with fingerprint 125FD8DB [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.0.0-rc0 [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours (Thursday 1st December 17:00 > UTC). It is adopted by majority approval, with at least 3 PMC affirmative > votes. > > Please note, this is a new connector and the first release. > > Thanks, > Danny > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352277 > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-3.0.0-rc0 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1552/ > [5] https://github.com/apache/flink-connector-aws/releases/tag/v3.0.0-rc0 > [6] https://github.com/apache/flink-web/pull/588 >
[jira] [Created] (FLINK-30268) HA metadata and other cluster submission related errors should not throw DeploymentFailedException
Gyula Fora created FLINK-30268:
--
Summary: HA metadata and other cluster submission related errors
should not throw DeploymentFailedException
Key: FLINK-30268
URL: https://issues.apache.org/jira/browse/FLINK-30268
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Reporter: Gyula Fora
Assignee: Peter Vary
Fix For: kubernetes-operator-1.3.0
Currently most critical cluster submission errors , and checks that validate HA
metadata before deployment, end up throwing DeploymentFailedException.
This causes the operator to go into a weird state and actually hide the error
in subsequent loops:
{noformat}
flink-kubernetes-operator 2022-12-01 21:55:03,978 o.a.f.k.o.l.AuditUtils
[INFO ][default/basic-checkpoint-ha-example] >>> Status | Info | UPGRADING
| The resource is being upgraded
flink-kubernetes-operator 2022-12-01 21:55:03,992 o.a.f.k.o.l.AuditUtils
[INFO ][default/basic-checkpoint-ha-example] >>> Event | Info | SUBMIT
| Starting deployment
flink-kubernetes-operator 2022-12-01 21:55:03,992
o.a.f.k.o.s.AbstractFlinkService [INFO ][default/basic-checkpoint-ha-example]
Deploying application cluster requiring last-state from HA metadata
flink-kubernetes-operator 2022-12-01 21:55:03,997
o.a.f.k.o.c.FlinkDeploymentController
[ERROR][default/basic-checkpoint-ha-example] Flink Deployment failed
flink-kubernetes-operator
org.apache.flink.kubernetes.operator.exception.DeploymentFailedException: HA
metadata not available to restore from last state. It is possible that the job
has finished or terminally failed, or the configmaps have been deleted. Manual
restore required.
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.validateHaMetadataExists(AbstractFlinkService.java:844)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.submitApplicationCluster(AbstractFlinkService.java:177)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:195)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:60)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.restoreJob(AbstractJobReconciler.java:210)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractJobReconciler.reconcileSpecChange(AbstractJobReconciler.java:142)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:161)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:62)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:123)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:54)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
flink-kubernetes-operator at
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
flink-kubernetes-operator at
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
flink-kubernetes-operator at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
flink-kubernetes-operator at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
flink-kubernetes-operator at java.base/java.lang.Thread.run(Unknown Source)
flink-kubernetes-operat
RE: [VOTE] Release flink-connector-dynamodb v3.0.0, release candidate #0
Hi Danny, +1 (non-binding) - Checked release notes - Verified hashes & signature - Build the source with Maven - Verified tag is present - Verified web PR - Verified streaming sink writes to DynamoDB Best regards, Yuri Gusev On 2022/11/28 16:42:42 Danny Cranmer wrote: > Hi everyone, > Please review and vote on the release candidate #0 for the version 3.0.0 as > follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org [2], > which are signed with the key with fingerprint 125FD8DB [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.0.0-rc0 [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours (Thursday 1st December 17:00 > UTC). It is adopted by majority approval, with at least 3 PMC affirmative > votes. > > Please note, this is a new connector and the first release. > > Thanks, > Danny > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352277 > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-3.0.0-rc0 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1552/ > [5] https://github.com/apache/flink-connector-aws/releases/tag/v3.0.0-rc0 > [6] https://github.com/apache/flink-web/pull/588 >
Re: [DISCUSS] assign SQL Table properties from environment variables
Hello devs,
I'd like to revive this discussion. There is also a ticket about this effort
for some time [1] and this thing also affects us as well. Right now we have a
custom solution that is similar to "environment variables", but it only can be
used in parts of our downstream product. The main thing for us to achieve would
be to be able to use variables in DDLs (not necessarily for hiding sensitive
props). I think it would be really handy to have the ability to reuse values in
multiple tables.
With that said, comes the temptation to hit two birds with one stone, although
a sensitive property requires much more care than a regular one, so I think
these two things should be handled separately. At least in the beginning. The
tricky part of the "environment variables" are their scope, and if they are not
coming from an external system, it will probably be necessary to persist them.
Or keep them in memory, but that may be insufficient according to what is the
scope of the "environment variables".
Considering the sensitive props, I think a small step forward could be to hide
the values in case of a "SHOW CREATE TABLE" op.
For a varible to be used in a DDL I'd imagine it could apply for a whole
catalog as starters. As long as the catalog is present, those variables would
be valid.
I did not check implementation details yet, so it is possible I'm missing
something important or wrong in some places, but I wanted to get some feedback
about the idea.
WDYT?
[1] https://issues.apache.org/jira/browse/FLINK-28028
Best,
F
--- Original Message ---
On Monday, April 4th, 2022 at 09:53, Timo Walther wrote:
>
>
> Hi Fred,
>
> thanks for starting this discussion. I totally agree that this an issue
> that the community should solve. It popped up before and is still
> unsolved today. Great that you offer your help here. So let's clarify
> the implementation details.
>
> 1) Global vs. Local solution
>
> Is this a DDL-only problem? If yes, it would be easier to solve it in
> the `FactoryUtil` that all Flink connectors and formats use.
>
> 2) Configruation vs. enviornment variables
>
> I agree with Qingsheng that environment variable are not always
> straightforward to identify if you have a "pre-flight phase" and a
> "cluster phase".
> In the DynamicTableFactory, one has access to Flink configuration and
> could resolve `${...}` variables.
>
>
> What do you think?
>
> Regards,
> Timo
>
>
> Am 01.04.22 um 12:26 schrieb Qingsheng Ren:
>
> > Hi Fred,
> >
> > Thanks for raising the discussion! I think the definition of “environment
> > variable” varies under different context. Under Flink on K8s it means the
> > environment variable for a container, and if you are a SQL client user it
> > could refer to environment variable of SQL client, or even the system
> > properties on JVM. So using “environment variable” is a bit vague under
> > different environments.
> >
> > A more generic solution in my mind is that we can take advantage of
> > configurations in Flink, to pass table options dynamically by adding
> > configs to TableConfig or even flink-conf.yaml. For example option
> > “table.dynamic.options.my_catalog.my_db_.my_table.accessId = foo” means
> > adding table option “accessId = foo” to table “my_catalog.my_db.my_table”.
> > By this way we could de-couple DDL statement with table options containing
> > secret credentials. What do you think?
> >
> > Best regards,
> >
> > Qingsheng
> >
> > > On Mar 30, 2022, at 16:25, Teunissen, F.G.J. (Fred)
> > > fred.teunis...@ing.com.INVALID wrote:
> > >
> > > Hi devs,
> > >
> > > Some SQL Table properties contain sensitive data, like passwords that we
> > > do not want to expose in the VVP ui to other users. Also, having them
> > > clear text in a SQL statement is not secure. For example,
> > >
> > > CREATE TABLE Orders (
> > > `user` BIGINT,
> > > product STRING,
> > > order_time TIMESTAMP(3)
> > > ) WITH (
> > > 'connector' = 'kafka',
> > >
> > > 'properties.bootstrap.servers' = 'kafka-host-1:9093,kafka-host-2:9093',
> > > 'properties.security.protocol' = 'SSL',
> > > 'properties.ssl.key.password' = 'should-be-a-secret',
> > > 'properties.ssl.keystore.location' = '/tmp/secrets/my-keystore.jks',
> > > 'properties.ssl.keystore.password' = 'should-also-be-a-secret',
> > > 'properties.ssl.truststore.location' = '/tmp/secrets/my-truststore.jks',
> > > 'properties.ssl.truststore.password' = 'should-again-be-a-secret',
> > > 'scan.startup.mode' = 'earliest-offset'
> > > );
> > >
> > > I would like to bring up for a discussion a proposal to provide these
> > > secrets values via environment variables since these can be populated
> > > from a K8s configMap or secrets.
> > >
> > > For implementing the SQL Table properties, the ConfigOption class is
> > > used in connectors and formatters. This class could be extended that it
> > > checks whether the config-value contains certain tokens, like
> > > ‘${env-var-name}’. If i
Re: [DISCUSS] Retroactively externalize some connectors for 1.16
Hello Chesney, The overall plan sounds good! Just to double check, is Dec 9th the proposed cutoff date for the release of those externalized connectors? Also, will we reserve time for users to verify that the drop-in replacement from Flink 1.16 to those externalized connectors can work as expected before removing their code from the master branch? Thanks, Dong On Thu, Dec 1, 2022 at 11:01 PM Chesnay Schepler wrote: > Hello, > > let me clarify the title first. > > In the original proposal for the connector externalization we said that > an externalized connector has to exist in parallel with the version > shipped in the main Flink release for 1 cycle. > > For example, 1.16.0 shipped with the elasticsearch connector, but at the > same time there's the externalized variant as a drop-in replacement, and > the 1.17.0 release will not include a ES connector. > > The rational was to give users some window to update their projects. > > > We are now about to externalize a few more connectors (cassandra, > pulsar, jdbc), targeting 1.16 within the next week. > The 1.16.0 release has now been about a month ago; so it hasn't been a > lot of time since then. > I'm now wondering if we could/should treat these connectors as > externalized for 1.16, meaning that we would remove them from the master > branch now, not ship them in 1.17 and move all further development into > the connector repos. > > The main benefit is that we won't have to bother with syncing changes > across repos all the time. > > We would of course need some sort-of cutoff date for this (December > 9th?), to ensure there's still some reasonably large gap left for users > to migrate. > > Let me know what you think. > > Regards, > Chesnay > >
Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax
+1 (non-binding) Best regards, Yuxia - 原始邮件 - 发件人: "Yaroslav Tkachenko" 收件人: "dev" 发送时间: 星期五, 2022年 12 月 02日 上午 12:27:24 主题: Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax +1 (non-binding). Looking forward to it! On Thu, Dec 1, 2022 at 5:06 AM Dong Lin wrote: > +1 (binding) > > Thanks for the FLIP! > > On Thu, Dec 1, 2022 at 12:20 PM Shengkai Fang wrote: > > > Hi All, > > > > Thanks for all the feedback so far. Based on the discussion[1] we seem > > to have a consensus, so I would like to start a vote on FLIP-273. > > > > The vote will last for at least 72 hours (Dec 5th at 13:00 GMT, > > excluding weekend days) unless there is an objection or insufficient > votes. > > > > Best, > > Shengkai > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-273%3A+Improve+the+Catalog+API+to+Support+ALTER+TABLE+syntax > > [2] https://lists.apache.org/thread/2v4kh2bpzvk049zdxb687q7o1pcmnnnw > > >
Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax
+1 (non-binding) Best, Paul Lam > 2022年12月2日 09:17，yuxia 写道： > > +1 (non-binding) > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Yaroslav Tkachenko" > 收件人: "dev" > 发送时间: 星期五, 2022年 12 月 02日 上午 12:27:24 > 主题: Re: [VOTE] FLIP-273: Improve Catalog API to Support ALTER TABLE syntax > > +1 (non-binding). > > Looking forward to it! > > On Thu, Dec 1, 2022 at 5:06 AM Dong Lin wrote: > >> +1 (binding) >> >> Thanks for the FLIP! >> >> On Thu, Dec 1, 2022 at 12:20 PM Shengkai Fang wrote: >> >>> Hi All, >>> >>> Thanks for all the feedback so far. Based on the discussion[1] we seem >>> to have a consensus, so I would like to start a vote on FLIP-273. >>> >>> The vote will last for at least 72 hours (Dec 5th at 13:00 GMT, >>> excluding weekend days) unless there is an objection or insufficient >> votes. >>> >>> Best, >>> Shengkai >>> >>> [1] >>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-273%3A+Improve+the+Catalog+API+to+Support+ALTER+TABLE+syntax >>> [2] https://lists.apache.org/thread/2v4kh2bpzvk049zdxb687q7o1pcmnnnw >>> >>
[jira] [Created] (FLINK-30269) Validate table name for metadata table
Shammon created FLINK-30269: --- Summary: Validate table name for metadata table Key: FLINK-30269 URL: https://issues.apache.org/jira/browse/FLINK-30269 Project: Flink Issue Type: Sub-task Components: Table Store Affects Versions: table-store-0.3.0 Reporter: Shammon Currently user can create tables `tablename` and `tablename$snapshots`, but can't insert into values into `tablename$snapshots` and execute query on it. At the same time, user can create table `tablename$aaa$bbb`, but cannot execute query on it and event cannot drop it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30270) FlinkJoinToMultiJoinRule incorrectly combines Left/Right outer join to MultiJoin
Yunhong Zheng created FLINK-30270: - Summary: FlinkJoinToMultiJoinRule incorrectly combines Left/Right outer join to MultiJoin Key: FLINK-30270 URL: https://issues.apache.org/jira/browse/FLINK-30270 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.16.0 Reporter: Yunhong Zheng Fix For: 1.17.0 FlinkJoinToMultiJoinRule incorrectly combines Left/Right outer join to MultiJoin. In some complex cases, it need to consider join conditions, -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30271) Introduce Table.copy from dynamic options
Jingsong Lee created FLINK-30271: Summary: Introduce Table.copy from dynamic options Key: FLINK-30271 URL: https://issues.apache.org/jira/browse/FLINK-30271 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.3.0 At present, our processing of dynamic options is relatively independent. In FileStoreTableFactory, this is not conducive to other engines configuring dynamic options. We should propose an interface on the Table, and dynamic options can be configured at any time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30272) Introduce a Predicate Visitor
Jingsong Lee created FLINK-30272: Summary: Introduce a Predicate Visitor Key: FLINK-30272 URL: https://issues.apache.org/jira/browse/FLINK-30272 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.3.0 At present, predicate is traversed in many places. We need a visitor mode, which can better traverse Predicate. -- This message was sent by Atlassian Jira (v8.20.10#820010)
