[jira] [Created] (FLINK-26176) Scala savepoint migration tests don't run as configured (not testing RocksDB savepoints)
Sebastian Mattheis created FLINK-26176: -- Summary: Scala savepoint migration tests don't run as configured (not testing RocksDB savepoints) Key: FLINK-26176 URL: https://issues.apache.org/jira/browse/FLINK-26176 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.14.3, 1.13.5, 1.12.7, 1.11.6, 1.10.3, 1.9.3 Reporter: Sebastian Mattheis The following Scala integration test cases in flink-tests: * flink-tests/src/test/scala/org/apache/flink/api/scala/migration/StatefulJobSavepointMigrationITCase.scala * flink-tests/src/test/scala/org/apache/flink/api/scala/migration/StatefulJobWBroadcastStateMigrationITCase.scala are supposed and configured to test migration of savepoints of the RocksDB, Memory, and HashMap state backend between minor version upgrades (1.x -> 1.y). However, the test methods (in both test cases), which are * testCreateSavepoint and testRestoreSavepoint * testCreateSavepointWithBroadcastState and testRestoreSavepointWithBroadcast do set the state backend to use only MemoryStateBackend and overwrite any previous (correct according to test configuration) setting of the state backend. The consequence is: * tests run only with MemoryStateBackend * previous test resources, i.e., savepoints for RocksDB/HashMap states of versions (1.3-1.14), are corrputed Proposed solution; * fix the configuration * delete corrupted savepoints * recreate savepoints for older versions OR omit test runs for older versions -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26177) PulsarSourceITCase.testScaleDown fails with timeout
Matthias Pohl created FLINK-26177:
-
Summary: PulsarSourceITCase.testScaleDown fails with timeout
Key: FLINK-26177
URL: https://issues.apache.org/jira/browse/FLINK-26177
Project: Flink
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Matthias Pohl
We observed a [build
failure|https://dev.azure.com/mapohl/flink/_build/results?buildId=742&view=logs&j=f3dc9b18-b77a-55c1-591e-264c46fe44d1&t=2d3cd81e-1c37-5c31-0ee4-f5d5cdb9324d&l=26553]
caused by {{PulsarSourceITCase.testScaleDown}}:
{code}
Feb 15 20:56:02 [ERROR] Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 431.023 s <<< FAILURE! - in
org.apache.flink.connector.pulsar.source.PulsarSourceITCase
Feb 15 20:56:02 [ERROR]
org.apache.flink.connector.pulsar.source.PulsarSourceITCase.testScaleDown(TestEnvironment,
DataStreamSourceExternalContext, CheckpointingMode)[2] Time elapsed: 138.444
s <<< FAILURE!
Feb 15 20:56:02 java.lang.AssertionError:
Feb 15 20:56:02
Feb 15 20:56:02 Expecting
Feb 15 20:56:02
Feb 15 20:56:02 to be completed within 2M.
Feb 15 20:56:02
Feb 15 20:56:02 exception caught while trying to get the future result:
java.util.concurrent.TimeoutException
Feb 15 20:56:02 at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
[...]
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26178) Use the same enum for expected and observed jobstate (JobState / JobStatus.state)
Gyula Fora created FLINK-26178: -- Summary: Use the same enum for expected and observed jobstate (JobState / JobStatus.state) Key: FLINK-26178 URL: https://issues.apache.org/jira/browse/FLINK-26178 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Reporter: Gyula Fora We should consolidate these two and maybe even use Flink's own job state enum here if makes sense -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26179) Support periodic savepointing in the operator
Gyula Fora created FLINK-26179: -- Summary: Support periodic savepointing in the operator Key: FLINK-26179 URL: https://issues.apache.org/jira/browse/FLINK-26179 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Reporter: Gyula Fora Automatic triggering of savepoints is a commonly requested feature. The configuration should be part of the job spec. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26180) Update docs to introduce the compaction for FileSink
Gen Luo created FLINK-26180: --- Summary: Update docs to introduce the compaction for FileSink Key: FLINK-26180 URL: https://issues.apache.org/jira/browse/FLINK-26180 Project: Flink Issue Type: Sub-task Reporter: Gen Luo -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26181) Support manual savepoint triggering in the operator
Gyula Fora created FLINK-26181: -- Summary: Support manual savepoint triggering in the operator Key: FLINK-26181 URL: https://issues.apache.org/jira/browse/FLINK-26181 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Reporter: Gyula Fora We should allow users to trigger savepoints manually. There have been different approaches to implement this in different operators: - savepoint generation field (Google/Spotify operator) - trigger savepoint via resource annotation [~wangyang0918] - native operator poc I think the annotation approach is more elegant but we should discuss this on the ML before proceeding with the implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26182) E2E test for PulsarSink on new Flink sink testing tools
Yufan Sheng created FLINK-26182: --- Summary: E2E test for PulsarSink on new Flink sink testing tools Key: FLINK-26182 URL: https://issues.apache.org/jira/browse/FLINK-26182 Project: Flink Issue Type: Sub-task Components: Connectors / Pulsar Reporter: Yufan Sheng -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26183) Support kubernetes-operator metrics using the Flink metric system
Gyula Fora created FLINK-26183: -- Summary: Support kubernetes-operator metrics using the Flink metric system Key: FLINK-26183 URL: https://issues.apache.org/jira/browse/FLINK-26183 Project: Flink Issue Type: Sub-task Components: Deployment / Kubernetes Reporter: Gyula Fora Assignee: Matyas Orhidi We should leverage the existing Metric and Reporter infrastructure of Flink to expose metrics of the Flink operator. Users should be able to pass configuration to the operator that will control the metric registry (like for regular Flink jobs) and should be able to access the built in metric reporter plugins. Initially we should expose standard JVM metrics, and later we can add operator specific metric groups if needed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26184) Migrate StoreSink to Sink V2
Jingsong Lee created FLINK-26184: Summary: Migrate StoreSink to Sink V2 Key: FLINK-26184 URL: https://issues.apache.org/jira/browse/FLINK-26184 Project: Flink Issue Type: Sub-task Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.1.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator
Hi All! Thank you all for reviewing the PR and already helping to make it better. I have opened a bunch of jira tickets under https://issues.apache.org/jira/browse/FLINK-25963 based on some comments and incomplete features in general. Given that there were no major objections about the prototype, I will merge it now so we can start collaborating together. Cheers, Gyula On Wed, Feb 16, 2022 at 3:52 AM Yang Wang wrote: > Thanks for the explanation. > Given that it is unrelated with java version in Flink. > Starting with java11 for the flink-kubernetes-operator makes sense to me. > > > Best, > Yang > > Thomas Weise 于2022年2月15日周二 23:57写道： > > > Hi, > > > > At this point I see no reason to support Java 8 for a new project. > > Java 8 is being phased out, we should start with 11. > > > > Also, since the operator isn't a library but effectively just a docker > > image, the ability to change the Java version isn't as critical as it > > is for Flink core, which needs to run in many different environments. > > > > Cheers, > > Thomas > > > > On Tue, Feb 15, 2022 at 4:50 AM Gyula Fóra wrote: > > > > > > Hi Devs, > > > > > > Yang Wang discovered that the current prototype is not compatible with > > Java > > > 8 but only 11 and upwards. > > > > > > The reason for this is that the java operator SDK itself is not java 8 > > > compatible unfortunately. > > > > > > Given that Java 8 is on the road to deprecation and that the operator > > runs > > > as a containerized deployment, are there any concerns regarding making > > the > > > target java version 11? > > > This should not affect deployed flink clusters and jobs, those should > > still > > > work with Java 8, but only the kubernetes operator itself. > > > > > > Cheers, > > > Gyula > > > > > > > > > On Tue, Feb 15, 2022 at 1:06 PM Yang Wang > wrote: > > > > > > > I also lean to not introduce the savepoint/checkpoint related fields > > to the > > > > job spec, especially in the very beginning of > > flink-kubernetes-operator. > > > > > > > > > > > > Best, > > > > Yang > > > > > > > > Gyula Fóra 于2022年2月15日周二 19:02写道： > > > > > > > > > Hi Peng Yuan! > > > > > > > > > > While I do agree that savepoint path is a very important production > > > > > configuration there are a lot of other things that come to my mind: > > > > > - savepoint dir > > > > > - checkpoint dir > > > > > - checkpoint interval/timeout > > > > > - high availability settings (provider/storagedir etc) > > > > > > > > > > just to name a few... > > > > > > > > > > While these are all production critical, they have nice clean Flink > > > > config > > > > > settings to go with them. If we stand introducing these to jobspec > we > > > > only > > > > > get confusion about priority order etc and it is going to be hard > to > > > > change > > > > > or remove them in the future. In any case we should validate that > > these > > > > > configs exist in cases where users use a stateful upgrade mode for > > > > example. > > > > > This is something we need to add for sure. > > > > > > > > > > As for the other options you mentioned like automatic savepoint > > > > generation > > > > > for instance, those deserve an independent discussion of their own > I > > > > > believe :) > > > > > > > > > > Cheers, > > > > > Gyula > > > > > > > > > > On Tue, Feb 15, 2022 at 11:23 AM K Fred > > wrote: > > > > > > > > > > > Hi Matyas! > > > > > > > > > > > > Thanks for your reply! > > > > > > For 1. and 3. scenarios，I couldn't agree more with the > podTemplate > > > > > solution > > > > > > , i missed this part. > > > > > > For savepoint related configuration, I think it's very important > > to be > > > > > > specified in JobSpec, Because savepoint is a very common > > configuration > > > > > for > > > > > > upgrading a job, if it has been placed in JobSpec can be > obviously > > > > > > configured by the user. In addition, other advanced properties > can > > be > > > > put > > > > > > into flinkConfiguration customized by expert users. > > > > > > A bunch of savepoint configuration as follows: > > > > > > > > > > > > > fromSavepoint——Job restart from > > > > > > > > > > > > autoSavepointSecond—— Automatically take a savepoint to the > > > > > `savepointsDir` > > > > > > > every n seconds. > > > > > > > > > > > > savepointsDir—— Savepoints dir where to store automatically taken > > > > > > > savepoints > > > > > > > > > > > > savepointGeneration—— Update savepoint generation of job status > > for a > > > > > > > running job (should be defined in JobStatus) > > > > > > > > > > > > > > > > > > Best wishes, > > > > > > Peng Yuan. > > > > > > > > > > > > On Tue, Feb 15, 2022 at 4:41 PM Őrhidi Mátyás < > > matyas.orh...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Peng, > > > > > > > > > > > > > > Thanks for your feedback. Regarding 1. and 3. scenarios, the > > > > > podTemplate > > > > > > > functionality in the operator could cover both. We also need to > > be > > > > > > careful > > > > > > > about in
Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator
Hi Gyula, sorry for joining late. One comment on the API design for consideration: we are using the job.state as kind of a "desired state", right? This is quite uncommon in Kubernetes to my knowledge. In Kubernetes almost always the fact that a resource exists means that it should be "running". The only API that I am aware of that has something like "suspended" is a Kubernetes Job ( https://kubernetes.io/docs/concepts/workloads/controllers/job/#suspending-a-job), which looks retrofitted to me. Cheers, Konstantin On Wed, Feb 16, 2022 at 10:52 AM Gyula Fóra wrote: > Hi All! > > Thank you all for reviewing the PR and already helping to make it better. I > have opened a bunch of jira tickets under > https://issues.apache.org/jira/browse/FLINK-25963 based on some comments > and incomplete features in general. > > Given that there were no major objections about the prototype, I will merge > it now so we can start collaborating together. > > Cheers, > Gyula > > On Wed, Feb 16, 2022 at 3:52 AM Yang Wang wrote: > > > Thanks for the explanation. > > Given that it is unrelated with java version in Flink. > > Starting with java11 for the flink-kubernetes-operator makes sense to me. > > > > > > Best, > > Yang > > > > Thomas Weise 于2022年2月15日周二 23:57写道： > > > > > Hi, > > > > > > At this point I see no reason to support Java 8 for a new project. > > > Java 8 is being phased out, we should start with 11. > > > > > > Also, since the operator isn't a library but effectively just a docker > > > image, the ability to change the Java version isn't as critical as it > > > is for Flink core, which needs to run in many different environments. > > > > > > Cheers, > > > Thomas > > > > > > On Tue, Feb 15, 2022 at 4:50 AM Gyula Fóra > wrote: > > > > > > > > Hi Devs, > > > > > > > > Yang Wang discovered that the current prototype is not compatible > with > > > Java > > > > 8 but only 11 and upwards. > > > > > > > > The reason for this is that the java operator SDK itself is not java > 8 > > > > compatible unfortunately. > > > > > > > > Given that Java 8 is on the road to deprecation and that the operator > > > runs > > > > as a containerized deployment, are there any concerns regarding > making > > > the > > > > target java version 11? > > > > This should not affect deployed flink clusters and jobs, those should > > > still > > > > work with Java 8, but only the kubernetes operator itself. > > > > > > > > Cheers, > > > > Gyula > > > > > > > > > > > > On Tue, Feb 15, 2022 at 1:06 PM Yang Wang > > wrote: > > > > > > > > > I also lean to not introduce the savepoint/checkpoint related > fields > > > to the > > > > > job spec, especially in the very beginning of > > > flink-kubernetes-operator. > > > > > > > > > > > > > > > Best, > > > > > Yang > > > > > > > > > > Gyula Fóra 于2022年2月15日周二 19:02写道： > > > > > > > > > > > Hi Peng Yuan! > > > > > > > > > > > > While I do agree that savepoint path is a very important > production > > > > > > configuration there are a lot of other things that come to my > mind: > > > > > > - savepoint dir > > > > > > - checkpoint dir > > > > > > - checkpoint interval/timeout > > > > > > - high availability settings (provider/storagedir etc) > > > > > > > > > > > > just to name a few... > > > > > > > > > > > > While these are all production critical, they have nice clean > Flink > > > > > config > > > > > > settings to go with them. If we stand introducing these to > jobspec > > we > > > > > only > > > > > > get confusion about priority order etc and it is going to be hard > > to > > > > > change > > > > > > or remove them in the future. In any case we should validate that > > > these > > > > > > configs exist in cases where users use a stateful upgrade mode > for > > > > > example. > > > > > > This is something we need to add for sure. > > > > > > > > > > > > As for the other options you mentioned like automatic savepoint > > > > > generation > > > > > > for instance, those deserve an independent discussion of their > own > > I > > > > > > believe :) > > > > > > > > > > > > Cheers, > > > > > > Gyula > > > > > > > > > > > > On Tue, Feb 15, 2022 at 11:23 AM K Fred > > > wrote: > > > > > > > > > > > > > Hi Matyas! > > > > > > > > > > > > > > Thanks for your reply! > > > > > > > For 1. and 3. scenarios，I couldn't agree more with the > > podTemplate > > > > > > solution > > > > > > > , i missed this part. > > > > > > > For savepoint related configuration, I think it's very > important > > > to be > > > > > > > specified in JobSpec, Because savepoint is a very common > > > configuration > > > > > > for > > > > > > > upgrading a job, if it has been placed in JobSpec can be > > obviously > > > > > > > configured by the user. In addition, other advanced properties > > can > > > be > > > > > put > > > > > > > into flinkConfiguration customized by expert users. > > > > > > > A bunch of savepoint configuration as follows: > > > > > > > > > > > > > > > fromSavepoint——Job restart from > > > > > > > > > > > > >
[jira] [Created] (FLINK-26185) E2E Elasticsearch Tests should use the new Sink interface
Alexander Preuss created FLINK-26185: Summary: E2E Elasticsearch Tests should use the new Sink interface Key: FLINK-26185 URL: https://issues.apache.org/jira/browse/FLINK-26185 Project: Flink Issue Type: Bug Components: Connectors / ElasticSearch, Tests Affects Versions: 1.15.0 Reporter: Alexander Preuss Currently the E2E tests for Elasticsearch (test_streaming_elasticsearch.sh) is testing the old Sink interface implementation. As we are now moving to the new Sink interface, we should update the tests to test the new implementation -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator
Hi Konstantin, I completely agree with the general philosophy that if a resource exists it should be "running" or I would rather say "do it's thing" whatever that means for a particular resource. We followed this design principle when we decided to have only 2 "desired states" running and suspended and not have states like canceled etc. Based on our own use cases and the feedback that we received from others temporarily suspending a streaming job deployment is part of it's regular lifecycle. The job still exist and will continue after suspension but it signifies a state where data processing should be paused for whatever reason. The current suspend mechanism is also difficult to do manually if we remove it from the operator: 1. We need to implement a cancel-with-savepoint operation and expose this to the user 2. The user needs to manually look up the savepoint 3. Create a new resource later Adding 1.) is basically equivalent to the current implementation but would actually expose an operation that feels much more unnatural compared to a supend. Cheers, Gyula On Wed, Feb 16, 2022 at 11:16 AM Konstantin Knauf wrote: > Hi Gyula, > > sorry for joining late. One comment on the API design for consideration: we > are using the job.state as kind of a "desired state", right? This is quite > uncommon in Kubernetes to my knowledge. In Kubernetes almost always the > fact that a resource exists means that it should be "running". The only API > that I am aware of that has something like "suspended" is a Kubernetes Job > ( > > https://kubernetes.io/docs/concepts/workloads/controllers/job/#suspending-a-job > ), > which looks retrofitted to me. > > Cheers, > > Konstantin > > On Wed, Feb 16, 2022 at 10:52 AM Gyula Fóra wrote: > > > Hi All! > > > > Thank you all for reviewing the PR and already helping to make it > better. I > > have opened a bunch of jira tickets under > > https://issues.apache.org/jira/browse/FLINK-25963 based on some comments > > and incomplete features in general. > > > > Given that there were no major objections about the prototype, I will > merge > > it now so we can start collaborating together. > > > > Cheers, > > Gyula > > > > On Wed, Feb 16, 2022 at 3:52 AM Yang Wang wrote: > > > > > Thanks for the explanation. > > > Given that it is unrelated with java version in Flink. > > > Starting with java11 for the flink-kubernetes-operator makes sense to > me. > > > > > > > > > Best, > > > Yang > > > > > > Thomas Weise 于2022年2月15日周二 23:57写道： > > > > > > > Hi, > > > > > > > > At this point I see no reason to support Java 8 for a new project. > > > > Java 8 is being phased out, we should start with 11. > > > > > > > > Also, since the operator isn't a library but effectively just a > docker > > > > image, the ability to change the Java version isn't as critical as it > > > > is for Flink core, which needs to run in many different environments. > > > > > > > > Cheers, > > > > Thomas > > > > > > > > On Tue, Feb 15, 2022 at 4:50 AM Gyula Fóra > > wrote: > > > > > > > > > > Hi Devs, > > > > > > > > > > Yang Wang discovered that the current prototype is not compatible > > with > > > > Java > > > > > 8 but only 11 and upwards. > > > > > > > > > > The reason for this is that the java operator SDK itself is not > java > > 8 > > > > > compatible unfortunately. > > > > > > > > > > Given that Java 8 is on the road to deprecation and that the > operator > > > > runs > > > > > as a containerized deployment, are there any concerns regarding > > making > > > > the > > > > > target java version 11? > > > > > This should not affect deployed flink clusters and jobs, those > should > > > > still > > > > > work with Java 8, but only the kubernetes operator itself. > > > > > > > > > > Cheers, > > > > > Gyula > > > > > > > > > > > > > > > On Tue, Feb 15, 2022 at 1:06 PM Yang Wang > > > wrote: > > > > > > > > > > > I also lean to not introduce the savepoint/checkpoint related > > fields > > > > to the > > > > > > job spec, especially in the very beginning of > > > > flink-kubernetes-operator. > > > > > > > > > > > > > > > > > > Best, > > > > > > Yang > > > > > > > > > > > > Gyula Fóra 于2022年2月15日周二 19:02写道： > > > > > > > > > > > > > Hi Peng Yuan! > > > > > > > > > > > > > > While I do agree that savepoint path is a very important > > production > > > > > > > configuration there are a lot of other things that come to my > > mind: > > > > > > > - savepoint dir > > > > > > > - checkpoint dir > > > > > > > - checkpoint interval/timeout > > > > > > > - high availability settings (provider/storagedir etc) > > > > > > > > > > > > > > just to name a few... > > > > > > > > > > > > > > While these are all production critical, they have nice clean > > Flink > > > > > > config > > > > > > > settings to go with them. If we stand introducing these to > > jobspec > > > we > > > > > > only > > > > > > > get confusion about priority order etc and it is going to be > hard > > > to > > > > > > change > > > > > > > or remove them
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi Jark, I think you are raising a very good point. I think we need an application mode for SQL that would work along the lines of executing a SQL script (incl. init scripts) located in a particular directory in the Docker Image. Details to be discussed. Do you think Zeppelin/SQL CLI could work with such a mode for non-interactive queries (interactive queries would use a session cluster)? Best, Konstantin On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > Hi David, > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > jobs. So the session cluster is not a fit mode. > > Best, > Jark > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > Hi Jark, can you please elaborate about the current need of the per-job > > mode for interactive clients (eg. Zeppelin that you've mentioned)? Aren't > > these a natural fit for the session cluster? > > > > D. > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > Hi Konstantin, > > > > > > I'm not very familiar with the implementation of per-job mode and > > > application mode. > > > But is there any instruction for users abou how to migrate > platforms/jobs > > > to application mode? > > > IIUC, the biggest difference between the two modes is where the main() > > > method is executed. > > > However, SQL jobs are not jar applications and don't have the main() > > > method. > > > For example, SQL CLI submits SQL jobs by invoking > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > How SQL Client and SQL platforms (e.g. Zeppelin) support application > > mode? > > > > > > Best, > > > Jark > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > wrote: > > > > > > > Hi everyone, > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > these limitations of per-job mode on YARN. It seems that there is a > > > general > > > > agreement to deprecate per-job mode and to drop it once the > limitations > > > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > > > > > Thanks again, > > > > > > > > Konstantin > > > > > > > > > > > > [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > wrote: > > > > > > > > > Hi Yang, > > > > > > > > > > Thank you for the clarification. In general I think we will have > time > > > to > > > > > experiment with this until it will be removed totally and migrate > our > > > > > solution to use application mode. > > > > > > > > > > Regards, > > > > > F > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > Hi all, > > > > > > > > > > > > I remember the application mode was initially named "cluster > mode". > > > As > > > > a > > > > > > contrast, the per-job mode is the "client mode". > > > > > > So I believe application mode should cover all the > functionalities > > of > > > > > > per-job except where we are running the user main code. > > > > > > In the containerized or the Kubernetes world, the application > mode > > is > > > > > more > > > > > > native and easy to use since all the Flink and user > > > > > > jars are bundled in the image. I am also in favor of deprecating > > and > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > IIRC, the YARN application mode could ship user jars and > > dependencies > > > > via > > > > > > "yarn.ship-files" config option. The only > > > > > > limitation is that we could not ship and load the user > dependencies > > > > with > > > > > > user classloader, not the parent classloader. > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > directory > > > > > > automatically. > > > > > > > > > > > > > > > > > > Best, > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > > > First of all, sorry for the delay. We at Cloudera are currently > > > > > relying on > > > > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > > > > > > > Specifically, we allow users to upload connector jars and other > > > > > artifacts. > > > > > > > There are also some default jars that we need to ship. These > are > > > all > > > > > stored > > > > > > > on the local file system of our service’s node. The Flink job > is > > > > > submitted > > > > > > > on the users’ behalf by our service, which also specifies the > > jars > > > to > > > > > ship. > > > > > > > The service runs on a single node, not on all nodes with Flink > > > TM/JM. > > > > > It > > > > > > > would thus be difficult to manage the jars on every node. > > > > > > > > > > > > > > We are not familiar with the reasoning behind why application > > mode > > > > > > > currently doesn’t ship the user jars, besides the deployment > > being > > > > > faster > > > > > > > this way. Would it be possible for the application mode to > >
[jira] [Created] (FLINK-26186) update the API_ANNOTATIONS rule
Jing Ge created FLINK-26186:
---
Summary: update the API_ANNOTATIONS rule
Key: FLINK-26186
URL: https://issues.apache.org/jira/browse/FLINK-26186
Project: Flink
Issue Type: Improvement
Reporter: Jing Ge
Assignee: Jing Ge
It seems that the excluding shaded classes filter does not work in this case
since the check is against a valid class whose return type is a shaded external
class which somehow was not handled by the filter.
Since there is no way to add Flink API annotation to the 3rd party classes, the
rule should be updated.
the main ideas is to extend the rule allow external classes without Flink API
annotation, like:
.or(
{_}resideInAnyPackage{_}(
"org.apache.flink.(*)..shaded.(**)"))
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26187) Chinese docs override english aliases
Chesnay Schepler created FLINK-26187: Summary: Chinese docs override english aliases Key: FLINK-26187 URL: https://issues.apache.org/jira/browse/FLINK-26187 Project: Flink Issue Type: Bug Components: Documentation Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0, 1.14.4 Various chinese pages define an alias for an URL to an english page. This results in redirects being set up that point to the chinese version of the docs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Re: [DISCUSS] Future of Per-Job Mode
I think this mode is still limited and maybe not easy to extend. Could the application mode provide an interface to execute? So that clients can implement the interface and pass arbitrary parameters (e.g. SQL scripts) ? Best, Jark On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf wrote: > Hi Jark, > > I think you are raising a very good point. I think we need an application > mode for SQL that would work along the lines of executing a SQL script > (incl. init scripts) located in a particular directory in the Docker Image. > Details to be discussed. > > Do you think Zeppelin/SQL CLI could work with such a mode for > non-interactive queries (interactive queries would use a session cluster)? > > Best, > > Konstantin > > > On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > > > Hi David, > > > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > > jobs. So the session cluster is not a fit mode. > > > > Best, > > Jark > > > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > > > Hi Jark, can you please elaborate about the current need of the per-job > > > mode for interactive clients (eg. Zeppelin that you've mentioned)? > Aren't > > > these a natural fit for the session cluster? > > > > > > D. > > > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > > > Hi Konstantin, > > > > > > > > I'm not very familiar with the implementation of per-job mode and > > > > application mode. > > > > But is there any instruction for users abou how to migrate > > platforms/jobs > > > > to application mode? > > > > IIUC, the biggest difference between the two modes is where the > main() > > > > method is executed. > > > > However, SQL jobs are not jar applications and don't have the main() > > > > method. > > > > For example, SQL CLI submits SQL jobs by invoking > > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > > How SQL Client and SQL platforms (e.g. Zeppelin) support application > > > mode? > > > > > > > > Best, > > > > Jark > > > > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > > these limitations of per-job mode on YARN. It seems that there is a > > > > general > > > > > agreement to deprecate per-job mode and to drop it once the > > limitations > > > > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > > > > > > > Thanks again, > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > [1] > https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > > > > wrote: > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > Thank you for the clarification. In general I think we will have > > time > > > > to > > > > > > experiment with this until it will be removed totally and migrate > > our > > > > > > solution to use application mode. > > > > > > > > > > > > Regards, > > > > > > F > > > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > I remember the application mode was initially named "cluster > > mode". > > > > As > > > > > a > > > > > > > contrast, the per-job mode is the "client mode". > > > > > > > So I believe application mode should cover all the > > functionalities > > > of > > > > > > > per-job except where we are running the user main code. > > > > > > > In the containerized or the Kubernetes world, the application > > mode > > > is > > > > > > more > > > > > > > native and easy to use since all the Flink and user > > > > > > > jars are bundled in the image. I am also in favor of > deprecating > > > and > > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > > IIRC, the YARN application mode could ship user jars and > > > dependencies > > > > > via > > > > > > > "yarn.ship-files" config option. The only > > > > > > > limitation is that we could not ship and load the user > > dependencies > > > > > with > > > > > > > user classloader, not the parent classloader. > > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > > directory > > > > > > > automatically. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > > > > > First of all, sorry for the delay. We at Cloudera are > currently > > > > > > relying on > > > > > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > > > > > > > > > Specifically, we allow users to upload connector jars and > other > > > > > > artifacts. > > > > > > > > There are also some default jars that we need to ship. These > > are > > > > all > > > > > > stored > > > > > > > > on the local file system of our service’s node. The Flink jo
[jira] [Created] (FLINK-26188) Refactor TestingHighAvailabilityServices to properly use the builder pattern.
Matthias Pohl created FLINK-26188:
-
Summary: Refactor TestingHighAvailabilityServices to properly use
the builder pattern.
Key: FLINK-26188
URL: https://issues.apache.org/jira/browse/FLINK-26188
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Coordination
Affects Versions: 1.16.0
Reporter: Matthias Pohl
We want to align the creation of {{{}TestingHighAvailabilityServices{}}}. This
is currently done differently in different places.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26189) Remove bundling of grizzled-slf4j from rpc-akka
Chesnay Schepler created FLINK-26189: Summary: Remove bundling of grizzled-slf4j from rpc-akka Key: FLINK-26189 URL: https://issues.apache.org/jira/browse/FLINK-26189 Project: Flink Issue Type: Technical Debt Components: Build System Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 This dependency is unnecessary nowadays. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release 1.13.6, release candidate #1
+1 (binding) - checked signatures and checksums of binaries OK - build from sources - run TopSpeedWindowing.jar in standalone application mode - ran examples/python/table/batch/word_count.py in a local mini cluster - ran examples/python/table/batch/word_count.py against standalone session cluster On Tue, Feb 15, 2022 at 4:53 PM Anton Kalashnikov wrote: > +1 (non-binding) > > - signatures OK > - checksums OK > - tag OK > - all artifacts OK > - PR OK > > run examples > > -- > > Best regards, > Anton Kalashnikov > > > 15.02.2022 09:30, Dawid Wysakowicz пишет: > > +1 (binding) > > > > > > - signatures OK > > - checksums OK > > - tag OK > > - PR looks good > > > > - built from sources > > > > - run example > > > > - checked dependency version changes from 1.13.5 and notice files > > > > Best, > > > > Dawid > > > > On 05/02/2022 21:06, Konstantin Knauf wrote: > >> Hi everyone, > >> > >> Please review and vote on the release candidate #1 for the version > >> 1.13.6, > >> as follows: > >> [ ] +1, Approve the release > >> [ ] -1, Do not approve the release (please provide specific comments) > >> > >> The complete staging area is available for your review, which > >> includes: > >> * JIRA release notes [1], > >> * the official Apache source release and binary convenience releases > >> to be > >> deployed to dist.apache.org [2], which are signed with the key with > >> fingerprint 8C3FB007FE60 DEFA [3], > >> * all artifacts to be deployed to the Maven Central Repository [4], > >> * source code tag "release-1.2.3-rc3" [5], > >> * website pull request listing the new release and adding > >> announcement blog > >> post [6]. > >> > >> The vote will be open for at least 72 hours. It is adopted by majority > >> approval, with at least 3 PMC affirmative votes. > >> > >> Thanks, > >> Konstantin > >> > >> [1] > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351074 > >> > >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.6-rc1/ > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >> [4] > >> https://repository.apache.org/content/repositories/orgapacheflink-1486/ > >> [5] https://github.com/apache/flink/tree/release-1.13.6-rc1 > >> [6] https://github.com/apache/flink-web/pull/505 > >> > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[RESULT] [VOTE] Release 1.13.6, release candidate #1
Hi everyone, I am pleased to announce that we have unanimously approved this release candidate: There are 4 approving votes, 3 of which are binding: - Chesnay Schepler (binding) - Dawid Wysakowicz (binding) - Anton Kalashnikov (non-binding) - Konstantin Knauf (binding) There are no disapproving votes. Thank you for verifying the release candidate. I will now (today or tomorrow) proceed to finalize the release and announce it once everything is published. Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[jira] [Created] (FLINK-26190) Remove TableConfig from ExecNodeConfiguration
Marios Trivyzas created FLINK-26190:
---
Summary: Remove TableConfig from ExecNodeConfiguration
Key: FLINK-26190
URL: https://issues.apache.org/jira/browse/FLINK-26190
Project: Flink
Issue Type: Technical Debt
Components: Table SQL / Planner
Reporter: Marios Trivyzas
Currently, *ExecNodeConfiguration* holds *TableConfig* on top of the
*plannerConfig* and the *persistedConfig,* since it's needed by the
*CodeGeneratorContext* basically for the {*}nullCheck{*}.
The *nullCheck* should be deprecated and removed which will facilitate the
usage of the simple iface: *ReadableConfig* in *CodeGeneratorContext* **
instead of *TableConfig* and the *TableConfig* could be removed
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26191) Incorrect license in Elasticsearch 7 connector
Chesnay Schepler created FLINK-26191: Summary: Incorrect license in Elasticsearch 7 connector Key: FLINK-26191 URL: https://issues.apache.org/jira/browse/FLINK-26191 Project: Flink Issue Type: Technical Debt Components: Connectors / ElasticSearch Reporter: Chesnay Schepler Fix For: 1.15.0 The sql-connector-elasticsearch7 NOTICE lists the elasticsearch dependencies as ASLv2, but they are nowadays (at least in part) licensed differently (dual-licensed under elastic license 2.0 & Server Side Public License (SSPL)). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Hi all, I would like to formally announce a few new Flink PMC members on the dev@ list. The PMC has not done a good job of always announcing new PMC members (and committers) recently. I'll try to keep an eye on this in the future to improve the situation. Nevertheless, I'm very happy to announce some very active community members as new PMC members: - Igal Shilman, added to the PMC in October 2021 - Konstantin Knauf, added to the PMC in January 2022 - Yun Gao, added to the PMC in February 2022 Please join me in welcoming them to the Flink PMC! Best, Robert
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congrats to all three of you, well deserved. Best, Fabian On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger wrote: > > Hi all, > > I would like to formally announce a few new Flink PMC members on the dev@ > list. The PMC has not done a good job of always announcing new PMC members > (and committers) recently. I'll try to keep an eye on this in the future to > improve the situation. > > Nevertheless, I'm very happy to announce some very active community members > as new PMC members: > > - Igal Shilman, added to the PMC in October 2021 > - Konstantin Knauf, added to the PMC in January 2022 > - Yun Gao, added to the PMC in February 2022 > > Please join me in welcoming them to the Flink PMC! > > Best, > Robert
[ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Hi everyone, On behalf of the PMC, I'm very happy to announce two new Flink committers: Feng Wang and Zhipeng Zhang! Feng is one of the most active Flink evangelists in China, with plenty of public talks, blog posts and other evangelization activities. The PMC wants to recognize and value these efforts by making Feng a committer! Zhipeng Zhang has made significant contributions to flink-ml, like most of the FLIPs for our ML efforts. Please join me in welcoming them as committers! Best, Robert
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations to all of you! Best regards, Martijn Visser https://twitter.com/MartijnVisser82 On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > Congrats to all three of you, well deserved. > > Best, > Fabian > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > wrote: > > > > Hi all, > > > > I would like to formally announce a few new Flink PMC members on the dev@ > > list. The PMC has not done a good job of always announcing new PMC > members > > (and committers) recently. I'll try to keep an eye on this in the future > to > > improve the situation. > > > > Nevertheless, I'm very happy to announce some very active community > members > > as new PMC members: > > > > - Igal Shilman, added to the PMC in October 2021 > > - Konstantin Knauf, added to the PMC in January 2022 > > - Yun Gao, added to the PMC in February 2022 > > > > Please join me in welcoming them to the Flink PMC! > > > > Best, > > Robert >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations Feng and Zhipeng! Best regards, Martijn https://twitter.com/MartijnVisser82 On Wed, 16 Feb 2022 at 14:32, Robert Metzger wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce two new Flink > committers: Feng Wang and Zhipeng Zhang! > > Feng is one of the most active Flink evangelists in China, with plenty of > public talks, blog posts and other evangelization activities. The PMC wants > to recognize and value these efforts by making Feng a committer! > > Zhipeng Zhang has made significant contributions to flink-ml, like most of > the FLIPs for our ML efforts. > > Please join me in welcoming them as committers! > > > Best, > Robert >
[jira] [Created] (FLINK-26192) PulsarOrderedSourceReaderTest fails with exit code 255
Dawid Wysakowicz created FLINK-26192:
Summary: PulsarOrderedSourceReaderTest fails with exit code 255
Key: FLINK-26192
URL: https://issues.apache.org/jira/browse/FLINK-26192
Project: Flink
Issue Type: Bug
Components: Connectors / Pulsar
Affects Versions: 1.15.0
Reporter: Dawid Wysakowicz
https://dev.azure.com/wysakowiczdawid/Flink/_build/results?buildId=1367&view=logs&j=f3dc9b18-b77a-55c1-591e-264c46fe44d1&t=2d3cd81e-1c37-5c31-0ee4-f5d5cdb9324d&l=26787
{code}
Feb 16 13:49:46 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-connector-pulsar: There are test failures.
Feb 16 13:49:46 [ERROR]
Feb 16 13:49:46 [ERROR] Please refer to
/__w/1/s/flink-connectors/flink-connector-pulsar/target/surefire-reports for
the individual test results.
Feb 16 13:49:46 [ERROR] Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
Feb 16 13:49:46 [ERROR] The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
Feb 16 13:49:46 [ERROR] Command was /bin/sh -c cd
/__w/1/s/flink-connectors/flink-connector-pulsar &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
-Dmvn.forkNumber=1
-XX:-UseGCOverheadLimit -Duser.country=US -Duser.language=en -jar
/__w/1/s/flink-connectors/flink-connector-pulsar/target/surefire/surefirebooter3139517882560779643.jar
/__w/1/s/flink-connectors/flink-connector-pulsar/target/surefire
2022-02-16T13-48-34_435-jvmRun1 surefire3358354372075396323tmp
surefire_08509996975514960300tmp
Feb 16 13:49:46 [ERROR] Error occurred in starting fork, check output in log
Feb 16 13:49:46 [ERROR] Process Exit Code: 255
Feb 16 13:49:46 [ERROR]
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
terminated without properly saying goodbye. VM crash or System.exit called?
Feb 16 13:49:46 [ERROR] Command was /bin/sh -c cd
/__w/1/s/flink-connectors/flink-connector-pulsar &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
-Dmvn.forkNumber=1
-XX:-UseGCOverheadLimit -Duser.country=US -Duser.language=en -jar
/__w/1/s/flink-connectors/flink-connector-pulsar/target/surefire/surefirebooter3139517882560779643.jar
/__w/1/s/flink-connectors/flink-connector-pulsar/target/surefire
2022-02-16T13-48-34_435-jvmRun1 surefire3358354372075396323tmp
surefire_08509996975514960300tmp
Feb 16 13:49:46 [ERROR] Error occurred in starting fork, check output in log
Feb 16 13:49:46 [ERROR] Process Exit Code: 255
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
Feb 16 13:49:46 [ERROR] at
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
Feb 16 13:49:46 [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
Feb 16 13:49:46 [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[jira] [Created] (FLINK-26193) Commit generated FlinkDeployment resource into Git repo
Gyula Fora created FLINK-26193: -- Summary: Commit generated FlinkDeployment resource into Git repo Key: FLINK-26193 URL: https://issues.apache.org/jira/browse/FLINK-26193 Project: Flink Issue Type: Sub-task Components: Kubernetes Operator Reporter: Gyula Fora Currently the generated resource lands under helm/flink-operator/templates/flinkdeployments.flink.apache.org-v1.yml and it is ignored by git. We should move it under helm/flink-operator/crds/ and remove it from .gitignore. This will allow helm to install the crd if necessary (and without running the build) and also make any changes visible easily. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations! Well deserved! Cheers, Till On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser wrote: > Congratulations to all of you! > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > > On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > > > Congrats to all three of you, well deserved. > > > > Best, > > Fabian > > > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > > wrote: > > > > > > Hi all, > > > > > > I would like to formally announce a few new Flink PMC members on the > dev@ > > > list. The PMC has not done a good job of always announcing new PMC > > members > > > (and committers) recently. I'll try to keep an eye on this in the > future > > to > > > improve the situation. > > > > > > Nevertheless, I'm very happy to announce some very active community > > members > > > as new PMC members: > > > > > > - Igal Shilman, added to the PMC in October 2021 > > > - Konstantin Knauf, added to the PMC in January 2022 > > > - Yun Gao, added to the PMC in February 2022 > > > > > > Please join me in welcoming them to the Flink PMC! > > > > > > Best, > > > Robert > > >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations Feng and Zhipeng :-) Cheers, Till On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser wrote: > Congratulations Feng and Zhipeng! > > Best regards, > > Martijn > https://twitter.com/MartijnVisser82 > > > On Wed, 16 Feb 2022 at 14:32, Robert Metzger wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce two new Flink > > committers: Feng Wang and Zhipeng Zhang! > > > > Feng is one of the most active Flink evangelists in China, with plenty of > > public talks, blog posts and other evangelization activities. The PMC > wants > > to recognize and value these efforts by making Feng a committer! > > > > Zhipeng Zhang has made significant contributions to flink-ml, like most > of > > the FLIPs for our ML efforts. > > > > Please join me in welcoming them as committers! > > > > > > Best, > > Robert > > >
[jira] [Created] (FLINK-26194) Deprecate nullCheck in TableConfig
Marios Trivyzas created FLINK-26194: --- Summary: Deprecate nullCheck in TableConfig Key: FLINK-26194 URL: https://issues.apache.org/jira/browse/FLINK-26194 Project: Flink Issue Type: Technical Debt Components: Table SQL / API Reporter: Marios Trivyzas Deprecate *nullCheck* in *TableConfig.* -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26195) Kafka connector tests are mixing JUnit4 and JUnit5
Alexander Preuss created FLINK-26195: Summary: Kafka connector tests are mixing JUnit4 and JUnit5 Key: FLINK-26195 URL: https://issues.apache.org/jira/browse/FLINK-26195 Project: Flink Issue Type: Technical Debt Components: Connectors / Kafka Affects Versions: 1.15.0 Reporter: Alexander Preuss In the tests for the Kafka connector there are multiple occurrences of mixing JUnit 4 and JUnit 5. This prevents proper logging from e.g. TestLoggerExtension. There are also tests that run on JUnit 4 but use Assertions or Annotations from JUnit 5 -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations to all of you :) On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > Congratulations! Well deserved! > > Cheers, > Till > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > wrote: > > > Congratulations to all of you! > > > > Best regards, > > > > Martijn Visser > > https://twitter.com/MartijnVisser82 > > > > > > On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > > > > > Congrats to all three of you, well deserved. > > > > > > Best, > > > Fabian > > > > > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > > > wrote: > > > > > > > > Hi all, > > > > > > > > I would like to formally announce a few new Flink PMC members on the > > dev@ > > > > list. The PMC has not done a good job of always announcing new PMC > > > members > > > > (and committers) recently. I'll try to keep an eye on this in the > > future > > > to > > > > improve the situation. > > > > > > > > Nevertheless, I'm very happy to announce some very active community > > > members > > > > as new PMC members: > > > > > > > > - Igal Shilman, added to the PMC in October 2021 > > > > - Konstantin Knauf, added to the PMC in January 2022 > > > > - Yun Gao, added to the PMC in February 2022 > > > > > > > > Please join me in welcoming them to the Flink PMC! > > > > > > > > Best, > > > > Robert > > > > >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations :-) Best, Matthias On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > Congratulations Feng and Zhipeng :-) > > Cheers, > Till > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > wrote: > > > Congratulations Feng and Zhipeng! > > > > Best regards, > > > > Martijn > > https://twitter.com/MartijnVisser82 > > > > > > On Wed, 16 Feb 2022 at 14:32, Robert Metzger > wrote: > > > > > Hi everyone, > > > > > > On behalf of the PMC, I'm very happy to announce two new Flink > > > committers: Feng Wang and Zhipeng Zhang! > > > > > > Feng is one of the most active Flink evangelists in China, with plenty > of > > > public talks, blog posts and other evangelization activities. The PMC > > wants > > > to recognize and value these efforts by making Feng a committer! > > > > > > Zhipeng Zhang has made significant contributions to flink-ml, like most > > of > > > the FLIPs for our ML efforts. > > > > > > Please join me in welcoming them as committers! > > > > > > > > > Best, > > > Robert > > > > >
Re: [DISCUSS]Support the merge statement in FlinkSQL
> In the theory aspect, incremental data should be carefully considered for streaming data. In this situation, the data flow from target_table to target_table will be a loop, and the incremental data with one key will keep going through the loop. It looks very strange. This is the same concern I have here, I don't see how MERGE can work in a streaming scenario without modifying its preliminary assumptions and semantics. Even assuming we put some hard constraint on the state size, for example requiring to specify a window definition (like in interval joins), I still think that the fundamental assumption of MERGE here is a problem: the target table is both a sink and a source. And I think this is a big issue, as we cannot reasonably assume that sink and sources are available for the same table definition or that they behave similarly. Also, talking about the batch implementation, I don't understand how you would implement this: from what I see in the "*validator*" paragraph of your document, you convert the merge statement to a bunch of other sql statements, but you omit the initial join, fundamental for the semantics of MERGE. Perhaps can you provide more details about it? On another note, I think we can take inspiration from MERGE and its "event driven" semantics, in order to have something that works both for batch and streaming, say a "Flink-ified" version of MERGE. For example, something that I can think of could be: PUSH TO target_table FROM source_table ON [window TVF] [when_clause [...]] Where when_clause looks like the ones from MERGE (looking at the pgsql). This has the window TVF constraint, so the state doesn't grow indefinitely, and the source_table is effectively any select you can think of, removing the assumption that the target is both a sink and a source. This statement at the end produces a changelog stream, pushed to the output table. A statement like this could then allow you to have something similar to the MERGE, just by replacing source_table with a select performing the join. Of course this is an example, and might not make much sense, but I hope it gives you the idea. FG On Mon, Feb 14, 2022 at 4:28 AM OpenInx wrote: > I'm currently maintaining the iceberg flink modules from apache iceberg > community. > > Currently, the spark has a great integration experience with iceberg format > v2 in batch mode. In this document [1], > The merge into syntax from spark sql extensions does really help a lot when > people want to change row-level data. > > We flink currently has a good integration with iceberg format v2 in > streaming mode, I mean people can export their > change log data into an iceberg table directly by writing a few sql. > This[2] is a good material to read if anybody want to > create a simple demo. > > But I'd say in the batch scenarios, we flink sql currently lack few > critical SQL syntax (for integrating iceberg format v2 in batch mode > better): > 1. ALTER TABLE to change columns. > 2. UPDATE/DELETE sql to change the unexpected rows in a given table. > 3. MERGE INTO to merge a batch changing row set (mixed with > insert/delete/update) into the given table. > > In short, if we want to provide better integration and user experience with > iceberg v2 in batch, then I think the support of the above syntax > is very important (from iceberg perspective). > > > I think it's better to make that time investment at Calcite's > implementation before bringing this to Flink. > > I find that there are some sql syntax which are critical for flink sql > while not for other generic sql parser. Is it possible to implement our > flink sql plugin/extensions which > extends the core calcite sql. Going a step further, is it possible for us > to achieve a better abstraction of the flink sql framework, so that > downstream components can implement > their own customized sql plugins based on this sql framework. In this way, > it is possible to meet the needs of different components to add their own > sql implementation on top of > flink sql. > > [1]. https://iceberg.apache.org/docs/latest/spark-writes/#merge-into > [2]. > > https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/build-real-time-data-lake-tutorial.html > > > On Fri, Feb 11, 2022 at 4:28 PM zhou chao wrote: > > > Hi, Martijn Visser, thanks for your reply. Firstly, I am sorry for > posting > > the > > discussion twice. I sent the message to the dev mail group from an unsub- > > scribed account, but the message was not shown for a while, and I > guessed > > that > > the dev mail group would not post an email coming from an unsubscribed > > account, such that I sent it again from a subscribed account. > > > > Q: How would you see merge work for streaming data? > > I think this is an interesting topic, especially for Flink, which is > > wanting to unify > > the streaming & batch processing. Back to the merge statement, there > exist > > two inputs, target_table and source_table(query). In the mer
[jira] [Created] (FLINK-26196) error when Incremental Checkpoints by RocksDb
hjw created FLINK-26196:
---
Summary: error when Incremental Checkpoints by RocksDb
Key: FLINK-26196
URL: https://issues.apache.org/jira/browse/FLINK-26196
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Runtime / State Backends
Affects Versions: 1.13.2
Reporter: hjw
When I use Incremental Checkpoints by RocksDb , errors happen occasionally.
Fortunately，Flink job is running normally
Log:
{code:java}
java.io.IOException: Could not perform checkpoint 2804 for operator
cc-rule-keyByAndReduceStream (2/8)#1.
at
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1045)
at
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:135)
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:250)
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:61)
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:431)
at
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:61)
at
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:227)
at
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:180)
at
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:158)
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:681)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:636)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:620)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not
complete snapshot 2804 for operator cc-rule-keyByAndReduceStream (2/8)#1.
Failure reason: Checkpoint was declined.
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:264)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:169)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:371)
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointStreamOperator(SubtaskCheckpointCoordinatorImpl.java:706)
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.buildOperatorSnapshotFutures(SubtaskCheckpointCoordinatorImpl.java:627)
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:590)
at
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:312)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$8(StreamTask.java:1089)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1073)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1029)
... 19 more
Caused by: org.rocksdb.RocksDBException: while link file to
/opt/flink/log/iodir/flink-io-1c4c28bd-c5ce-4c07-9d33-81d480ec5216/job_b9a574334a212349298b39d98567b519_op_WindowOperator_306d8342cb5b2ad8b53f1be57f65bee8__2_8__uuid_c9adab75-3696-4342-9019-e8477cf0a7ca/chk-2804.tmp/000279.sst:
/opt/flink/log/iodir/flink-io-1c4c28bd-c5ce-4c07-9d33-81d480ec5216/job_b9a574334a212349298b39d98567b519_op_WindowOperator_306d8342cb5b2ad8b53f1be57f65bee8__2_8__uuid_c9adab75-3696-4342-9019-e8477cf0a7ca/db/000279.sst:
File exists
at org.rocksdb.Checkpoint.cr
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations! Regards, Roman On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl wrote: > > Congratulations to all of you :) > > On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > > > Congratulations! Well deserved! > > > > Cheers, > > Till > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > > wrote: > > > > > Congratulations to all of you! > > > > > > Best regards, > > > > > > Martijn Visser > > > https://twitter.com/MartijnVisser82 > > > > > > > > > On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > > > > > > > Congrats to all three of you, well deserved. > > > > > > > > Best, > > > > Fabian > > > > > > > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > I would like to formally announce a few new Flink PMC members on the > > > dev@ > > > > > list. The PMC has not done a good job of always announcing new PMC > > > > members > > > > > (and committers) recently. I'll try to keep an eye on this in the > > > future > > > > to > > > > > improve the situation. > > > > > > > > > > Nevertheless, I'm very happy to announce some very active community > > > > members > > > > > as new PMC members: > > > > > > > > > > - Igal Shilman, added to the PMC in October 2021 > > > > > - Konstantin Knauf, added to the PMC in January 2022 > > > > > - Yun Gao, added to the PMC in February 2022 > > > > > > > > > > Please join me in welcoming them to the Flink PMC! > > > > > > > > > > Best, > > > > > Robert > > > > > > >
[ANNOUNCE] Flink 1.15 Feature Freeze
Hi everyone, The deadline for merging new features for Flink 1.15 has passed. * From now on, only bug-fixes and documentation fixes / improvements are allowed to be merged into the master branch. * New features merged after this point can be reverted. If you need an exception to this rule, please open a discussion on dev@ list and reach out to us. We plan to wait for the master branch to get a bit more stabilized before cutting the "release-1.15" branch, in order to reduce the overhead of having to manage two branches. That also means potentially delaying merging new features for Flink 1.16 into the master branch. If you are blocked on this, please let us know and we can come up with a compromise for the branch cutting time. What you can do to help with the release testing phase: * The first release testing sync will be on *February 22, 9am CET*. Everyone is welcome to join. The link can be found on the release wiki page [1]. * Please prepare for the release testing by creating Jira tickets for documentation and testing tasks for the new features. Tickets should be opened with Priority Blocker, FixVersion 1.15.0 and Label release-testing (testing tasks only). * There are currently 92 test-stability issues affecting the 1.15.0 release [2]. It is greatly appreciated if you can help address some of them. Cheers, Joe, Yun & Till [1] https://cwiki.apache.org/confluence/display/FLINK/1.15+Release [2] https://issues.apache.org/jira/issues/?filter=12351363
[jira] [Created] (FLINK-26197) Allow playground egress to keep connection open and push messages
Till Rohrmann created FLINK-26197: - Summary: Allow playground egress to keep connection open and push messages Key: FLINK-26197 URL: https://issues.apache.org/jira/browse/FLINK-26197 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: statefun-3.2.0, statefun-3.3.0 Reporter: Till Rohrmann In order to improve the getting started experience it would be nice if the playground egress can keep connections to clients open and push new messages eagerly. Moreover, we could keep all messages stored in memory to allow serving old messages. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26198) ArchitectureTest fails on AZP (table.api.StatementSet)
Roman Khachatryan created FLINK-26198:
-
Summary: ArchitectureTest fails on AZP (table.api.StatementSet)
Key: FLINK-26198
URL: https://issues.apache.org/jira/browse/FLINK-26198
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Roman Khachatryan
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31681&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=26849
{code}
[INFO] Running org.apache.flink.architecture.rules.ApiAnnotationRules
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 48.583
s <<< FAILURE! - in org.apache.flink.architecture.rules.ApiAnnotationRules
[ERROR]
ApiAnnotationRules.PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES
Time elapsed: 0.282 s <<< FAILURE!
java.lang.AssertionError:
Architecture Violation [Priority: MEDIUM] - Rule 'Return and argument types of
methods annotated with @PublicEvolving must be annotated with
@Public(Evolving).' was violated (1 times):
org.apache.flink.table.api.StatementSet.compilePlan(): Returned leaf type
org.apache.flink.table.api.CompiledPlan does not satisfy: reside outside of
package 'org.apache.flink..' or annotated with @Public or annotated with
@PublicEvolving or annotated with @Deprecated
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26199) StatementSet.compilePlan doesn't pass the archiunit tests
David Morávek created FLINK-26199:
-
Summary: StatementSet.compilePlan doesn't pass the archiunit tests
Key: FLINK-26199
URL: https://issues.apache.org/jira/browse/FLINK-26199
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: David Morávek
Fix For: 1.15.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31678&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=26852
{code}
Feb 16 18:10:30 Architecture Violation [Priority: MEDIUM] - Rule 'Return and
argument types of methods annotated with @PublicEvolving must be annotated with
@Public(Evolving).' was violated (1 times):
Feb 16 18:10:30 org.apache.flink.table.api.StatementSet.compilePlan(): Returned
leaf type org.apache.flink.table.api.CompiledPlan does not satisfy: reside
outside of package 'org.apache.flink..' or annotated with @Public or annotated
with @PublicEvolving or annotated with @Deprecated
Feb 16 18:10:30 at
com.tngtech.archunit.lang.ArchRule$Assertions.assertNoViolation(ArchRule.java:94)
Feb 16 18:10:30 at
com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:82)
Feb 16 18:10:30 at
com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:96)
Feb 16 18:10:30 at
com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:159)
Feb 16 18:10:30 at
com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:142)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
Feb 16 18:10:30 at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
Feb 16 18:10:30 at java.util.ArrayList.forEach(ArrayList.java:1259)
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
question about StatefunContext in golang Statefun SDK
When stateful functions are invoked, they are passed an instance of statefun.Context, which wraps the context.Context received by the HTTP request. Is there any way to customize that context.Context to, say, hold custom values, using ctx.WithValue()? I don't see a way but I wanted to ask. If not, would you be interested in a PR to add this functionality? A simple way might be to add a property to StatefulFunctionSpec, say: TransformContext func(ctx context.Context) context.Context ... that, if supplied, would be called to create a customized context that would be used downstream? Thanks.
Re: question about StatefunContext in golang Statefun SDK
Sorry, that should be context.WithValue, not ctx.WithValue ... On Wed, Feb 16, 2022 at 5:35 PM Galen Warren wrote: > When stateful functions are invoked, they are passed an instance of > statefun.Context, which wraps the context.Context received by the HTTP > request. Is there any way to customize that context.Context to, say, hold > custom values, using ctx.WithValue()? I don't see a way but I wanted to ask. > > If not, would you be interested in a PR to add this functionality? A > simple way might be to add a property to StatefulFunctionSpec, say: > > TransformContext func(ctx context.Context) context.Context > > ... that, if supplied, would be called to create a customized context that > would be used downstream? > > Thanks. >
Re: question about StatefunContext in golang Statefun SDK
Hi Galen,
No, that is not currently supported, the current idiomatic way would be to
pass those values to the struct implementing the Statefun interface.
type MyFunc struct { someRuntimeInfo string } func (m *MyFunc) Invoke(ctx
statefun.Context, message statefun.Message) error { } func main() { builder
:= statefun.StatefulFunctionsBuilder()
f := MyFunc { someRuntimeInfo: "runtime-provided" } builder.WithSpec
(statefun.StatefulFunctionSpec{ FunctionType: statefun.TypeNameFrom(
"example/my-func"), Function: f })
http.Handle("/statefun", builder.AsHandler())
_ = http.ListenAndServe(":8000", nil) }
Would this work for you? Or what is the context (pun intended) you are
looking for?
Seth
On Wed, Feb 16, 2022 at 4:35 PM Galen Warren
wrote:
> When stateful functions are invoked, they are passed an instance of
> statefun.Context, which wraps the context.Context received by the HTTP
> request. Is there any way to customize that context.Context to, say, hold
> custom values, using ctx.WithValue()? I don't see a way but I wanted to
> ask.
>
> If not, would you be interested in a PR to add this functionality? A simple
> way might be to add a property to StatefulFunctionSpec, say:
>
> TransformContext func(ctx context.Context) context.Context
>
> ... that, if supplied, would be called to create a customized context that
> would be used downstream?
>
> Thanks.
>
Discussion: Kafka Consumer and client.id.prefix usage
Hey everyone! I'm currently investigating the way Flink configures client.id values for the Kafka consumer and I'd appreciate some help. *Old FlinkKafkaConsumer* It doesn't look like client.id receives any special treatment when using deprecated FlinkKafkaConsumer: if client.id is not specified, the default Kafka consumer behaviour is used, which relies on the provided group.id and a sequence number (based on at AtomicInteger), e.g. in Kafka 2.8: https://github.com/apache/kafka/blob/2.8/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java#L587-L599 This approach allows to: - Fully control the way client.id is configured, if needed - Fallback to the default Kafka consumer logic, which guarantees unique client.id values in the same JVM, even if we have multiple sources with the same group.id *New KafkaSource* Now, the new KafkaSource introduces a new client.id.prefix option. If this option is not specified, the group.id value is used instead. If this option is specific, the value is still overridden by Flink (basically ignored). When Flink overrides client.id it uses the client.id.prefix and a subtask id. So the final client.id value may not be unique if the same group id is used for multiple sources (which is a normal practice). This can result in many "javax.management.InstanceAlreadyExistsException" exceptions when the underlying Kafka Consumer tries to register its own metrics with JMX. Question: why does Flink need to override client.id and use a subtask id as a part of it? This forces users to specify a different group.id / client.id.prefix per Kafka source, which can explode the number of generated metrics. I'd love to learn about the reasons that led to this decision. In my opinion, Flink still should allow users to pass the client.id they choose and I'm thinking if I should open a PR for that. Thank you!
[jira] [Created] (FLINK-26200) Redirect statefun directly to docs
Seth Wiesman created FLINK-26200: Summary: Redirect statefun directly to docs Key: FLINK-26200 URL: https://issues.apache.org/jira/browse/FLINK-26200 Project: Flink Issue Type: Improvement Components: Project Website Reporter: Seth Wiesman Assignee: Seth Wiesman -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congrats to all of you! Best, Paul Lam > 2022年2月17日 01:24，Roman Khachatryan 写道： > > Congratulations! > > Regards, > Roman > > On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl wrote: >> >> Congratulations to all of you :) >> >> On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: >> >>> Congratulations! Well deserved! >>> >>> Cheers, >>> Till >>> >>> On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser >>> wrote: >>> Congratulations to all of you! Best regards, Martijn Visser https://twitter.com/MartijnVisser82 On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > Congrats to all three of you, well deserved. > > Best, > Fabian > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > wrote: >> >> Hi all, >> >> I would like to formally announce a few new Flink PMC members on the dev@ >> list. The PMC has not done a good job of always announcing new PMC > members >> (and committers) recently. I'll try to keep an eye on this in the future > to >> improve the situation. >> >> Nevertheless, I'm very happy to announce some very active community > members >> as new PMC members: >> >> - Igal Shilman, added to the PMC in October 2021 >> - Konstantin Knauf, added to the PMC in January 2022 >> - Yun Gao, added to the PMC in February 2022 >> >> Please join me in welcoming them to the Flink PMC! >> >> Best, >> Robert >
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations! Well deserved. Best, Yangze Guo On Thu, Feb 17, 2022 at 10:15 AM Paul Lam wrote: > > Congrats to all of you! > > Best, > Paul Lam > > > 2022年2月17日 01:24，Roman Khachatryan 写道： > > > > Congratulations! > > > > Regards, > > Roman > > > > On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl > > wrote: > >> > >> Congratulations to all of you :) > >> > >> On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > >> > >>> Congratulations! Well deserved! > >>> > >>> Cheers, > >>> Till > >>> > >>> On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > >>> wrote: > >>> > Congratulations to all of you! > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > > On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > > > Congrats to all three of you, well deserved. > > > > Best, > > Fabian > > > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > > wrote: > >> > >> Hi all, > >> > >> I would like to formally announce a few new Flink PMC members on the > dev@ > >> list. The PMC has not done a good job of always announcing new PMC > > members > >> (and committers) recently. I'll try to keep an eye on this in the > future > > to > >> improve the situation. > >> > >> Nevertheless, I'm very happy to announce some very active community > > members > >> as new PMC members: > >> > >> - Igal Shilman, added to the PMC in October 2021 > >> - Konstantin Knauf, added to the PMC in January 2022 > >> - Yun Gao, added to the PMC in February 2022 > >> > >> Please join me in welcoming them to the Flink PMC! > >> > >> Best, > >> Robert > > > >
[jira] [Created] (FLINK-26201) FileStoreScanTest is not stable
Jane Chan created FLINK-26201:
-
Summary: FileStoreScanTest is not stable
Key: FLINK-26201
URL: https://issues.apache.org/jira/browse/FLINK-26201
Project: Flink
Issue Type: Bug
Components: Table Store
Affects Versions: table-store-0.1.0
Reporter: Jane Chan
FileStoreScanTest#testWithSnapshot and FileStoreScanTest#testWithManifestList
are randomly failed with the following stacktrace.
h3. How to reproduce
You can reproduce this issue either by `mvn clean package/install` or run the
individual test in IDE.
h3. Details
h6. FileStoreScanTest#testWithSnapshot
{code:java}
java.lang.IllegalStateException: Trying to add file
{org.apache.flink.table.data.binary.BinaryRowData@ddd21057, 5, 0,
sst-b60934a5-eb23-4e2d-9b07-155d2ef29e15-0} which is already added. Manifest
might be corrupted. at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
at
org.apache.flink.table.store.file.operation.FileStoreScanImpl.scan(FileStoreScanImpl.java:184)
at
org.apache.flink.table.store.file.operation.FileStoreScanImpl.plan(FileStoreScanImpl.java:145)
at
org.apache.flink.table.store.file.operation.FileStoreWriteImpl.createWriter(FileStoreWriteImpl.java:59)
at
org.apache.flink.table.store.file.operation.OperationTestUtils.lambda$writeAndCommitData$1(OperationTestUtils.java:174)
at java.base/java.util.HashMap.compute(HashMap.java:1228)
at
org.apache.flink.table.store.file.operation.OperationTestUtils.writeAndCommitData(OperationTestUtils.java:169)
at
org.apache.flink.table.store.file.operation.FileStoreScanTest.writeData(FileStoreScanTest.java:202)
at
org.apache.flink.table.store.file.operation.FileStoreScanTest.testWithSnapshot(FileStoreScanTest.java:135)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector
[jira] [Created] (FLINK-26202) Support Table API in Pulsar Source Connector
Yufei Zhang created FLINK-26202: --- Summary: Support Table API in Pulsar Source Connector Key: FLINK-26202 URL: https://issues.apache.org/jira/browse/FLINK-26202 Project: Flink Issue Type: New Feature Components: Connectors / Pulsar Affects Versions: 1.15.0 Reporter: Yufei Zhang Currently pulsar connector only supports DataStream API, we plan to support Table API. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations! Well deserved! Best, Yangze Guo On Thu, Feb 17, 2022 at 12:22 AM Matthias Pohl wrote: > > Congratulations :-) > > Best, Matthias > > On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > > > Congratulations Feng and Zhipeng :-) > > > > Cheers, > > Till > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > > wrote: > > > > > Congratulations Feng and Zhipeng! > > > > > > Best regards, > > > > > > Martijn > > > https://twitter.com/MartijnVisser82 > > > > > > > > > On Wed, 16 Feb 2022 at 14:32, Robert Metzger > > wrote: > > > > > > > Hi everyone, > > > > > > > > On behalf of the PMC, I'm very happy to announce two new Flink > > > > committers: Feng Wang and Zhipeng Zhang! > > > > > > > > Feng is one of the most active Flink evangelists in China, with plenty > > of > > > > public talks, blog posts and other evangelization activities. The PMC > > > wants > > > > to recognize and value these efforts by making Feng a committer! > > > > > > > > Zhipeng Zhang has made significant contributions to flink-ml, like most > > > of > > > > the FLIPs for our ML efforts. > > > > > > > > Please join me in welcoming them as committers! > > > > > > > > > > > > Best, > > > > Robert > > > > > > >
[jira] [Created] (FLINK-26203) Support Table API in Pulsar Sink Connector
Yufei Zhang created FLINK-26203: --- Summary: Support Table API in Pulsar Sink Connector Key: FLINK-26203 URL: https://issues.apache.org/jira/browse/FLINK-26203 Project: Flink Issue Type: New Feature Components: Connectors / Pulsar Reporter: Yufei Zhang Currently Pulsar connector only supports DataStream API. We plan to support Table API as well. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations to Feng and Zhipeng! Best, Yun Tang From: Yangze Guo Sent: Thursday, February 17, 2022 10:58 To: dev Subject: Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang Congratulations! Well deserved! Best, Yangze Guo On Thu, Feb 17, 2022 at 12:22 AM Matthias Pohl wrote: > > Congratulations :-) > > Best, Matthias > > On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > > > Congratulations Feng and Zhipeng :-) > > > > Cheers, > > Till > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > > wrote: > > > > > Congratulations Feng and Zhipeng! > > > > > > Best regards, > > > > > > Martijn > > > https://twitter.com/MartijnVisser82 > > > > > > > > > On Wed, 16 Feb 2022 at 14:32, Robert Metzger > > wrote: > > > > > > > Hi everyone, > > > > > > > > On behalf of the PMC, I'm very happy to announce two new Flink > > > > committers: Feng Wang and Zhipeng Zhang! > > > > > > > > Feng is one of the most active Flink evangelists in China, with plenty > > of > > > > public talks, blog posts and other evangelization activities. The PMC > > > wants > > > > to recognize and value these efforts by making Feng a committer! > > > > > > > > Zhipeng Zhang has made significant contributions to flink-ml, like most > > > of > > > > the FLIPs for our ML efforts. > > > > > > > > Please join me in welcoming them as committers! > > > > > > > > > > > > Best, > > > > Robert > > > > > > >
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations to all of you! Best, Yun Tang From: Yangze Guo Sent: Thursday, February 17, 2022 10:44 To: dev Cc: ro...@apache.org Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao Congratulations! Well deserved. Best, Yangze Guo On Thu, Feb 17, 2022 at 10:15 AM Paul Lam wrote: > > Congrats to all of you! > > Best, > Paul Lam > > > 2022年2月17日 01:24，Roman Khachatryan 写道： > > > > Congratulations! > > > > Regards, > > Roman > > > > On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl > > wrote: > >> > >> Congratulations to all of you :) > >> > >> On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > >> > >>> Congratulations! Well deserved! > >>> > >>> Cheers, > >>> Till > >>> > >>> On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > >>> wrote: > >>> > Congratulations to all of you! > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > > On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > > > Congrats to all three of you, well deserved. > > > > Best, > > Fabian > > > > On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger > > wrote: > >> > >> Hi all, > >> > >> I would like to formally announce a few new Flink PMC members on the > dev@ > >> list. The PMC has not done a good job of always announcing new PMC > > members > >> (and committers) recently. I'll try to keep an eye on this in the > future > > to > >> improve the situation. > >> > >> Nevertheless, I'm very happy to announce some very active community > > members > >> as new PMC members: > >> > >> - Igal Shilman, added to the PMC in October 2021 > >> - Konstantin Knauf, added to the PMC in January 2022 > >> - Yun Gao, added to the PMC in February 2022 > >> > >> Please join me in welcoming them to the Flink PMC! > >> > >> Best, > >> Robert > > > >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congrats! Well deserved! Best, Paul Lam > 2022年2月16日 21:30，Robert Metzger 写道： > > Hi everyone, > > On behalf of the PMC, I'm very happy to announce two new Flink > committers: Feng Wang and Zhipeng Zhang! > > Feng is one of the most active Flink evangelists in China, with plenty of > public talks, blog posts and other evangelization activities. The PMC wants > to recognize and value these efforts by making Feng a committer! > > Zhipeng Zhang has made significant contributions to flink-ml, like most of > the FLIPs for our ML efforts. > > Please join me in welcoming them as committers! > > > Best, > Robert
[jira] [Created] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time is gre
Bo Huang created FLINK-26204: Summary: set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time is greater than the value of the next step window time Key: FLINK-26204 URL: https://issues.apache.org/jira/browse/FLINK-26204 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.14.3, 1.14.0 Reporter: Bo Huang Attachments: TestApp.java, test.png set table.optimizer.distinct-agg.split.enabled to true using Window TVF CUMULATE to count users It will appear that the value of the current step window time is greater than the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26205) Support Online Model Save in FlinkML
weibo zhao created FLINK-26205: -- Summary: Support Online Model Save in FlinkML Key: FLINK-26205 URL: https://issues.apache.org/jira/browse/FLINK-26205 Project: Flink Issue Type: New Feature Components: Library / Machine Learning Reporter: weibo zhao Support Online Model Save in FlinkML. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations ! Best, Leonard > 2022年2月17日 上午11:09，Yun Tang 写道： > > Congratulations to all of you! > > Best, > Yun Tang > > From: Yangze Guo > Sent: Thursday, February 17, 2022 10:44 > To: dev > Cc: ro...@apache.org > Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf > and Yun Gao > > Congratulations! Well deserved. > > Best, > Yangze Guo > > On Thu, Feb 17, 2022 at 10:15 AM Paul Lam wrote: >> >> Congrats to all of you! >> >> Best, >> Paul Lam >> >>> 2022年2月17日 01:24，Roman Khachatryan 写道： >>> >>> Congratulations! >>> >>> Regards, >>> Roman >>> >>> On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl >>> wrote: Congratulations to all of you :) On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann wrote: > Congratulations! Well deserved! > > Cheers, > Till > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser > wrote: > >> Congratulations to all of you! >> >> Best regards, >> >> Martijn Visser >> https://twitter.com/MartijnVisser82 >> >> >> On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: >> >>> Congrats to all three of you, well deserved. >>> >>> Best, >>> Fabian >>> >>> On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger >>> wrote: Hi all, I would like to formally announce a few new Flink PMC members on the >> dev@ list. The PMC has not done a good job of always announcing new PMC >>> members (and committers) recently. I'll try to keep an eye on this in the >> future >>> to improve the situation. Nevertheless, I'm very happy to announce some very active community >>> members as new PMC members: - Igal Shilman, added to the PMC in October 2021 - Konstantin Knauf, added to the PMC in January 2022 - Yun Gao, added to the PMC in February 2022 Please join me in welcoming them to the Flink PMC! Best, Robert >>> >> >>
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations ! Best, Leonard > 2022年2月17日 上午11:17，Paul Lam 写道： > > Congrats! Well deserved! > > Best, > Paul Lam > >> 2022年2月16日 21:30，Robert Metzger 写道： >> >> Hi everyone, >> >> On behalf of the PMC, I'm very happy to announce two new Flink >> committers: Feng Wang and Zhipeng Zhang! >> >> Feng is one of the most active Flink evangelists in China, with plenty of >> public talks, blog posts and other evangelization activities. The PMC wants >> to recognize and value these efforts by making Feng a committer! >> >> Zhipeng Zhang has made significant contributions to flink-ml, like most of >> the FLIPs for our ML efforts. >> >> Please join me in welcoming them as committers! >> >> >> Best, >> Robert >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations :-D On Thu, Feb 17, 2022 at 12:15 PM Leonard Xu wrote: > Congratulations ! > > > Best, > Leonard > > > 2022年2月17日 上午11:17，Paul Lam 写道： > > > > Congrats! Well deserved! > > > > Best, > > Paul Lam > > > >> 2022年2月16日 21:30，Robert Metzger 写道： > >> > >> Hi everyone, > >> > >> On behalf of the PMC, I'm very happy to announce two new Flink > >> committers: Feng Wang and Zhipeng Zhang! > >> > >> Feng is one of the most active Flink evangelists in China, with plenty > of > >> public talks, blog posts and other evangelization activities. The PMC > wants > >> to recognize and value these efforts by making Feng a committer! > >> > >> Zhipeng Zhang has made significant contributions to flink-ml, like most > of > >> the FLIPs for our ML efforts. > >> > >> Please join me in welcoming them as committers! > >> > >> > >> Best, > >> Robert > > > >
[jira] [Created] (FLINK-26206) refactor web ui's service layer and app config
Junhan Yang created FLINK-26206: --- Summary: refactor web ui's service layer and app config Key: FLINK-26206 URL: https://issues.apache.org/jira/browse/FLINK-26206 Project: Flink Issue Type: Sub-task Components: Runtime / Web Frontend Reporter: Junhan Yang Add abstractions for api service layer and refactor app.config.ts to a config service. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations to all of you! Best, Lijie Leonard Xu 于2022年2月17日周四 12:13写道： > Congratulations ! > > Best, > Leonard > > > 2022年2月17日 上午11:09，Yun Tang 写道： > > > > Congratulations to all of you! > > > > Best, > > Yun Tang > > > > From: Yangze Guo > > Sent: Thursday, February 17, 2022 10:44 > > To: dev > > Cc: ro...@apache.org > > Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin > Knauf and Yun Gao > > > > Congratulations! Well deserved. > > > > Best, > > Yangze Guo > > > > On Thu, Feb 17, 2022 at 10:15 AM Paul Lam wrote: > >> > >> Congrats to all of you! > >> > >> Best, > >> Paul Lam > >> > >>> 2022年2月17日 01:24，Roman Khachatryan 写道： > >>> > >>> Congratulations! > >>> > >>> Regards, > >>> Roman > >>> > >>> On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl > wrote: > > Congratulations to all of you :) > > On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann > wrote: > > > Congratulations! Well deserved! > > > > Cheers, > > Till > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser < > mart...@ververica.com> > > wrote: > > > >> Congratulations to all of you! > >> > >> Best regards, > >> > >> Martijn Visser > >> https://twitter.com/MartijnVisser82 > >> > >> > >> On Wed, 16 Feb 2022 at 14:29, Fabian Paul wrote: > >> > >>> Congrats to all three of you, well deserved. > >>> > >>> Best, > >>> Fabian > >>> > >>> On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger < > rmetz...@apache.org> > >>> wrote: > > Hi all, > > I would like to formally announce a few new Flink PMC members on > the > >> dev@ > list. The PMC has not done a good job of always announcing new PMC > >>> members > (and committers) recently. I'll try to keep an eye on this in the > >> future > >>> to > improve the situation. > > Nevertheless, I'm very happy to announce some very active > community > >>> members > as new PMC members: > > - Igal Shilman, added to the PMC in October 2021 > - Konstantin Knauf, added to the PMC in January 2022 > - Yun Gao, added to the PMC in February 2022 > > Please join me in welcoming them to the Flink PMC! > > Best, > Robert > >>> > >> > >> > >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations to all of you! Best, Lijie Zakelly Lan 于2022年2月17日周四 12:21写道： > Congratulations :-D > > On Thu, Feb 17, 2022 at 12:15 PM Leonard Xu wrote: > > > Congratulations ! > > > > > > Best, > > Leonard > > > > > 2022年2月17日 上午11:17，Paul Lam 写道： > > > > > > Congrats! Well deserved! > > > > > > Best, > > > Paul Lam > > > > > >> 2022年2月16日 21:30，Robert Metzger 写道： > > >> > > >> Hi everyone, > > >> > > >> On behalf of the PMC, I'm very happy to announce two new Flink > > >> committers: Feng Wang and Zhipeng Zhang! > > >> > > >> Feng is one of the most active Flink evangelists in China, with plenty > > of > > >> public talks, blog posts and other evangelization activities. The PMC > > wants > > >> to recognize and value these efforts by making Feng a committer! > > >> > > >> Zhipeng Zhang has made significant contributions to flink-ml, like > most > > of > > >> the FLIPs for our ML efforts. > > >> > > >> Please join me in welcoming them as committers! > > >> > > >> > > >> Best, > > >> Robert > > > > > > > >
[jira] [Created] (FLINK-26207) refactor web ui's components to depend on module's injected config object
Junhan Yang created FLINK-26207: --- Summary: refactor web ui's components to depend on module's injected config object Key: FLINK-26207 URL: https://issues.apache.org/jira/browse/FLINK-26207 Project: Flink Issue Type: Sub-task Components: Runtime / Web Frontend Reporter: Junhan Yang By making components depend on injected config object, one is able to config corresponding module's view, styles and/or business logic. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26208) Introduce implementation of ManagedTableFactory
Jane Chan created FLINK-26208: - Summary: Introduce implementation of ManagedTableFactory Key: FLINK-26208 URL: https://issues.apache.org/jira/browse/FLINK-26208 Project: Flink Issue Type: Sub-task Components: Table Store Affects Versions: table-store-0.1.0 Reporter: Jane Chan Introduce impl for `org.apache.flink.table.factories.ManagedTableFactory`(#enrichOptions, #onCreateTable, #onDropTable and #onCompactTable) to support interaction with Flink's TableEnv -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26209) Possibility of Command Injection attack
Iman Sharafaldin created FLINK-26209: Summary: Possibility of Command Injection attack Key: FLINK-26209 URL: https://issues.apache.org/jira/browse/FLINK-26209 Project: Flink Issue Type: Bug Components: Library / Machine Learning Reporter: Iman Sharafaldin As you can see in line 134 command line is built using string concatenation. An attacker who has control over args can execute malicious commands. |final String cmd = discoveryScript.getAbsolutePath() + " " + gpuAmount + " " + args;| || [https://github.com/apache/flink/blob/0d29b23f892714e4936b8af2f896e3040ddc9e89/flink-external-resources/flink-external-resource-gpu/src/main/java/org/apache/flink/externalresource/gpu/GPUDriver.java#L134] Reference: https://owasp.org/www-community/attacks/Command_Injection -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: Re: [DISCUSS] Future of Per-Job Mode
I am not sure whether the sql script could also be submitted like python. We will need a sql-runner jar, which plays as the user jar and has the sql script as the argument. ./bin/flink run-application \ --target kubernetes-application \ -Dkubernetes.cluster-id= \ -Dkubernetes.container.image= \ --sqlFiles /opt/flink/examples/sql/word_count.sql Best, Yang Jark Wu 于2022年2月16日周三 20:00写道： > I think this mode is still limited and maybe not easy to extend. > Could the application mode provide an interface to execute? > So that clients can implement the interface and pass arbitrary parameters > (e.g. SQL scripts) ? > > Best, > Jark > > On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf wrote: > > > Hi Jark, > > > > I think you are raising a very good point. I think we need an application > > mode for SQL that would work along the lines of executing a SQL script > > (incl. init scripts) located in a particular directory in the Docker > Image. > > Details to be discussed. > > > > Do you think Zeppelin/SQL CLI could work with such a mode for > > non-interactive queries (interactive queries would use a session > cluster)? > > > > Best, > > > > Konstantin > > > > > > On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > > > > > Hi David, > > > > > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > > > jobs. So the session cluster is not a fit mode. > > > > > > Best, > > > Jark > > > > > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > > > > > Hi Jark, can you please elaborate about the current need of the > per-job > > > > mode for interactive clients (eg. Zeppelin that you've mentioned)? > > Aren't > > > > these a natural fit for the session cluster? > > > > > > > > D. > > > > > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > > > > > Hi Konstantin, > > > > > > > > > > I'm not very familiar with the implementation of per-job mode and > > > > > application mode. > > > > > But is there any instruction for users abou how to migrate > > > platforms/jobs > > > > > to application mode? > > > > > IIUC, the biggest difference between the two modes is where the > > main() > > > > > method is executed. > > > > > However, SQL jobs are not jar applications and don't have the > main() > > > > > method. > > > > > For example, SQL CLI submits SQL jobs by invoking > > > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > > > How SQL Client and SQL platforms (e.g. Zeppelin) support > application > > > > mode? > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > > > these limitations of per-job mode on YARN. It seems that there > is a > > > > > general > > > > > > agreement to deprecate per-job mode and to drop it once the > > > limitations > > > > > > around YARN are resolved. I've started a corresponding vote in > [1]. > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > > > Thank you for the clarification. In general I think we will > have > > > time > > > > > to > > > > > > > experiment with this until it will be removed totally and > migrate > > > our > > > > > > > solution to use application mode. > > > > > > > > > > > > > > Regards, > > > > > > > F > > > > > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I remember the application mode was initially named "cluster > > > mode". > > > > > As > > > > > > a > > > > > > > > contrast, the per-job mode is the "client mode". > > > > > > > > So I believe application mode should cover all the > > > functionalities > > > > of > > > > > > > > per-job except where we are running the user main code. > > > > > > > > In the containerized or the Kubernetes world, the application > > > mode > > > > is > > > > > > > more > > > > > > > > native and easy to use since all the Flink and user > > > > > > > > jars are bundled in the image. I am also in favor of > > deprecating > > > > and > > > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > > > IIRC, the YARN application mode could ship user jars and > > > > dependencies > > > > > > via > > > > > > > > "yarn.ship-files" config option. The only > > > > > > > > limitation is that we could not ship and load the user > > > dependencies > > > > > > with > > > > > > > > user classloader, not the parent classloader. > > > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > > > directory > > > > > > > > automatically. > > > >
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations to all of you! Best, Yang Lijie Wang 于2022年2月17日周四 14:20写道： > Congratulations to all of you! > > Best, > Lijie > > Leonard Xu 于2022年2月17日周四 12:13写道： > > > Congratulations ! > > > > Best, > > Leonard > > > > > 2022年2月17日 上午11:09，Yun Tang 写道： > > > > > > Congratulations to all of you! > > > > > > Best, > > > Yun Tang > > > > > > From: Yangze Guo > > > Sent: Thursday, February 17, 2022 10:44 > > > To: dev > > > Cc: ro...@apache.org > > > Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin > > Knauf and Yun Gao > > > > > > Congratulations! Well deserved. > > > > > > Best, > > > Yangze Guo > > > > > > On Thu, Feb 17, 2022 at 10:15 AM Paul Lam > wrote: > > >> > > >> Congrats to all of you! > > >> > > >> Best, > > >> Paul Lam > > >> > > >>> 2022年2月17日 01:24，Roman Khachatryan 写道： > > >>> > > >>> Congratulations! > > >>> > > >>> Regards, > > >>> Roman > > >>> > > >>> On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl < > matth...@ververica.com> > > wrote: > > > > Congratulations to all of you :) > > > > On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann > > > wrote: > > > > > Congratulations! Well deserved! > > > > > > Cheers, > > > Till > > > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser < > > mart...@ververica.com> > > > wrote: > > > > > >> Congratulations to all of you! > > >> > > >> Best regards, > > >> > > >> Martijn Visser > > >> https://twitter.com/MartijnVisser82 > > >> > > >> > > >> On Wed, 16 Feb 2022 at 14:29, Fabian Paul > wrote: > > >> > > >>> Congrats to all three of you, well deserved. > > >>> > > >>> Best, > > >>> Fabian > > >>> > > >>> On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger < > > rmetz...@apache.org> > > >>> wrote: > > > > Hi all, > > > > I would like to formally announce a few new Flink PMC members on > > the > > >> dev@ > > list. The PMC has not done a good job of always announcing new > PMC > > >>> members > > (and committers) recently. I'll try to keep an eye on this in > the > > >> future > > >>> to > > improve the situation. > > > > Nevertheless, I'm very happy to announce some very active > > community > > >>> members > > as new PMC members: > > > > - Igal Shilman, added to the PMC in October 2021 > > - Konstantin Knauf, added to the PMC in January 2022 > > - Yun Gao, added to the PMC in February 2022 > > > > Please join me in welcoming them to the Flink PMC! > > > > Best, > > Robert > > >>> > > >> > > >> > > > > >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations! Best, Yang Lijie Wang 于2022年2月17日周四 14:21写道： > Congratulations to all of you! > > Best, > Lijie > > Zakelly Lan 于2022年2月17日周四 12:21写道： > > > Congratulations :-D > > > > On Thu, Feb 17, 2022 at 12:15 PM Leonard Xu wrote: > > > > > Congratulations ! > > > > > > > > > Best, > > > Leonard > > > > > > > 2022年2月17日 上午11:17，Paul Lam 写道： > > > > > > > > Congrats! Well deserved! > > > > > > > > Best, > > > > Paul Lam > > > > > > > >> 2022年2月16日 21:30，Robert Metzger 写道： > > > >> > > > >> Hi everyone, > > > >> > > > >> On behalf of the PMC, I'm very happy to announce two new Flink > > > >> committers: Feng Wang and Zhipeng Zhang! > > > >> > > > >> Feng is one of the most active Flink evangelists in China, with > plenty > > > of > > > >> public talks, blog posts and other evangelization activities. The > PMC > > > wants > > > >> to recognize and value these efforts by making Feng a committer! > > > >> > > > >> Zhipeng Zhang has made significant contributions to flink-ml, like > > most > > > of > > > >> the FLIPs for our ML efforts. > > > >> > > > >> Please join me in welcoming them as committers! > > > >> > > > >> > > > >> Best, > > > >> Robert > > > > > > > > > > > > >
Re: [ANNOUNCE] New Apache Flink Committers: Feng Wang, Zhipeng Zhang
Congratulations to Feng and Zhipeng! Best, Guowei On Thu, Feb 17, 2022 at 3:30 PM Yang Wang wrote: > Congratulations! > > Best, > Yang > > Lijie Wang 于2022年2月17日周四 14:21写道： > > > Congratulations to all of you! > > > > Best, > > Lijie > > > > Zakelly Lan 于2022年2月17日周四 12:21写道： > > > > > Congratulations :-D > > > > > > On Thu, Feb 17, 2022 at 12:15 PM Leonard Xu wrote: > > > > > > > Congratulations ! > > > > > > > > > > > > Best, > > > > Leonard > > > > > > > > > 2022年2月17日 上午11:17，Paul Lam 写道： > > > > > > > > > > Congrats! Well deserved! > > > > > > > > > > Best, > > > > > Paul Lam > > > > > > > > > >> 2022年2月16日 21:30，Robert Metzger 写道： > > > > >> > > > > >> Hi everyone, > > > > >> > > > > >> On behalf of the PMC, I'm very happy to announce two new Flink > > > > >> committers: Feng Wang and Zhipeng Zhang! > > > > >> > > > > >> Feng is one of the most active Flink evangelists in China, with > > plenty > > > > of > > > > >> public talks, blog posts and other evangelization activities. The > > PMC > > > > wants > > > > >> to recognize and value these efforts by making Feng a committer! > > > > >> > > > > >> Zhipeng Zhang has made significant contributions to flink-ml, like > > > most > > > > of > > > > >> the FLIPs for our ML efforts. > > > > >> > > > > >> Please join me in welcoming them as committers! > > > > >> > > > > >> > > > > >> Best, > > > > >> Robert > > > > > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao
Congratulations Best, Guowei On Thu, Feb 17, 2022 at 3:29 PM Yang Wang wrote: > Congratulations to all of you! > > Best, > Yang > > Lijie Wang 于2022年2月17日周四 14:20写道： > > > Congratulations to all of you! > > > > Best, > > Lijie > > > > Leonard Xu 于2022年2月17日周四 12:13写道： > > > > > Congratulations ! > > > > > > Best, > > > Leonard > > > > > > > 2022年2月17日 上午11:09，Yun Tang 写道： > > > > > > > > Congratulations to all of you! > > > > > > > > Best, > > > > Yun Tang > > > > > > > > From: Yangze Guo > > > > Sent: Thursday, February 17, 2022 10:44 > > > > To: dev > > > > Cc: ro...@apache.org > > > > Subject: Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, > Konstantin > > > Knauf and Yun Gao > > > > > > > > Congratulations! Well deserved. > > > > > > > > Best, > > > > Yangze Guo > > > > > > > > On Thu, Feb 17, 2022 at 10:15 AM Paul Lam > > wrote: > > > >> > > > >> Congrats to all of you! > > > >> > > > >> Best, > > > >> Paul Lam > > > >> > > > >>> 2022年2月17日 01:24，Roman Khachatryan 写道： > > > >>> > > > >>> Congratulations! > > > >>> > > > >>> Regards, > > > >>> Roman > > > >>> > > > >>> On Wed, Feb 16, 2022 at 5:22 PM Matthias Pohl < > > matth...@ververica.com> > > > wrote: > > > > > > Congratulations to all of you :) > > > > > > On Wed, Feb 16, 2022 at 4:04 PM Till Rohrmann < > trohrm...@apache.org > > > > > > wrote: > > > > > > > Congratulations! Well deserved! > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Feb 16, 2022 at 2:51 PM Martijn Visser < > > > mart...@ververica.com> > > > > wrote: > > > > > > > >> Congratulations to all of you! > > > >> > > > >> Best regards, > > > >> > > > >> Martijn Visser > > > >> https://twitter.com/MartijnVisser82 > > > >> > > > >> > > > >> On Wed, 16 Feb 2022 at 14:29, Fabian Paul > > wrote: > > > >> > > > >>> Congrats to all three of you, well deserved. > > > >>> > > > >>> Best, > > > >>> Fabian > > > >>> > > > >>> On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger < > > > rmetz...@apache.org> > > > >>> wrote: > > > > > > Hi all, > > > > > > I would like to formally announce a few new Flink PMC members > on > > > the > > > >> dev@ > > > list. The PMC has not done a good job of always announcing new > > PMC > > > >>> members > > > (and committers) recently. I'll try to keep an eye on this in > > the > > > >> future > > > >>> to > > > improve the situation. > > > > > > Nevertheless, I'm very happy to announce some very active > > > community > > > >>> members > > > as new PMC members: > > > > > > - Igal Shilman, added to the PMC in October 2021 > > > - Konstantin Knauf, added to the PMC in January 2022 > > > - Yun Gao, added to the PMC in February 2022 > > > > > > Please join me in welcoming them to the Flink PMC! > > > > > > Best, > > > Robert > > > >>> > > > >> > > > >> > > > > > > > > >
