[jira] [Created] (FLINK-26881) Stop with savepoint should pick up the targetDirectory from Flink configuration

[Yang Wang (Jira)]

Yang Wang created FLINK-26881:
-

 Summary: Stop with savepoint should pick up the targetDirectory 
from Flink configuration
 Key: FLINK-26881
 URL: https://issues.apache.org/jira/browse/FLINK-26881
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Yang Wang
 Fix For: kubernetes-operator-0.1.0


Upgrading stateless FlinkDeployment to savepoint still could not work. Because 
{{FlinkService#cancelJob}} does not pick up the target directory in the flink 
configuration.

 
{code:java}
case SAVEPOINT:
String savepoint =
clusterClient
.stopWithSavepoint(jobID, false, null)
.get(1, TimeUnit.MINUTES); {code}
We should use configured savepoint directory instead of {{null}} in the above 
implementation.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

FLIP-216 Decouple Hive connector with Flink planner

[??????]

Hi, everyone




I would like to open a discussion about decoupling Hive connector with Flink table planner.
 
 It's a follow-up discussion after Hive syntax discussion[1], but only focus on how to decouple Hive connector. The origin doc is here[2], from which you can see the details work and heated discussion about exposing Calcite API or reuse Operation tree to decouple. 
I have created FLIP-216: Decouple Hive connector with Flink planner[3] for it.




Thanks and looking forward to a lively discussion!



[1] https://lists.apache.org/thread/2w046dwl46tf2wy750gzmt0qrcz17z8t

[2] https://docs.google.com/document/d/1LMQ_mWfB_mkYkEBCUa2DgCO2YdtiZV7YRs2mpXyjdP4/edit?usp=sharing

[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A+Decouple+Hive+connector+with+Flink+planner




Best regards,
Yuxia


 

[jira] [Created] (FLINK-26882) Unaligned checkpoint with 0s timeout would fail RescaleCheckpointManuallyITCase

[Yun Tang (Jira)]

Yun Tang created FLINK-26882:


 Summary: Unaligned checkpoint with 0s timeout would fail 
RescaleCheckpointManuallyITCase
 Key: FLINK-26882
 URL: https://issues.apache.org/jira/browse/FLINK-26882
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Tests
Reporter: Yun Tang
 Fix For: 1.16.0


Once we make {{execution.checkpointing.unaligned: true}} and 
{{execution.checkpointing.alignment-timeout: PT0S}}, the 
RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState would fail 
then.

Borken instances:

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33776&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5623
 

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33787&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5626
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33787&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=12409

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=5c8e7682-d68f-54d1-16a2-a09310218a49&t=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba&l=5629
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=12409
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=5733
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=a549b384-c55a-52c0-c451-00e0477ab6db&t=eef5922c-08d9-5ba3-7299-8393476594e7&l=12575
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=5838
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=b0a398c0-685b-599c-eb57-c8c2a771138e&t=747432ad-a576-5911-1e2a-68c6bedc248a&l=12931
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=33779&view=logs&j=8fd9202e-fd17-5b26-353c-ac1ff76c8f28&t=ea7cf968-e585-52cb-e0fc-f48de023a7ca&l=5682



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[ANNOUNCE] Kubernetes Operator release-0.1 branch cut

[Gyula Fóra]

Hi Flink devs!

The release-0.1 branch has been forked from main and version numbers have
been upgraded accordingly.

https://github.com/apache/flink-kubernetes-operator/tree/release-0.1

The version on the main branch remains 1.0-SNAPSHOT for now.

>From now on, for PRs that should be presented in 0.1.0, please make sure:
- Merge the PR into both main and release-0.1 branches
- The JIRA ticket should be closed with the correct fix-versions
(kubernetes-operator-0.1.0 &  kubernetes-operator-1.0.0).

There is currently one outstanding blocker issue:
 - Use savepoint dir config for savepoint upgrades (
https://issues.apache.org/jira/browse/FLINK-26881)

We are working together with Yang and Marton to fix the blocker and prepare
rc1 today.

Cheers,
Gyula

回复：FLIP-216 Decouple Hive connector with Flink planner

[罗宇侠(莫辞)]

Sorry for this email, seems there's some format issue in my email client.  Just 
ignore it for it's a duplicate of  [DISCUSS] FLIP-216 Decouple Hive connector 
with Flink planner [1]


[1] https://lists.apache.org/thread/6xg33nxrnow5zy7xwqk5nwp00h9gcsbc

Best regards,
Yuxia--
发件人：罗宇侠
日　期：2022年03月25日 20:19:37
收件人：dev
主　题：FLIP-216 Decouple Hive connector with Flink planner

Hi, everyone




I would like to open a discussion about decoupling Hive connector with Flink table planner.
 
 It's a follow-up discussion after Hive syntax discussion[1], but only focus on how to decouple Hive connector. The origin doc is here[2], from which you can see the details work and heated discussion about exposing Calcite API or reuse Operation tree to decouple. 
I have created FLIP-216: Decouple Hive connector with Flink planner[3] for it.




Thanks and looking forward to a lively discussion!



[1] https://lists.apache.org/thread/2w046dwl46tf2wy750gzmt0qrcz17z8t

[2] https://docs.google.com/document/d/1LMQ_mWfB_mkYkEBCUa2DgCO2YdtiZV7YRs2mpXyjdP4/edit?usp=sharing

[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A+Decouple+Hive+connector+with+Flink+planner




Best regards,
Yuxia


 

Re: [ANNOUNCE] Kubernetes Operator release-0.1 branch cut

[Martijn Visser]

Hi Gyula,

The documentation on https://flink.apache.org/ currently shows 'Flink
Kubernetes Operator 1.0 (Latest stable release)'.

Best regards,

Martijn

On Mon, 28 Mar 2022 at 10:54, Gyula Fóra  wrote:

> Hi Flink devs!
>
> The release-0.1 branch has been forked from main and version numbers have
> been upgraded accordingly.
>
> https://github.com/apache/flink-kubernetes-operator/tree/release-0.1
>
> The version on the main branch remains 1.0-SNAPSHOT for now.
>
> From now on, for PRs that should be presented in 0.1.0, please make sure:
> - Merge the PR into both main and release-0.1 branches
> - The JIRA ticket should be closed with the correct fix-versions
> (kubernetes-operator-0.1.0 &  kubernetes-operator-1.0.0).
>
> There is currently one outstanding blocker issue:
>  - Use savepoint dir config for savepoint upgrades (
> https://issues.apache.org/jira/browse/FLINK-26881)
>
> We are working together with Yang and Marton to fix the blocker and prepare
> rc1 today.
>
> Cheers,
> Gyula
>

Re: [ANNOUNCE] Kubernetes Operator release-0.1 branch cut

[Gyula Fóra]

Thanks Martijn, a few of these administrative bits are still work in
progress :)

Gyula

On Mon, 28 Mar 2022 at 11:31, Martijn Visser 
wrote:

> Hi Gyula,
>
> The documentation on https://flink.apache.org/ currently shows 'Flink
> Kubernetes Operator 1.0 (Latest stable release)'.
>
> Best regards,
>
> Martijn
>
> On Mon, 28 Mar 2022 at 10:54, Gyula Fóra  wrote:
>
> > Hi Flink devs!
> >
> > The release-0.1 branch has been forked from main and version numbers have
> > been upgraded accordingly.
> >
> > https://github.com/apache/flink-kubernetes-operator/tree/release-0.1
> >
> > The version on the main branch remains 1.0-SNAPSHOT for now.
> >
> > From now on, for PRs that should be presented in 0.1.0, please make sure:
> > - Merge the PR into both main and release-0.1 branches
> > - The JIRA ticket should be closed with the correct fix-versions
> > (kubernetes-operator-0.1.0 &  kubernetes-operator-1.0.0).
> >
> > There is currently one outstanding blocker issue:
> >  - Use savepoint dir config for savepoint upgrades (
> > https://issues.apache.org/jira/browse/FLINK-26881)
> >
> > We are working together with Yang and Marton to fix the blocker and
> prepare
> > rc1 today.
> >
> > Cheers,
> > Gyula
> >
>

[jira] [Created] (FLINK-26883) Bump dependency-check-maven to 2.10.1

[zl (Jira)]

zl created FLINK-26883:
--

 Summary: Bump dependency-check-maven to 2.10.1
 Key: FLINK-26883
 URL: https://issues.apache.org/jira/browse/FLINK-26883
 Project: Flink
  Issue Type: Improvement
  Components: Build System
Reporter: zl


when running *_mvn org.owasp:dependency-check-maven:aggregate ,_* the following 
error occurred:

 
{code:java}
IO Exception connecting to 
https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2019.json.gz: HEAD request 
returned a non-200 status code: 
https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2019.json.gz 
.. {code}
 

That's because org.owasp:dependency-check-maven:5.0.0-M2 in 
_*flink-parent/pom.xml*_ is outdated and the data is unavailable. we may need 
to bump dependency-check-maven to newer version, like 7.0.1.

I rerun *_mvn org.owasp:dependency-check-maven:aggregate_* with 
org.owasp:dependency-check-maven:7.0.1, it works well.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26884) Move Elasticsearch connector to external connector repository

[Jing Ge (Jira)]

Jing Ge created FLINK-26884:
---

 Summary: Move Elasticsearch connector to external connector 
repository
 Key: FLINK-26884
 URL: https://issues.apache.org/jira/browse/FLINK-26884
 Project: Flink
  Issue Type: Improvement
Reporter: Jing Ge
Assignee: Jing Ge






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26885) Cleanup MetricRegistryImplTest

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26885:


 Summary: Cleanup MetricRegistryImplTest
 Key: FLINK-26885
 URL: https://issues.apache.org/jira/browse/FLINK-26885
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Metrics, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.16.0


While working on FLINK-21585 I stumbled on a large number of issues in this 
test and want to clean it up to ease testing in FLINK-21585.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26886) Document reporter behavior w.r.t. scopes & push/pull

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26886:


 Summary: Document reporter behavior w.r.t. scopes & push/pull
 Key: FLINK-26886
 URL: https://issues.apache.org/jira/browse/FLINK-26886
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / Metrics
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.16.0


The docs are lacking information for whether a reporter uses a metric 
identifier or the logical scope + tags, and whether they are push or pull based.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Gyula Fóra]

Hi everyone,

Please review and vote on the release candidate #1 for the version 0.1.0 of
Apache Flink Kubernetes Operator,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

**Release Overview**

As an overview, the release consists of the following:
a) Kubernetes Operator canonical source distribution (including the
Dockerfile), to be deployed to the release repository at dist.apache.org
b) Kubernetes Operator Helm Chart to be deployed to the release repository
at dist.apache.org
c) Maven artifacts to be deployed to the Maven Central Repository

**Staging Areas to Review**

The staging areas containing the above mentioned artifacts are as follows,
for your review:
* All artifacts for a,b) can be found in the corresponding dev repository
at dist.apache.org [1]
* All artifacts for c) can be found at the Apache Nexus Repository [2]

All artifacts are signed with the
key 911F218F79ACEA8EB453799EEE325DDEBFED467D [3]

Other links for your review:
* JIRA release notes [4]
* source code tag "release-0.1.0-rc1" [5]
* PR to update the website Downloads page to include Kubernetes Operator
links [6]

**Vote Duration**

The voting time will run for at least 72 hours.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

**Note for Functional Verification**
Please use the source distribution and helm chart directly from [1] to
build and deploy the operator in your testing environment.

Thanks,
Gyula

[1]
https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-0.1.0-rc1/
[2] https://repository.apache.org/content/repositories/orgapacheflink-1490/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351499
[5]
https://github.com/apache/flink-kubernetes-operator/tree/release-0.1.0-rc1
[6] https://github.com/apache/flink-web/pull/519

[jira] [Created] (FLINK-26887) Drop Jepsen tests

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26887:


 Summary: Drop Jepsen tests
 Key: FLINK-26887
 URL: https://issues.apache.org/jira/browse/FLINK-26887
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.16.0


Following our discussion in 
https://lists.apache.org/thread/hnkvrtlvwjo7mf47ybdv3dd5x723b7yl, drop the 
Jepsen tests (which are currently broken) as no one objected.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26888) Add e2e tests for network partitions

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26888:


 Summary: Add e2e tests for network partitions
 Key: FLINK-26888
 URL: https://issues.apache.org/jira/browse/FLINK-26888
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Reporter: Chesnay Schepler


Following the removal of the Jepsen tests in FLINK-26887 we should investigate 
a new approach for testing network partitions, for example based on Kubernetes 
and network policies.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26889) Eliminate the duplicated construct for FlinkOperatorConfiguration in test

[Aitozi (Jira)]

Aitozi created FLINK-26889:
--

 Summary: Eliminate the duplicated construct for 
FlinkOperatorConfiguration in test
 Key: FLINK-26889
 URL: https://issues.apache.org/jira/browse/FLINK-26889
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Aitozi


A minor improvement to reduce the boilerplate code



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Chesnay Schepler]

There's some strange stuff in here.

What exactly is the purpose of flink-kubernetes-shaded? You're just 
re-packaging flink-kubernetes without making any changes.


The uploaded flink-kubernetes-operator jar isn't bundling any 
dependencies. Why is the fat jar not uploaded? Is it used anywhere else 
(e.g., directly embedded into a docker image)
If it is used, where do you have the appropriate licensing information 
for the bundled dependencies?


On 28/03/2022 16:14, Gyula Fóra wrote:

Hi everyone,

Please review and vote on the release candidate #1 for the version 0.1.0 of
Apache Flink Kubernetes Operator,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

**Release Overview**

As an overview, the release consists of the following:
a) Kubernetes Operator canonical source distribution (including the
Dockerfile), to be deployed to the release repository at dist.apache.org
b) Kubernetes Operator Helm Chart to be deployed to the release repository
at dist.apache.org
c) Maven artifacts to be deployed to the Maven Central Repository

**Staging Areas to Review**

The staging areas containing the above mentioned artifacts are as follows,
for your review:
* All artifacts for a,b) can be found in the corresponding dev repository
at dist.apache.org [1]
* All artifacts for c) can be found at the Apache Nexus Repository [2]

All artifacts are signed with the
key 911F218F79ACEA8EB453799EEE325DDEBFED467D [3]

Other links for your review:
* JIRA release notes [4]
* source code tag "release-0.1.0-rc1" [5]
* PR to update the website Downloads page to include Kubernetes Operator
links [6]

**Vote Duration**

The voting time will run for at least 72 hours.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

**Note for Functional Verification**
Please use the source distribution and helm chart directly from [1] to
build and deploy the operator in your testing environment.

Thanks,
Gyula

[1]
https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-0.1.0-rc1/
[2] https://repository.apache.org/content/repositories/orgapacheflink-1490/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351499
[5]
https://github.com/apache/flink-kubernetes-operator/tree/release-0.1.0-rc1
[6] https://github.com/apache/flink-web/pull/519

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Gyula Fóra]

Hi Chesnay,

Let me try to explain the "strange stuff"

flink-kubernetes-shaded relocates some classes found in flink-kubernetes in
order to not conflict with some of the operator dependencies.
This is necessary as flink-kubernetes packages almost everything in the
fat-jar as-is without relocation. I think this should be fine from a
release perspective, as flink-kubernetes already contains the
relevant notice files.

For  flink-kubernetes-operator we are not releasing a fat-jar as we don't
have any client binaries etc. It is not necessary for the users therefore
it's not part of the release.
We release the Dockerfile instead so that users can build the image. Since
the fatjar is not part of the release we do not have bundled dependencies,
and we do not need extra licensing information I believe.

Cheers,
Gyula

On Mon, Mar 28, 2022 at 4:40 PM Chesnay Schepler  wrote:

> There's some strange stuff in here.
>
> What exactly is the purpose of flink-kubernetes-shaded? You're just
> re-packaging flink-kubernetes without making any changes.
>
> The uploaded flink-kubernetes-operator jar isn't bundling any
> dependencies. Why is the fat jar not uploaded? Is it used anywhere else
> (e.g., directly embedded into a docker image)
> If it is used, where do you have the appropriate licensing information
> for the bundled dependencies?
>
> On 28/03/2022 16:14, Gyula Fóra wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version 0.1.0
> of
> > Apache Flink Kubernetes Operator,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > **Release Overview**
> >
> > As an overview, the release consists of the following:
> > a) Kubernetes Operator canonical source distribution (including the
> > Dockerfile), to be deployed to the release repository at dist.apache.org
> > b) Kubernetes Operator Helm Chart to be deployed to the release
> repository
> > at dist.apache.org
> > c) Maven artifacts to be deployed to the Maven Central Repository
> >
> > **Staging Areas to Review**
> >
> > The staging areas containing the above mentioned artifacts are as
> follows,
> > for your review:
> > * All artifacts for a,b) can be found in the corresponding dev repository
> > at dist.apache.org [1]
> > * All artifacts for c) can be found at the Apache Nexus Repository [2]
> >
> > All artifacts are signed with the
> > key 911F218F79ACEA8EB453799EEE325DDEBFED467D [3]
> >
> > Other links for your review:
> > * JIRA release notes [4]
> > * source code tag "release-0.1.0-rc1" [5]
> > * PR to update the website Downloads page to include Kubernetes Operator
> > links [6]
> >
> > **Vote Duration**
> >
> > The voting time will run for at least 72 hours.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > **Note for Functional Verification**
> > Please use the source distribution and helm chart directly from [1] to
> > build and deploy the operator in your testing environment.
> >
> > Thanks,
> > Gyula
> >
> > [1]
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-0.1.0-rc1/
> > [2]
> https://repository.apache.org/content/repositories/orgapacheflink-1490/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351499
> > [5]
> >
> https://github.com/apache/flink-kubernetes-operator/tree/release-0.1.0-rc1
> > [6] https://github.com/apache/flink-web/pull/519
> >
>
>

[jira] [Created] (FLINK-26890) [Kinesis][Consumer] DynamoDB consumer error consuming partitions close to retention

[Danny Cranmer (Jira)]

Danny Cranmer created FLINK-26890:
-

 Summary: [Kinesis][Consumer] DynamoDB consumer error consuming 
partitions close to retention
 Key: FLINK-26890
 URL: https://issues.apache.org/jira/browse/FLINK-26890
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: 1.15.0
Reporter: Danny Cranmer
 Fix For: 1.15.0, 1.16.0, 1.13.7, 1.14.5


*Background*

The Amazon Kinesis Data Streams consumer supports consuming from Amazon 
DynamoDB via the [DynamoDB Streams Kinesis 
Adapter|https://github.com/awslabs/dynamodb-streams-kinesis-adapter]. 

*Problem*

We have seen instances of consumer throwing {{ResouceNotFoundException}} when 
attempting to invoke {{GetShardIterator}}.

{code}
com.amazonaws.services.kinesis.model.ResourceNotFoundException: Requested 
resource not found: Shard does not exist 
{code}

According to the DynamoDB team, the {{DescribeStream}} call may return shard 
IDs that are no longer valid, and this exception needs to be handled by the 
client. 

*Solution*

Modify the DynamoDB consumer to treat {{ResourceNotFoundException}} as a shard 
closed signal.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Chesnay Schepler]

I don't think having users build the fat-jar & docker image absolves us 
of all responsibility w.r.t. the licensing of used products.


At the very least we need to inform users what licenses the fat-jar & 
docker image fall under such that they can make an informed decision as 
to whether they can adhere to said restrictions.
In particular since building it yourself is (apparently) a hard 
requirement for using said product.


Even beyond that though, as /we/ push images to ghcr.io we still need to 
adhere to the licensing requirements in any case afaict.


On 28/03/2022 17:07, Gyula Fóra wrote:

Hi Chesnay,

Let me try to explain the "strange stuff"

flink-kubernetes-shaded relocates some classes found in 
flink-kubernetes in order to not conflict with some of the operator 
dependencies.
This is necessary as flink-kubernetes packages almost everything in 
the fat-jar as-is without relocation. I think this should be fine from 
a release perspective, as flink-kubernetes already contains the 
relevant notice files.


For  flink-kubernetes-operator we are not releasing a fat-jar as we 
don't have any client binaries etc. It is not necessary for the users 
therefore it's not part of the release.
We release the Dockerfile instead so that users can build the image. 
Since the fatjar is not part of the release we do not have bundled 
dependencies, and we do not need extra licensing information I believe.


Cheers,
Gyula

On Mon, Mar 28, 2022 at 4:40 PM Chesnay Schepler  
wrote:


There's some strange stuff in here.

What exactly is the purpose of flink-kubernetes-shaded? You're just
re-packaging flink-kubernetes without making any changes.

The uploaded flink-kubernetes-operator jar isn't bundling any
dependencies. Why is the fat jar not uploaded? Is it used anywhere
else
(e.g., directly embedded into a docker image)
If it is used, where do you have the appropriate licensing
information
for the bundled dependencies?

On 28/03/2022 16:14, Gyula Fóra wrote:
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the
version 0.1.0 of
> Apache Flink Kubernetes Operator,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific
comments)
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution (including the
> Dockerfile), to be deployed to the release repository at
dist.apache.org 
> b) Kubernetes Operator Helm Chart to be deployed to the release
repository
> at dist.apache.org 
> c) Maven artifacts to be deployed to the Maven Central Repository
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are
as follows,
> for your review:
> * All artifacts for a,b) can be found in the corresponding dev
repository
> at dist.apache.org  [1]
> * All artifacts for c) can be found at the Apache Nexus
Repository [2]
>
> All artifacts are signed with the
> key 911F218F79ACEA8EB453799EEE325DDEBFED467D [3]
>
> Other links for your review:
> * JIRA release notes [4]
> * source code tag "release-0.1.0-rc1" [5]
> * PR to update the website Downloads page to include Kubernetes
Operator
> links [6]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours.
> It is adopted by majority approval, with at least 3 PMC
affirmative votes.
>
> **Note for Functional Verification**
> Please use the source distribution and helm chart directly from
[1] to
> build and deploy the operator in your testing environment.
>
> Thanks,
> Gyula
>
> [1]
>

https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-0.1.0-rc1/
> [2]
https://repository.apache.org/content/repositories/orgapacheflink-1490/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
>

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351499


> [5]
>
https://github.com/apache/flink-kubernetes-operator/tree/release-0.1.0-rc1
> [6] https://github.com/apache/flink-web/pull/519
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Gyula Fóra]

Thanks for the input!

I am not an expert on this topic and have been contemplating this myself
also.
We are basically trying to follow the precedent set by Flink and Statefun
projects where the docker builds that we use to publish images to
dockerhub do not declare any notices.

We will not use ghcr.io for the final release but will use dockerhub like
flink and other apache projects.

If I look at it from a strictly technical point of view, the docker image
is not part of the official release (as it's also not part of the
flink/statefun release).

It would be good to get some input from others on this. It's not impossible
to add the notices but it's considerable work and maintenance overhead.
By extending the logic would you then also add license information for the
base images of the docker container (and so on so forth)?

My gut feeling would be that we could highlight this in the NOTICE of the
main project  (or some other appropriate place) but we do not explicitly
list the dependencies.

Would be good to hear how others feel about this!

Gyula

On Mon, Mar 28, 2022 at 5:26 PM Chesnay Schepler  wrote:

> I don't think having users build the fat-jar & docker image absolves us of
> all responsibility w.r.t. the licensing of used products.
>
> At the very least we need to inform users what licenses the fat-jar &
> docker image fall under such that they can make an informed decision as to
> whether they can adhere to said restrictions.
> In particular since building it yourself is (apparently) a hard
> requirement for using said product.
>
> Even beyond that though, as *we* push images to ghcr.io we still need to
> adhere to the licensing requirements in any case afaict.
>
> On 28/03/2022 17:07, Gyula Fóra wrote:
>
> Hi Chesnay,
>
> Let me try to explain the "strange stuff"
>
> flink-kubernetes-shaded relocates some classes found in flink-kubernetes
> in order to not conflict with some of the operator dependencies.
> This is necessary as flink-kubernetes packages almost everything in the
> fat-jar as-is without relocation. I think this should be fine from a
> release perspective, as flink-kubernetes already contains the
> relevant notice files.
>
> For  flink-kubernetes-operator we are not releasing a fat-jar as we don't
> have any client binaries etc. It is not necessary for the users therefore
> it's not part of the release.
> We release the Dockerfile instead so that users can build the image. Since
> the fatjar is not part of the release we do not have bundled dependencies,
> and we do not need extra licensing information I believe.
>
> Cheers,
> Gyula
>
> On Mon, Mar 28, 2022 at 4:40 PM Chesnay Schepler 
> wrote:
>
>> There's some strange stuff in here.
>>
>> What exactly is the purpose of flink-kubernetes-shaded? You're just
>> re-packaging flink-kubernetes without making any changes.
>>
>> The uploaded flink-kubernetes-operator jar isn't bundling any
>> dependencies. Why is the fat jar not uploaded? Is it used anywhere else
>> (e.g., directly embedded into a docker image)
>> If it is used, where do you have the appropriate licensing information
>> for the bundled dependencies?
>>
>> On 28/03/2022 16:14, Gyula Fóra wrote:
>> > Hi everyone,
>> >
>> > Please review and vote on the release candidate #1 for the version
>> 0.1.0 of
>> > Apache Flink Kubernetes Operator,
>> > as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> > **Release Overview**
>> >
>> > As an overview, the release consists of the following:
>> > a) Kubernetes Operator canonical source distribution (including the
>> > Dockerfile), to be deployed to the release repository at
>> dist.apache.org
>> > b) Kubernetes Operator Helm Chart to be deployed to the release
>> repository
>> > at dist.apache.org
>> > c) Maven artifacts to be deployed to the Maven Central Repository
>> >
>> > **Staging Areas to Review**
>> >
>> > The staging areas containing the above mentioned artifacts are as
>> follows,
>> > for your review:
>> > * All artifacts for a,b) can be found in the corresponding dev
>> repository
>> > at dist.apache.org [1]
>> > * All artifacts for c) can be found at the Apache Nexus Repository [2]
>> >
>> > All artifacts are signed with the
>> > key 911F218F79ACEA8EB453799EEE325DDEBFED467D [3]
>> >
>> > Other links for your review:
>> > * JIRA release notes [4]
>> > * source code tag "release-0.1.0-rc1" [5]
>> > * PR to update the website Downloads page to include Kubernetes Operator
>> > links [6]
>> >
>> > **Vote Duration**
>> >
>> > The voting time will run for at least 72 hours.
>> > It is adopted by majority approval, with at least 3 PMC affirmative
>> votes.
>> >
>> > **Note for Functional Verification**
>> > Please use the source distribution and helm chart directly from [1] to
>> > build and deploy the operator in your testing environment.
>> >
>> > Thanks,
>> > Gyula
>> >
>> > [1]
>> >
>> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-oper

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Chesnay Schepler]

One difference to Flink is that the distribution bundled in the docker 
image still contains the NOTICE covering the contents of it.


It may admittedly not be the most discoverable place, but a reasonable 
one I think.


Docker as a whole is very weird when it comes to licensing.
Think of all the things are are shipped in an image; I don't think we 
can (nor should) try to document everything in there.
For the most part this is also not necessary as the Flink images are 
based on Debian,
where (al)most every installed package already embeds licensing 
information into the image.


However, for content that we copy into the image (i.e., the jars), I 
think it would be reasonable to document that.
(and based on experience from the Flink side has also shown other 
advantages beyond licensing...)


On 28/03/2022 17:41, Gyula Fóra wrote:

Thanks for the input!

I am not an expert on this topic and have been contemplating this 
myself also.
We are basically trying to follow the precedent set by Flink and 
Statefun projects where the docker builds that we use to publish 
images to dockerhub do not declare any notices.


We will not use ghcr.io  for the final release but 
will use dockerhub like flink and other apache projects.


If I look at it from a strictly technical point of view, the docker 
image is not part of the official release (as it's also not part of 
the flink/statefun release).


It would be good to get some input from others on this. It's not 
impossible to add the notices but it's considerable work and 
maintenance overhead.
By extending the logic would you then also add license information for 
the base images of the docker container (and so on so forth)?


My gut feeling would be that we could highlight this in the NOTICE of 
the main project  (or some other appropriate place) but we do not 
explicitly list the dependencies.


Would be good to hear how others feel about this!

Gyula

On Mon, Mar 28, 2022 at 5:26 PM Chesnay Schepler  
wrote:


I don't think having users build the fat-jar & docker image
absolves us of all responsibility w.r.t. the licensing of used
products.

At the very least we need to inform users what licenses the
fat-jar & docker image fall under such that they can make an
informed decision as to whether they can adhere to said restrictions.
In particular since building it yourself is (apparently) a hard
requirement for using said product.

Even beyond that though, as /we/ push images to ghcr.io
 we still need to adhere to the licensing
requirements in any case afaict.

On 28/03/2022 17:07, Gyula Fóra wrote:

Hi Chesnay,

Let me try to explain the "strange stuff"

flink-kubernetes-shaded relocates some classes found in
flink-kubernetes in order to not conflict with some of the
operator dependencies.
This is necessary as flink-kubernetes packages almost everything
in the fat-jar as-is without relocation. I think this should be
fine from a release perspective, as flink-kubernetes already
contains the relevant notice files.

For  flink-kubernetes-operator we are not releasing a fat-jar as
we don't have any client binaries etc. It is not necessary for
the users therefore it's not part of the release.
We release the Dockerfile instead so that users can build the
image. Since the fatjar is not part of the release we do not have
bundled dependencies, and we do not need extra licensing
information I believe.

Cheers,
Gyula

On Mon, Mar 28, 2022 at 4:40 PM Chesnay Schepler
 wrote:

There's some strange stuff in here.

What exactly is the purpose of flink-kubernetes-shaded?
You're just
re-packaging flink-kubernetes without making any changes.

The uploaded flink-kubernetes-operator jar isn't bundling any
dependencies. Why is the fat jar not uploaded? Is it used
anywhere else
(e.g., directly embedded into a docker image)
If it is used, where do you have the appropriate licensing
information
for the bundled dependencies?

On 28/03/2022 16:14, Gyula Fóra wrote:
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the
version 0.1.0 of
> Apache Flink Kubernetes Operator,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific
comments)
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution
(including the
> Dockerfile), to be deployed to the release repository at
dist.apache.org 
> b) Kubernetes Operator Helm Chart to be deployed to the
release repository
> at dist.apache.org 
 

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Gyula Fóra]

I see your point and the value for having such a notice added.

I think there are 2 completely distinct questions at play here:

a) Is there a legal requirement for a NOTICE file for the docker image?
b) If not, should we block the release on this and add it immediately?

For a)
I think from a legal (and ASF policy) perspective there is one question
that decides whether this is a requirement for this release or not:
Is the docker image part of the release?

I think the answer here is clearly no, the image is not part of the
release. Only the Dockerfile is part of the release.

For b)
I think adding the NOTICE is a good idea but it will take some work so I
would not block the preview release on it.
If someone has some handy utility or experience generating it, I don't mind
including it in later RCs of course.
Otherwise we can aim for the next release.

Gyula


On Mon, Mar 28, 2022 at 6:03 PM Chesnay Schepler  wrote:

> One difference to Flink is that the distribution bundled in the docker
> image still contains the NOTICE covering the contents of it.
>
> It may admittedly not be the most discoverable place, but a reasonable one
> I think.
>
> Docker as a whole is very weird when it comes to licensing.
> Think of all the things are are shipped in an image; I don't think we can
> (nor should) try to document everything in there.
> For the most part this is also not necessary as the Flink images are based
> on Debian,
> where (al)most every installed package already embeds licensing
> information into the image.
>
> However, for content that we copy into the image (i.e., the jars), I think
> it would be reasonable to document that.
> (and based on experience from the Flink side has also shown other
> advantages beyond licensing...)
>
> On 28/03/2022 17:41, Gyula Fóra wrote:
>
> Thanks for the input!
>
> I am not an expert on this topic and have been contemplating this myself
> also.
> We are basically trying to follow the precedent set by Flink and Statefun
> projects where the docker builds that we use to publish images to
> dockerhub do not declare any notices.
>
> We will not use ghcr.io for the final release but will use dockerhub like
> flink and other apache projects.
>
> If I look at it from a strictly technical point of view, the docker image
> is not part of the official release (as it's also not part of the
> flink/statefun release).
>
> It would be good to get some input from others on this. It's not
> impossible to add the notices but it's considerable work and maintenance
> overhead.
> By extending the logic would you then also add license information for the
> base images of the docker container (and so on so forth)?
>
> My gut feeling would be that we could highlight this in the NOTICE of the
> main project  (or some other appropriate place) but we do not explicitly
> list the dependencies.
>
> Would be good to hear how others feel about this!
>
> Gyula
>
> On Mon, Mar 28, 2022 at 5:26 PM Chesnay Schepler 
> wrote:
>
>> I don't think having users build the fat-jar & docker image absolves us
>> of all responsibility w.r.t. the licensing of used products.
>>
>> At the very least we need to inform users what licenses the fat-jar &
>> docker image fall under such that they can make an informed decision as to
>> whether they can adhere to said restrictions.
>> In particular since building it yourself is (apparently) a hard
>> requirement for using said product.
>>
>> Even beyond that though, as *we* push images to ghcr.io we still need to
>> adhere to the licensing requirements in any case afaict.
>>
>> On 28/03/2022 17:07, Gyula Fóra wrote:
>>
>> Hi Chesnay,
>>
>> Let me try to explain the "strange stuff"
>>
>> flink-kubernetes-shaded relocates some classes found in flink-kubernetes
>> in order to not conflict with some of the operator dependencies.
>> This is necessary as flink-kubernetes packages almost everything in the
>> fat-jar as-is without relocation. I think this should be fine from a
>> release perspective, as flink-kubernetes already contains the
>> relevant notice files.
>>
>> For  flink-kubernetes-operator we are not releasing a fat-jar as we don't
>> have any client binaries etc. It is not necessary for the users therefore
>> it's not part of the release.
>> We release the Dockerfile instead so that users can build the image.
>> Since the fatjar is not part of the release we do not have bundled
>> dependencies, and we do not need extra licensing information I believe.
>>
>> Cheers,
>> Gyula
>>
>> On Mon, Mar 28, 2022 at 4:40 PM Chesnay Schepler 
>> wrote:
>>
>>> There's some strange stuff in here.
>>>
>>> What exactly is the purpose of flink-kubernetes-shaded? You're just
>>> re-packaging flink-kubernetes without making any changes.
>>>
>>> The uploaded flink-kubernetes-operator jar isn't bundling any
>>> dependencies. Why is the fat jar not uploaded? Is it used anywhere else
>>> (e.g., directly embedded into a docker image)
>>> If it is used, where do you have the appropriate licensin

[jira] [Created] (FLINK-26891) Emit events for important Deployment / Job changes

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26891:
--

 Summary: Emit events for important Deployment / Job changes
 Key: FLINK-26891
 URL: https://issues.apache.org/jira/browse/FLINK-26891
 Project: Flink
  Issue Type: New Feature
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


We should try capturing the important deployment states, such as RUNNING, 
FAILING, DEPLOYING



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26892) Observe current status before validating CR changes

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26892:
--

 Summary: Observe current status before validating CR changes
 Key: FLINK-26892
 URL: https://issues.apache.org/jira/browse/FLINK-26892
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


Currently validation is the first step in the controller loop which means that 
when there is a validation error we fail to observe the status of currently 
running deployments.

We should change the order of operations and observe before validation. 

Furthermore observe should use the previous configuration not the one after the 
CR change.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26893) Validate checkpoint config with last-state upgrade mode

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26893:
--

 Summary: Validate checkpoint config with last-state upgrade mode
 Key: FLINK-26893
 URL: https://issues.apache.org/jira/browse/FLINK-26893
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


We should consider validating if checkpointing is turned on (in addition to the 
HA settings) when last-state upgrade mode is selected.

Unfortunately it is not clear how to do this properly as the user can also 
programmatically override this from the main class.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26894) Support custom validator implementations

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26894:
--

 Summary: Support custom validator implementations
 Key: FLINK-26894
 URL: https://issues.apache.org/jira/browse/FLINK-26894
 Project: Flink
  Issue Type: New Feature
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


We should allow users to specify custom validator implementations through the 
operator configuration. This can be very useful in production environments to 
implement custom validation logic that is specific to certain envs.

We should also expose a simple way for users to add extra libraries when 
deploying the operator



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26895) Improve cancelWithSavepoint timeout error message

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26895:
--

 Summary: Improve cancelWithSavepoint timeout error message
 Key: FLINK-26895
 URL: https://issues.apache.org/jira/browse/FLINK-26895
 Project: Flink
  Issue Type: New Feature
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


If a cancelwithsavepoint operation times out during a savepoint upgrade step, 
the user gets an error but the savepoint information is not persisted in the 
status.

We should add an informative error telling the user how to resolve this 
situation (switch to last-state mode or look up the savepoint manually and 
recreate the deployment)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Flink 1.15 Stabilisation Sync

[Johannes Moser]

Dear Community,

We are fairly on track with stabilising the 1.15 release, that’s why Yun Gao 
and me think we don’t need the sync anymore.

So we skip it for now, starting from tomorrow. If the situation might change. 
We will let you know.

Best,
Joe

[jira] [Created] (FLINK-26896) Logging - Support yaml/xml/json format in Log4j2

[Koala Lam (Jira)]

Koala Lam created FLINK-26896:
-

 Summary: Logging - Support yaml/xml/json format in Log4j2
 Key: FLINK-26896
 URL: https://issues.apache.org/jira/browse/FLINK-26896
 Project: Flink
  Issue Type: Improvement
Reporter: Koala Lam


Only log4j.properties is currenlty supported:

[https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/bin/flink-daemon.sh#L89]

 

https://logging.apache.org/log4j/2.x/manual/configuration.html



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26897) Provide an additional way to add dependency files to k8s

[liuzhuo (Jira)]

liuzhuo created FLINK-26897:
---

 Summary: Provide an additional way to add dependency files to k8s
 Key: FLINK-26897
 URL: https://issues.apache.org/jira/browse/FLINK-26897
 Project: Flink
  Issue Type: New Feature
  Components: Deployment / Kubernetes
Reporter: liuzhuo


In the yarn environment, we can submit the dependency files of the task through 
the parameter of 'yarn.ship-files', but in two modes of k8s:
1.Application mode: dependencies must be put into the image, resulting in 
different images for each different task
2.Session mode: No relevant parameters have been found to add dependent files

Do we need to provide a way to add some dependencies of the task itself when 
the job is submitted in session mode?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Thomas Weise]

I believe if we as the PMC distribute a docker image (which is optional,
"convenience"), then that image has to follow the rules for binary packages
[1]. (And I would assume that applies regardless where we host the images.)

Now to say that we only publish sources kind of side steps that problem. At
the same time it probably also defeats the purpose of the preview release,
which is to make it easier for folks that are not active contributors to
take the operator for a test drive.

So I'm leaning towards publishing the images with respective NOTICE
requirements (for the layers that we add).

We are also planning to publish the jar files in the future as it helps to
build clients and those would need to have the binary NOTICE also.

Cheers,
Thomas

[1] https://infra.apache.org/licensing-howto.html#binary

On Mon, Mar 28, 2022 at 9:20 AM Gyula Fóra  wrote:

> I see your point and the value for having such a notice added.
>
> I think there are 2 completely distinct questions at play here:
>
> a) Is there a legal requirement for a NOTICE file for the docker image?
> b) If not, should we block the release on this and add it immediately?
>
> For a)
> I think from a legal (and ASF policy) perspective there is one question
> that decides whether this is a requirement for this release or not:
> Is the docker image part of the release?
>
> I think the answer here is clearly no, the image is not part of the
> release. Only the Dockerfile is part of the release.
>
> For b)
> I think adding the NOTICE is a good idea but it will take some work so I
> would not block the preview release on it.
> If someone has some handy utility or experience generating it, I don't mind
> including it in later RCs of course.
> Otherwise we can aim for the next release.
>
> Gyula
>
>
> On Mon, Mar 28, 2022 at 6:03 PM Chesnay Schepler 
> wrote:
>
> > One difference to Flink is that the distribution bundled in the docker
> > image still contains the NOTICE covering the contents of it.
> >
> > It may admittedly not be the most discoverable place, but a reasonable
> one
> > I think.
> >
> > Docker as a whole is very weird when it comes to licensing.
> > Think of all the things are are shipped in an image; I don't think we can
> > (nor should) try to document everything in there.
> > For the most part this is also not necessary as the Flink images are
> based
> > on Debian,
> > where (al)most every installed package already embeds licensing
> > information into the image.
> >
> > However, for content that we copy into the image (i.e., the jars), I
> think
> > it would be reasonable to document that.
> > (and based on experience from the Flink side has also shown other
> > advantages beyond licensing...)
> >
> > On 28/03/2022 17:41, Gyula Fóra wrote:
> >
> > Thanks for the input!
> >
> > I am not an expert on this topic and have been contemplating this myself
> > also.
> > We are basically trying to follow the precedent set by Flink and Statefun
> > projects where the docker builds that we use to publish images to
> > dockerhub do not declare any notices.
> >
> > We will not use ghcr.io for the final release but will use dockerhub
> like
> > flink and other apache projects.
> >
> > If I look at it from a strictly technical point of view, the docker image
> > is not part of the official release (as it's also not part of the
> > flink/statefun release).
> >
> > It would be good to get some input from others on this. It's not
> > impossible to add the notices but it's considerable work and maintenance
> > overhead.
> > By extending the logic would you then also add license information for
> the
> > base images of the docker container (and so on so forth)?
> >
> > My gut feeling would be that we could highlight this in the NOTICE of the
> > main project  (or some other appropriate place) but we do not explicitly
> > list the dependencies.
> >
> > Would be good to hear how others feel about this!
> >
> > Gyula
> >
> > On Mon, Mar 28, 2022 at 5:26 PM Chesnay Schepler 
> > wrote:
> >
> >> I don't think having users build the fat-jar & docker image absolves us
> >> of all responsibility w.r.t. the licensing of used products.
> >>
> >> At the very least we need to inform users what licenses the fat-jar &
> >> docker image fall under such that they can make an informed decision as
> to
> >> whether they can adhere to said restrictions.
> >> In particular since building it yourself is (apparently) a hard
> >> requirement for using said product.
> >>
> >> Even beyond that though, as *we* push images to ghcr.io we still need
> to
> >> adhere to the licensing requirements in any case afaict.
> >>
> >> On 28/03/2022 17:07, Gyula Fóra wrote:
> >>
> >> Hi Chesnay,
> >>
> >> Let me try to explain the "strange stuff"
> >>
> >> flink-kubernetes-shaded relocates some classes found in flink-kubernetes
> >> in order to not conflict with some of the operator dependencies.
> >> This is necessary as flink-kubernetes packages almost everything in the
> >> fat-jar as-is w

[jira] [Created] (FLINK-26898) Introduce creating table document for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26898:


 Summary: Introduce creating table document for table store
 Key: FLINK-26898
 URL: https://issues.apache.org/jira/browse/FLINK-26898
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.1.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26899) Introduce query table document for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26899:


 Summary: Introduce query table document for table store
 Key: FLINK-26899
 URL: https://issues.apache.org/jira/browse/FLINK-26899
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.1.0


* Batch or streaming
 * file store continuous and log store
 * retention



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26900) Introduce Best Practices document for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26900:


 Summary: Introduce Best Practices document for table store
 Key: FLINK-26900
 URL: https://issues.apache.org/jira/browse/FLINK-26900
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.1.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26901) Introduce config options document for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26901:


 Summary: Introduce config options document for table store
 Key: FLINK-26901
 URL: https://issues.apache.org/jira/browse/FLINK-26901
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.1.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26902) Introduce benchmark document for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26902:


 Summary: Introduce benchmark document for table store
 Key: FLINK-26902
 URL: https://issues.apache.org/jira/browse/FLINK-26902
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.1.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Yang Wang]

Given that *flink-kubernetes-shaded* already contains a NOTICE file, and
*flink-kubernetes-webhook* does not bundle any dependencies.
IIUC, what we should do now is to add a correct NOTICE file only for the
*flink-kubernetes-operator* module.

If I do not miss anything, this would not be very difficult and I would
like to fix it.


Best,
Yang

Thomas Weise  于2022年3月29日周二 11:25写道：

> I believe if we as the PMC distribute a docker image (which is optional,
> "convenience"), then that image has to follow the rules for binary packages
> [1]. (And I would assume that applies regardless where we host the images.)
>
> Now to say that we only publish sources kind of side steps that problem. At
> the same time it probably also defeats the purpose of the preview release,
> which is to make it easier for folks that are not active contributors to
> take the operator for a test drive.
>
> So I'm leaning towards publishing the images with respective NOTICE
> requirements (for the layers that we add).
>
> We are also planning to publish the jar files in the future as it helps to
> build clients and those would need to have the binary NOTICE also.
>
> Cheers,
> Thomas
>
> [1] https://infra.apache.org/licensing-howto.html#binary
>
> On Mon, Mar 28, 2022 at 9:20 AM Gyula Fóra  wrote:
>
> > I see your point and the value for having such a notice added.
> >
> > I think there are 2 completely distinct questions at play here:
> >
> > a) Is there a legal requirement for a NOTICE file for the docker image?
> > b) If not, should we block the release on this and add it immediately?
> >
> > For a)
> > I think from a legal (and ASF policy) perspective there is one question
> > that decides whether this is a requirement for this release or not:
> > Is the docker image part of the release?
> >
> > I think the answer here is clearly no, the image is not part of the
> > release. Only the Dockerfile is part of the release.
> >
> > For b)
> > I think adding the NOTICE is a good idea but it will take some work so I
> > would not block the preview release on it.
> > If someone has some handy utility or experience generating it, I don't
> mind
> > including it in later RCs of course.
> > Otherwise we can aim for the next release.
> >
> > Gyula
> >
> >
> > On Mon, Mar 28, 2022 at 6:03 PM Chesnay Schepler 
> > wrote:
> >
> > > One difference to Flink is that the distribution bundled in the docker
> > > image still contains the NOTICE covering the contents of it.
> > >
> > > It may admittedly not be the most discoverable place, but a reasonable
> > one
> > > I think.
> > >
> > > Docker as a whole is very weird when it comes to licensing.
> > > Think of all the things are are shipped in an image; I don't think we
> can
> > > (nor should) try to document everything in there.
> > > For the most part this is also not necessary as the Flink images are
> > based
> > > on Debian,
> > > where (al)most every installed package already embeds licensing
> > > information into the image.
> > >
> > > However, for content that we copy into the image (i.e., the jars), I
> > think
> > > it would be reasonable to document that.
> > > (and based on experience from the Flink side has also shown other
> > > advantages beyond licensing...)
> > >
> > > On 28/03/2022 17:41, Gyula Fóra wrote:
> > >
> > > Thanks for the input!
> > >
> > > I am not an expert on this topic and have been contemplating this
> myself
> > > also.
> > > We are basically trying to follow the precedent set by Flink and
> Statefun
> > > projects where the docker builds that we use to publish images to
> > > dockerhub do not declare any notices.
> > >
> > > We will not use ghcr.io for the final release but will use dockerhub
> > like
> > > flink and other apache projects.
> > >
> > > If I look at it from a strictly technical point of view, the docker
> image
> > > is not part of the official release (as it's also not part of the
> > > flink/statefun release).
> > >
> > > It would be good to get some input from others on this. It's not
> > > impossible to add the notices but it's considerable work and
> maintenance
> > > overhead.
> > > By extending the logic would you then also add license information for
> > the
> > > base images of the docker container (and so on so forth)?
> > >
> > > My gut feeling would be that we could highlight this in the NOTICE of
> the
> > > main project  (or some other appropriate place) but we do not
> explicitly
> > > list the dependencies.
> > >
> > > Would be good to hear how others feel about this!
> > >
> > > Gyula
> > >
> > > On Mon, Mar 28, 2022 at 5:26 PM Chesnay Schepler 
> > > wrote:
> > >
> > >> I don't think having users build the fat-jar & docker image absolves
> us
> > >> of all responsibility w.r.t. the licensing of used products.
> > >>
> > >> At the very least we need to inform users what licenses the fat-jar &
> > >> docker image fall under such that they can make an informed decision
> as
> > to
> > >> whether they can adhere to said restrictions.
> >

[jira] [Created] (FLINK-26903) Add correct NOTICE file to flink-kubernetes-operator

[Yang Wang (Jira)]

Yang Wang created FLINK-26903:
-

 Summary: Add correct NOTICE file to flink-kubernetes-operator
 Key: FLINK-26903
 URL: https://issues.apache.org/jira/browse/FLINK-26903
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Yang Wang
 Fix For: kubernetes-operator-0.1.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #1

[Gyula Fóra]

Thank you all for the input, let’s consider this a blocker.

As soon as we have the NOTICE fix , I will prepare the new RC.

Gyula

On Tue, 29 Mar 2022 at 06:51, Yang Wang  wrote:

> Given that *flink-kubernetes-shaded* already contains a NOTICE file, and
> *flink-kubernetes-webhook* does not bundle any dependencies.
> IIUC, what we should do now is to add a correct NOTICE file only for the
> *flink-kubernetes-operator* module.
>
> If I do not miss anything, this would not be very difficult and I would
> like to fix it.
>
>
> Best,
> Yang
>
> Thomas Weise  于2022年3月29日周二 11:25写道：
>
> > I believe if we as the PMC distribute a docker image (which is optional,
> > "convenience"), then that image has to follow the rules for binary
> packages
> > [1]. (And I would assume that applies regardless where we host the
> images.)
> >
> > Now to say that we only publish sources kind of side steps that problem.
> At
> > the same time it probably also defeats the purpose of the preview
> release,
> > which is to make it easier for folks that are not active contributors to
> > take the operator for a test drive.
> >
> > So I'm leaning towards publishing the images with respective NOTICE
> > requirements (for the layers that we add).
> >
> > We are also planning to publish the jar files in the future as it helps
> to
> > build clients and those would need to have the binary NOTICE also.
> >
> > Cheers,
> > Thomas
> >
> > [1] https://infra.apache.org/licensing-howto.html#binary
> >
> > On Mon, Mar 28, 2022 at 9:20 AM Gyula Fóra  wrote:
> >
> > > I see your point and the value for having such a notice added.
> > >
> > > I think there are 2 completely distinct questions at play here:
> > >
> > > a) Is there a legal requirement for a NOTICE file for the docker image?
> > > b) If not, should we block the release on this and add it immediately?
> > >
> > > For a)
> > > I think from a legal (and ASF policy) perspective there is one question
> > > that decides whether this is a requirement for this release or not:
> > > Is the docker image part of the release?
> > >
> > > I think the answer here is clearly no, the image is not part of the
> > > release. Only the Dockerfile is part of the release.
> > >
> > > For b)
> > > I think adding the NOTICE is a good idea but it will take some work so
> I
> > > would not block the preview release on it.
> > > If someone has some handy utility or experience generating it, I don't
> > mind
> > > including it in later RCs of course.
> > > Otherwise we can aim for the next release.
> > >
> > > Gyula
> > >
> > >
> > > On Mon, Mar 28, 2022 at 6:03 PM Chesnay Schepler 
> > > wrote:
> > >
> > > > One difference to Flink is that the distribution bundled in the
> docker
> > > > image still contains the NOTICE covering the contents of it.
> > > >
> > > > It may admittedly not be the most discoverable place, but a
> reasonable
> > > one
> > > > I think.
> > > >
> > > > Docker as a whole is very weird when it comes to licensing.
> > > > Think of all the things are are shipped in an image; I don't think we
> > can
> > > > (nor should) try to document everything in there.
> > > > For the most part this is also not necessary as the Flink images are
> > > based
> > > > on Debian,
> > > > where (al)most every installed package already embeds licensing
> > > > information into the image.
> > > >
> > > > However, for content that we copy into the image (i.e., the jars), I
> > > think
> > > > it would be reasonable to document that.
> > > > (and based on experience from the Flink side has also shown other
> > > > advantages beyond licensing...)
> > > >
> > > > On 28/03/2022 17:41, Gyula Fóra wrote:
> > > >
> > > > Thanks for the input!
> > > >
> > > > I am not an expert on this topic and have been contemplating this
> > myself
> > > > also.
> > > > We are basically trying to follow the precedent set by Flink and
> > Statefun
> > > > projects where the docker builds that we use to publish images to
> > > > dockerhub do not declare any notices.
> > > >
> > > > We will not use ghcr.io for the final release but will use dockerhub
> > > like
> > > > flink and other apache projects.
> > > >
> > > > If I look at it from a strictly technical point of view, the docker
> > image
> > > > is not part of the official release (as it's also not part of the
> > > > flink/statefun release).
> > > >
> > > > It would be good to get some input from others on this. It's not
> > > > impossible to add the notices but it's considerable work and
> > maintenance
> > > > overhead.
> > > > By extending the logic would you then also add license information
> for
> > > the
> > > > base images of the docker container (and so on so forth)?
> > > >
> > > > My gut feeling would be that we could highlight this in the NOTICE of
> > the
> > > > main project  (or some other appropriate place) but we do not
> > explicitly
> > > > list the dependencies.
> > > >
> > > > Would be good to hear how others feel about this!
> > > >
> > > > Gyula
> > > >
> > 

[jira] [Created] (FLINK-26904) Update load(...) of all Stage subclasses to use StreamTableEnvironment

[Dong Lin (Jira)]

Dong Lin created FLINK-26904:


 Summary: Update load(...) of all Stage subclasses to use 
StreamTableEnvironment
 Key: FLINK-26904
 URL: https://issues.apache.org/jira/browse/FLINK-26904
 Project: Flink
  Issue Type: Bug
Reporter: Dong Lin


Currently every Stage subclass uses static `load(StreamExecutionEnvironment, 
String)` to load model data from the given path. Algorithm developers are 
expected to use StreamExecutionEnvironment.create(env) to instantiate a new 
StreamTableEnvironment and uses it to create Table instances for model data.

This approach is problematic. Use KMeansModel as example. Users will use 
KMeansModel::load(env, path) to instantiate the model and call 
model.transform(inputDataTable) to do inference, where modelDataTable (created 
from load(...)) and inputDataTable are created using different 
StreamTableEnvironment instances. 

Having multiple Table instances in the same job where instances are created 
from different StreamTableEnvironment instances are in general error prone, as 
they can not sure information such as table catalog.

In order to fix this problem, we will need to consistently use 
StreamTableEnvironment for load(...) and similar public APIs in Flink ML.




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26905) Re-add FlinkDeploymentList and FlinkSessionJobList classes

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26905:
--

 Summary: Re-add FlinkDeploymentList and FlinkSessionJobList classes
 Key: FLINK-26905
 URL: https://issues.apache.org/jira/browse/FLINK-26905
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.0.0
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


Seems like removing these was a bad idea as it breaks the fabric8 java client 
when using the POJO classes



--
This message was sent by Atlassian Jira
(v8.20.1#820001)