Re: [VOTE] FLIP-454: New Apicurio Avro format
+1 I believe expanding the supported list of schema registries is a valuable addition. -- Kind regards Nic Townsend Sofware Developer – IBM Event Automation Slack: @nictownsend X: @nict0wnsend From: David Radley Date: Thursday, 2 May 2024 at 10:43 To: dev@flink.apache.org Subject: [EXTERNAL] [VOTE] FLIP-454: New Apicurio Avro format Hi everyone, I'd like to start a vote on the FLIP-454: New Apicurio Avro format [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format [2] https://lists.apache.org/thread/wtkl4yn847tdd0wrqm5xgv9wc0cb0kr8 Kind regards, David. Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
RE: Flink technical debt.
Hi Robert, David I am very much interested in a backlog review call – personally for two reasons: 1. Learning a bit more about the state of Flink, understanding how each issue would be triaged (and understanding what/why the severity is) 2. Seeing if there are any obvious places that I can help, it’s often easier for me hearing people describe the problem -- Nic Townsend IBM Event Processing Senior Engineer / Technical Lead Slack: @nictownsend From: Robert Metzger Date: Wednesday, 6 November 2024 at 15:22 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Flink technical debt. Hey David, Thanks a lot for this initiative. What do you think about setting up a call weekly or every second week, open to the public, where we collaboratively review a backlog of Jira tickets and triage them into: - Closing, - following up, - assigning a committer owner? The purpose of the call is not to hide anything from anybody, it is just for quicker collaboration. All decisions from the calls need to end up in Jira and on the mailing lists On Mon, Nov 4, 2024 at 4:07 PM David Radley wrote: > Hello, > I have been looking at the Flink Jira and git. I see a large number of > Flink Jira issues that are open and critical or blockers > https://issues.apache.org/jira/browse/FLINK-36655?jql=project%20%3D%20FLINK%20AND%20priority%20in%20(Blocker%2C%20Critical) > I realise some of these issues may not actually be critical as they have > been labelled by the submitter. > > I see 1239 open unmerged PRs > https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen. Some of these > are not associated with assigned issues, so may never be merged. This > amount of unmerged PRs, means that many people have put a lot of time and > effort into creating code that has not made it into the codebase, so they > do not get the credit for the contribution, which must be disheartening and > the codebase does not get the benefit of the contribution. > > This is a large amount of technical debt. I would like to help address > this problem by setting up a workgroup, with others in the community who > would like this addressed. The scope of the workgroup would be to improve > these numbers by activities such as: > > * Triaging PRs so it is easier for committers to merge or close them. > * Identifying prs that could be closed out as no longer relevant. > * Getting committer buy in. > > Are there other ideas from the community around how this could be improved > with or without a workgroup, or whether the existing processes should be > sufficient or enhanced? > > Is there an appetite to address this in the community? I am happy to drive > this as a community workgroup, with my team in IBM, if there is community > support. > > We could call the community workgroup ?Community Health Initiative? CHI to > energise the Flink community. > > WDYT? > > Kind regards, David. > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN > Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
Java truststores
Hi, I have a proposal for a Flink operator change – but I’m unsure whether it’s a) valuable, and b) large enough for a FLIP vs a Jira issue. Scenario: * I am using a selection of connectors that will communicate via TLS * I would like to be able to connect to both internal and external services (i.e some use company issued certificates, others are signed by public authorities) * I don’t want to have to configure each connector with PEM certificates (and then edit the program whenever a certificate expires) The problem is that all I believe I can do is override the Flink JVM truststore to point to a truststore with the internal CA certificates: env.java.opts.taskmanager: >- -Djavax.net.ssl.trustStore=/certs/truststore. -Djavax.net.ssl.trustStorePassword= env.java.opts.jobmanager: >- -Djavax.net.ssl.trustStore=/certs/truststore. -Djavax.net.ssl.trustStorePassword= The problem here is that it overrides the default cacerts truststore in Java – so I cannot connect to externally signed endpoints. I know that I can merge cacerts and my truststore into a new truststore as a manual process. But it means that I have to remember to extract the cacerts every time I have a minor java change. Proposed scenario: * New property on the CR for a secret containing an “external truststore” (possibly an “external keystore” as well for MTLS) * The flink operator deploys Flink with an init container and mounts the secrets * The init container then combines the default cacerts on the Flink job/task manager with the “external truststore” and supplies that to the Flink runtime This means that you can enable connecting to both internal and externally signed endpoints from a Flink job. Thoughts as to whether this is big enough for a FLIP (if valuable)? -- Nic Townsend IBM Event Processing Senior Engineer / Technical Lead Slack: @nictownsend Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
RE: Java truststores
Hi Gabor – thank you for the prompt reply! I appreciate that building a new Flink image means the Dockerfile would allow me to merge certs into the JVM truststore at build time. However, the reason for the question is that our company (IBM) has a monthly release cycle which includes new Flink images (maybe due to Java version change, dependency change etc.) I would prefer that all our customers do not have to re-build Flink images each month to add in the external certificates. If this was something the operator supported, they could update to new versions without needing to build a new image each time. In addition, it would also mean that in the case of certificate expiry, you would just update the single k8s secret and then stop/resume clusters for the new certificates to be picked up. I hope that clarifies the usecase? -- Nic Townsend IBM Event Processing Senior Engineer / Technical Lead Slack: @nictownsend From: Gabor Somogyi Date: Thursday, 19 December 2024 at 11:07 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Java truststores Hi Nic, Not sure I understand all of the road blockers but truststore is normally not per application but per company/org which is relatively static (or at least not short lived). Keystores are normally propagated through volume mounts. Is it not possible to import the truststore into the JVM truststore[1] ? Such can be done during docker image creation... [1] https://connect2id.com/blog/importing-ca-root-cert-into-jvm-trust-store BR, G On Thu, Dec 19, 2024 at 11:43 AM Nic Townsend wrote: > Hi, I have a proposal for a Flink operator change – but I’m unsure whether > it’s a) valuable, and b) large enough for a FLIP vs a Jira issue. > > Scenario: > > * I am using a selection of connectors that will communicate via TLS > * I would like to be able to connect to both internal and external > services (i.e some use company issued certificates, others are signed by > public authorities) > * I don’t want to have to configure each connector with PEM > certificates (and then edit the program whenever a certificate expires) > > The problem is that all I believe I can do is override the Flink JVM > truststore to point to a truststore with the internal CA certificates: > > env.java.opts.taskmanager: >- >-Djavax.net.ssl.trustStore=/certs/truststore. >-Djavax.net.ssl.trustStorePassword= > env.java.opts.jobmanager: >- >-Djavax.net.ssl.trustStore=/certs/truststore. >-Djavax.net.ssl.trustStorePassword= > > The problem here is that it overrides the default cacerts truststore in > Java – so I cannot connect to externally signed endpoints. > > I know that I can merge cacerts and my truststore into a new truststore as > a manual process. But it means that I have to remember to extract the > cacerts every time I have a minor java change. > > Proposed scenario: > > * New property on the CR for a secret containing an “external > truststore” (possibly an “external keystore” as well for MTLS) > * The flink operator deploys Flink with an init container and mounts > the secrets > * The init container then combines the default cacerts on the Flink > job/task manager with the “external truststore” and supplies that to the > Flink runtime > > This means that you can enable connecting to both internal and externally > signed endpoints from a Flink job. > > Thoughts as to whether this is big enough for a FLIP (if valuable)? > > -- > > Nic Townsend > IBM Event Processing > Senior Engineer / Technical Lead > > Slack: @nictownsend > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN > Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
RE: [VOTE] FLIP-532: Donate GetInData HTTP Connector to Flink
+ 1 (non-binding) Particularly excited by the idea of the community backing to help mature the connector towards other RPC protocols as well (e.g websockets source/sink) -- Nic Townsend IBM Event Processing Architect / Senior Engineer Slack: @nictownsend Bluesky: @nict0wnsend.bsky.social From: Sharath Date: Friday, 6 June 2025 at 18:32 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [VOTE] FLIP-532: Donate GetInData HTTP Connector to Flink +1 (non-binding) On Fri, Jun 6, 2025 at 9:38 AM Poorvank Bhatia wrote: > +1 (non-binding) > > On Fri, Jun 6, 2025 at 9:39 PM David Radley > wrote: > > > Hi everyone, > > > > I'd like to start a vote on FLIP-532: Donate GetInData HTTP Connector to > > Flink [1], which has been discussed in this thread [2]. The formal > > discussion thread [3] did not have any more input. > > > > The vote will be open for at least 72 hours. > > > > kind regards, David > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-532%3A+Donate+GetInData+HTTP+Connector+to+Flink > > [2] > > https://lists.apache.org/thread/fdpkhqxpf0mo4gdpjwnz4yhposlxycgf > > [3] > > https://lists.apache.org/thread/vcfj5xgd1olj4cph6rtys01wzdtsvryd > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > > Winchester, Hampshire SO21 2JN > > > Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
RE: Re: Flink Kubenetes Operator client version
Hi Gyula, Tom I appreciate that the details are a bit wooly – but what class of incompatibility problems were you envisioning? My first guess Is it that the operator maybe might start to put annotations/labels etc on resources that then conflict with the native integration? That said – our reasons for upgrading are to support kube 1.33 – which I realise means we have to upgrade both; it’s not worth having an operator on 1.33 that deploys a Flink native that won’t work! Thanks, Nic On 2025/07/08 16:06:19 Gyula Fóra wrote: > Hi David! > > I think we should definitely pick up the latest version if possible, we > have to make sure that this is compatible with how Flink uses the client > for native integration though. > > Cheers > Gyula > > On Tue, Jul 8, 2025 at 6:02 PM David Radley > mailto:da...@uk.ibm.com>> wrote: > > > Hi, > > I can see that the Flink operator uses the 6.13.2 Kubenetes client . We > > are looking to move up to use 7.20 to pick up security fixes. I see > > https://github.com/fabric8io/kubernetes-client/blob/main/doc/MIGRATION-v7.md > > . > > > > Any thoughts on the impact and whether there is a community support for > > this upgrade? > >Kind regards, David. > > > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > > Winchester, Hampshire SO21 2JN > > > -- Nic Townsend IBM Event Processing Architect / Senior Engineer Slack: @nictownsend Bluesky: @nict0wnsend.bsky.social Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
[jira] [Created] (FLINK-38162) Use of SQL functions with time attributes causes downstream temporal functions to fail
Nic Townsend created FLINK-38162:
Summary: Use of SQL functions with time attributes causes
downstream temporal functions to fail
Key: FLINK-38162
URL: https://issues.apache.org/jira/browse/FLINK-38162
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 1.20.0
Reporter: Nic Townsend
In the Flink docs for window join -
[https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-join/#innerleftrightfull-outer]
- the outer join example does not include the use of {{window_time.}}
This means that you cannot perform a window aggregate for example on the result
of the join - perhaps for example you want to count how many rows came from the
left, right, or both in a given window.
If you use the {{COALESCE}} like in the example, then although a {{DESCRIBE}}
will show
{code:java}
+--++--+-++--+
| name | type | null | key | extras |
watermark |
+--++--+-++--+
| window_time | TIMESTAMP(3) *ROWTIME* | true | || | {code}
it will not work with a Window TVF and throws an error that {{window_time}} is
not a time attribute.
Even worse - if you build the following statement set (pseduo):
* CREATE left table
* CREATE right table
* CREATE temporary view joined
* CREATE temporary view aggregate
* CREATE blackhole joined_sink
* CREATE blackhole aggregate_sink
* INSERT into joined_sink
* INSERT into aggregate_sink
Then you end up with a stack trace from the planner of:
{code:java}
java.lang.RuntimeException: Error while applying rule ProjectToCalcRule, args
[rel#7305:LogicalProject.NONE.any.None:
0.[NONE].[NONE](input=RelSubset#7304,exprs=[$4, $5, $6, $0, $1])] at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
~[flink-table-planner_2.12-1.20.1.jar:1.20.1]
[...]
Caused by: java.lang.IllegalArgumentException: Type mismatch:
rel rowtype: RecordType(TIMESTAMP(3) windowStartTime, TIMESTAMP(3)
windowEndTime, TIMESTAMP(3) windowResultTime, VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" user_id, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" product_id)
NOT NULL
equiv rowtype: RecordType(TIMESTAMP(3) windowStartTime, TIMESTAMP(3)
windowEndTime, TIMESTAMP(3) *ROWTIME* windowResultTime, VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" user_id, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
product_id) NOT NULL
Difference:
windowResultTime: TIMESTAMP(3) -> TIMESTAMP(3) *ROWTIME* {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
