Re: [VOTE] Release 1.14.4, release candidate #1

[Konstantin Knauf]

+1 (binding)

- checked signatures and checksums of binaries OK
- build from sources
- checked for suspicious changes in pom.xml between 1.14.3 and 1.14.4
- ran examples/python/table/batch/word_count.py in a local mini cluster
- ran examples/python/table/batch/word_count.py against standalone session
cluster

On Wed, Mar 9, 2022 at 3:07 PM Timo Walther  wrote:

> +1 (binding)
>
> - I scanned the commit diff and affected files.
> - I could not find major API changes or otherwise problematic changes.
> - The biggest changes I could spot were around Kubernetes (see
> FLINK-20830).
>
> Thanks for taking care of this Konstantin.
>
> Timo
>
>
> Am 02.03.22 um 10:04 schrieb Yun Tang:
> > +1 non-binding
> >
> > - Checked the signatures for pending release artifacts.
> > - Download the pre-built flink-dist of both scala_2.11 and scala_2.12
> versions and run them locally with the StateMachine example.
> > - Reviewed the flink-web PR
> >
> > Best
> > Yun Tang
> > 
> > From: Seth Wiesman 
> > Sent: Monday, February 28, 2022 23:02
> > To: dev 
> > Subject: Re: [VOTE] Release 1.14.4, release candidate #1
> >
> > +1 non-binding
> >
> > - built from source
> > - checked hashes and signatures
> > - started locally and deployed wordcount / stopped with savepoint /
> > restarted
> > - reviewed announcement post
> >
> > Thanks for managing the release!
> >
> > Seth
> >
> > On Fri, Feb 25, 2022 at 7:30 AM Konstantin Knauf 
> wrote:
> >
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #1 for the version
> 1.14.4,
> >> as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific comments)
> >>
> >>   The complete staging area is available for your review, which
> includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release and binary convenience releases to
> be
> >> deployed to dist.apache.org [2], which are signed with the key with
> >> fingerprint 8C3FB007FE60 DEFA [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag "release-1.14.4-rc1" [5],
> >> * website pull request listing the new release and adding announcement
> blog
> >> post [6].
> >>
> >> The vote will be open for at least 72 hours. It is adopted by majority
> >> approval, with at least 3 PMC affirmative votes.
> >>
> >> Thanks,
> >> Konstantin
> >>
> >> [1]
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351231
> >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.4-rc1/
> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1487/
> >> [5] https://github.com/apache/flink/tree/release-1.14.4-rc1
> >> [6] https://github.com/apache/flink-web/pull/510
> >>
> >> --
> >>
> >> Konstantin Knauf
> >>
> >> https://twitter.com/snntrable
> >>
> >> https://github.com/knaufk
> >>
>
>

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

[RESULT] [VOTE] Release 1.14.4, release candidate #1

[Konstantin Knauf]

Hi everyone,

I am pleased to announce that we have unanimously approved this release
candidate:

There are 5 approving votes, 3 of which are binding:
- Seth Wiesman (non-binding)
- Yun Tang (non-binding)
- Yun Gao (binding)
- Timo Walther (binding)
- Konstantin Knauf (binding)

There are no disapproving votes.

Thank you for verifying the release candidate. I will now proceed to
finalize the release and announce it once everything is published.

Cheers,
Konstantin

-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [RESULT] [VOTE] Release 1.14.4, release candidate #1

[Martijn Visser]

Thanks to all the testers and Konstantin for driving the release!

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82


On Fri, 11 Mar 2022 at 09:36, Konstantin Knauf  wrote:

> Hi everyone,
>
> I am pleased to announce that we have unanimously approved this release
> candidate:
>
> There are 5 approving votes, 3 of which are binding:
> - Seth Wiesman (non-binding)
> - Yun Tang (non-binding)
> - Yun Gao (binding)
> - Timo Walther (binding)
> - Konstantin Knauf (binding)
>
> There are no disapproving votes.
>
> Thank you for verifying the release candidate. I will now proceed to
> finalize the release and announce it once everything is published.
>
> Cheers,
> Konstantin
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[jira] [Created] (FLINK-26602) The Rocksdb task failed savepoint, and then checkpoint failed several times

[liuchenhong (Jira)]

liuchenhong created FLINK-26602:
---

 Summary: The Rocksdb task failed savepoint, and then checkpoint 
failed several times
 Key: FLINK-26602
 URL: https://issues.apache.org/jira/browse/FLINK-26602
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.11.2
Reporter: liuchenhong


The Rocksdb task failed savepoint (2022-03-10 19:55:**), and then checkpoint 
failed several times (2022-03-11)。Savepoint fails because it is Out Of Memory. 
But I'd like to know why checkpoint fails and why it goes “beyond physical 
Memory limits”. I checked the number of data sources and there was no exception 
. Could it be that savePoint failed, but memory was never freed?
{code:java}
//代码占位符
job manager log
2022-03-11 00:58:24,891 INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering 
checkpoint 108412 (type=CHECKPOINT) @ 1646931504738 for job 
d90b4aca73c5802e0dbbd50ca8af97e0.
2022-03-11 00:58:27,605 INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed 
checkpoint 108412 for job d90b4aca73c5802e0dbbd50ca8af97e0 (9815989304 bytes in 
2801 ms).
2022-03-11 01:00:06,603 INFO  org.apache.flink.yarn.YarnResourceManager         
           [] - Closing TaskExecutor connection 
container_e06_1603181034156_0493_01_23 because: Container 
[pid=177263,containerID=container_e06_1603181034156_0493_01_23] is running 
beyond physical memory limits. Current usage: 12.0 GB of 12 GB physical memory 
used; 14.3 GB of 25.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_e06_1603181034156_0493_01_23 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 177263 177261 177263 177263 (bash) 2 2 116015104 357 /bin/bash -c 
/usr/jdk64/jdk1.8.0_152/bin/java -Xmx2786359756 -Xms2786359756 
-XX:MaxDirectMemorySize=1744830464 -XX:MaxMetaspaceSize=268435456 -XX:+UseG1GC 
-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-Xloggc:/mnt/ssd/3/yarn/log/application_1603181034156_0493/container_e06_1603181034156_0493_01_23/jobmanager-gc.log
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=128M 
-Dlog4j2.formatMsgNoLookups=true 
-Dlog.file=/mnt/ssd/3/yarn/log/application_1603181034156_0493/container_e06_1603181034156_0493_01_23/taskmanager.log
 -Dlog4j.configuration=file:./log4j.properties 
-Dlog4j.configurationFile=file:./log4j.properties 
org.apache.flink.yarn.YarnTaskExecutorRunner -D 
taskmanager.memory.framework.off-heap.size=134217728b -D 
taskmanager.memory.network.max=1073741824b -D 
taskmanager.memory.network.min=1073741824b -D 
taskmanager.memory.framework.heap.size=134217728b -D 
taskmanager.memory.managed.size=6796786004b -D taskmanager.cpu.cores=1.0 -D 
taskmanager.memory.task.heap.size=2652142028b -D 
taskmanager.memory.task.off-heap.size=536870912b --configDir . 
-Djobmanager.rpc.address='' 
-Dweb.tmpdir='/tmp/flink-web-cd3b923f-86f9-463c-9524-40f357bd9afc' 
-Dsecurity.kerberos.login.keytab='/mnt/ssd/8/yarn/local/usercache/portal/appcache/application_1603181034156_0493/container_e06_1603181034156_0493_01_01/krb5.keytab'
 -Dweb.port='0' -Djobmanager.rpc.port='41239' -Drest.address='' 1> 
/mnt/ssd/3/yarn/log/application_1603181034156_0493/container_e06_1603181034156_0493_01_23/taskmanager.out
 2> 
/mnt/ssd/3/yarn/log/application_1603181034156_0493/container_e06_1603181034156_0493_01_23/taskmanager.err
 
    |- 177416 177263 177263 177263 (java) 484303004 122930506 15252447232 
3145560 /usr/jdk64/jdk1.8.0_152/bin/java -Xmx2786359756 -Xms2786359756 
-XX:MaxDirectMemorySize=1744830464 -XX:MaxMetaspaceSize=268435456 -XX:+UseG1GC 
-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-Xloggc:/mnt/ssd/3/yarn/log/application_1603181034156_0493/container_e06_1603181034156_0493_01_23/jobmanager-gc.log
 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=128M 
-Dlog4j2.formatMsgNoLookups=true 
-Dlog.file=/mnt/ssd/3/yarn/log/application_1603181034156_0493/container_e06_1603181034156_0493_01_23/taskmanager.log
 -Dlog4j.configuration=file:./log4j.properties 
-Dlog4j.configurationFile=file:./log4j.properties 
org.apache.flink.yarn.YarnTaskExecutorRunner -D 
taskmanager.memory.framework.off-heap.size=134217728b -D 
taskmanager.memory.network.max=1073741824b -D 
taskmanager.memory.network.min=1073741824b -D 
taskmanager.memory.framework.heap.size=134217728b -D 
taskmanager.memory.managed.size=6796786004b -D taskmanager.cpu.cores=1.0 -D 
taskmanager.memory.task.heap.size=2652142028b -D 
taskmanager.memory.task.off-heap.size=536870912b --configDir . 
-Djobmanager.rpc.address= 
-Dweb.tmpdir=/tmp/flink-web-cd3b923f-86f9-463c-9524-40f357bd9afc 
-Dsecurity.kerberos.login.keytab=/mnt/ssd/8/yarn/local/usercache/portal/appcache/application_1603181034156_0493/container_e06_1603181034156_0493_01_01

[jira] [Created] (FLINK-26603) [Umbrella] Decouple Hive with Flink itself

[luoyuxia (Jira)]

luoyuxia created FLINK-26603:


 Summary: [Umbrella] Decouple Hive with Flink itself
 Key: FLINK-26603
 URL: https://issues.apache.org/jira/browse/FLINK-26603
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive, Table SQL / Planner
Reporter: luoyuxia
 Fix For: 1.16.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26604) update AvroParquet format user-facing document

[Jing Ge (Jira)]

Jing Ge created FLINK-26604:
---

 Summary: update AvroParquet format user-facing document
 Key: FLINK-26604
 URL: https://issues.apache.org/jira/browse/FLINK-26604
 Project: Flink
  Issue Type: Improvement
Reporter: Jing Ge
Assignee: Jing Ge


* add minimal mvn dependency setup
 * describe the namespace use case in the Avro schema
 * reduce the redundant information w.r.t. the bunded/unbunded data



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Add libatomic1 to flink image

[Almeida, Julius]

Hi Developers,

I wish to add a new library to existing flink docker image since it’s a 
dependency requirement in my processor.
Is it possible to add?

apt-get install libatomic1

Thanks,
Julius

[jira] [Created] (FLINK-26605) JobManagerDeploymentStatus.READY does not correctly reflect the real Flink cluster status

[Yang Wang (Jira)]

Yang Wang created FLINK-26605:
-

 Summary: JobManagerDeploymentStatus.READY does not correctly 
reflect the real Flink cluster status
 Key: FLINK-26605
 URL: https://issues.apache.org/jira/browse/FLINK-26605
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Yang Wang


Follow the discussion in the PR[1].

{{JobManagerDeploymentStatus.READY}} does not mean the Flink cluster is ready 
for accepting REST API calls. This will cause problems when we are running 
session cluster.

For example, if something is wrong or very slow with the leader election, 
{{JobManagerDeploymentStatus}} is {{READY}} but the session cluster is not 
ready for accepting job submission. What I mean is to update the 
{{JobManagerDeploymentStatus}} to {{READY}} only when the flink cluster is 
actually working.

 

[1]. 
https://github.com/apache/flink-kubernetes-operator/pull/51#discussion_r824359419



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26606) CompletedCheckpoints that failed to be discarded are not stored in the CompletedCheckpointStore

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-26606:
-

 Summary: CompletedCheckpoints that failed to be discarded are not 
stored in the CompletedCheckpointStore
 Key: FLINK-26606
 URL: https://issues.apache.org/jira/browse/FLINK-26606
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl


We introduced a repeatable per-job cleanup after the job reached a 
globally-terminated state. It also tries to clean up the 
{{CompletedCheckpointStore}}. But we missed one code path where 
{{CompletedCheckpoints}} are tried to be discarded in the 
{{CheckpointsCleaner}}. The {{CompletedCheckpointStore}} does not hold any 
references to these {{CompletedCheckpoints}} anymore. The shutdown at the end 
is not able to clean these checkpoints up.

We should not remove the {{CompletedCheckpoints}} from the 
{{CompletedCheckpointStore}} if the deletion failed. This would enable us to 
retry deleting these artifacts at the end of the job and consider them in the 
retryable cleanup as well.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26607) There are multiple MAX_LONG_VALUE value errors in pyflink code

[zhangjingcun (Jira)]

zhangjingcun created FLINK-26607:


 Summary: There are multiple MAX_LONG_VALUE value errors in pyflink 
code
 Key: FLINK-26607
 URL: https://issues.apache.org/jira/browse/FLINK-26607
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.14.4
Reporter: zhangjingcun


There are multiple MAX_LONG_VALUE values sys. In pyflink code maxsize

MAX_LONG_VALUE = sys.maxsize

*maxsize* attribute of the *sys* module fetches the largest value a variable of 
data type *Py_ssize_t* ** can store. It is the Python platform’s pointer that 
dictates the maximum size of lists and strings in Python. The size value 
returned by maxsize depends on the platform architecture:
 * *32-bit:* the value will be 2^31 – 1, i.e. 2147483647
 * *64-bit:* the value will be 2^63 – 1, i.e. 9223372036854775807



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26609) Add a "sum" operator In PyFlink KeyedStream API

[CaoYu (Jira)]

CaoYu created FLINK-26609:
-

 Summary: Add a "sum" operator In PyFlink KeyedStream API
 Key: FLINK-26609
 URL: https://issues.apache.org/jira/browse/FLINK-26609
 Project: Flink
  Issue Type: New Feature
  Components: API / Python
Affects Versions: 1.14.3
Reporter: CaoYu


In Java, KeyedStream has a "sum" operator, But when i using in PyFlink. that 
operator is not found.

 

So i implemented it in Python Flink KeyedStream And test has passed.

 

But i don't know if this feature is necessary in PyFlink.

so i send a jira task in here, we can discuss it.

 

If that feature is necessary, i have already implemented that feature so i can 
summit a PR in github.

 

Thansks.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26608) Add a "sum" operator In PyFlink KeyedStream API

[CaoYu (Jira)]

CaoYu created FLINK-26608:
-

 Summary: Add a "sum" operator In PyFlink KeyedStream API
 Key: FLINK-26608
 URL: https://issues.apache.org/jira/browse/FLINK-26608
 Project: Flink
  Issue Type: New Feature
  Components: API / Python
Affects Versions: 1.14.3
Reporter: CaoYu


In Java, KeyedStream has a "sum" operator, But when i using in PyFlink. that 
operator is not found.

 

So i implemented it in Python Flink KeyedStream And test has passed.

 

But i don't know if this feature is necessary in PyFlink.

so i send a jira task in here, we can discuss it.

 

If that feature is necessary, i have already implemented that feature so i can 
summit a PR in github.

 

Thansks.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] FLIP-213: TaskManager's Flame Graphs

[Yufei Liu]

Hi Jacky,
I think it’s a very useful feature, can get flame graph easily and avoid
run script personally.
But I think can package async-profiler library into flink, and use JNI
execute the profiling, doesn’t depend on local environment.
We did this way in out internal version, and provide api to download
generated html file.

Jacky Lau <281293...@qq.com.invalid> 于2022年1月24日周一 16:48写道：

> Hi All,
>     I would like to start the discussion on FLIP-213 <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-213%3A+TaskManager%27s+Flame+Graphs>
> ;
>  which aims to provide taskmanager level(process level) flame graph
> by async profiler, which is most popular tool in java performance. and the
> arthas and intellij both use it. 
> And we support it in our ant group company.
>    And Flink supports FLIP-165: Operator's Flame Graphs
> now. and it draw flame graph by the front-end
> libraries d3-flame-graph, which has some problem in  jobs
> of large of parallelism.
>    Please be aware that the FLIP wiki area is not fully done
> since i don't konw whether it will accept by flink community. 
>    Feel free to add your thoughts to make this feature better! i
> am looking forward  to all your response. Thanks too much!
>
>
>
>
> Best Jacky Lau

[jira] [Created] (FLINK-26610) FileSink can not upgrade from 1.13 if the uid of the origin sink is not set.

[Gen Luo (Jira)]

Gen Luo created FLINK-26610:
---

 Summary: FileSink can not upgrade from 1.13 if the uid of the 
origin sink is not set.
 Key: FLINK-26610
 URL: https://issues.apache.org/jira/browse/FLINK-26610
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.15.0
Reporter: Gen Luo
 Fix For: 1.15.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Flink 1.15 Stabilisation Sync skipped tomorrow

[Gabor Somogyi]

Any news on the branch cut? The listed jira-s are closed as I see.

G


On Tue, Mar 8, 2022 at 3:02 PM Galen Warren  wrote:

> You're welcome. Fwiw, I've also run through testing steps on my
> contribution -- [FLINK-26451] Test GCS FileSystem w/ RecoverableWriter
> manually - ASF JIRA (apache.org)
>  -- but I don't think
> I'm supposed to be officially doing manually testing on my own stuff.
>
> On Tue, Mar 8, 2022 at 3:46 AM Johannes Moser  wrote:
>
> > Thanks so much Gabor and Galen.
> >
> > Very much appreciated.
> >
> > > On 07.03.2022, at 19:00, Galen Warren  wrote:
> > >
> > > I just ran through the testing steps on [FLINK-26462] Release Testing:
> > > Running Python UDF in different Execution Mode - ASF JIRA (apache.org)
> > > .
> > >
> > > Everything worked but I think there might be a typo in the steps to set
> > up
> > > the tests, I left a comment on the issue.
> > >
> > > On Mon, Mar 7, 2022 at 12:01 PM Gabor Somogyi <
> gabor.g.somo...@gmail.com
> > >
> > > wrote:
> > >
> > >> Since I know this area I help to test FLINK-26468
> > >> .
> > >>
> > >> G
> > >>
> > >> On Mon, Mar 7, 2022 at 5:15 PM Johannes Moser 
> > wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> We will skip tomorrows weekly stabilisation sync.
> > >>> It looks like all blockers are assigned and worked on. A significant
> > >>> number of contributors are on holiday tomorrow.
> > >>>
> > >>> Currently we plan to cut off the 1.15 branch on Wednesday of Thursday
> > >> this
> > >>> week.
> > >>>
> > >>> In case you want to contribute something to the release there are
> some
> > >>> features to perform manual tests:
> > >>> * https://issues.apache.org/jira/browse/FLINK-26451 <
> > >>> https://issues.apache.org/jira/browse/FLINK-26451>
> > >>> * https://issues.apache.org/jira/browse/FLINK-26462 <
> > >>> https://issues.apache.org/jira/browse/FLINK-26462>
> > >>> * https://issues.apache.org/jira/browse/FLINK-26468 <
> > >>> https://issues.apache.org/jira/browse/FLINK-26468>
> > >>>
> > >>>
> > >>> Best, Yun Gao and Joe
> > >>
> >
> >
>

Re: Add libatomic1 to flink image

[Konstantin Knauf]

Moving dev@ to bcc, adding user@

Hi Julius,

the recommended approach would be to build your own Docker images from the
official images along the lines of

FROM apache/flink:1.14.3
RUN apt-get install -y libatomic1

Cheers,

Konstantin


On Fri, Mar 11, 2022 at 11:07 AM Almeida, Julius
 wrote:

> Hi Developers,
>
> I wish to add a new library to existing flink docker image since it’s a
> dependency requirement in my processor.
> Is it possible to add?
>
> apt-get install libatomic1
>
> Thanks,
> Julius
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

[jira] [Created] (FLINK-26611) Generate documentation for OperatorConfigOptions

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26611:
--

 Summary: Generate documentation for OperatorConfigOptions
 Key: FLINK-26611
 URL: https://issues.apache.org/jira/browse/FLINK-26611
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Gyula Fora


Similar to how other flink configs are documented we should use the maven 
plugin to generate the doc content for the operator specific configs



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26612) Create docs webpages like other Flink projects

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26612:
--

 Summary: Create docs webpages like other Flink projects
 Key: FLINK-26612
 URL: https://issues.apache.org/jira/browse/FLINK-26612
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Gyula Fora


We should create the necessary documents webpages so we can integrate with 
other flink websites like flink, statefun, ml do.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26613) Stateful unified Sink V2 upgrades only work when operator uids are given

[Fabian Paul (Jira)]

Fabian Paul created FLINK-26613:
---

 Summary: Stateful unified Sink V2 upgrades only work when operator 
uids are given
 Key: FLINK-26613
 URL: https://issues.apache.org/jira/browse/FLINK-26613
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.0
Reporter: Fabian Paul
 Fix For: 1.15.0


As part of documentation [1]we guarantee that in case a stateful migration 
fails and no uids are used that it is possible to bind the recovered state via 
the uidHash. 

This ticket should add a uidHash setter for the operators that are affected by 
the migration (writer, committer, global committer).

 

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#preconditions



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Martijn Visser

[Marios Trivyzas]

Congrats Martijn!!

Best,
Marios

On Thu, Mar 10, 2022 at 4:50 PM yu'an huang  wrote:

> Congrats, Martijn!
>
>
> Best
> Yuan
>
>
> > On 3 Mar 2022, at 11:49 PM, Robert Metzger  wrote:
> >
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Martijn Visser as a new
> > Flink committer.
> >
> > Martijn is a very active Flink community member, driving a lot of efforts
> > on the dev@flink mailing list. He also pushes projects such as replacing
> > Google Analytics with Matomo, so that we can generate our web analytics
> > within the Apache Software Foundation.
> >
> > Please join me in congratulating Martijn for becoming a Flink committer!
> >
> > Cheers,
> > Robert
>
>

-- 
Marios

[jira] [Created] (FLINK-26614) Upsert Kafka SQL Connector:Support startup mode of timestamp and specific offsets

[Ning Ma (Jira)]

Ning Ma created FLINK-26614:
---

 Summary: Upsert Kafka SQL Connector:Support startup mode of 
timestamp and specific offsets
 Key: FLINK-26614
 URL: https://issues.apache.org/jira/browse/FLINK-26614
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.14.4
Reporter: Ning Ma
 Fix For: 1.14.4


When we run Flink SQL using upsert-kafka connector, it cannot support startup 
mode of timestamp or specific offsets. It is determined to read data from the 
earliest.

I don't think it's reasonable. Because the data stored in Kafka is incomplete 
that it only saves data for 7 days in most cases. So why don't we consume data 
from the specified timestamp or offsets, and this can improve the efficiency of 
reading data from Kafka.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26615) BatchingStateChangeUploadSchedulerTest.testRetryOnTimeout fails on azure

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-26615:
-

 Summary: BatchingStateChangeUploadSchedulerTest.testRetryOnTimeout 
fails on azure
 Key: FLINK-26615
 URL: https://issues.apache.org/jira/browse/FLINK-26615
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends, Tests
Affects Versions: 1.15.0
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.15.0


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32896&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=24724]

{code}
[ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.103 
s <<< FAILURE! - in 
org.apache.flink.changelog.fs.BatchingStateChangeUploadSchedulerTest
[ERROR] 
org.apache.flink.changelog.fs.BatchingStateChangeUploadSchedulerTest.testRetryOnTimeout
  Time elapsed: 0.042 s  <<< FAILURE!
java.lang.AssertionError: expected:<[0]> but was:<[]>
   at org.junit.Assert.fail(Assert.java:89)
   at org.junit.Assert.failNotEquals(Assert.java:835)
   at org.junit.Assert.assertEquals(Assert.java:120)
   at org.junit.Assert.assertEquals(Assert.java:146)
   at 
org.apache.flink.changelog.fs.BatchingStateChangeUploadSchedulerTest.testRetryOnTimeout(BatchingStateChangeUploadSchedulerTest.java:240)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
   at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
   at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
   at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
   at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
   at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
   at 
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
   at 
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
   at 
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
   at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrato
 r.java:107)
   at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrato
 r.java:88)
{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26616) AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI failed with a timeout

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-26616:
-

 Summary: 
AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI failed 
with a timeout
 Key: FLINK-26616
 URL: https://issues.apache.org/jira/browse/FLINK-26616
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl


{{AdaptiveSchedulerITCase.}} failed in [this 
build|https://dev.azure.com/mapohl/flink/_build/results?buildId=855&view=logs&j=0a15d512-44ac-5ba5-97ab-13a5d066c22c&t=9a028d19-6c4b-5a4e-d378-03fca149d0b1&l=5778]
 due to a timeout.
{code}
Mar 11 14:41:36 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 76.177 s <<< FAILURE! - in 
org.apache.flink.test.scheduling.AdaptiveSchedulerITCase
Mar 11 14:41:36 [ERROR] 
org.apache.flink.test.scheduling.AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI
  Time elapsed: 60.146 s  <<< ERROR!
Mar 11 14:41:36 java.util.concurrent.TimeoutException: Condition was not met in 
given timeout.
Mar 11 14:41:36 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:167)
Mar 11 14:41:36 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
Mar 11 14:41:36 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:137)
Mar 11 14:41:36 at 
org.apache.flink.test.scheduling.AdaptiveSchedulerITCase.testExceptionHistoryIsRetrievableFromTheRestAPI(AdaptiveSchedulerITCase.java:268)

{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Preview release for Flink Kubernetes Operator

[Gyula Fóra]

Hi Team!

I would like to discuss the timeline for the initial preview/milestone
release of the flink-kubernetes-operator
 project.

The last few weeks we have been working very hard with the community to
stabilize the initial feature set and I think we have made great progress.
While we are still far from a production ready-state, a preview release
will give us the opportunity to reach more people and gather much needed
input to take this project to the next level.

There are still a couple missing features that we need to iron out and we
need to make sure we have proper documentation but after that I think it
would be a good time for the preview release.

I propose to aim for the first release candidate around the 25-27th of
March after which we should dedicate a few days for some extensive testing
and bugfixing.

What do you think?

Gyula