[jira] [Created] (FLINK-35872) Fix the incorrect partition generation during period refresh in Full Mode
Feng Jin created FLINK-35872: Summary: Fix the incorrect partition generation during period refresh in Full Mode Key: FLINK-35872 URL: https://issues.apache.org/jira/browse/FLINK-35872 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Gateway Affects Versions: 1.20.0 Reporter: Feng Jin During the period refresh, we currently default to using scheduler time to generate the target partition, but we should use the T - 1 partition instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] CLI action deprecation process
> > As I think about it, most likely orthogonal to the current > discussion, but this idea seems to be applicable for the > submitted job JARs as well. The same check could be done for all > the submitted files and their deps. IDK if that was your original > thought Xintong or not? > IIUC, you mean applying the strict mode for the programming APIs as well? I'm not entirely sure about this. The programming APIs are much less stable compared to CLI interfaces. There are APIs being deprecated in almost every release. My concern is that this may end up with users never turning the strict mode on due to too many breaks. I'm a bit leaning towards starting with only applying the strict mode to CLI interfaces, and see how it goes. But I'm also open to other opinions. Best, Xintong On Wed, Jul 17, 2024 at 7:59 PM Ferenc Csaky wrote: > Hi Xintong, Muhammet, > > Thank you for your thoughts! > > I agree with Xintong regarding 1), and 2). > > About making users aware of these compatibility guarantees, > documentation and CLI print deprecation is what I already aimed > for when I was merging the "run" and "run-application" actions > functionality [1], but I missed the CLI page, which is a very good > point to include. I will address that in a follow-up change. > > I also like the idea of a strict, and compatibility mode. I guess > that should kick in when a "@Public" AND "@Deprecated" annotation > combination is present on the called CLI action. > > With something like this at hand, I think applying the same > deprecation process and annotations for CLI actions that we already > have would make sense. Regarding the strict/compatibility strategy > itself, the straightforward and easy way IMO: > > - Strict mode: In case a @Public entity gets @Deprecated, execution > breaks right away. > - Compatibility mode: This should allow execution until the code > exists. > > I am not sure if bringing in more complexity would be beneficial, > as it would also make it harder for the users to understand. > > As I think about it, most likely orthogonal to the current > discussion, but this idea seems to be applicable for the > submitted job JARs as well. The same check could be done for all > the submitted files and their deps. IDK if that was your original > thought Xintong or not? > > Best, > Ferenc > > [1] > https://github.com/apache/flink/commit/e56b54db40a2afac420d8d8952707c2644ba633a > > > > On Tuesday, 9 July 2024 at 06:02, Xintong Song > wrote: > > > > > > > I think there are three questions to be anwsered, and here are my two > cents. > > > > 1) How do we define the compatibility guarantees for cli interfaces? > > > > I'd be +1 for reusing the existing @Public and @PublicEvolving > annotations, > > as suggested by Ferenc. Having multiple sets of rules in the same project > > may easily confuse users. > > > > 2) What deprecation process is required for cli interfaces? > > > > If we are reusing the existing annotations, I think we'd better to have > the > > same deprecation process as well, for the same reason not to confuse > users. > > > > 3) How do we make user aware of the compatibility guarantees and > > deprecations of cli interfaces? > > > > I think there are several things that we can do. > > - In addition to codes and JavaDocs, we should also display the > annotations > > (Public / PublicEvolving / Experimental Dprecated) in documentation [1] > and > > cli helps (things you see when execute `/bin/flink -h`). > > > > - Print a warning message if a deprecated command / argument is used, as > > suggested by Muhammet. > > - We can also provide a strict / compatible mode. E.g., we use strict > mode > > by default, which fails if any deprecated interface is used. In the error > > message, we hint user that he/she can manually enable the compatible > mode, > > which allows deprecated interfaces. This should draw users' attention to > > the deprecation of the interfaces, while not block the adoption of a new > > Flink version with breaking changes. There are probably more details to > be > > considered, e.g., should we fail immediately once an interface is > > deprecated in strict mode, how long do we support a deprecated interface > in > > compatible mode, how to properly suggest users about the compatible mode > > while not encourage them to always stay in that mode, etc. > > > > Best, > > > > Xintong > > > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/ > > > > > > > > On Mon, Jul 8, 2024 at 5:54 PM Muhammet Orazov > > mor+fl...@morazow.com.invalid wrote: > > > > > Hey Ferenc, > > > > > > Yes correct. My thoughts were based on finding tradeoff between > > > following fully deprecation process and leaner one for CLIs. > > > > > > Since cli are not like APIs, I think users would be aware of > > > deprecation only when were remove the commands. That is they > > > try to run their jobs with upgrade and it fails with action > > > not available. > > > > > > So maybe we don't hav
Re: [VOTE] Release 1.20.0, release candidate #1
Hi, weijie -1 (non-binding) During our testing process, we discovered a critical bug that impacts the correctness of the materialized table. A fix pr [1] is now prepared and will be merged within the next two days. I apologize for any inconvenience during the release process. [1]. https://issues.apache.org/jira/browse/FLINK-35872 Best, Feng On Fri, Jul 19, 2024 at 5:45 PM Xintong Song wrote: > +1 (binding) > > - reviewed flink-web PR > - verified checksum and signature > - verified source archives don't contain binaries > - built from source > - tried example jobs on a standalone cluster, and everything looks fine > > Best, > > Xintong > > > > On Thu, Jul 18, 2024 at 4:25 PM Rui Fan <1996fan...@gmail.com> wrote: > > > +1(binding) > > > > - Reviewed the flink-web PR (Left some comments) > > - Checked Github release tag > > - Verified signatures > > - Verified sha512 (hashsums) > > - The source archives don't contain any binaries > > - Build the source with Maven 3 and java8 (Checked the license as well) > > - Start the cluster locally with jdk8, and run the StateMachineExample > job, > > it works fine. > > > > Best, > > Rui > > > > On Mon, Jul 15, 2024 at 10:59 PM weijie guo > > wrote: > > > > > Hi everyone, > > > > > > > > > Please review and vote on the release candidate #1 for the version > > 1.20.0, > > > > > > as follows: > > > > > > > > > [ ] +1, Approve the release > > > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > The complete staging area is available for your review, which includes: > > > > > > * JIRA release notes [1], and the pull request adding release note for > > > > > > users [2] > > > > > > * the official Apache source release and binary convenience releases to > > be > > > > > > deployed to dist.apache.org [3], which are signed with the key with > > > > > > fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [4], > > > > > > * all artifacts to be deployed to the Maven Central Repository [5], > > > > > > * source code tag "release-1.20.0-rc1" [6], > > > > > > * website pull request listing the new release and adding announcement > > blog > > > > > > post [7]. > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > [1] > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354210 > > > > > > [2] https://github.com/apache/flink/pull/25091 > > > > > > [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc1/ > > > > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > > [5] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1744/ > > > > > > [6] https://github.com/apache/flink/releases/tag/release-1.20.0-rc1 > > > > > > [7] https://github.com/apache/flink-web/pull/751 > > > > > > > > > Best, > > > > > > Robert, Rui, Ufuk, Weijie > > > > > >
Re: [DISCUSS] FLIP-XXX Amazon SQS Source Connector
Hi Dhingra, thanks for driving this. I am not very familiar with SQS, but this should be some kind of message queue. So could we add metric of currentFetchEventTimeLag and currentEmitEventTimeLag from FLIP-33[1] in SQS source? and I want to know what metrics do we provide in SQS source. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics Saurabh Singh 于2024年7月20日周六 02:43写道： > Hi Fink Devs, > > Our team has been working on migrating various data pipelines to Flink to > leverage the benefits of exactly-once processing, checkpointing, and > stateful computing. We have several use cases built around the AWS SQS > Service. For this migration, we have developed an SQS Source Connector, > which enables us to run both stateless and stateful Flink-based jobs. > > We believe that this SQS Source Connector would be a valuable addition to > the existing connector set. Therefore, we propose a FLIP to include it. > > For more information, please refer to the FLIP document. > > > https://docs.google.com/document/d/1lreo27jNh0LkRs1Mj9B3wj3itrzMa38D4_XGryOIFks/edit?usp=sharing > > Thanks > Saurabh & Abhi >
Re: [VOTE] FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2
Thanks Wencong for driving this proposal! +1(binding) Best, Rui On Fri, Jul 19, 2024 at 3:58 PM Xintong Song wrote: > +1(binding) > > Best, > > Xintong > > > > On Thu, Jul 18, 2024 at 9:40 AM weijie guo > wrote: > > > +1(binding) > > > > Best regards, > > > > Weijie > > > > > > Wencong Liu 于2024年7月17日周三 21:31写道： > > > > > Hi dev, > > > > > > I'd like to start a vote on FLIP-466. > > > > > > Discussion thread: > > > https://lists.apache.org/thread/sw2or62299w0hw9jm5kdqjqj3j8rnrdt > > > FLIP: > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-466%3A+Introduce+ProcessFunction+Attribute+in+DataStream+API+V2 > > > > > > Best regards, > > > Wencong Liu > > >
