Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

Congratulations Zhijiang! Good work! Best, Andrey On Tue, Jul 23, 2019 at 1:40 PM Hequn Cheng wrote: > Congratulations Zhijiang! > > Best, Hequn > > On Tue, Jul 23, 2019 at 6:17 PM JingsongLee .invalid> > wrote: > > > Congratulations Zhijiang! > > > > Best, Jingsong Lee > > > > > > ---

Re: [Requirement] CI report

Hi, @Tison I guess it was done because there were many associated comments notified by everyone involved in PR. I agree that it would be actually useful for a PR's author to get notifications about failed/succeeded CI builds, not sure how easy it is though. Best, Andrey On Tue, Jul 23, 2019 at

[DISCUSS][CODE STYLE] Create collections always with initial capacity

Hi all, As you probably already noticed, Stephan has triggered a discussion thread about code style guide for Flink [1]. Recently we were discussing internally some smaller concerns and I would like start separate threads for them. This thread is about creating collections always with initial cap

[DISCUSS][CODE STYLE] Usage of Java Optional

Hi all, This is the next follow up discussion about suggestions for the recent thread about code style guide in Flink [1]. In general, one could argue that any variable, which is nullable, can be replaced by wrapping it with Optional to explicitly show that it can be null. Examples are: - ret

[DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

Hi all, This is one more small suggestion for the recent thread about code style guide in Flink [1]. We already have a note about using a new line for each chained call in Scala, e.g. either: *values**.stream()**.map(...)**,collect(...);* or *values* *.stream()* *.map(*...*)* *.col

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

pache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroFactory.java#L95 On Thu, Aug 1, 2019 at 6:00 PM Andrey Zagrebin wrote: > Hi all, > > This is the next follow up discussion about suggestions for the recent > thread about code sty

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

+1 (non-binding) Tested in AWS EMR Yarn: 1 master and 4 worker nodes (m5.xlarge: 4 vCore, 16 GiB). EMR runs only on Java 8. Fine-grained recovery is enabled by default. Modified E2E test scripts can be found here (asserting output): https://github.com/azagrebin/flink/commits/FLINK-13597 Batch S

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

t; >> > >> Congratulations Andrey! >> > >> >> > >> On Wed, Aug 14, 2019 at 10:14 PM chaojianok >> wrote: >> > >> >> > >> > Congratulations Andrey! >> > >> > At 2019-08-14 21:26:37, "Till Rohrm

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

n the > > same > > > line > > > > > > That way, parentheses are aligned in a similar way to braces and > > exceptions > > > can be well aligned. > > > > > > *public **void func(* > > > *int arg1,* > > > *int arg2,* &g

Re: [DISCUSS][CODE STYLE] Create collections always with initial capacity

It’s not about whether we can set it, but whether it makes sense > to do so (to avoid the before mentioned "maintenance burden”). > > Piotrek > > > On 1 Aug 2019, at 14:41, Xintong Song wrote: > > > > +1 on setting initial capacity only when have good expectat

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

gt;> use of @Nullable/@Nonnull annotation or Flink's > >>>>>> SerializableOptional. It would be sufficient. > >>>>>> > >>>>>> Best, > >>>>>> tison. > >>>>>> > >>>&

Re: [DISCUSS][CODE STYLE] Create collections always with initial capacity

add this to the code style? > > On Mon, Aug 19, 2019 at 11:51 AM Andrey Zagrebin > wrote: > > > Hi All, > > > > It looks like this proposal has an approval and we can conclude this > > discussion. > > Additionally, I agree with Piotr we should really force

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

t; > Just wanted to quickly elaborate on my opinion. I wouldn't say I am -1, > > > just -0 for the Optionals in private methods. I am ok with not > > > forbidding them there. I just think in all cases there is a better > > > solution than passing the Optionals a

Re: [DISCUSS][CODE STYLE] Create collections always with initial capacity

FYI the PR: https://github.com/apache/flink-web/pull/249 A review from the discussion participants would be appreciated. On Tue, Aug 20, 2019 at 5:29 PM Andrey Zagrebin wrote: > I created an umbrella issue for the code style guide effort and a subtask > for this discussion: &

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

8.19 um 11:57 schrieb Andrey Zagrebin: > > Hi All, > > > > It looks like we have reached a consensus regarding the last left > question. > > > > I suggest the following final summary: > > > > - Use @Nullable annotation where you do not use Option

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

elopers can break before that if they feel it helps with > > > readability. > > > > > > This should be similar to what we have, except for enforcing the line > > > length limit. > > > > > > I think our Java guide originally suggested 120 ch

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

declaration and function body is clear. > > > Zili Chen 于2019年8月22日周四 上午9:53写道： > > > Thanks Andrey for driving the discussion. Just for clarification, > > what we conclude here are several guidelines without automatic > > checker/tool guard them, right? > > >

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

FYI PR: https://github.com/apache/flink-web/pull/254 On Thu, Aug 22, 2019 at 10:11 AM Andrey Zagrebin wrote: > Hi Tison, > > Regarding the automatic checks. > Yes, I suggest we conclude the discussion without the automatic checks. > As soon as we have more ideas/investig

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

Hi All, @Xitong thanks a lot for driving the discussion. I also reviewed the FLIP and it looks quite good to me. Here are some comments: - One thing I wanted to discuss is the backwards-compatibility with the previous user setups. We could list which options we plan to deprecate. From

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks for starting the vote Xintong Also +1 for the proposed FLIP-49. @Stephan regarding namings: network vs shuffle. My understanding so far was that the network memory is what we basically give to Shuffle implementations and default netty implementation uses it in particular mostly for network

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

or this. > > See > > > above. > > > > > > @Zhijiang > > > - Thanks for the input. My understanding is that 'shuffle memory' is a > > > portion of the task executor memory reserved for the shuffle component. > > The > &g

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Thanks for updating the FLIP Xintong. It looks good to me. I would be ok to start a vote for it. Best, Andrey On Wed, Sep 4, 2019 at 10:03 AM Xintong Song wrote: > @all > > The FLIP document [1] has been updated. > > Thank you~ > > Xintong Song > > > [1] > > https://cwiki.apache.org/confluence/

Re: [VOTE] FLIP-53: Fine Grained Operator Resource Management

Thanks for starting the vote @Xintong +1 for the FLIP-53 Best, Andrey On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann wrote: > Hi Xintong, > > thanks for starting this vote. The proposal looks good and, hence, +1 for > it. > > One comment I have is concerning the first implementation step. I woul

Re: [DISCUSS] FLIP-56: Dynamic Slot Allocation

Hi Xintong, Thanks for sharing the implementation steps. I also think they makes sense with the feature option. I was wondering if we could order the steps in a way that each change does not affect other components too much, always having a working system then maybe the feature option does not al

Re: [DISCUSS] FLIP-56: Dynamic Slot Allocation

But if it > > cannot be easily done, I would rather not to put too much efforts on > having > > a good abstraction and deduplication between the new code path and the > > original one that we are removing soon. > > > > What do you think? > > > > Thank you~

Re: [DISCUSS] FLIP-56: Dynamic Slot Allocation

then it would make step 6 easier. We can start to > have > > some IT/E2E tests, with the default slot resource profiles being > available. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Sep 16, 2019 at 9:50 PM Andrey Zagrebin >

Re: [VOTE] FLIP-56: Dynamic Slot Allocation

Hi Xintong, Thanks for starting the vote, +1 from my side. Best, Andrey On Tue, Sep 17, 2019 at 4:26 PM Xintong Song wrote: > Hi all, > > I would like to start the vote for FLIP-56 [1], on which a consensus is > reached in this discussion thread [2]. > > The vote will be open for at least 72 h

Re: [VOTE] Release 1.10.0, release candidate #2

Hi Benchao, Do you observe this issue FLINK-15938 with 1.9 or 1.10? If with 1.9, I suggest to check with 1.10. Thanks, Andrey On Thu, Feb 6, 2020 at 4:07 PM Benchao Li wrote: > Hi all, > > I found another issue[1], I don't know if it should be a blocker. But it > does affects joins without win

Re: [VOTE] Release 1.10.0, release candidate #2

. We can move to the > Jira ticket to discuss further more. > > Andrey Zagrebin 于2020年2月6日周四 下午11:30写道： > > > Hi Benchao, > > > > Do you observe this issue FLINK-15938 with 1.9 or 1.10? > > If with 1.9, I suggest to check with 1.10. > > > > Thanks,

Re: [VOTE] Release 1.10.0, release candidate #3

Hi, @Jingsong Lee Regarding "OutOfMemoryError: Direct buffer memory" in FileChannelBoundedData$FileBufferReader I saw you created a specific issue issue: https://issues.apache.org/jira/browse/FLINK-15981 In general, I think we could rewrap this error in MemorySegmentFactory#allocateUnpooledOffHea

[DISCUSS] FLIP-111: Docker image unification

Hi All, If you have ever touched the docker topic in Flink, you probably noticed that we have multiple places in docs and repos which address its various concerns. We have prepared a FLIP [1] to simplify the perception of docker topic in Flink by users. It mostly advocates for an approach of exte

Re: [DISCUSS] FLIP-111: Docker image unification

onvenient for users without any learning curve. >> `docker run flink session_jobmanager -D rest.bind-port=8081` >> >> >> > About the logging >> >> Updating the `log4j-console.properties` to support multiple appender is a >> better option. >> Currently,

Re: [DISCUSS] Releasing Flink 1.10.1

Hi Yu, Thanks for kicking off the 1.10.1 release discussion! Apart from - FLINK-16406 Increase default value for JVM Metaspace to minimise its OutOfMemoryError which should be merged soon I think we should also try to get in the following issues: - [FLINK-16225] Metaspace Out Of Memory should b

[DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

Hi All, As you may have noticed, 1.10 release included an extensive improvements to memory management and configuration of Task Managers, FLIP-49: [1]. The memory configuration of Job Managers has not been touched in 1.10. Although, Job Manager's memory model does not look so sophisticated as for

Re: [DISCUSS] Releasing Flink 1.10.1

xperience when using SQL CLI >>> >>> Best, >>> Jark >>> >>> >>> On Wed, 11 Mar 2020 at 20:37, Yu Li wrote: >>> >>>> Thanks for the suggestion Andrey! I've added 1.10.1 into FLINK-16225 >> fix >>>> versions

Re: [DISCUSS] Releasing Flink 1.10.1

Ewen wrote: > > @Andrey about the increase in metaspace size > - I have no concerns for 1.11.0. > - For 1.10.1 I am not completely sure, because users expect to upgrade > that without config adjustments. That might not be possible with that > change. > > On Thu, Mar

Re: [DISCUSS] Releasing Flink 1.10.1

probably not much). > > > > > > > > Thank you~ > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > On Thu, Mar 12, 2020 at 10:09 PM Stephan Ewen > > wrote: > > > > > > >

Re: [DISCUSS] FLIP-111: Docker image unification

> > > I would second that it is desirable to support Java 11 and in general > use a base image that allows the (straightforward) use of more recent > versions of other software (Python etc.) > > > > > https://github.com/apache/flink-docker/blob/d3416e720377e9b4c07a2d0f4

Re: Question about RocksDBStateBackend Compaction Filter state cleanup

Hi Lake, When the Flink doc mentions a state entry in RocksDB, we mean one key/value pair stored by user code over any keyed state API (keyed context in keyed operators obtained e.g. from keyBy() transformation). In case of map or list, the doc means map key/value and list element. - value/aggreg

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

o trigger GC in most cases, and the max > direct > > memory limit can act as a safe net. Moreover, I think the cases should be > > very rare that we need native memory for user codes. Therefore, we > probably > > should not break the JM/TM consistency for potential risks i

[VOTE] FLIP 116: Unified Memory Configuration for Job Managers

Hi All, The discussion for FLIP-116 looks to be resolved [1]. Therefore, I start the vote for it. The vote will end at 6pm CET on Monday, 23 March. Best, Andrey [1] http://mail-archives.apache.org/mod_mbox/flink-dev/202003.mbox/%3CCAJNyZN7AJAU_RUVhnWa7r%2B%2BtXpmUqWFH%2BG0hfoLVBzgRMmAO2w%40mail.

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

fulfilled. We could experiment with what is the working minimum but It is hard to come up with this limit because it again can depend on the job and environment. Best, Andrey On Wed, Mar 18, 2020 at 5:03 PM Andrey Zagrebin wrote: > Hi all, > > Thanks for the feedback, Xintong

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

p is not fulfilled. As already >> mentioned, it would be hard to determine the min heap size. And if we make >> the min heap configurable, then in any case that users need to configure >> the min heap, they can configure the heap size directly. >> >> Thank you~ >>

Re: [DISCUSS] FLIP-111: Docker image unification

l > configuration options (e.g., flink_docker_utils set_web_ui_port 8081). It > should be fine enough to do it via flink_docker-utils conifgure rest.port > 8081 if we cannot solve it via the general configuration mechanism. > > Cheers, > Till > > On Wed, Mar 18, 2020 a

Re: [VOTE] FLIP 116: Unified Memory Configuration for Job Managers

:16 PM Gary Yao wrote: > +1 (binding) > > Best, > Gary > > On Wed, Mar 18, 2020 at 3:16 PM Andrey Zagrebin > wrote: > > > Hi All, > > > > The discussion for FLIP-116 looks to be resolved [1]. > > Therefore, I start the vote for it. > > The vote

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

Hi All, Thanks for this FLIP, Yadong. This is a very good improvement to the Flink's UI. It looks like there are still couple of things to resolve before the final vote. - I also find the non-heap title in configuration confusing because there are also other non-heap types of memory. The "off-hea

Re: [DISCUSS] FLIP-111: Docker image unification

Hi everyone, Patrick and Ufuk, thanks a lot for more ideas and suggestions! I have updated the FLIP according to the current state of discussion. Now it also contains the implementation steps and future follow-ups. Please, review if there are any concerns. The order of the steps aims for keeping

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

Hi guys, Thanks for more details Zhijiang. It also looks to me that mapped memory size is mostly driven by OS limits and bit-ness of JVM (32/64). Thinking more about the 'Metrics' tab layout, couple of more things have come into my mind. # 'Metrics' tab -> 'Memory': 'Metrics' and 'Configuration'

Re: [DISCUSS] FLIP-111: Docker image unification

t;> >on the application this should 'just work'. >> > >> > Additional note: >> > >> >1. Job/Task attempt logging instead of task manager logging. >> >*I realize this has nothing to do with the docker images* >> >I found som

[VOTE] FLIP-111: Docker image unification

Hi All, As discussed in these threads [1] and [2], we suggest to unify the docker topic in Flink for users [3]. This mainly means refactoring of the existing code and introducing more docs as a first step. The effort should enable further improvements and follow-ups for the docker integration wit

Re: [VOTE] FLIP-111: Docker image unification

日周二 下午11:04写道： > > > > > > > > > Thanks for driving this effort Andrey. > > > > > > > > > > +1 (binding) > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On

Re: [VOTE] Accept Stateful Functions into Apache Flink

+1 (binding) On Tue, Oct 29, 2019 at 10:09 AM Igal Shilman wrote: > +1 (non-binding) > > Thanks, > Igal Shilman > > On Sat, Oct 26, 2019 at 12:25 AM Ufuk Celebi wrote: > > > +1 (binding) > > > > On Fri, Oct 25, 2019 at 6:39 PM Maximilian Michels > wrote: > > > > > +1 (binding) > > > > > > On 2

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

Congrats Becket! On Tue, Oct 29, 2019 at 6:11 PM Xuannan Su wrote: > Congratulations Becket! :-) > > Xuannan > > > On Oct 28, 2019, at 6:07 AM, Fabian Hueske wrote: > > > > Hi everyone, > > > > I'm happy to announce that Becket Qin has joined the Flink PMC. > > Let's congratulate and welcome B

Re: [VOTE] Accept Stateful Functions into Apache Flink

sorry, my +1 was non-binding, confused that it was not a committer vote but PMC. On Wed, Oct 30, 2019 at 4:43 PM Chesnay Schepler wrote: > +1 (binding) > > On 30/10/2019 15:25, Vasiliki Kalavri wrote: > > +1 (binding) from me. I hope this is not too late :) > > > > Thank you for this great contr

Re: set job level TTL

Hi Chen, if I understand correctly, your question is not about the state with TTL, correct me if I am wrong. We could consider adding this feature to Flink: run x time => save point => shutdown job but it seems to me that it is something quite application specific, not sure how high priority it w

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

Congrats Zhu Zhu! On Mon, Dec 16, 2019 at 8:10 AM Xintong Song wrote: > Congratulations Zhu Zhu~ > > Thank you~ > > Xintong Song > > > > On Mon, Dec 16, 2019 at 12:34 PM Danny Chan wrote: > > > Congrats Zhu Zhu! > > > > Best, > > Danny Chan > > 在 2019年12月14日 +0800 AM12:51，dev@flink.apache.org，写

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

Hi Stephan and Xintong, Thanks for the further FLIP-49 feedbacks. > - "taskmanager.memory.size" (old main config option) is replaced by > "taskmanager.memory.total-process.size" which has a different meaning in > standalone setups. The old option did not subtract metaspace and other > overhe

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

t; much. >>>>> - We do not have a good way to know the exact size needed for >>>>> jvm overhead / metaspace and framework / task off-heap memory, thus >>>>> having >>>>> to conservatively reserve slightly more memory th

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

ds >> compatibility for standalone setups more important than for active setups? >> >> IMO, "taskmanager.memory.process.size" is easier for the new comers. For >> standalone setups, users can simply configure it to their machines' >> available memory size, wi

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

ell in most cases. > > > > - In the JVM world, users are used to configuring the heap size and > know > > that there is additional memory overhead. The > > "taskmanager.memory.flink.size" option fits well with that mindset. > > > > - One you start to th

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Hi Michaël, Thanks for mentioning this, users can benefit from this for sure. This API is not exposed (although there is some code prepared for this) because community believes that it would require more extensive design and discussion before the implementation, basically another FLIP. Although

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

It also looks to me that we should only swap network and memory in the option names: 'taskmanager.memory.network.*'. There is no strong consensus towards using new 'shuffle' naming so we can just rename it to 'taskmanager.memory.network.*' as 'shuffle' naming has never been released. When we have

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

; > > > Thank you~ > > > > Xintong Song > > > > > > > > On Wed, Jan 8, 2020 at 9:39 PM Andrey Zagrebin > > wrote: > > > > > It also looks to me that we should only swap network and memory in the > > > option

Re: [Discuss] Tuning FLIP-49 configuration default values.

Hi all! Great that we have already tried out new FLIP-49 with the bigger jobs. I am also +1 for the JVM metaspace and overhead changes. Regarding 0.3 vs 0.4 for managed memory, +1 for having more managed memory for Rocksdb limiting case. In general, this looks mostly to be about memory distribu

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

ephan Ewen > wrote: > > > > > > > > > I think we need an interpretation of "-tm" regardless of what is in > > the > > > > > default configuration, because we can always have a modified > > > > configuration > > > >

Re: [Discuss] Tuning FLIP-49 configuration default values.

the config.yaml. > Maybe we don't need to double it, but something like 1280m would be okay? > > On Tue, Jan 14, 2020 at 3:47 PM Andrey Zagrebin > wrote: > > > Hi all! > > > > Great that we have already tried out new FLIP-49 with the bigger jobs. > >

Re: [DISCUSS] decentralized scheduling strategy is needed

HI HuWeihua, I think your issue should resolve with 1.9.2 and 1.10 (not released but in progress). You can check the related Jira ticket [1]. Best, Andrey [1] https://jira.apache.org/jira/browse/FLINK-12122 On Wed, Jan 15, 2020 at 10:08 AM HuWeihua wrote: > Hi, All > We encountered some probl

Re: [DISCUSS] Change default for RocksDB timers: Java Heap => in RocksDB

Hi Stephan, Thanks for starting this discussion. I am +1 for this change. In general, number of timer state keys can have the same order as number of main state keys. So if RocksDB is used for main state for scalability, it makes sense to have timers there as well unless timers are used for only v

Re: Increasing Disk Read Throughput and IOPS

Hi, I just wanted to add that if you are using EBS you could consider to switch to IO provisioned type of it (io1: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html ) if it is ok from the cost prospe

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi everybody, We have been recently brainstorming ideas around state TTL in Flink and compiled our thoughts in the following design doc: https://docs.google.com/document/d/1SI_WoXAfOd4_NKpGyk4yh3mf59g12pSGNXRtNFi-tgM

Re: Ordering of stream from different kafka partitions

Hi Amol, I think you could try (based on your stack overflow code) org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor like this: DataStream streamSource = env .addSource(kafkaConsumer) .setParallelism(4) .assignTimestampsAndWatermarks(

Re: Ordering of stream from different kafka partitions

Hi Amol, I think you could try (based on your stack overflow code) org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor like this: DataStream streamSource = env .addSource(kafkaConsumer) .setParallelism(4) .assignTimestampsAndWatermarks( new

Re: Ordering of stream from different kafka partitions

? > > --- > *Amol Suryawanshi* > Java Developer > am...@iprogrammer.com > > > *iProgrammer Solutions Pvt. Ltd.* > > > > *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society, > Bahiratwadi,Near Hot

Re: Ordering of stream from different kafka partitions

iott, Off Senapati Bapat Road, Pune - 411016, > MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer* > www.iprogrammer.com <http://www.iprogrammer.com/> <mailto:sac...@iprogrammer.com>> > > > On W

Re: [VOTE] Release 1.6.0, release candidate #3

Hi everyone, After careful consideration, I think that it makes sense to change the order of serialisation for state values with TTL. It will simplify further optimisations and feature improvements. If it is still possible, I would ask to include this PR to the release candidate: https://githu

Re: Why documentation always say checkpoint does not support Flink specific features like rescaling.

As I understand, this is more about the documented semantics of checkpoint feature that Flink community agreed to guarantee for users so far in the last releases. Although technically the rescaling is currently possible for full and incremental checkpoints, it can change in future if checkpoints

Re: [ANNOUNCE] New committer Gary Yao

Congratulations Gary! > On 7 Sep 2018, at 15:45, Stefan Richter wrote: > > Congrats Gary! > >> Am 07.09.2018 um 15:14 schrieb Till Rohrmann : >> >> Hi everybody, >> >> On behalf of the PMC I am delighted to announce Gary Yao as a new Flink >> committer! >> >> Gary started contributing to the

Re: Question over Incremental Snapshot vs Full Snapshot in rocksDb state backend

Hi Chandan, > 1. Why did we took 2 different approaches using different RocksDB apis ? > We could have used Checkpoint api of RocksDB for fullSnapshot as well . The reason here is partially historical. Full snapshot in RocksDB backend was implemented before incremental and rescaling for incremen

Re: [DISCUSS] Proposal of external shuffle service

Hi Zhijiang, Thanks for sharing the document Zhijiang. I decided to compile my thoughts to consider here, not to overload document comments any more :) I think I still have question about job level configuration for the shuffle service. You mentioned that we can keep several shuffle manager ob

Re: [DISCUSS] Proposal of external shuffle service

for the implementation aspects. And I agree with the above points, > especially for above 2.2 you might need double check if I understand > correctly. > Wish your further feedbacks then I can adjust the docs based on it. Also > welcome any other person's feedbacks! > > > Best, > Zhijiang &

Re: [DISCUSS] Proposal of external shuffle service

rnal shuffle cases, they can reuse the same unified netty-based > shuffle service if we wrap the related componenets into current shuffle > service well. If we want to furtherextend other implementations of shuffle > service, like http-based shuffle service, then we can define an interface > for it, the way as current RpcService interface

[SURVEY] Custom RocksDB branch

Dear Flink users and developers! I start this discussion to collect feedback about maintaining a custom RocksDB branch for Flink, if anyone sees any problems with this approach. Are there people who already uses a custom RocksDB client build with RocksDB state backend? As you might already know,

Re: [RESULT][VOTE] Clean up and reorganize the JIRA components

Hi all, Thank you for such useful effort, Robert! I would also suggest a separate tag for state TTL, like queryable state. There might be not so many issues atm but it might help in future. Thanks, Andrey On Mon, Feb 25, 2019 at 8:02 PM Robert Metzger wrote: > The proposal has been approved w

[Proposal] Shuffle resources lifecycle management

Hi All, While working on pluggable shuffle architecture, looking into interactive programming and fine-grained recovery efforts, we released that lifecycle management of intermediate result partitions needs more detailed discussion to enable envisioned use cases. Here I prepared a design document

[Discuss] Semantics of event time for state TTL

Hi All, As you might have already seen there is an effort tracked in FLINK-12005 [1] to support event time scale for state with time-to-live (TTL) [2]. While thinking about design, we realised that there can be multiple options for semantics of this feature, depending on use case. There is also so

Re: [DISCUSS] A more restrictive JIRA workflow

Hi all, +1 for option 2. I also do not mind option 2, after 1-2 contributions, any contributor could just ask the committer (who merged those contributions) about contributor permissions. Best, Andrey On Wed, Apr 10, 2019 at 3:58 AM Robert Metzger wrote: > I'm +1 on option 1. > > On Tue, Apr

Re: [Discuss][FLINK-8297]A solution for FLINK-8297 Timebased RocksDBListState

Hi Faxian, Thanks for thinking on this new approach. Here are my thoughts: - In case of event time, although, this approach changes semantics of original list state, it could be a good fit for certain use cases. The main advantage is that it is deterministic in event time. The list should end up

Re: [DISCUSS] A more restrictive JIRA workflow

, > Feng > > Le lun. 15 avr. 2019 à 09:17, Robert Metzger a > écrit : > > > @Andrey: You mention "option 2" two times, I guess one of the two uses of > > "option 2" contains a typo? > > > > On Wed, Apr 10, 2019 at 10:33 AM Andrey Zagrebin &g

Re: [Discuss][FLINK-8297]A solution for FLINK-8297 Timebased RocksDBListState

Hi Faxian, True, we can resolve timestamp conflicts putting values into the same row, good point. Then re-ordering in case of internal clock jump changes behaviour comparing with the list state we have now. In this case, it can be similar to dispersing elements by hash and we can call it a bag, no

Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing

+1 Best, Andrey On Tue, Sep 8, 2020 at 2:16 PM Yu Li wrote: > +1 > > Best Regards, > Yu > > > On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek wrote: > > > +1 > > > > We just need to make sure to find a good name before the release but > > shouldn't block any work on this. > > > > Aljoscha > > >

Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing

For the option name, maybe: *flink.main* or *flink.managed* (this may be a bit confusing for existing users as we said that the overall managed memory is managed by Flink) On Wed, Sep 9, 2020 at 9:56 AM Andrey Zagrebin wrote: > +1 > > Best, > Andrey > > On Tue, Sep 8, 202

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

Thanks a lot for this nice UI guys! +1 and for closed issues that is just because many steps have been already done. Best, Andrey On Fri, Oct 23, 2020 at 11:12 AM Till Rohrmann wrote: > Thanks for reviving this Flip Yadong! The changes look good to me and the > new memory UI looks awesome :-)

Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation

Hi lsyldliu, You can try to tune the StateTtlConfig. As the documentation suggests [1] the TTL incremental cleanup can decrease the per record performance. This is the price of the automatic cleanup. If the only thing, which happens mostly in your operator, is working with state then even checking

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

Welcome to committers and congrats, Xintong! Cheers, Andrey On Fri, Jun 5, 2020 at 4:22 PM Till Rohrmann wrote: > Congratulations! > > Cheers, > Till > > On Fri, Jun 5, 2020 at 10:00 AM Dawid Wysakowicz > wrote: > > > Congratulations! > > > > Best, > > > > Dawid > > > > On 05/06/2020 09:10, ti

[REMINDER] Use 'starter' labels for Jira issues where it makes sense

Hi Flink Devs, I would like to remind you that we have a 'starter' label [1] to annotate Jira issues which need a contribution and which are not very complicated for the new contributors. The starter issues can be a good opportunity for the new contributors who want to learn about Flink but do not

Re: Adding a new "Docker Images" component to Jira

+1 for the consolidation Best, Andrey On Fri, Aug 7, 2020 at 3:38 PM Till Rohrmann wrote: > +1 for unifying Deployment / Docker, Dockerfiles and Release System / > Docker into Docker. > > Cheers, > Till > > On Fri, Aug 7, 2020 at 12:18 PM Robert Metzger > wrote: > > > Hi all, > > > > we now ha

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

Hi All, Thanks for reviving the discussion, Matthias! This would mean that we could adapt the current proposal to replace the > Nonheap usage pane by a pane displaying the Metaspace usage. > I do not know the value of having the Nonheap usage in metrics. I can see that the metaspace metric can be

Re: How jobmanager and task manager communicates with each other ?

Hi Sidhant, (1) If we are not using Flink's HA services then how we can dynamically > configure task manager nodes to connect to job manager? Any suggestions or > best practices? Not sure what you mean by 'dynamically'. I think you have to restart the task manager with the new configuration to co

[jira] [Created] (FLINK-13169) IT test for fine-grained recovery (task executor failures)

Andrey Zagrebin created FLINK-13169: --- Summary: IT test for fine-grained recovery (task executor failures) Key: FLINK-13169 URL: https://issues.apache.org/jira/browse/FLINK-13169 Project: Flink

[jira] [Created] (FLINK-13371) Release partitions in JM of producer gets restarted

Andrey Zagrebin created FLINK-13371: --- Summary: Release partitions in JM of producer gets restarted Key: FLINK-13371 URL: https://issues.apache.org/jira/browse/FLINK-13371 Project: Flink

  1   2   3   >