Thanks for the feedbacks, @Stephan
- There is a push to make as much as possible configurable via the main > configuration, and not only in code. Specifically values for operations and > tuning. > I think it would be more important to have such memory weights in the > config, compared to in the program API. /cc Aljoscha I can see the benefit that having memory weights in the main configuration makes tuning easier, which makes great sense to me. On the other hand, what we lose is the flexibility to have different weights for jobs running in the same Flink cluster. It seems to me the problem is that we don't have an easy way to overwrite job-specific configurations without touching the codes. Given the current status, what if we make the memory weights configurable through both the main configuration and the programming API? The main configuration should take effect iff the weights are not explicitly specified through the programming API. In this way, job cluster users can easily tune the weight through the main configuration, while session cluster users, if they want to have different weights for jobs, can still overwrite the weight through execution configs. - My recommendation would be to keep this as simple as possible. This > will make a lot of configuration code harder, and make it harder for users > to understand Flink's memory model. > Making things as easy for users to understand is very important in my > opinion. In that regard, the main proposal in the FLIP seems better than > the alternative proposal listed at the end of the FLIP page. +1 from my side. - For the simplicity, we could go even further and simply have two memory > users at the moment: The operator algorithm/data-structure and the external > language process (Python for now). > We never have batch algos and RocksDB mixed, having this as separate > options is confusing as it suggests this can be combined arbitrarily. I > also think that a slim possibility that we may ever combine this in the > future is not enough reason to make it more complex/confusing. Good point. +1 for combining batch/rocksdb weights, for they're never mixed together. We can even just name it "StateBackend-BatchAlgorithm" for explicitly. For "external language process", I'm not entirely sure. Future external languages are possibly mixed with python processes. To avoid later considering how to share external language memory across different languages, I would suggest to present the concept as "python memory" rather than "external language process memory". Thank you~ Xintong Song On Sun, Aug 30, 2020 at 10:19 PM Stephan Ewen <se...@apache.org> wrote: > Thanks for driving this proposal. A few thoughts on the current design: > > - There is a push to make as much as possible configurable via the main > configuration, and not only in code. Specifically values for operations and > tuning. > I think it would be more important to have such memory weights in the > config, compared to in the program API. /cc Aljoscha > > - My recommendation would be to keep this as simple as possible. This > will make a lot of configuration code harder, and make it harder for users > to understand Flink's memory model. > Making things as easy for users to understand is very important in my > opinion. In that regard, the main proposal in the FLIP seems better than > the alternative proposal listed at the end of the FLIP page. > > - For the simplicity, we could go even further and simply have two memory > users at the moment: The operator algorithm/data-structure and the external > language process (Python for now). > We never have batch algos and RocksDB mixed, having this as separate > options is confusing as it suggests this can be combined arbitrarily. I > also think that a slim possibility that we may ever combine this in the > future is not enough reason to make it more complex/confusing. > > - I am also not aware of any plans to combine the network and operator > memory. Not that it would be infeasible to do this, but I think this would > also be orthogonal to this change, and I am not sure this would be solved > with static weights. So trying to get network memory into this proposal > seems pre-mature to me. > > Best, > Stephan > > > On Fri, Aug 28, 2020 at 10:48 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > > > > > A quick question, does network memory treated as managed memory now? Or > > in > > > the future? > > > > > No, network memory is independent from managed memory ATM. And I'm not > > aware of any plan to combine these two. > > > > Any insights there? > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Fri, Aug 28, 2020 at 4:35 PM Kurt Young <ykt...@gmail.com> wrote: > > > > > A quick question, does network memory treated as managed memory now? Or > > in > > > the future? > > > > > > Best, > > > Kurt > > > > > > > > > On Wed, Aug 26, 2020 at 5:32 PM Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > Hi devs, > > > > > > > > I'd like to bring the discussion over FLIP-141[1], which proposes how > > > > managed memory should be shared by various use cases within a slot. > > This > > > is > > > > an extension to FLIP-53[2], where we assumed that RocksDB state > backend > > > and > > > > batch operators are the only use cases of managed memory for > streaming > > > and > > > > batch jobs respectively, which is no longer true with the > introduction > > of > > > > Python UDFs. > > > > > > > > Please notice that we have not reached consensus between two > different > > > > designs. The major part of this FLIP describes one of the candidates, > > > while > > > > the alternative is discussed in the section "Rejected Alternatives". > We > > > are > > > > hoping to borrow intelligence from the community to help us resolve > the > > > > disagreement. > > > > > > > > Any feedback would be appreciated. > > > > > > > > Thank you~ > > > > > > > > Xintong Song > > > > > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility > > > > > > > > [2] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > > > > > > > > > >
