+1 and thanks for volunteering for an initial implementation. Looking forward to auto-scaling of network buffers.
On Tue, Feb 7, 2017 at 3:04 AM, Ufuk Celebi <u...@apache.org> wrote: > I fully agree with you Greg. > > Since this is doomed to get out of sync again very shortly after clean up, > I vote to automate this. Stephan introduced the ConfigOption type, which > makes it easy to define the options. It's already planned to migrate all > configuration options from ConfigConstants to this approach. > > For an example see here: https://github.com/apache/flink/blob/master/ > flink-core/src/main/java/org/apache/flink/configuration/ > HighAvailabilityOptions.java > > I think that it is possible to build the configuration docs page from this > with reasonable effort. > > This would translate the task to: > 1) Automate ConfigOption to HTML/Markdown generation > 2) Extend ConfigOption with description fields > 3) Migrate ConfigConstants to ConfigOptions > > I would also volunteer to take a first stab at this. > > Regarding the network buffers: +1 to your suggestion. Nico (cc'd) is > starting to work on automating the network buffer configuration in order to > get rid of any manual tuning for most users (because of the issues you > described + streaming and batch jobs require different tuning, which > complicates things even more). > > – Ufuk > > On 6 February 2017 at 19:21:28, Greg Hogan (c...@greghogan.com) wrote: > > > Hi devs, > > > > Flink's Configuration page [1] has grown intimidatingly long > > and complex. > > Options are described across three main sections: common options > > (single > > section), advanced options (multiple sections), and full reference. > > The > > trailing "background" section further describes the most impactful > > options > > in much greater detail. > > > > Several recent tickets, and a few outstanding, have added missing > > options > > to the configuration documentation. I'd like to propose a goal > > of > > organizing all options in the full reference into alphabetized, > > tabular > > form (one table per section), much like the system metrics [2]. > > Columns > > would be option name, description, and default value. > > > > The common and advanced sections could also be converted to tabular > > form > > with the exception of Kerberos-based Security. Missing options > > would be > > added to the full reference. > > > > Lastly, the simple heuristic for configuring network buffers > > has prompted > > many questions on the mailing list. With the 1.3 release the total > > and > > number of available buffers is reported through metrics and > > in the web > > dashboard. My experience has been that the number of required > > buffers is > > highly dependent on job topology and cluster performance. I > > propose keeping > > the simple heuristic and description while directing users > > to monitor the > > balance of available buffers. > > > > Greg > > > > [1] https://ci.apache.org/projects/flink/flink-docs- > master/setup/config.html > > [2] > > https://ci.apache.org/projects/flink/flink-docs- > master/monitoring/metrics.html#system-metrics > > [3] > > https://ci.apache.org/projects/flink/flink-docs- > master/setup/config.html#configuring-the-network-buffers > >
