The downside of change to the default behavior is the breakage of existing k8s or other container-based production setups.
Logs that are emitted to stdout are visible through kubectl logs and infrastructure is setup for log collection. Thanks, Thomas On Tue, May 5, 2020 at 6:31 AM Arvid Heise <ar...@ververica.com> wrote: > From my previous experience with K8s, I'd assume that the cluster itself > already has some ELK attached to it and all stdout/err is collected > automatically. > > So if you want to also add logs, I'd make that configurable and I'm torn > what the default should be as both make sense. > > However, thinking a bit further, it sounds as if the current way is already > very inconsistent. We should probably either have log files for everything > or log everything to stdout/err and let K8s deal with it. And then we > should have some configuration to toggle between the modes. > > On Tue, May 5, 2020 at 2:50 PM Stephan Ewen <se...@apache.org> wrote: > > > @Patrick could you chime in? > > > > We should at least understand the original motivation before simply > > changing the way it works. > > > > On Tue, May 5, 2020 at 2:49 PM Stephan Ewen <se...@apache.org> wrote: > > > > > I think Patrick originally introduced the foreground mode, and I > believe > > > it had indeed something to do with container use and logging. > > > > > > IIRC the default assumption in docker and Kubernetes is that the logs > > come > > > on stdout (or stderr) so after "principle of least astonishment" the > idea > > > was to give a similar experience with Flink. > > > > > > On Tue, May 5, 2020 at 10:49 AM David Anderson <da...@alpinegizmo.com> > > > wrote: > > > > > >> I like this idea because it should improve the experience (and reduce > > >> confusion) for folks having their first flink experience via one of > the > > >> docker playgrounds. Right now it gives the impression that something > is > > >> broken out-of-the-box. > > >> > > >> Regards, > > >> David > > >> > > >> On Mon, May 4, 2020 at 6:01 PM Till Rohrmann <trohrm...@apache.org> > > >> wrote: > > >> > > >> > Hi everyone, > > >> > > > >> > thanks for starting this discussion Chesnay. > > >> > > > >> > I think it would be nice if we also displayed the logs when starting > > the > > >> > process in the foreground. > > >> > > > >> > The repercussions could be mitigated if the default logger > > >> configurations > > >> > would contain file rolling with a max log file size. > > >> > > > >> > @Yang I think there are solutions how to redirect stdout and stderr > > into > > >> > separate files using tee without duplication [1]. > > >> > > > >> > [1] http://www.softpanorama.org/Tools/tee.shtml > > >> > > > >> > Cheers, > > >> > Till > > >> > > > >> > On Wed, Apr 29, 2020 at 4:28 AM Yang Wang <danrtsey...@gmail.com> > > >> wrote: > > >> > > > >> > > Thanks for Chesnay starting this discussion. > > >> > > > > >> > > In FLINK-17166 implementation[1], we are trying to use "tee" > instead > > >> of > > >> > > introducing the stream redirection(redirect the out/err to files). > > >> > However, > > >> > > a side effect is that the logging will be duplicated both in .log > > and > > >> > .out > > >> > > files. > > >> > > Then it may consume more disk space. However it is not a very > > critical > > >> > > problem since we could use log4j/logback configuration to control > > the > > >> > > rolling > > >> > > files and max size. > > >> > > > > >> > > Also, it only happens in docker/K8s deployment. For YARN/Mesos > > >> > deployment, > > >> > > the behavior is just same as before. > > >> > > > > >> > > > > >> > > [1]. https://github.com/apache/flink/pull/11839 > > >> > > > > >> > > Best, > > >> > > Yang > > >> > > > > >> > > Chesnay Schepler <ches...@apache.org> 于2020年4月29日周三 上午12:30写道： > > >> > > > > >> > > > Currently, processes started in the foreground (like in the case > > of > > >> > > > Docker) output all logging/stdout directly to the console, > without > > >> > > > creating any logging files. > > >> > > > > > >> > > > The downside of this approach, as outlined in FLIP-111, is that > > the > > >> > > > WebUI is not able to display the logs since it relies on these > > very > > >> > > > files to exist. > > >> > > > > > >> > > > In FLINK-17166 (part of FLIP-111) we are trying to change this > > such > > >> > that > > >> > > > we always created .log/.out files. It seems like a reasonable > > >> change to > > >> > > > do, but it could have repercussions on existing deployments > since > > we > > >> > > > will naturally use more disk space (logs gotta go somewhere). > > >> > > > > > >> > > > I'm curious what people think about this. > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
