Hi, Rui. Thanks for the proposal. It sounds reasonable. I have some questions, PTAL: 1. I have a similar concern as Feng. Will we redirect to another log file not taskManager.log ? taskManager.log contains lots of important information like init log. It will be rolled quickly if we redirect out and error here. 2. Since we have redirected to LOG mode, Could we also log the subtask info ? It may help us to debug granularly.
On Thu, Nov 9, 2023 at 9:47 AM Feng Jin <jinfeng1...@gmail.com> wrote: > Hi, Rui. > > Thank you for initiating this proposal. > > I have a question regarding redirecting stdout and stderr to LOG: > > Will they be written to the taskManager.log file by default or the > taskManager.out file? > If we can make taskmanager.out splittable and rolling, would it be easier > for users to use this feature? > > Best, > Feng > > On Thu, Nov 9, 2023 at 3:15 AM Archit Goyal <argo...@linkedin.com.invalid> > wrote: > > > Hi Rui, > > > > Thanks for the proposal. > > > > The proposed solution of supporting System out and err to be redirected > to > > LOG or discarded and introducing an enum and two options to manage this, > > seems reasonable. > > > > +1 > > > > Thanks, > > Archit Goyal > > > > > > From: Piotr Nowojski <pnowoj...@apache.org> > > Date: Wednesday, November 8, 2023 at 7:38 AM > > To: dev@flink.apache.org <dev@flink.apache.org> > > Subject: Re: [DISCUSS] FLIP-390: Support System out and err to be > > redirected to LOG or discarded > > Hi Rui, > > > > Thanks for the proposal. > > > > +1 I don't have any major comments :) > > > > One nit. In `SystemOutRedirectToLog` in this code: > > > > System.arraycopy(buf, count - LINE_SEPARATOR_LENGTH, bytes, 0, > > LINE_SEPARATOR_LENGTH); > > return Arrays.equals(LINE_SEPARATOR_BYTES, bytes) > > > > Is there a reason why you are suggesting to copy out bytes from `buf` to > > `bytes`, > > instead of using `Arrays.equals(int[] a, int aFromIndex, int aToIndex, > > int[] b, int bFromIndex, int bToIndex)`? > > > > Best, > > Piotrek > > > > śr., 8 lis 2023 o 11:53 Rui Fan <1996fan...@gmail.com> napisał(a): > > > > > Hi all! > > > > > > I would like to start a discussion of FLIP-390: Support System out and > > err > > > to be redirected to LOG or discarded[1]. > > > > > > In various production environments, either cloud native or physical > > > machines, the disk space that Flink TaskManager can use is limited. > > > > > > In general, the flink users shouldn't use the `System.out.println` in > > > production, > > > however this may happen when the number of Flink jobs and job > developers > > > is very large. Flink job may use System.out to output a large amount of > > > data > > > to the taskmanager.out file. This file will not roll, it will always > > > increment. > > > Eventually the upper limit of what the TM can be used for is reached. > > > > > > We can support System out and err to be redirected to LOG or discarded, > > > the LOG can roll and won't increment forever. > > > > > > This feature is useful for SREs who maintain Flink environments, they > can > > > redirect System.out to LOG by default. Although the cause of this > problem > > > is > > > that the user's code is not standardized, for SRE, pushing users to > > modify > > > the code one by one is usually a very time-consuming operation. It's > also > > > useful for job stability where System.out is accidentally used. > > > > > > Looking forward to your feedback, thanks~ > > > > > > [1] > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F4guZE&data=05%7C01%7Cargoyal%40linkedin.com%7C937821de7bd846e6b97408dbe070beae%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638350547072823674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zEv6B0Xiq2SNuU6Fm%2BAXnH%2BRILbm6Q0uDRbN7h6iaPM%3D&reserved=0 > > <https://cwiki.apache.org/confluence/x/4guZE> > > > > > > Best, > > > Rui > > > > > > -- Best, Hangxiang.
