flink session job retention time

2020-10-08 Thread Richard Moorhead
Is there a configuration that controls how long jobs are retained in a flink session?

Re: FlinkCounterWrapper

2020-10-02 Thread Richard Moorhead
Furthermore, it looks like the rest of the dropwizard wrappers all have the mutators implemented. https://issues.apache.org/jira/browse/FLINK-19497 On Fri, Oct 2, 2020 at 2:30 PM Richard Moorhead wrote: > We have a use case wherein counters emitted by flink are decremented after >

FlinkCounterWrapper

2020-10-02 Thread Richard Moorhead
We have a use case wherein counters emitted by flink are decremented after being reported. In this way we report only the change in the counter. Currently it seems that FlinkCounterWrapper doesnt mutate the wrapped counter when either inc or dec is called; would this be a valid improvement? https

Flink 1.10 classpath

2020-09-25 Thread Richard Moorhead
We're submitting jobs to a flink session running on YARN. Our yarn-site.xml does not have our mapreduce libs in yarn.application.classpath. Our application code has some provided dependencies from hadoop-mapreduce-client-core. Without editing our yarn-site, or adding those libraries to the flink li

Fwd: Flink Session TM Logs

2020-07-24 Thread Richard Moorhead
-- Forwarded message - From: Robert Metzger Date: Fri, Jul 24, 2020 at 1:09 PM Subject: Re: Flink Session TM Logs To: Richard Moorhead I accidentally replied to you directly, not to the user@ mailing list. Is it okay for you to publish the thread on the list again? On Fri

Flink Session TM Logs

2020-07-24 Thread Richard Moorhead
When running a flink session on YARN, task manager logs for a job are not available after completion. How do we locate these logs?

Re: job history server

2020-02-18 Thread Richard Moorhead
` 508k 465k 43k 92% /dev/shm Thanks for the tip. On Mon, Feb 17, 2020 at 8:08 PM Richard Moorhead wrote: > I did not know that. > > I have since wiped the directory. I will post when I see this error again. > > On Mon, Feb 17, 2020 at 8:03 PM Benchao Li wro

Re: job history server

2020-02-17 Thread Richard Moorhead
I did not know that. I have since wiped the directory. I will post when I see this error again. On Mon, Feb 17, 2020 at 8:03 PM Benchao Li wrote: > `df -H` only gives the sizes, not inodes information. Could you also show > us the result of `df -iH`? > > Richard Moorhead 于2020年2月1

Re: job history server

2020-02-17 Thread Richard Moorhead
% /dev/shm On Mon, Feb 17, 2020 at 7:13 PM Benchao Li wrote: > Hi Richard, > > Have you checked that inodes of the disk partition were full or not? > > Richard Moorhead 于2020年2月18日周二 上午8:16写道: > >> I see the following exception often: >> &g

job history server

2020-02-17 Thread Richard Moorhead
I see the following exception often: 2020-02-17 18:13:26,796 ERROR org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher - Failure while fetching/processing job archive for job eaf0639027aca1624adaa100bdf1332e. java.nio.file.FileSystemException: /dev/shm/flink-history-server/job

Re: UI stability at high parallelism

2020-02-14 Thread Richard Moorhead
d out, you can try to find > some evidence in the jobmanager log. > > You can provide the full log to help us find the root cause. > > > Best > Weihua Hu > > 2020年2月13日 09:40,Richard Moorhead 写道: > > When I submit a job to flink session with parallelism higher than

group by optimizations with sorted input

2020-02-13 Thread Richard Moorhead
In batch mode, if input is sorted prior to a group by operation; does flink forward the aggregate data early? Is there a way to prevent grouping operations from buffering all data in a GBK operation in batch mode?

UI stability at high parallelism

2020-02-12 Thread Richard Moorhead
When I submit a job to flink session with parallelism higher than 128, the job is submitted and renders in the UI but when I view the job itself the UI starts to rapidly emit errors in the upper right: Server Response: Unable to load requested file /bad-request. Is this a known issue? Is there a