Flink on K8s job submission best practices

loads/controllers/jobs-run-to-completion/ -- Maximilian Bode * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Müller Sitz: Unterföhring * Amtsgericht München * HRB 135082

Re: Get EOF from PrometheusReporter in JM

Hi Tony, thanks for troubleshooting this. I have added a commit to https://github.com/apache/flink/pull/4586 that should enable you to use the reporter with 1.3.2 as well. Best regards, Max > Tony Wei > 23. September 2017 um 13:11 > Hi Chesnay, > > I built another

Re: flink testing

Hi Georg, Have a look at (Streaming)MultipleProgramsTestBase (https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/StreamingMultipleProgramsTestBase.java | https://github.com/apache/flink/blob/master/flink-test-utils-

Re: Daily/hourly TumblingEventTimeWindows

I have just noticed that this is exactly what it currently does. Reading the docs I assumed all windows would be of the same size. > Am 22.09.2016 um 13:35 schrieb Maximilian Bode : > > Hi everyone, > > is there an easy way to implement a tumbling event time window that tumbles

Daily/hourly TumblingEventTimeWindows

Hi everyone, is there an easy way to implement a tumbling event time window that tumbles at a certain time? Examples could be daily or hourly (tumbling at exactly 00:00, 01:00, 02:00 etc.) windows. So in particular, for a daily window, the first window would be shorter than the rest, tumble at

Prevent job/operator from spilling to disk

preferable to see the job fail and then be able to grant sufficient memory and run it again. Cheers, Max — Maximilian Bode * Software Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr

OOME PermGen in URLClassLoader

suspect there might be a memory leak caused by the Classloader, any ideas? Best regards, Max — Maximilian Bode * Software Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz

Re: Jobmanager HA with Rolling Sink in HDFS

mentioned in an earlier mail then leads to a number smaller than 2 million. We are using an FsStateBackend in HDFS with a checkpoint interval of 10s. Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring

Re: Jobmanager HA with Rolling Sink in HDFS

ator in a map right before writing into HDFS. Also, the RollingSink has a DataTimeBucketer which makes it transparent when no new data is arriving anymore as the last bucket is from some minutes ago. > > Cheers, > Aljoscha >> On 08 Mar 2016, at 13:19, Maximilian Bode >> wrot

Re: Jobmanager HA with Rolling Sink in HDFS

2. So it looks to us as if one of the files was not even created in the second attempt. This behavior seems to be what somewhat reproducible, cf. my earlier email where the part-11 file disappeared as well. Thanks again for your help. Cheers, Max — Maximilian Bode * Junior Consultant

Re: Jobmanager HA with Rolling Sink in HDFS

Hi Aljoscha, thank you very much, I will try if this fixes the problem and get back to you. I am using 1.0.0 as of today :) Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik

Re: Jobmanager HA with Rolling Sink in HDFS

, one of the part files (11 in the example below) is missing completely. We are able to reproduce this behavior by killing a task manager. Can you make sense of that? Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a

Re: Jobmanager HA with Rolling Sink in HDFS

from this what is going wrong? Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz: Unterföhring * Amtsgericht München * HRB 135082

Re: Jobmanager HA with Rolling Sink in HDFS

Just for the sake of completeness: this also happens when killing a task manager and is therefore probably unrelated to job manager HA. > Am 03.03.2016 um 14:17 schrieb Maximilian Bode : > > Hi everyone, > > unfortunately, I am running into another problem trying to establish

Jobmanager HA with Rolling Sink in HDFS

ault value. Then again, as the NonRollingBucketer is used, there should not be any timing issues, right? Cheers, Max [1] https://issues.apache.org/jira/browse/FLINK-2979 — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 1

Re: YARN JobManager HA using wrong network interface

the problem as now the new job manager is available in sufficient time. I still would agree with Stephan on taking the local hostname being the preferred strategy. Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com TNG Technology Consulting GmbH, Betastr. 13a, 85774

YARN JobManager HA using wrong network interface

t able to choose the right interface. In particular, it seems that iteration order over the network interfaces differs between the HEURISTIC and SLOW strategy, which then leads to the wrong interface being selected. Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.

Re: Backpressure in the context of JDBCOutputFormat update

ation, as the WHERE clause only contains the table's primary key. Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com * 0176 1000 75 50 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert D

Backpressure in the context of JDBCOutputFormat update

steps? Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com * 0176 1000 75 50 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke Sitz: Unterföhring * Amtsgericht München * HRB 135082

Re: JDBCInputFormat GC overhead limit exceeded error

concerned) is disabling autocommits and letting users decide the fetchSize (somewhat in parallel to batchInterval in JDBCOutputFormat). Cheers, Max — Maximilian Bode * Junior Consultant * maximilian.b...@tngtech.com * 0176 1000 75 50 TNG Technology Consulting GmbH, Betastr. 13a, 85774

Re: JDBCInputFormat GC overhead limit exceeded error

Hi Robert, I am using 0.10.1. > Am 19.01.2016 um 17:42 schrieb Robert Metzger : > > Hi Max, > > which version of Flink are you using? > > On Tue, Jan 19, 2016 at 5:35 PM, Maximilian Bode <mailto:maximilian.b...@tngtech.com>> wrote: > Hi everyone, >

JDBCInputFormat GC overhead limit exceeded error

util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)" Any ideas what is going wrong here? Cheers, Max — Maximilian Bode * Junior Consultant * maxi

Re: Flink on EMR Question

Hi everyone, Regarding Q1, I believe I have witnessed a comparable phenomenon in a (3-node, non-EMR) YARN cluster. After shutting down the yarn session via `stop`, one container seems to linger around. `yarn application -list` is empty, whereas `bin/yarn-session.sh -q` lists the left-over conta

Re: Checkpoints in batch processing & JDBC Output Format

ing output format to give you exactly-once semantics. The way to > do that would be to commit the upserts only on completed checkpoints (and > buffer them in the sink between checkpoints). This may be interesting if your > database cannot deduplicate insertions (no deterministic pr

Checkpoints in batch processing & JDBC Output Format

Hi everyone, I am considering using Flink in a project. The setting would be a YARN cluster where data is first read in from HDFS, then processed and finally written into an Oracle database using an upsert command. If I understand the documentation correctly, the DataSet API would be the natura