Anyone trying to adopt Scotty on the recent Flink versions?

Hi community, We're recently trying to adopt Scotty to overcome the poor performance caused by too many sliding windows. We're facing the following exception on the latest Flink-1.14.2: switched from RUNNING to FAILED with failure cause: java.lang.ArrayIndexOutOfBoundsException: -1 at ja

Re: How to reduce interval between Uptime Metric meaasurements?

Hi! Have you tried metrics.reporter.promgateway.interval? See [1] for more detail. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/metric_reporters/#prometheuspushgateway Geldenhuys, Morgan Karl 于2021年12月28日周二 02:43写道： > Hello everyone, > > > I have a flink 1.14 job ru

How to reduce interval between Uptime Metric meaasurements?

Hello everyone, I have a flink 1.14 job running and im looking at the uptime metric (flink_jobmanager_job_uptime) together with prometheus (scrape every second). It looks as if this metric is updated every 60 seconds, is there a way of decreasing this interval? A fixed delay recovery strategy

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

Hi Arvid, Thanks for the suggestion! Sorry for the late reply. I just finished investigating the PulsarSource/StopCursor as you suggested. Please see my reply inline. On Sun, Dec 19, 2021 at 6:53 PM Arvid Heise wrote: > Hi Dong, > > I see your point. The main issue with dynamic EOF is that we c

Re: How to handle java.lang.OutOfMemoryError: Metaspace

Ok all settings above are for smaller dev cluster and I'm experimenting to set metasize to 2GB. It runs same jobs as production just less volume in terms of data. The below snapshot of JCMD are of a slightly bigger task manager and the active cluster... It also once in a while does metaspace so th

Re: How to handle java.lang.OutOfMemoryError: Metaspace

Yes standalone cluster. 3 zoo, 3 job, 3 tasks. The task managers have taskslots at double core. So 2*4 I think metaspace of 2GB is ok. I'll try to get some jcmd stats. The jobs are fairly straight forward ETL they read from Kafka, do some json parsing, using vertx.io json parser and either Inser

Re: Parquet files in streaming mode

Hi, Have you looked into File Compaction (which is supported in the Table/SQL side)? [1] Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#file-compaction On Mon, 27 Dec 2021 at 16:10, Deepak Sharma wrote: > I would suggest takin

Re: Parquet files in streaming mode

I would suggest taking a look at CheckpointRollingPolicy. You need to extend it and override the default behviors in your FileSink. HTH. Thanks Deepak On Mon, Dec 27, 2021 at 8:13 PM Mathieu D wrote: > Hello, > > We’re trying to use a Parquet file sink to output files in s3. > > When running i

Parquet files in streaming mode

Hello, We’re trying to use a Parquet file sink to output files in s3. When running in Streaming mode, it seems that parquet files are flushed and rolled at each checkpoint. The result is a crazy high number of very small parquet files which completely defeats the purpose of that format. Is ther

Remove stackTrace from error response

Hi, as part of security improvement we need to make sure APIs do not return stack traces as part of the error response, The JobManager REST API does currently return the stack trace on error, For example: ❯ curl http://localhost:8081/jobs/123 {"errors":["org.apache.flink.runtime.rest.handler.RestH

Re: Avoiding Dynamic Classloading for User Code

And everything happens at launch From: Lior Liviev Sent: Sunday, December 26, 2021 7:38 PM To: David Morávek Cc: user Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Hey David, I placed the jar in flink folder but now I see so