Re: [External] A random checkpoint failure creates an avalanche of restarts

max parallelism? I don't think I have changed anything here and am running with defaults. Finally, since I stay in India, I will not be able to attend the FlinkForward Conference in Barcelona. Thanks Sachin On Mon, Sep 22, 2025 at 1:03 PM Schwalbe Matthias < matthias.schwa...@viseca.ch>

A random checkpoint failure creates an avalanche of restarts

Hi, So we are running Flink 19.1 on AWS EMR using Yarn as resource manager. We have a fairly large cluster with 120 parallelism and 30 task managers running with 4 task slots. Here are some of the important configs: taskmanager.numberOfTaskSlots: 4 jobmanager.memory.process.size: 12g taskmanager.m

Flink autoscaler is not working

e back the autoscaler was working when I was going with default values for intervals, and I found it too aggressive. So I relaxed these intervals and after that I don't see this working. Is there any issue with my configs causing it to not work ? Thanks Sachin

Flink autoscaler on Kubernetes

as resource manager. There also Flink autoscaler is enabled. So once I add more task nodes to my EMR cluster, job automatically scales up. What could be the issue here? Thanks Sachin

Query: Geo-Redundancy with Apache Flink on Kubernetes & Replicated Checkpoints !

y guidance, references, or community experience around this type of architecture. Thanks in advance for your time and support. Looking forward to your insights. Best regards, *Sachin*

Re: Flink CDC for mongoDB, need to understand MongoDBStreamFetchTask better

Hi, I have raised a JIRA for this: https://issues.apache.org/jira/browse/FLINK-37909 And also a PR which I feel should fix this issue: https://github.com/apache/flink-cdc/pull/4039 Can someone from the Flink community take an initiative for this and get this fixed. Thanks Sachin On Thu, Jun 5

Flink CDC for mongoDB, need to understand MongoDBStreamFetchTask better

ay be that a competing application thread wrote an older timestamp. Please let me know if my findings and what is reported by MongoDB support are somewhat related. What it looks like is that heartbeatManager runs on an older changeStreamCursor which is kind of not updating resume tokes or getting rewounded. Thanks Sachin

Re: Is there anyway to control the size of my ListState in my KeyedProcessFunction

idual keys in the map. Please let me know if there is a better way to do this. Thanks Sachin On Fri, May 30, 2025 at 9:32 AM Zakelly Lan wrote: > Hi Sachin, > > I assume you are using the rocksdb state backend. The TTL for ListState is > applied for each list entry, if you are usi

Re: Is there anyway to control the size of my ListState in my KeyedProcessFunction

Hi, I think ttl would be applied for the entire list, I would like the ListState to restrict the entries by size and automatically purge older added entries as new ones get added. Something similar to a bounded list. Thanks Sachin On Thu, May 29, 2025 at 6:51 PM Sigalit Eliazov wrote: >

Is there anyway to control the size of my ListState in my KeyedProcessFunction

something possible using Flink constructs. If not, is there any way I can achieve this ? Thanks Sachin

Flink CDC 3.1 for mongodb repeatedly failing in streaming mode after running for few days despite setting heartbeat

oned in docs, I have added heartbeat so for infrequently changed oplogs, it does not fail, however it looks like these settings have no effect on the failure and still fails when I did not have the heartbeat. Can anyone tell me what else I could fix here? Are there any settings missing from the mongodb side? Thanks Sachin

Re: Flink CDC is failing for MongoDB with Resume of change stream was not possible, as the resume point may no longer be in the oplog error

entalSourceStreamFetcher.java:89) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

Re: Best approach to aggregate state with idle timeout and periodic output

Second approach is good to try out. I am also solving for a similar problem using this approach only. Thanks Sachin On Mon, 12 May 2025 at 3:52 PM, Ehud Lev wrote: > Hi Flink community, > > We're building a Flink topology that aggregates events by key. When a key > is seen f

Re: Inquiry Regarding Flink and Amazon S3 Integration for Externalized Checkpoints

I am using s3 as checkpoint storage for Flink running as part of EMR (EC2) + YARN setup and also running on EKS. There should not be any problem with it. Thanks Sachin On Thu, Apr 24, 2025 at 12:09 PM Anuj Jain wrote: > Dear Apache Flink Community, > > > > I hope this message

Flink CDC is failing for MongoDB with Resume of change stream was not possible, as the resume point may no longer be in the oplog error

ernal.connection.DefaultConnectionPool$PooledConnection .sendAndReceive(DefaultConnectionPool.java:643) at com.mongodb.internal.connection.CommandProtocolImpl.execute( CommandProtocolImpl.java:73) I think the error is mainly for MongoDB ops log side. Any idea how I can approach the team handling MongoDB to get them to resolve at their end, so such errors are not propagated at Flink side. Anything I can fix at Flink or Flink CDC side to stop application from continuously restarting due to these errors ? Thanks Sachin

Re: How can we read checkpoint data for debugging state

debug the keyed state one of my operators writes to, to understand if there is a state leak and why this state becomes very huge. Thanks Sachin On Wed, Feb 26, 2025 at 5:01 PM Gabor Somogyi wrote: > State processor API is now not supporting changelog. Please see here [1]. > > [1

Re: How can we read checkpoint data for debugging state

: s3:/{bucket}/changelog dstl.dfs.compression.enabled: 'true' Thanks Sachin On Fri, Feb 21, 2025 at 3:37 PM Gabor Somogyi wrote: > The UID must match in the Flink app `events.uid("my-uid")` and in the > reader app `forUid("my-uid")`. > In general it's

Re: How can we read checkpoint data for debugging state

? Thanks Sachin On Fri, Feb 21, 2025 at 3:37 PM Gabor Somogyi wrote: > The UID must match in the Flink app `events.uid("my-uid")` and in the > reader app `forUid("my-uid")`. > In general it's a good practice to set uid in the Flink app for each and > every operator

Re: How can we read checkpoint data for debugging state

he right way to access it ? Thanks Sachin On Fri, Feb 21, 2025 at 11:09 AM Xuyang wrote: > Hi, > > FYI that hopes helpful: FLIP-496: SQL connector for keyed savepoint data[1] > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+sa

How can we read checkpoint data for debugging state

my KeyedState class, I can access state1 and state2. Would this work? Please let me know if I am on the right track or this is something not possible to read checkpointed states via any external application for debugging. Thanks Sachin

Flink future release and upgrade plan

performance we get in Flink 1.20.0 as compared to Flink 1.19.1. Thanks Sachin

Re: How to register pojo type information for third party pojo classes

ate.update(a); I get: java.lang.ClassCastException: null Any idea how can I make this work for null values of primitive array ? Thanks Sachin On Thu, Feb 13, 2025 at 2:56 PM Zhanghao Chen wrote: > Hi Sachin, > > Sorry that I misunderstood your question before. It seems like there's

Re: How to register pojo type information for third party pojo classes

yway I can check via logs if right serializers are registered with my custom POJOs. Thanks Sachin On Thu, Feb 13, 2025 at 12:06 PM Zhanghao Chen wrote: > What you are doing is registering two third-party classes as POJO types, > and this is actually the default behavior of Flink even witho

Re: How to register pojo type information for third party pojo classes

Context.java:101) at org.apache.flink.api.common.state.StateDescriptor .initializeSerializerUnlessSet(StateDescriptor.java:336) at org.apache.flink.streaming.api.operators.StreamingRuntimeContext .getState(StreamingRuntimeContext.java:202) So my question is if I have defined a serializer for A class, why is it still c

How to register pojo type information for third party pojo classes

new HashMap<>() { { ... } }; return Types.POJO(A.class, fields); } } Now I want to register this type information whenever A class's object is serialized or de-serialized in Flink state. How can I register this to the StreamExecutionEnvironment. Thanks Sachin

Flink autoscaler is throwing exception and shutting down the entire job

mpletion.run(CompletableFuture.java:482) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] Can anyone please tell me what area to look into to see how to fix this ? Thanks Sachin

For the FileSystem-based Changelog storage, for how long these changelogs are retained

allocate for the HDFS ? I am performing incremental checkpointing using rocksdb and using HDFS as the store. A checkpoint state can easily go to 50 GB. Thanks Sachin

Re: [External] Is there a way to get consolidated metrics at task level

number of subtasks that are running which can vary with time. Thanks Sachin On Mon, Jan 6, 2025 at 4:09 PM Schwalbe Matthias < matthias.schwa...@viseca.ch> wrote: > Hi Sachin, > > > > Happy new year … 😊 > > > > On Yarn we setup prometheus push gateway on o

Generic log-based incremental checkpoint seems to be not working

Protos$ClientNamenodeProtocol$2.callBlockingMethod( ClientNamenodeProtocolProtos.java) Can anyone help me to understand why I am getting this error and how I can fix this? Also note that when I do: hdfs dfs -ls /changelog or hdfs dfs -ls /flink-checkpoints I do get directories inside it, so it looks like settings are working. Thanks Sachin

Is there a way to get consolidated metrics at task level

task level and provide my once consolidated value. This way I can provide uniform reporting irrespective of the number of sub tasks that get created at different points in time. Thanks Sachin

Late data stream is not progressing after re-watermarking

I may be doing wrong. If I don't assign fresh timestamp and watermarking to the *reducedLateData* stream then I notice that this *lateReduce* step now drops late records which were before dropped by *reduce* step. Thanks Sachin

What could be the reason for failure - Key group is not in KeyGroupRange.

Data2) .keyBy( e -> MurmurHash3.hash128x64((e.field1 + e.field2 + e.field3).getBytes())[0]) .process(new UnionKeyedProcessFunction()) .name("Task"); Any idea where I should look into as this code was working before. Thanks Sachin

What would be the best serialization mechanism in flink

pojos to be serialized by our process functions. I was just wondering is there any better serialization mechanism we can use which is simple and offer much higher performance as compared to POJO. Thanks Sachin

Kafka sink producing record at event timestamp

same as its event time. Ideally I was expecting that the record's time should be the timestamp when the record was inserted in the Kafka topic and not the same as its event time. How can I control this behaviour? Thanks Sachin

Stopping the flink 1.18 program with savepoint seems to fail with timeout

see is that actually a savepoint does get generated at the specified path and my flink job is also stopped after a while. Is there any setting which is making the cli program to timeout and is there a way we can verify that the entire savepoint got generated on the specified path ? Thanks Sachin

Re: Status of ClickHouseSink

-ru/flink-clickhouse-sink/blob/master/pom.xml#L31C24-L31C29> > version of flink. Did you make the version up yourself? > > чт, 3 окт. 2024 г. в 15:17, Sachin Mittal : > >> I have been using: >> https://github.com/ivi-ru/flink-clickhouse-sink >> >> >> >>

Re: Status of ClickHouseSink

I have been using: https://github.com/ivi-ru/flink-clickhouse-sink On Thu, Oct 3, 2024 at 4:54 PM Ilya Karpov wrote: > Hi, > I've been searching for an implementation of kafka to clickhouse sink and > found FLIP >

Flink 1.15.4 JobGraphs not compatible with 1.19.1

e.completeThrowable(Unknown > Source)\n\tat > java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown > Source)\n\tat java.util.concurrent.CompletableFuture$Completion.run(Unknown > Source)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source)\n\t... 2 frames truncated > What should I do to make it compatible? Thanks & Regards, Sachin Sharma +1-669-278-5239

Re: Flink 1.19.1 Java 17 Compatibility

Thanks Zhanghao and Andreas. Thanks & Regards, Sachin Sharma +1-669-278-5239 On Tue, Sep 17, 2024 at 6:04 AM Zhanghao Chen wrote: > In our production environment, it works fine. > > Best, > Zhanghao Chen > ------ > *From:* Sachin Sharma >

Dynamic configuration FlinkDeployment with Kubernetes Operator 1.9

along with many other properties. Thanks & Regards, Sachin Sharma +1-669-278-5239

Flink 1.19.1 Java 17 Compatibility

Hi, We are planning to use Flink 1.19.1 with kubernetes operator, I wanted to check if 1.19.1 is Java 17 compatible or not. In documentation it says in version 1.18 we added experimental support for it but nothing concrete is said whether it supports Java 17 completely. Thanks & Regards, Sa

Re: When to use scan.incremental.snapshot.enabled

Thanks for the explanation. One quick question how do I enable: scan.incremental.snapshot.enabled = true for my MongoDBSource ? I don't see any option in the builder for the same. Regards Sachin On Mon, Aug 19, 2024 at 8:00 PM Jiabao Sun wrote: > Sorry, in my previous reply, I mi

Re: How to set the number formatter for json convertor for mongo cdc connector

This worked!!! Thanks Sachin On Mon, Aug 19, 2024 at 3:06 PM Jiabao Sun wrote: > Hi Sachin, > > It is recommended to use org.bson.Document to convert MongoDB Extended > JSON into Java types, and then perform further f

Re: When to use scan.incremental.snapshot.enabled

this version of flink ? Thanks Sachin On Mon, Aug 19, 2024 at 2:29 PM Jiabao Sun wrote: > Hi Sachin, > > Incremental snapshot reading is a new feature introduced in Flink 2.0. > > It has the following capabilities: > - Source can be parallel during snapshot reading to impr

How to set the number formatter for json convertor for mongo cdc connector

document like a regular JSON, but I am not sure how and where I can specify this in my configuration. Can anyone help me with this? Thanks Sachin

When to use scan.incremental.snapshot.enabled

property help with ? Please note that I am configuring my connector with: .startupOptions(StartupOptions.initial()) .batchSize(2048) Thanks Sachin

Re: Mongo flink CDC connector not reading from the source

This worked. Thanks On Sat, 17 Aug 2024 at 6:02 PM, Jiabao Sun wrote: > Hi Sachin, > > The 'collectionList' needs to be filled with fully qualified names. > > For example, > database: test_db > collection: test_collection > > MongoDBSource.builder() &

Mongo flink CDC connector not reading from the source

onds=1723896025, inc=1} >From the logs it seems that we are able to connect to the CDC stream and it should start by loading existing records in the collections as snapshot is set as initial. However I don't see any records being read or even any error in my Flink UI/logs. Any idea what may be going wrong. Thanks Sachin

Re: Integrating flink CDC with flink

Ok. This makes sense. I will try it out. One more question in terms of performance which of the two connector would scan the existing collection faster. Say existing collection has 10 million records and in terms of storage size it is 1GB. Thanks Sachin On Fri, 16 Aug 2024 at 4:09 PM, Jiabao

Re: Integrating flink CDC with flink

or I can use flink-connector-mongodb-cdc to process both existing and new data ? Thanks Sachin On Fri, Aug 16, 2024 at 3:46 PM Jiabao Sun wrote: > Hi Sachin, > > flink-connector-mongodb supports batch reading and writing to MongoDB, > similar to flink-connector-jdbc, while fli

Integrating flink CDC with flink

Flink job. When researching this I found Flink CDC and they have a CDC connector for MongoDB - flink-connector-mongodb-cdc However I am not able to figure out how to stream those changes also to my Job which is also reading from the same collection. Thanks Sachin

Re: Can we share states across tasks/operators

Hi, Thanks for pointing me in the right direction. I think I can union the two streams as they have the same type and in the process function have a shared state since now both are the same stream. I will figure this around connecting streams and using keyed state stores. Sachin On Wed, Aug 7

Can we share states across tasks/operators

hieve this in Flink by connecting to the same state store ? Is there any concept of global state in Flink if I cannot achieve this by using keyed states associated with an operator's process function ? Any other way you can think of achieving the same ? Thanks Sachin

Re: Is there any way to perform left outer join using flink's data stream APIs

;t be able to use the window operator. Now loading entire collections in flink memory may not be feasible, so how best I can do this using coGroup. Thanks Sachin On Wed, Aug 7, 2024 at 10:41 AM Kirill Ternovsky < kirill.ternov...@interchecks.com> wrote: > Hi Sachin, > > You s

Is there any way to perform left outer join using flink's data stream APIs

field values from A and empty for ones supposed to be joined from B. Please let me know if there is any way I can do this, using the CoProcess function or something. Thanks Sachin

Re: How can I debug Assigned key must not be null error when reading from Mongodb source

Yes, this fixed the issue. Thanks Sachin On Mon, Aug 5, 2024 at 6:38 PM Jiabao Sun wrote: > Hi Sachin, > > Could you please check if you have used the keyBy operator and ensure that > the keyBy field is not null? > > Best, > Jiabao > > On 2024/08/05 12:33:27 Sachin

Re: How can I debug Assigned key must not be null error when reading from Mongodb source

ark strategy is WatermarkStrategy.*noWatermarks.* Thanks Sachin On Mon, Aug 5, 2024 at 5:47 PM Xiqian YU wrote: > Hi Sachin, > > > > Seems KeyGroupStreamPartitioner is complaining about receiving a null > StreamRecord, which is abnormal since MongoDeserializationSchema ensures > non-nulla

How can I debug Assigned key must not be null error when reading from Mongodb source

1610ad6715a8aa23323aee6:?] ... Any idea where should I look and how can I fix this ? Thanks Sachin

Re: Understanding flink-autoscaler behavior

) controls the parallelism or the Reactive mode in the Flink job controls it. Thanks & Regards, Sachin Sharma On Fri, Jun 7, 2024 at 4:55 AM Gyula Fóra wrote: > Hi! > > To simplify things you can generally look at TRUE_PROCESSING_RATE, > SCALUE_UP_RATE_THRESHOLD and SCALE_DOWN_RATE_THR

How to start a flink job on a long running yarn cluster from a checkpoint (with arguments)

mples/streaming/WordCount-Updated.jar --input file:///input1.txt --output file:///output1/ --newarg value123 Thanks Sachin

Re: Task Manager memory usage

Hi Where are you storing the state. Try rocksdb. Thanks Sachin On Thu, 23 May 2024 at 6:19 PM, Sigalit Eliazov wrote: > Hi, > > I am trying to understand the following behavior in our Flink application > cluster. Any assistance would be appreciated. > > We are running a

Re: What is the best way to aggregate data over a long window

so clean the state. This way each state would always keep only one record, no matter for what period we aggregate data for. Is this a better approach than windowing ? Thanks Sachin On Fri, May 17, 2024 at 1:14 PM gongzhongqiang wrote: > Hi Sachin, > > We can optim

What is the best way to aggregate data over a long window

by say 10 fold I would have no option but to again increase the memory. Is there a better way to perform long window aggregation so overall this step has a small memory footprint. Thanks Sachin

Re: how to reduce read times when many jobs read the same kafka topic?

. Another thing you can look into is if your job step is keyed by some key, then make sure they are keyed as a string or any other Java primitive types since Object keys are much slower when reading from and writing to a state store. Thanks Sachin On Wed, May 15, 2024 at 7:58 AM longfeng Xu wrote

How can we exclude operator level metrics from getting reported

etting for metrics.reporter.stsd.filter.excludes is correct in achieving the same. Thanks Sachin

Re: Understanding default firings in case of allowed lateness

educedData(late d4, late d5, late d6) ] Thanks Sachin On Wed, Apr 17, 2024 at 4:05 PM Xuyang wrote: > Hi, Sachin. > > IIUC, it is in the second situation you listed, that is: > [ reduceData (d1, d2, d3), reducedData(ld1, d2, d3, late d4, late d5, late > d6) ]. > However, beca

Understanding default firings in case of allowed lateness

) ] or something else ? Also if I want the reduced data from late records to not include the data emitted within the window bounds, how can I do the same ? or if this is handled as default case ? Thanks Sachin

Re: Understanding event time wrt watermarking strategy in flink

luded in a particular window. Basically having allowed lateness A greater than the out of order bound B won't make sense as records later than T - B would have got dropped at the source itself. Please let me know if I am understanding this correctly or am I missing something? Thanks Sachi

Re: Understanding event time wrt watermarking strategy in flink

than T - B would have already been dropped at the source. If this is not the case than how does lateness work with our of order boundedness ? Thanks Sachin On Fri, Apr 12, 2024 at 12:30 PM Yunfeng Zhou wrote: > Hi Sachin, > > 1. When your Flink job performs an operation like map o

Understanding event time wrt watermarking strategy in flink

atermark timestamp would be T - B ? How can we control the watermarking period ? Thanks Sachin

How are window's boundaries decided in flink

can I achieve this ? Thanks Sachin

Re: How to debug window step in flink

Hi, Yes it was a watermarking issue. There were few out of order records in my stream and as per watermarking strategy the watermark was advanced to the future and hence current events were getting discarded. I have fixed this by not processing future timestamped records. Thanks Sachin On Mon

How to debug window step in flink

problem in this function it would have been thrown to the pipeline and I would have known. Please let me know how I can debug this better. Thanks Sachin

How to handle tuple keys with null values

at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:31) ~[flink-dist-1.17.1.jar:1.17.1] Thanks Sachin

Re: Understanding RocksDBStateBackend in Flink on Yarn on AWS EMR

So, when we create an EMR cluster the NN service runs on the primary node of the cluster. Now at the time of creating the cluster, how can we specify the name of this NN in format hdfs://*namenode-host*:8020/. Is there a standard name by which we can identify the NN server ? Thanks Sachin On

Re: Flink unable to read from kafka source due to starting offset strategy

. Any idea why we the consumer network client is getting disconnected. Is this because this thread is not getting enough resources or something ? Thanks Sachin On Fri, Mar 22, 2024 at 12:48 PM Sachin Mittal wrote: > Hi, > I was experimenting with different starting offset strategies for my

Re: Flink unable to read from kafka source due to starting offset strategy

offset will not longer be available in Kafka topic, as the data would have been discarded based on topic retention policy. Hence just using the committed offsets strategy does not always work. Thanks Sachin

Understanding RocksDBStateBackend in Flink on Yarn on AWS EMR

: *state.checkpoints.dir*: hdfs://namenode-host:port/flink-checkpoints vs *state.checkpoints.dir*: file:///tmp Also if we decide to use HDFS then from where we can get the value for *namenode-host:port* given we are running Flink on an EMR. Thanks Sachin

Re: Need help in understanding PojoSerializer

ListSerializer and *MapSerializer.* Thanks Sachin On Wed, Mar 20, 2024 at 10:47 PM Ken Krugler wrote: > Flink doesn’t have built-in support for serializing Sets. > > See this (stale) issue about the same: > https://issues.apache.org/jira/browse/FLINK-16729 > > You could create a custom s

Need help in understanding PojoSerializer

lds and even when I have disabled generics types. why I am getting message that it will be processed as GenericType? Any help in understanding what I need to do to ensure all the fields of my object are handled using PojoSerializer. Thanks Sachin

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

simply downgrade it to flink version 1.17 Does this ring a bell ? Is there an issue with Flink 1.18 where we try to submit jobs via yarn on aws EMR ? Thanks Sachin On Wed, Mar 13, 2024 at 8:02 AM Hang Ruan wrote: > Hi, Sachin. > > I use the command `jar -tf flink-dist-1.18.0.jar| grep Outp

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 4 more Thanks Sachin On Tue, Mar 12, 2024 at 2:11 PM Sachin Mittal wrote: >

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

Ok. Actually it’s version 1.18. I will try to remove flink-core from the fat jar. On Tue, 12 Mar 2024 at 1:51 PM, Hang Ruan wrote: > Hi, Sachin. > > This error occurs when there is class conflict. There is no need to > package flink-core in your own jar. It is already contained i

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

I miss wrote. It’s version 1.18. This is latest and works locally but not on aws emr and I get class not found exception. On Tue, 12 Mar 2024 at 1:25 PM, Zhanghao Chen wrote: > Hi Sachin, > > Flink 1.8 series have already been out of support, have you tried with a > newer vers

Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

Hi, We have installed a flink cluster version 1.8.0 on AWS EMR. However when we submit a job we get the following error: (Do note that when we submit the same job on a local instance of Flink 1.8.1 it is working fine. The fat jar we submit has all the flink dependencies from 1.8.0 including the cl

Re: REST api: how to upload jar?

Hey Cliff You can upload a jar file using http post with the file data sent under a form field 'jarfile'. Can you also please open a jira for fixing the documentation? - Sachin On Jan 25, 2017 06:55, "Cliff Resnick" wrote: > The 1.2 release documentation (https://ci

Running into memory issues while running on Yarn

7:00:08,825 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1483603191971_0002_01_02 is : 143 Is the memory available on my pc not enough or are there any known issues which might lead to this? Also, this doesn't occur every time I start a flink session. Thanks Sachin

Re: Running continuously on yarn with kerberos

Usually, if all the dependencies are being downloaded, i.e., on the first build, it'll likely take 30-40 minutes. Subsequent builds might take 10 minutes approx. [I have the same PC configuration.] -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Sun, Nov 8, 2015 at 2:05 AM,

Re: Bug in Hybrid Hash Join

Okay. Great! Please re-open the jira in case the issue hasn't been resolved. Cheers! Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Tue, Oct 13, 2015 at 4:00 PM, Flavio Pompermaier wrote: > I ensured that I was using the latest version and I showed to pr

Re: Bug in Hybrid Hash Join

Hi Flavio This was addressed in Flink-2763. Can you check if you're using the latest version? Cheers! Sachin On Oct 13, 2015 5:04 AM, "Flavio Pompermaier" wrote: > Hi guys, > my job fails on Flink 0.10-snapshot with the following message: Bug in > Hybrid Hash Join: Requ

Re: Stuck builds on travis

Found another one: https://travis-ci.org/apache/flink/jobs/84473635 -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Fri, Oct 9, 2015 at 7:06 PM, Sachin Goel wrote: > These two builds are stuck on travis. It seems to a travis issue and is > limiting the number of conc

Stuck builds on travis

These two builds are stuck on travis. It seems to a travis issue and is limiting the number of concurrent builds to 3. https://travis-ci.org/apache/flink/jobs/84317317 https://travis-ci.org/apache/flink/jobs/84405887 Perhaps someone from infra should cancel them. -- Sachin Goel Computer Science

Re: Distribute DataSet to subset of nodes

Of course, someone else might have better ideas in re the partitioner. :) On Sep 14, 2015 1:12 AM, "Sachin Goel" wrote: > Hi Stefan > Just a clarification : The output corresponding to an element based on the > whole data will be a union of the outputs based on the two hal

Re: "Not enough free slots available to run the job" for word count example

Hi Daniel Your problem did get solved, I assume. As for the -p flag, it determines the default parallelism of operators at runtime. If you end up specifying a value more than the slots available, that's an issue. Hope that helped. Cheers Sachin On Sep 13, 2015 9:13 PM, "Daniel Blazevs

Re: Distribute DataSet to subset of nodes

, one solution could be to simply have two flatMap operations based on parts of the *broadcast* data set, and take a union. Cheers Sachin On Sep 13, 2015 7:04 PM, "Stefan Bunk" wrote: > Hi! > > Following problem: I have 10 nodes on which I want to execute a flatMap > operato

Re: Convergence Criterion in IterativeDataSet

Hi Andres Does something like this solve what you're trying to achieve? https://github.com/apache/flink/pull/918/files Regards Sachin On Sep 4, 2015 6:24 PM, "Stephan Ewen" wrote: > I think you can do this with the current interface. The convergence > criterion object

Re: Multiple restarts of Local Cluster

I'm not sure what you mean by "Crucial cleanup is in shutdown hooks". Could you elaborate? -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Wed, Sep 2, 2015 at 10:25 PM, Stephan Ewen wrote: > You can always shut down a cluster manually (via shutdown()) and i

Re: Multiple restarts of Local Cluster

If I'm right, all Tests use either the MultipleProgramTestBase or JavaProgramTestBase​. Those shut down the cluster explicitly anyway. I will make sure if this is the case. Regards Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Wed, Sep 2, 2015 at 9:40 PM, Till Roh

Re: Multiple restarts of Local Cluster

to start up the cluster only once. Regards Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Wed, Sep 2, 2015 at 9:33 PM, Till Rohrmann wrote: > Oh sorry, then I got the wrong context. I somehow thought it was about > test cases because I read `MultipleProgramTestBase` etc.

Re: Multiple restarts of Local Cluster

user to terminate the cluster after execution, while in local mode. -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Wed, Sep 2, 2015 at 9:19 PM, Till Rohrmann wrote: > Why is it not possible to shut down the local cluster? Can’t you shut it > down in the @AfterClass

  1   2   >