Re: Utilising EMR's master node

Hi Averell, There is no general answer to your question. If you are running more TMs, you get better isolation between different Flink jobs because one TM is backed by one JVM [1]. However, every TMs brings additional overhead (heartbeating, running more threads, etc.) [1]. It also depends on the

Re: Utilising EMR's master node

Thank you Gary. Regarding your previous suggestion to to change the configuration regarding to the number of vcores on the EMR master node, I tried and found one funny/bad behaviour as following: * hardware onfiguration: master node: 4vcores + 8GB ram, 2x executors with 16vcores + 32GB ram each.

Re: Scheduling sources

Hi Averell, If the 2a fits in memory, then you can load the data to all TMs in the open() method of any rich function, eg. ProcessFunction [1]. The open() runs before any data is allowed to flow in your pipeline from the sources. Cheers, Kostas [1] https://ci.apache.org/projects/flink/flin

Re: Scheduling sources

Hi Kostas, So that means my 2a will be broadcasted to all TMs? Is that possible to partition that? As I'm using CoProcessFunction to join 1 with 2. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Could not find a suitable table factory for 'org.apache.flink.table.factories.DeserializationSchemaFactory' in the classpath.

Hi, actually it should not be neccessary to put the flink-json format into /lib. Is the class `org.apache.flink.formats.json.JsonRowFormatFactory` present in the jar file you are creating with Maven? There should also be an entry in `META_INF/services/org.apache.flink.table.factories.TableFac

Re: Rocksdb Metrics

Hi Sayat Before this future is on, you could also find some metrics information, such as hit/miss count, file status from RocksDB itself. By default, RocksDB will dump its stats to its information LOG file every 10 minutes (you could call DBOptions.setStatsDumpPeriodSec to reduce the time inter

Re: Should Queryable State Server be listening on 127.0.1.1?

Hi Andrew, I think you ran into the same problem we discussed here [1]. I think it is a bug and the KvStateServerImpl or more specifically the AbstractServerBase should bind to any address. Or at least it should be configurable similar to RestOptions#BIND_ADDRESS. I've opened a JIRA issue to fix

Re: Scheduling sources

Hi Averell, As Till pointed out, currently Flink doesn't provide such a flexible schedule strategy. However, our team internally implemented a mechanism that allow user-define schedule plugin(flexible schedule strategy). It could fit your case by setting a timer on schedule start and trigger (1)

Re: Flink TaskManagers do not start until job is submitted in YARN

Hi Till, What I was ideally looking for was to have a completely managed service for Flink via AWS EMR in which YARN Cluster would be completely dedicated to only one Flink Session and as the EMR scales in and out, EMR/YARN would add/remove TMs accordingly. I could then get the value of total task

Measure task execution time

Hello, I am trying to measure the total time my flink app takes. I've read it can be done with this: val r = env.execute("NAME") System.out.println("The job took " + r.getNetRuntime(TimeUnit.SECONDS) + " s to execute") The thing is, it works for some task, where all computations occur in the sa

Re: Flink 1.5.4 -- issues w/ TaskManager connecting to ResourceManager

Hey Jamie, we've been facing the same issue with dA Platform, when running Flink 1.6.1. I assume a lot of people will be affected by this. On Tue, Sep 25, 2018 at 11:18 PM Till Rohrmann wrote: > Hi Jamie, > > thanks for the update on how to fix the problem. This is very helpful for > the rest

Re: Scheduling sources

Hi Tison, "/setting a timer on schedule start and trigger (1) after a fixed delay/" would be quite sufficient for me. Looking forward to the change of that Jira ticket's status. Thanks for your help. Regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabbl

Re: Flink 1.5.4 -- issues w/ TaskManager connecting to ResourceManager

Should we add a warning to the release announcements? Fabian Am Mi., 26. Sep. 2018 um 10:22 Uhr schrieb Robert Metzger < rmetz...@apache.org>: > Hey Jamie, > > we've been facing the same issue with dA Platform, when running Flink > 1.6.1. > I assume a lot of people will be affected by this. > >

Re: Flink takes 40ms ~ 100ms to proceed from one operator to another?

Hi Stefan, Sorry for jumping in the discussion. I’ve seen a blog post [1] of dataArtisans which says that object reuse has not much influence on data streams. > For Flink’s DataStream API, this setting does in fact not even result in > reusing of objects, but only in avoiding additional obje

Re: Flink takes 40ms ~ 100ms to proceed from one operator to another?

Hi Paul, sure, what I mean is basically what this comment in the blogpost says: „For Flink’s DataStream API, this setting does in fact not even result in reusing of objects, but only in *avoiding additional object copying* on the way, which happens by default as an additional safety net for use

Re: Question about Window Tigger

You configure the ExecutionConfig#setAutoWatermarkInterval on the StreamExecutionEnvironment by calling getConfig(). Cheers, Till On Tue, Sep 25, 2018 at 11:54 PM Chang Liu wrote: > Hi Till, > > You mean use AssignerWithPeriodicWatermarks but combine the logic of Event > time and Processing Tim

Re: Flink 1.5.4 -- issues w/ TaskManager connecting to ResourceManager

Yes, that would be a good idea. I think it should go into the release notes. Will add it. On Wed, Sep 26, 2018 at 10:24 AM Fabian Hueske wrote: > Should we add a warning to the release announcements? > > Fabian > > Am Mi., 26. Sep. 2018 um 10:22 Uhr schrieb Robert Metzger < > rmetz...@apache.org

Re: Help: how to get latency value comfortable in Flink1.5?

Hi Rui, 1) Flink should always fetch records from Kafka if there some independent of the parallelism of the consumer. The only problem which could appear is that if you set the parallelism higher than the number of partitions, some of the source operators won't get a partition assigned. Due to tha

Re: Flink takes 40ms ~ 100ms to proceed from one operator to another?

Hi Stefan, Thanks for your detailed explanation! It helps a lot! I think I misunderstood the sentence. I thought “avoiding additional object copying” was the default behavior. Best, Paul Lam > 在 2018年9月26日，17:22，Stefan Richter 写道： > > Hi Paul, > > sure, what I mean is basically what this c

Re: Rocksdb Metrics

Thank you for this information. @Yun is there an easy way to expose a number of records in rockdsdb? On Wed, Sep 26, 2018 at 9:47 AM Yun Tang wrote: > Hi Sayat > > Before this future is on, you could also find some metrics information, > such as hit/miss count, file status from RocksDB itself. B

Re: Rocksdb Metrics

actually, once I wrote my question I've realized that I can do it with custom metrics and getting easily the size of the state map. On Wed, Sep 26, 2018 at 11:57 AM Sayat Satybaldiyev wrote: > Thank you for this information. @Yun is there an easy way to expose a > number of records in rockdsdb?

Re: RocksDB Read IOPs

Hi Yun, > From your description, I think you actually concern more about the overall > performance instead of the high disk IOPs. Maybe you should first ensure > whether the job performance degradation is related to RocksDB's performance. You are right that my main concern is the overall perfor

Re: FlinkCEP, circular references and checkpointing failures

Hi, I think I've hit this same issue on a 3 node standalone cluster (1.4.2) using HDFS (2.8.4) as state backend. 2018-09-26 17:07:39,370 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to fail task externally SelectCepOperator (1/1) (3bec4aa1ef2226c4e0c5ff7b3860d34

OpenSSL use in Flink

Hello, I have a query regarding OpenSSL usage in Flink. Please let me know if Flink uses OpenSSL and SSL_CTX API's. Thanks, Suchithra

Re: FlinkCEP, circular references and checkpointing failures

Hi Shailesh, Are you sure you are using version 1.4.2? Do you run a vanilla flink, or have you introduced some changes? I am asking cause the lines in stacktrace does not align with the source code for 1.4.2. Also it is a different exception than the one in the issue you've linked, so if it is a

Question about sharing resource among slots with a TM

We use Hbase extensively and the general pattern we follow is acquiring a Connection in the open() method of a RichFunction and closing in the close() method. Of course that implies that if we have a parallelism of n, there will be n Hbase Connections. We want to use the fact that Hbase connection

Can't get the FlinkKinesisProducer to work against Kinesalite for tests

Hi, We have started to use Kinesis with Flink and we need to be able to test when a Flink jobs writes to Kinesis. For that, we use a docker image with Kinesalite. To configure the producer, we do like it is explained in the docs [1]. However, if we use this code, the job submission is going to f

Re: OpenSSL use in Flink

Hi Suchithra, did you take a look at the documentation [1] about the SSL setup? Regards, Timo [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/security-ssl.html Am 26.09.18 um 14:08 schrieb V N, Suchithra (Nokia - IN/Bangalore): Hello, I have a query regarding OpenSSL u

Re: Question about sharing resource among slots with a TM

any one? On Wed, Sep 26, 2018 at 9:15 AM Vishal Santoshi wrote: > We use Hbase extensively and the general pattern we follow is acquiring a > Connection in the open() method of a RichFunction and closing in the > close() method. Of course that implies that if we have a parallelism of n, > there

Re: Question about sharing resource among slots with a TM

Hi vishal, Yes, we can define a static connection to reuse it or implement a connection pool. Maybe we can also ask the problem in hbase community and see if there are other better ways. Best, Hequn On Thu, Sep 27, 2018 at 12:40 AM Vishal Santoshi wrote: > any one? > > On Wed, Sep 26, 2018 at

Re: Help: how to get latency value comfortable in Flink1.5?

Hi Till, I have copy the flink-metrics-graphite-1.5.2.jar to lib/, and restart flink.It seems make no difference. When I checked the jobmanager log, the metrics configured in flink-conf.yaml, could be load since it printed as below: "2018-09-27 09:50:23,953 INFO org.apache.flink.configuration.Gl

Re: FlinkCEP, circular references and checkpointing failures

Hi Dawid, Yes, it is version 1.4.2. We are running vanilla flink, but have added a couple of changes in the CEP operator specifically (top 3 commits here: https://github.com/jainshailesh/flink/commits/poc_on_1.4.2). Changes I've made to CEP operators do not touch the checkpointing path, just overl

Metrics: (1) Histogram in prometheus and (2) meter by event_time

Good day everyone, I have a stream with two timestamps (ts1 and ts2) inside each record. My event time is ts1. This ts1 has value truncated to a quarter (like 23:30, 23:45, 00:00,...) I want to report two metrics: 1. A meter which counts number of records per value of ts1. (fig.1)