Organize env using files

I have 3 different files for env: test, staging and production. Each of those has different parameters like: kafka host, endpoint urls, redis connection host… I read about `https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#register-the-parameters-globally`

Re: How would I use OneInputStreamOperator to deal with data skew?

I think you might mixed some test codes with the operator. "List getOutputs()" is from "TestMapBundleFunction" and only used for validation. For the real usage, you need to write whatever records you want to emit to the "collector" which passed in during "finishBundle". Best, Kurt On Wed, Apr

Re: 回复：Is it possible to handle late data when using table API?

Hi Lasse, > some devices can deliver data days back in time and I would like to have the results as fast as possible. What JingsongLee said is correct. However, it's possible to handle your problem with Table API according to your description above. You can use the non-window(or unbounded) aggre

Re: Retain metrics counters across task restarts

Hi Zhijiang Thanks for the clarification we were thinking about the very same solution, we'll then go in this direction. Best Peter zhijiang ezt írta (időpont: 2019. ápr. 15., H, 4:28): > Hi Peter, > > The lifecycle of these metrics are coupled with lifecycle of task, So the > metrics would be

Log Management

Hello! I am working on logging for our Flink/Kubernetes infrastructure to our external corporate ElasticSearch cluster. I have a few ideas to explore and wondered if anyone had any feedback/experience to share. Ideas I am exploring right now: 1) Add a K8s configmap that contains an updated log4j

Re: inconsistent module descriptor for hadoop uber jar (Flink 1.8)

I am using sbt and sbt-assembly. In build.sbt libraryDependencies ++= Seq("org.apache.flink" % "flink-shaded-hadoop2-uber" % "2.8.3-1.8.0") Hao Sun On Tue, Apr 16, 2019 at 12:07 AM Gary Yao wrote: > Hi, > > Can you describe how to reproduce this? > > Best, > Gary > > On Mon, Apr 15, 2019 at

Re: How would I use OneInputStreamOperator to deal with data skew?

Hi Kurt, How do you make the finishBundle method returns the combined tuples? I saw that there is a method "List getOutputs()" which is

Re: Apache Flink - CEP vs SQL detecting patterns

Hi Mans, Yes you are right. That feature is not available in SQL, as there is no such feature in SQL standard. The only addition to SQL standard we introduce so far is the WITHIN clause. We might introduce the timed out patterns some time in the future, but personally I am not aware of such plans.

Re: Queryable state support in Flink 1.9

Thanks thats it. Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Apr 16, 2019, at 8:31 AM, Guowei Ma wrote: > > AbstractQueryableStateTestBase

Checkpoint/Savepoint error: Could not find any valid local directory

When trying to savepoint or checkpoint we've begun getting this error: org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for s3ablock-0001- I believe this error can occur when there's no space available. Task and Jo

Re: Queryable state support in Flink 1.9

Hi, 1. I think Mini cluster supports queryable state. 2. You could set queryable-state.enable to true and try again. You could check AbstractQueryableStateTestBase and there are some tests. :) Best, Guowei Boris Lublinsky 于2019年4月16日周二 下午9:09写道： > Thanks Guowei > The questions that I am askin

Re: Queryable state support in Flink 1.9

Thanks Guowei The questions that I am asking is slightly different: 1. Does Mini cluster support queryable state? 2. If the answer is yes, how to set it up? Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Apr 15, 2019, at 12:07 AM, Guowei Ma wrote: >

Re: How to calculate moving average result using flink sql ?

Thanks, I'll check it out. Fabian Hueske 于2019年4月16日周二 下午6:37写道： > hi Lifei, > > This sounds to me like you need an OVER window aggregation. > OVER is a standard SQL clause to compute aggregates for each row over a > group of surrounding rows (defined by ordering and partitioning). > > Check out

Re: How to calculate moving average result using flink sql ?

hi Lifei, This sounds to me like you need an OVER window aggregation. OVER is a standard SQL clause to compute aggregates for each row over a group of surrounding rows (defined by ordering and partitioning). Check out the documentation [1]. The example only shows ROW based windows, but Flink also

Re: Is it possible to handle late data when using table API?

To set rowtime watermarks delay of source you can: val desc = Schema() .field("a", Types.INT) .field("e", Types.LONG) .field("f", Types.STRING) .field("t", Types.SQL_TIMESTAMP) .rowtime(Rowtime().timestampsFromField("t").watermarksPeriodicBounded(1000)) Use watermarksPeriodicBounded a

Re: 回复：Is it possible to handle late data when using table API?

Hi Thanks for the fast reply. Unfortunately it not an option as some devices can deliver data days back in time and I would like to have the results as fast as possible. I have to convert my implementation to use streaming API instead. Med venlig hilsen / Best regards Lasse Nedergaard > Den

回复：Is it possible to handle late data when using table API?

Hi @Lasse Nedergaard, Table API don't have allowedLateness api. But you can set rowtime.watermarks.delay of source to slow down the watermark clock. -- 发件人：Lasse Nedergaard 发送时间：2019年4月16日(星期二) 16:20 收件人：user 主　题：Is it possible to

Is it possible to handle late data when using table API?

Hi. I have a simple tumble window working on eventtime like this. Table res30MinWindows = machineInsights .window(Tumble.over("30.minutes").on("UserActionTime").as("w")) // define window .groupBy("machineId, machineInsightId, w") // group by key and window .select("machine

How to calculate moving average result using flink sql ?

Hi, all: I have a `Order` table as follow: rowtime item price === == 09:00 item1 10 09:01 item2 15 09:03 item1 20 I want to calculate the moving average price in past 5 minutes, and emit the result for every record. how to do this using flink sql? R

Re: inconsistent module descriptor for hadoop uber jar (Flink 1.8)

Hi, Can you describe how to reproduce this? Best, Gary On Mon, Apr 15, 2019 at 9:26 PM Hao Sun wrote: > Hi, I can not find the root cause of this, I think hadoop version is mixed > up between libs somehow. > > --- ERROR --- > java.text.ParseException: inconsistent module descriptor file found