Re: error: Static methods in interface require -target:jvm-1.8 using scala 2.11

2019-09-10 Thread Ben Yan
thanks a lot!It works. Jeff Zhang 于2019年9月11日周三 下午12:58写道: > Add this to your scala-maven-plugin > > > > > -target:jvm-1.8 > > > > > > > > Ben Yan 于2019年9月11日周三 下午12:07写道: > >> The following is the environment I use: >> 1. flink.version: 1.9.0 >> 2. java version "1.8.0_212" >> 3. scal

Problem starting taskexecutor daemons in 3 node cluster

2019-09-10 Thread Komal Mariam
I'm trying to set up a 3 node Flink cluster (version 1.9) on the following machines: Node 1 (Master) : 4 GB (3.8 GB) Core2 Duo 2.80GHz, Ubuntu 16.04 LTS Node 2 (Slave) : 16 GB, Core i7-3.40GHz, Ubuntu 16.04 LTS Node 3 (Slave) : 16 GB, Core i7-3,40GHz, Ubuntu 16.04 LTS I have followed the instruc

Re: error: Static methods in interface require -target:jvm-1.8 using scala 2.11

2019-09-10 Thread Jeff Zhang
Add this to your scala-maven-plugin -target:jvm-1.8 Ben Yan 于2019年9月11日周三 下午12:07写道: > The following is the environment I use: > 1. flink.version: 1.9.0 > 2. java version "1.8.0_212" > 3. scala version: 2.11.12 > > When I wrote the following code in the scala programming language

Re: Checkpointing is not performing well

2019-09-10 Thread Vijay Bhaskar
You crossed the upper limits of the check point system of Flink a way high. Try to distribute events equally over time by adding some sort of controlled back pressure after receiving data from kinesis streams. Otherwise the spike coming during 5 seconds time would always create problems. Tomorrow

error: Static methods in interface require -target:jvm-1.8 using scala 2.11

2019-09-10 Thread Ben Yan
The following is the environment I use: 1. flink.version: 1.9.0 2. java version "1.8.0_212" 3. scala version: 2.11.12 When I wrote the following code in the scala programming language, I found the following error: // set up the batch execution environment val bbSettings = EnvironmentSettings.newI

[no subject]

2019-09-10 Thread Ben Yan
The following is the environment I use: 1. flink.version: 1.9.0 2. java version "1.8.0_212" 3. scala version: 2.11.12 When I wrote the following code in the scala programming language, I found the following error: // set up the batch execution environment val bbSettings = EnvironmentSettings.newI

Filter events based on future events

2019-09-10 Thread theo.diefent...@scoop-software.de
Hi there, I have the following use case:I get transaction logs from multiple servers. Each server puts its logs into its own Kafka partition so that within each partition the elements are monothonically ordered by time. Within the stream of transactions, we have some special events. Let's call them

Re: Checkpointing is not performing well

2019-09-10 Thread Ravi Bhushan Ratnakar
@Rohan - I am streaming data to kafka sink after applying business logic. For checkpoint, I am using s3 as a distributed file system. For local recovery, I am using Optimized iops ebs volume. @Vijay - I forget to mention that incoming data volume is ~ 10 to 21GB per minute compressed(lz4) avro mes

Re: Flink SQL: How to tag a column as 'rowtime' from a Avro based DataStream?

2019-09-10 Thread Fabian Hueske
Hi, that would be regular SQL cast syntax: SELECT a, b, c, CAST(eventTime AS TIMESTAMP) FROM ... Am Di., 10. Sept. 2019 um 18:07 Uhr schrieb Niels Basjes : > Hi. > > Can you give me an example of the actual syntax of such a cast? > > On Tue, 10 Sep 2019, 16:30 Fabian Hueske, wrote: > >> Hi Ni

Re: Error While Building Flink From Source

2019-09-10 Thread Yuval Itzchakov
Never mind, turns out it was an error on my part. Somehow I managed do add an "S" to an attribute mistakenly :) On Tue, Sep 10, 2019 at 7:29 PM Yuval Itzchakov wrote: > Still getting the same error message using your command. Which Maven > version are you using? > > On Tue, Sep 10, 2019 at 6:39

Re: Error While Building Flink From Source

2019-09-10 Thread Yuval Itzchakov
Still getting the same error message using your command. Which Maven version are you using? On Tue, Sep 10, 2019 at 6:39 PM Debasish Ghosh wrote: > I could build using the following command .. > > mvn clean install -Dcheckstyle.skip -DskipTests -Dscala-2.12 > -Drat.skip=true > > regards. > > On

Re: Flink SQL: How to tag a column as 'rowtime' from a Avro based DataStream?

2019-09-10 Thread Niels Basjes
Hi. Can you give me an example of the actual syntax of such a cast? On Tue, 10 Sep 2019, 16:30 Fabian Hueske, wrote: > Hi Niels, > > I think (not 100% sure) you could also cast the event time attribute to > TIMESTAMP before you emit the table. > This should remove the event time property (and t

Re: Error While Building Flink From Source

2019-09-10 Thread Debasish Ghosh
I could build using the following command .. mvn clean install -Dcheckstyle.skip -DskipTests -Dscala-2.12 -Drat.skip=true regards. On Tue, Sep 10, 2019 at 9:06 PM Yuval Itzchakov wrote: > Hi, > I'm trying to build Flink from source. I'm using Maven 3.6.1 and executing > the following command:

Error While Building Flink From Source

2019-09-10 Thread Yuval Itzchakov
Hi, I'm trying to build Flink from source. I'm using Maven 3.6.1 and executing the following command: mvn clean install -DskipTests -Dfast -Dscala-2.12 Running both on the master branch and the release-1.9.0 tag, I get the following error: [ERROR] Failed to execute goal org.apache.maven.plugins:

Re: Join with slow changing dimensions/ streams

2019-09-10 Thread Fabian Hueske
Hi Hanan, BroadcastState and CoMap (or CoProcessFunction) have both advantages and disadvantages. Broadcast state is better if the broadcasted side is small (only low data rate). Its records are replicated to each instance but the other (larger) stream does not need to be partitioned and stays on

Re: Flink SQL: How to tag a column as 'rowtime' from a Avro based DataStream?

2019-09-10 Thread Fabian Hueske
Hi Niels, I think (not 100% sure) you could also cast the event time attribute to TIMESTAMP before you emit the table. This should remove the event time property (and thereby the TimeIndicatorTypeInfo) and you wouldn't know to fiddle with the output types. Best, Fabian Am Mi., 21. Aug. 2019 um 1

How to implement grouping set in stream

2019-09-10 Thread 刘建刚
I want to implement grouping set in stream. I am new to flink sql. I want to find a example to teach me how to self define rule and implement corresponding operator. Can anyone give me any suggestion?

Re: Checkpointing is not performing well

2019-09-10 Thread Vijay Bhaskar
For me task count seems to be huge in number with the mentioned resource count. To rule out the possibility of issue with state backend can you start writing sink data as , i.e., data ignore sink. And try whether you could run it for longer duration without any issue. You can start decreasing the

Re: How do I start a Flink application on my Flink+Mesos cluster?

2019-09-10 Thread Felipe Gutierrez
I managed to find what was going wrong. I will write here just for the record. First, the master machine was not login automatically at itself. So I had to give permission for it. chmod og-wx ~/.ssh/authorized_keys chmod 750 $HOME Then I put the number of "mesos.resourcemanager.tasks.cpus" to be

flinkML for production

2019-09-10 Thread pyh
Hello, Is there any production practice on using flinkML for machine learning? If so, where is the link? Thanks.