Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Thanks Just to do a test I have under root directory md_streaming the following hduser@rhes75: /home/hduser/dba/bin/flink/md_streaming> ltr total 24 drwxr-xr-x 3 hduser hadoop 4096 Jun 30 16:10 src drwxr-xr-x 4 hduser hadoop 4096 Jun 30 16:13 .. -rw-r--r-- 1 hduser hadoop 7732 Jul 1 22:40 pom.xm

Re: Let BucketingSink roll file on each checkpoint

Thank you Minglei, I should describe my current flow and my requirement more clearly. 1. any data we collect have a send-time 2. when collect data, we also send another counter message, says we have collect 45 message whose send-time is 2018-07-02 10:02:00 3. data is sent to kafka(or other message

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Hi, Mich From your jar contents, it seems not correct. There is not md_streaming class in your jar. You should put this to your jar file and kafka lib also. By the way, you need to use maven-shade-plugin or maven-assembly-plugin to help you. Cheers Minglei > 在 2018年7月2日，上午2:18，Mich Talebzadeh

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Hi, I am still not there. This is the simple Scala program that I created a jar file for using mvn import java.util.Properties import java.util.Arrays import org.apache.flink.api.common.functions.MapFunction import org.apache.flink.api.java.utils.ParameterTool import org.apache.flink.streaming.a

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Thanks Franke. That did the trick! [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar. [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar. [DEBUG] Processing JAR /home/hduser/dba/bin/flink/md_streaming

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Shouldn’t it be 1.5.0 instead of 1.5? > On 1. Jul 2018, at 18:10, Mich Talebzadeh wrote: > > Ok some clumsy work by me not creating the pom.xml file in flink > sub-directory (it was putting it in spark)! > > Anyhow this is the current issue I am facing > > [INFO] > --

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Ok some clumsy work by me not creating the pom.xml file in flink sub-directory (it was putting it in spark)! Anyhow this is the current issue I am facing [INFO] [INFO] BUILD FAILURE [INFO] ---

Re: Passing type information to JDBCAppendTableSink

Hi Chris, Looking at the code, seems like JDBCTypeUtil [1] is used for converting Flink TypeInformation into JDBC Type (Java.sql.type), and SQL_TIMESTAMP and SQL_TIME are both listed in the conversion mapping. However the JDBC types are different. Regarding the question whether your insert is cor

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Hi, Mich > [WARNING] Expected all dependencies to require Scala version: 2.10.4 > [WARNING] spark:scala:1.0 requires scala version: 2.11.7 > [WARNING] Multiple versions of scala libraries detected! I think you should make your scala version to 2.11 first. And try again. Cheers Minglei > 在 201

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Hi Minglei, Many thanks My flink version is 1.5 This is the pom.xml from GitHub as suggested http://maven.apache.org/POM/4.0.0"; xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";>

Re: compiling flink job with maven is failing with error: object flink is not a member of package org.apache

Hi, Mich. > Is there a basic MVN pom file for flink? The default one from GitHub does not > seem to be working! Please take a look on https://github.com/apache/flink/blob/master/flink-quickstart/flink-quickstart-java/src/main/resources/archetype-resources/pom.xml

Re: Passing type information to JDBCAppendTableSink

Full Source except for mapper and timestamp assigner. Sample Input Stream record: 1530447316589,Mary,./home What are the correct parameters to pass for data types in the JDBCAppendTableSink? Am I doing this correctly? // Get Execution Environment StreamExecut

compiling flink job with maven is failing with error: object flink is not a member of package org.apache

I have done many times with sbt or maven for spark streaming. Trying to compile a simple program that compiles ok in flink-scala.sh The imports are as follows import java.util.Properties import java.util.Arrays import org.apache.flink.api.common.functions.MapFunction import org.apache.flink.api.j

Re: Passing type information to JDBCAppendTableSink

can you share the full code . On Sun, Jul 1, 2018 at 12:49 PM chrisr123 wrote: > > I'm trying to determine if I'm specifying type information properly when > doing an INSERT using > the JDBCAppendTableSink API. Specifically, how do I specify timestamp and > date types? It looks like > I need t

Passing type information to JDBCAppendTableSink

I'm trying to determine if I'm specifying type information properly when doing an INSERT using the JDBCAppendTableSink API. Specifically, how do I specify timestamp and date types? It looks like I need to use Type.SQL_TIMESTAMP for a timestamp but BasicTypeInfo for types like varchar, etc? I a

Flink cluster on physical vs virtualization environment

Hi all, Is there any use case in virtualized environment? Is it a good choice? Please share your thought. Thank you. -Yow