Hi again Kurt, could you please help me with the pom.xml file? I have included all Table ecosystem dependencies and the flink-table-runtime-blink as well. However the class org.apache.flink.table.runtime.context.ExecutionContext is still not found. I guess I am missing some dependency, but I do not know which. This is my pom.xml file.
<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <groupId>org.sense.flink</groupId> <artifactId>explore-flink</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>explore-flink</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <jdk.version>1.8</jdk.version> <scala.binary.version>2.11</scala.binary.version> <!-- <flink.version>1.8.0</flink.version> --> <flink.version>1.9-SNAPSHOT</flink.version> <junit.version>4.12</junit.version> </properties> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-metrics-dropwizard</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <!-- Table ecosystem --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-scala-bridge_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-runtime-blink</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.fusesource.mqtt-client</groupId> <artifactId>mqtt-client</artifactId> <version>1.15</version> <!-- <scope>provided</scope> --> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>1.7.26</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>${junit.version}</version> </dependency> </dependencies> <build> <finalName>explore-flink</finalName> <plugins> <!-- download source code in Eclipse, best practice --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-eclipse-plugin</artifactId> <version>2.10</version> <configuration> <downloadSources>true</downloadSources> <downloadJavadocs>false</downloadJavadocs> </configuration> </plugin> <!-- Set a compiler level --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.8.0</version> <configuration> <source>${jdk.version}</source> <target>${jdk.version}</target> </configuration> </plugin> <!-- Maven Shade Plugin --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.2.0</version> <!-- Run shade goal on package phase --> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <exclude>org.apache.flink:*</exclude> <!-- Also exclude very big transitive dependencies of Flink WARNING: You have to remove these excludes if your code relies on other versions of these dependencies. --> <exclude>org.slf4j:*</exclude> <exclude>log4j:*</exclude> <exclude>com.typesafe:config:*</exclude> <exclude>junit:junit:*</exclude> <exclude>com.codahale.metrics:*</exclude> </excludes> </artifactSet> <filters> <filter> <artifact>org.apache.flink:*</artifact> <excludes> <!-- exclude shaded google but include shaded curator --> <exclude>org/apache/flink/shaded/com/**</exclude> <exclude>web-docs/**</exclude> </excludes> </filter> <filter> <!-- Do not copy the signatures in the META-INF folder. Otherwise, this might cause SecurityExceptions when using the JAR. --> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> <filter> <artifact>*:*</artifact> <includes> <include>org/apache/calcite/**</include> <include>org/apache/flink/calcite/shaded/**</include> <include>org/apache/flink/table/**</include> <include>org.codehaus.commons.compiler.properties</include> <include>org/codehaus/janino/**</include> <include>org/codehaus/commons/**</include> </includes> </filter> </filters> <!-- If you want to use ./bin/flink run <quickstart jar> uncomment the following lines. This will add a Main-Class entry to the manifest file --> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>org.sense.flink.App</mainClass> </transformer> </transformers> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Thanks *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Mon, Apr 15, 2019 at 3:25 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > oh, yes. I just saw. I will use 1.9 then. thanks > > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > > On Mon, Apr 15, 2019 at 3:23 PM Kurt Young <ykt...@gmail.com> wrote: > >> It's because all blink codes are not shipped with 1.8.0, they current >> only available in 1.9-SNAPSHOT. >> >> Best, >> Kurt >> >> >> On Mon, Apr 15, 2019 at 7:18 PM Felipe Gutierrez < >> felipe.o.gutier...@gmail.com> wrote: >> >>> Hi, >>> >>> what are the artifacts that I have to import on maven in order to use >>> Blink Api? >>> >>> I am using Flink 1.8.0 and I am trying to import blink code to use its >>> ExecutionContext >>> <https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/context/ExecutionContext.java>. >>> I want to do this in order to implement my own operator like it is >>> implemented here >>> <https://github.com/apache/flink/blob/blink/flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/bundle/BundleOperator.java>. >>> I guess if I import flink-table everything should come inside the same jar >>> as it is done here >>> <https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/pom.xml>. >>> However, I cannot import "flink-table-runtime-blink". Eclipse says that it >>> is a missing artifact. >>> >>> <dependency> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-table-planner_2.11</artifactId> >>> <version>1.8.0</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-table-api-java-bridge_2.11</artifactId> >>> <version>1.8.0</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-streaming-scala_2.11</artifactId> >>> <version>1.8.0</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-table-common</artifactId> >>> <version>1.8.0</version> >>> </dependency> >>> <dependency> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-table</artifactId> >>> <version>1.8.0</version> >>> <type>pom</type> >>> <scope>provided</scope> >>> </dependency> >>> <dependency> <!-- THIS IS NOT POSSIBLE TO IMPORT --> >>> <groupId>org.apache.flink</groupId> >>> <artifactId>flink-table-runtime-blink</artifactId> >>> <version>1.8.0</version> >>> </dependency> >>> >>> >>> *--* >>> *-- Felipe Gutierrez* >>> >>> *-- skype: felipe.o.gutierrez* >>> *--* *https://felipeogutierrez.blogspot.com >>> <https://felipeogutierrez.blogspot.com>* >>> >>> >>> On Mon, Apr 15, 2019 at 9:49 AM Felipe Gutierrez < >>> felipe.o.gutier...@gmail.com> wrote: >>> >>>> Cool, thanks Kurt! >>>> *-* >>>> *- Felipe Gutierrez* >>>> >>>> *- skype: felipe.o.gutierrez* >>>> *- **https://felipeogutierrez.blogspot.com >>>> <https://felipeogutierrez.blogspot.com>* * >>>> <https://felipeogutierrez.blogspot.com>* >>>> >>>> >>>> On Mon, Apr 15, 2019 at 6:06 AM Kurt Young <ykt...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> You can checkout the bundle operator which used in Blink to perform >>>>> similar thing you mentioned: >>>>> https://github.com/apache/flink/blob/blink/flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/bundle/BundleOperator.java >>>>> >>>>> Best, >>>>> Kurt >>>>> >>>>> >>>>> On Fri, Apr 12, 2019 at 8:05 PM Felipe Gutierrez < >>>>> felipe.o.gutier...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I was trying to implement a better way to handle data skew using >>>>>> Flink and I found this talk from #FlinkForward SF 2017: "Cliff >>>>>> Resnick & Seth Wiesman - From Zero to Streaming >>>>>> <https://youtu.be/mSLesPzWplA?t=835>" [1] which says that they used >>>>>> OneInputStreamOperator [2]. Through it, they could implement the >>>>>> "combiner" >>>>>> in Hadoop (execute part of the reduce tasks on the Map phase, before >>>>>> shuffling). >>>>>> >>>>>> I need some help here. What are some of the Flink source-code >>>>>> operators that I can peek up to implement my on operator that deals with >>>>>> data skew? Or maybe, is there someone that have an example of a use case >>>>>> similar to this? >>>>>> >>>>>> [1] https://youtu.be/mSLesPzWplA?t=835 >>>>>> [2] >>>>>> https://ci.apache.org/projects/flink/flink-docs-master/api/java/index.html?org/apache/flink/streaming/api/functions/source/ContinuousFileReaderOperator.html >>>>>> >>>>>> Thanks! >>>>>> Felipe >>>>>> >>>>>> *--* >>>>>> *-- Felipe Gutierrez* >>>>>> >>>>>> *-- skype: felipe.o.gutierrez* >>>>>> *--* *https://felipeogutierrez.blogspot.com >>>>>> <https://felipeogutierrez.blogspot.com>* >>>>>> >>>>>
