Github user bzz commented on the issue:
https://github.com/apache/zeppelin/pull/1334
Few questions after another pass:
1. Right now Beam interpreter uses **Beam version 0.1.0-incubating**, but
didn't Beam project release 0.2.0-incubating on 2016-08-09? Do you think it
would be useful to update it right here, before the merge or do you plan to do
that in a separate PR later on?
2. AFAIK there are multiple "runners" for Beam, and right only be reading
the doc you can see how to switch them. Do you think it would fit Zeppelin use
case better if there are 3 different interpreters `%beam.flink`, `%beam.spark`,
etc each with it's own appropriate configuration options and a user could just
skip creating a class and defining `main() {}` in every paragraph?
3. Here is the list of dependencies from `mvn dependency:tree`. All of
them will become part of the convenience binary for the next Zeppelin release.
What is needed to be done, in order for us to merge it is:
- make sure that ALL these dependencies
- and ALL their transitive dependencies
- have an appropriate licence, and are logged under the
`./zeppelin-distribution/src/bin_license/LICENSE`
- and for non-Apache license, full text of the license need to be added
to ./zeppelin-distribution/src/license/<dependency-name>`
For your convenience I have listed all current dependencies of `beam`
module below, please feel free to verify and mark those that satisfy the above
criterias.
```
- [ ] org.apache.zeppelin:zeppelin-beam:jar:0.7.0-SNAPSHOT
- [ ] +- org.apache.spark:spark-core_2.10:jar:1.4.1:compile
- [ ] | +- com.twitter:chill_2.10:jar:0.5.0:compile
- [ ] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:compile
- [ ] | | +-
com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile
- [ ] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
- [ ] | | \- org.objenesis:objenesis:jar:1.2:compile
- [ ] | +- com.twitter:chill-java:jar:0.5.0:compile
- [ ] | +- org.apache.spark:spark-launcher_2.10:jar:1.4.1:compile
- [ ] | +- org.apache.spark:spark-network-common_2.10:jar:1.4.1:compile
- [ ] | +- org.apache.spark:spark-network-shuffle_2.10:jar:1.4.1:compile
- [ ] | +- org.apache.spark:spark-unsafe_2.10:jar:1.4.1:compile
- [ ] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
- [ ] | +- org.apache.curator:curator-recipes:jar:2.4.0:compile
- [ ] | | \- org.apache.curator:curator-framework:jar:2.4.0:compile
- [ ] | | \- org.apache.curator:curator-client:jar:2.4.0:compile
- [ ] | +-
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
- [ ] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
- [ ] | +- org.apache.commons:commons-math3:jar:3.4.1:compile
- [ ] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
- [ ] | +- org.slf4j:slf4j-api:jar:1.7.10:compile
- [ ] | +- org.slf4j:jul-to-slf4j:jar:1.7.10:compile
- [ ] | +- org.slf4j:jcl-over-slf4j:jar:1.7.10:compile
- [ ] | +- log4j:log4j:jar:1.2.17:compile
- [ ] | +- com.ning:compress-lzf:jar:1.0.3:compile
- [ ] | +- org.xerial.snappy:snappy-java:jar:1.1.1.7:compile
- [ ] | +- net.jpountz.lz4:lz4:jar:1.2.0:compile
- [ ] | +- org.roaringbitmap:RoaringBitmap:jar:0.4.5:compile
- [ ] | +- commons-net:commons-net:jar:2.2:compile
- [ ] | +- org.scala-lang:scala-library:jar:2.10.4:compile
- [ ] | +- org.json4s:json4s-jackson_2.10:jar:3.2.10:compile
- [ ] | | \- org.json4s:json4s-core_2.10:jar:3.2.10:compile
- [ ] | | +- org.json4s:json4s-ast_2.10:jar:3.2.10:compile
- [ ] | | \- org.scala-lang:scalap:jar:2.10.0:compile
- [ ] | | \- org.scala-lang:scala-compiler:jar:2.10.0:compile
- [ ] | +- com.sun.jersey:jersey-server:jar:1.9:compile
- [ ] | | \- asm:asm:jar:3.1:compile
- [ ] | +- com.sun.jersey:jersey-core:jar:1.9:compile
- [ ] | +- org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1:compile
- [ ] | +- com.clearspring.analytics:stream:jar:2.7.0:compile
- [ ] | +- io.dropwizard.metrics:metrics-core:jar:3.1.0:compile
- [ ] | +- io.dropwizard.metrics:metrics-jvm:jar:3.1.0:compile
- [ ] | +- io.dropwizard.metrics:metrics-json:jar:3.1.0:compile
- [ ] | +- io.dropwizard.metrics:metrics-graphite:jar:3.1.0:compile
- [ ] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.4.4:compile
- [ ] | +-
com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.4:compile
- [ ] | | +- org.scala-lang:scala-reflect:jar:2.10.4:compile
- [ ] | | \- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
- [ ] | +- org.apache.ivy:ivy:jar:2.4.0:compile
- [ ] | +- oro:oro:jar:2.0.8:compile
- [ ] | +- org.tachyonproject:tachyon-client:jar:0.6.4:compile
- [ ] | | \- org.tachyonproject:tachyon:jar:0.6.4:compile
- [ ] | +- net.razorvine:pyrolite:jar:4.4:compile
- [ ] | +- net.sf.py4j:py4j:jar:0.8.2.1:compile
- [ ] | \- org.spark-project.spark:unused:jar:1.0.0:compile
- [ ] +- org.apache.spark:spark-streaming_2.10:jar:1.4.1:compile
- [ ] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.3.0:compile
- [ ] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
- [ ] | +- org.apache.avro:avro:jar:1.7.4:compile
- [ ] | +- com.google.inject.extensions:guice-servlet:jar:3.0:compile
- [ ] | \- io.netty:netty:jar:3.6.2.Final:compile
- [ ] +- org.apache.hadoop:hadoop-common:jar:2.3.0:compile
- [ ] | +- com.google.guava:guava:jar:15.0:compile
- [ ] | +- commons-cli:commons-cli:jar:1.2:compile
- [ ] | +- xmlenc:xmlenc:jar:0.52:compile
- [ ] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
- [ ] | +- commons-codec:commons-codec:jar:1.5:compile
- [ ] | +- commons-io:commons-io:jar:2.4:compile
- [ ] | +- commons-collections:commons-collections:jar:3.2.1:compile
- [ ] | +- javax.servlet:servlet-api:jar:2.5:compile
- [ ] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
- [ ] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
- [ ] | +- com.sun.jersey:jersey-json:jar:1.9:compile
- [ ] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
- [ ] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
- [ ] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
- [ ] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
- [ ] | +- tomcat:jasper-compiler:jar:5.5.23:compile
- [ ] | +- tomcat:jasper-runtime:jar:5.5.23:compile
- [ ] | +- javax.servlet.jsp:jsp-api:jar:2.1:compile
- [ ] | +- commons-el:commons-el:jar:1.0:compile
- [ ] | +- commons-logging:commons-logging:jar:1.1.1:compile
- [ ] | +- commons-lang:commons-lang:jar:2.5:compile
- [ ] | +- commons-configuration:commons-configuration:jar:1.9:compile
- [ ] | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
- [ ] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
- [ ] | +- org.apache.hadoop:hadoop-auth:jar:2.3.0:compile
- [ ] | | \- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
- [ ] | | \- org.apache.httpcomponents:httpcore:jar:4.3.3:compile
- [ ] | +- com.jcraft:jsch:jar:0.1.42:compile
- [ ] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
- [ ] | \- org.apache.commons:commons-compress:jar:1.4.1:compile
- [ ] | \- org.tukaani:xz:jar:1.0:compile
- [ ] +- org.apache.hadoop:hadoop-hdfs:jar:2.3.0:compile
- [ ] | \- commons-daemon:commons-daemon:jar:1.0.13:compile
- [ ] +- org.apache.hadoop:hadoop-client:jar:2.3.0:compile
- [ ] | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0:compile
- [ ] | | \-
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.3.0:compile
- [ ] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile
- [ ] | \-
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0:compile
- [ ] +- org.apache.hadoop:hadoop-annotations:jar:2.3.0:compile
- [ ] | \- jdk.tools:jdk.tools:jar:1.7:system
- [ ] +- org.apache.hadoop:hadoop-yarn-common:jar:2.3.0:compile
- [ ] | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
- [ ] | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
- [ ] | | \- javax.activation:activation:jar:1.1:compile
- [ ] | +- com.google.inject:guice:jar:3.0:compile
- [ ] | | +- javax.inject:javax.inject:jar:1:compile
- [ ] | | \- aopalliance:aopalliance:jar:1.0:compile
- [ ] | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
- [ ] +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.3.0:compile
- [ ] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.3.0:compile
- [ ] | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.3.0:compile
- [ ] +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
- [ ] | +- org.apache.commons:commons-math:jar:2.1:compile
- [ ] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
- [ ] | | \- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
- [ ] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
- [ ] | | \- ant:ant:jar:1.6.5:compile
- [ ] | +- hsqldb:hsqldb:jar:1.8.0.10:compile
- [ ] | \- org.eclipse.jdt:core:jar:3.1.1:compile
- [ ] +- com.thoughtworks.qdox:qdox:jar:2.0-M3:compile
- [ ] +- org.apache.beam:beam-runners-parent:pom:0.1.0-incubating:compile
- [ ] +-
org.apache.beam:beam-runners-core-java:jar:0.1.0-incubating:compile
- [ ] | \-
org.apache.beam:beam-sdks-java-core:jar:0.1.0-incubating:compile
- [ ] | +- io.grpc:grpc-all:jar:0.12.0:compile
- [ ] | | +- io.grpc:grpc-core:jar:0.12.0:compile
- [ ] | | +- io.grpc:grpc-okhttp:jar:0.12.0:compile
- [ ] | | | +- com.squareup.okio:okio:jar:1.6.0:compile
- [ ] | | | \- com.squareup.okhttp:okhttp:jar:2.5.0:compile
- [ ] | | +- io.grpc:grpc-protobuf-nano:jar:0.12.0:compile
- [ ] | | | \-
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-4:compile
- [ ] | | +- io.grpc:grpc-netty:jar:0.12.0:compile
- [ ] | | | +- io.netty:netty-codec-http2:jar:4.1.0.Beta8:compile
- [ ] | | | | \- io.netty:netty-codec-http:jar:4.1.0.Beta8:compile
- [ ] | | | \- com.twitter:hpack:jar:0.10.1:compile
- [ ] | | +- io.grpc:grpc-protobuf:jar:0.12.0:compile
- [ ] | | +- io.grpc:grpc-stub:jar:0.12.0:compile
- [ ] | | \- io.grpc:grpc-auth:jar:0.12.0:compile
- [ ] | +-
com.google.auth:google-auth-library-oauth2-http:jar:0.4.0:compile
- [ ] | | \-
com.google.auth:google-auth-library-credentials:jar:0.4.0:compile
- [ ] | +- io.netty:netty-handler:jar:4.1.0.Beta8:compile
- [ ] | | +- io.netty:netty-buffer:jar:4.1.0.Beta8:compile
- [ ] | | | \- io.netty:netty-common:jar:4.1.0.Beta8:compile
- [ ] | | +- io.netty:netty-transport:jar:4.1.0.Beta8:compile
- [ ] | | | \- io.netty:netty-resolver:jar:4.1.0.Beta8:compile
- [ ] | | \- io.netty:netty-codec:jar:4.1.0.Beta8:compile
- [ ] | +- com.google.api.grpc:grpc-pubsub-v1:jar:0.0.2:compile
- [ ] | | \- com.google.api.grpc:grpc-core-proto:jar:0.0.3:compile
- [ ] | +-
com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
- [ ] | +-
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
- [ ] | +-
com.google.http-client:google-http-client-jackson:jar:1.22.0:runtime
- [ ] | +-
com.google.oauth-client:google-oauth-client-java6:jar:1.22.0:compile
- [ ] | \- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
- [ ] +-
org.apache.beam:beam-runners-direct-java:jar:0.1.0-incubating:compile
- [ ] | +-
com.google.http-client:google-http-client-protobuf:jar:1.22.0:runtime
- [ ] | +- joda-time:joda-time:jar:2.4:compile
- [ ] | \-
com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2:compile
- [ ] +-
org.apache.beam:beam-runners-flink_2.10:jar:0.1.0-incubating:compile
- [ ] | +- org.apache.flink:flink-streaming-java_2.10:jar:1.0.0:compile
- [ ] | | +- org.apache.flink:flink-core:jar:1.0.0:compile
- [ ] | | | \- org.apache.flink:flink-annotations:jar:1.0.0:compile
- [ ] | | +- org.apache.flink:flink-runtime_2.10:jar:1.0.0:compile
- [ ] | | | +- org.javassist:javassist:jar:3.18.2-GA:compile
- [ ] | | | +- com.typesafe.akka:akka-actor_2.10:jar:2.3.7:compile
- [ ] | | | | \- com.typesafe:config:jar:1.2.1:compile
- [ ] | | | +- com.typesafe.akka:akka-remote_2.10:jar:2.3.7:compile
- [ ] | | | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile
- [ ] | | | +- com.typesafe.akka:akka-slf4j_2.10:jar:2.3.7:compile
- [ ] | | | +- org.clapper:grizzled-slf4j_2.10:jar:1.0.2:compile
- [ ] | | | \- com.github.scopt:scopt_2.10:jar:3.2.0:compile
- [ ] | | +-
org.apache.sling:org.apache.sling.commons.json:jar:2.0.6:compile
- [ ] | | \- org.apache.flink:force-shading:jar:1.0.0:compile
- [ ] | +- org.apache.flink:flink-java:jar:1.0.0:compile
- [ ] | | \- org.apache.flink:flink-shaded-hadoop2:jar:1.0.0:compile
- [ ] | | +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
- [ ] | | +- commons-digester:commons-digester:jar:1.8.1:compile
- [ ] | | \-
commons-beanutils:commons-beanutils-bean-collections:jar:1.8.3:compile
- [ ] | \- org.apache.flink:flink-clients_2.10:jar:1.0.0:compile
- [ ] | \- org.apache.flink:flink-optimizer_2.10:jar:1.0.0:compile
- [ ] +-
org.apache.beam:beam-runners-flink_2.10-examples:jar:0.1.0-incubating:compile
- [ ] | \-
org.apache.flink:flink-connector-kafka-0.8_2.10:jar:1.0.0:compile
- [ ] | +-
org.apache.flink:flink-connector-kafka-base_2.10:jar:1.0.0:compile
- [ ] | | \- com.101tec:zkclient:jar:0.7:compile
- [ ] | \- org.apache.kafka:kafka_2.10:jar:0.8.2.2:compile
- [ ] | +- org.apache.kafka:kafka-clients:jar:0.8.2.2:compile
- [ ] | \- com.yammer.metrics:metrics-core:jar:2.2.0:compile
- [ ] +-
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:0.1.0-incubating:compile
- [ ] | +- com.google.api-client:google-api-client:jar:1.22.0:compile
- [ ] | +-
com.google.apis:google-api-services-datastore-protobuf:jar:v1beta2-rev1-4.0.0:compile
- [ ] | +- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
- [ ] | +- com.google.http-client:google-http-client:jar:1.22.0:compile
- [ ] | +-
com.google.apis:google-api-services-dataflow:jar:v1b3-rev26-1.22.0:compile
- [ ] | +-
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
- [ ] | +-
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0:compile
- [ ] | +- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
- [ ] | | +-
com.google.api-client:google-api-client-java6:jar:1.20.0:compile
- [ ] | | \-
com.google.api-client:google-api-client-jackson2:jar:1.20.0:compile
- [ ] | \- com.fasterxml.jackson.core:jackson-core:jar:2.7.2:compile
- [ ] +- org.apache.beam:beam-runners-spark:jar:0.1.0-incubating:compile
- [ ] | +- org.apache.beam:beam-examples-java:jar:0.1.0-incubating:compile
- [ ] | | \- javax.servlet:javax.servlet-api:jar:3.1.0:compile
- [ ] | \- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
- [ ] | +- org.apache.avro:avro-ipc:jar:1.7.7:compile
- [ ] | | \- org.apache.velocity:velocity:jar:1.7:compile
- [ ] | \- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
- [ ] +- org.apache.commons:commons-exec:jar:1.3:compile
```
Hope this helps, and please let me know if you have any questions!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
