There are several open Jiras to add new Sinks OpenTSDB https://issues.apache.org/jira/browse/SPARK-12194 StatsD https://issues.apache.org/jira/browse/SPARK-11574 Kafka https://issues.apache.org/jira/browse/SPARK-13392
Some have PRs from 2015 so I'm assuming there is not the desire to integrate these into core Spark. Opening up the Sink/Source interfaces would at least allow these to exist somewhere such as spark-packages without having to pollute the o.a.s namespace On Sat, 19 Mar 2016 at 13:05 Gerard Maas <gerard.m...@gmail.com> wrote: > +1 > On Mar 19, 2016 08:33, "Pete Robbins" <robbin...@gmail.com> wrote: > >> This seems to me to be unnecessarily restrictive. These are very useful >> extension points for adding 3rd party sources and sinks. >> >> I intend to make an Elasticsearch sink available on spark-packages but >> this will require a single class, the sink, to be in the org.apache.spark >> package tree. I could submit the package as a PR to the Spark codebase, and >> I'd be happy to do that but it could be a completely separate add-on. >> >> There are similar issues with writing a 3rd party metrics source which >> may not be of interest to the community at large so would probably not >> warrant inclusion in the Spark codebase. >> >> Any thoughts? >> >
