Hello, Apache Streams (incubating) maintains and publishes json-schemas and jackson-compatible POJOs for Twitter and other popular third-party APIs.
http://streams.apache.org/site/0.5.1-incubating-SNAPSHOT/streams-project/streams-contrib/streams-provider-twitter/index.html We also have a repository of public examples, one of which demonstrates how to embed various twitter data collectors into Flink. http://streams.apache.org/site/0.5.1-incubating-SNAPSHOT/streams-examples/streams-examples-flink/flink-twitter-collection/index.html We’d welcome support of anyone from Flink project to help us maintain and improve these examples. Potentially, Flink could maintain the benefit of the existence of useful, ready-to-run examples for new Flink users, while getting the boring code out of your code base. Also, our examples have integration tests that actually connect to twitter and check that everything continues to work :) if anyone wants to know more about this, feel free to reach out to the team on d...@streams.incubator.apache.org Steve sblack...@apache.org On June 12, 2017 at 7:18:08 AM, Aljoscha Krettek (aljos...@apache.org) wrote: Bumpety-bump. I would be in favour or removing this: - It can be implemented as a MapFunction parser after a TextInputFormat - Additions, changes, fixes that happen on TextInputFormat are not reflected to SimpleTweetInputFormat - SimpleTweetInput format overrides nextRecord(), which is not something DelimitedInputFormats are normally supposed to do, I think - The Tweet POJO has a very strange naming scheme Best, Aljoscha > On 7. Jun 2017, at 11:15, Chesnay Schepler <ches...@apache.org> wrote: > > Hello, > > I'm proposing to remove the Twitter-InputFormat in FLINK-6710 > <https://issues.apache.org/jira/browse/FLINK-6710>, with an open PR you can > find here <https://github.com/apache/flink/pull/3984>. > The PR currently has a +1 from Robert, but Timo raised some concerns saying > that it is useful for prototyping and > advised me to start a discussion on the ML. > > This format is a DelimitedInputFormat that reads JSON objects and turns them > into a custom tweet class. > I believe this format doesn't provide much value to Flink; there's nothing > interesting about it as an InputFormat, > as it is purely an exercise in manually converting a JSON object into a POJO. > > This is apparent since you could just as well use > ExecutionEnvironment#readTextFile(...) and throw the parsing logic > into a subsequent MapFunction. > > In the PR i suggested to replace this with a JsonInputFormat, but this was a > misguided attempt at getting Timo to agree > to the removal. This format has the same problem outlined above, as it could > be effectively implemented with a one-liner map function. > > So the question now is whether we want to keep it, remove it, or replace it > with something more general. > > Regards, > Chesnay
