Dear Janani, Apache Kafka as a source is supported by our system, check out the documentation for details. [1]
You can use UDP as a source if you wish, just bear in mind the standard disadvantages of it: the possibility of losing messages and that you will have to manually deal with the serialization of datagrams. For adding your custom source all you need to implement is an instance of the SourceFunction interface. [2] [1] http://ci.apache.org/projects/flink/flink-docs-master/streaming_guide.html#apache-kafka [2] http://ci.apache.org/projects/flink/flink-docs-master/streaming_guide.html#sources Best, Marton On Thu, Mar 26, 2015 at 1:15 PM, Janani Chakkaradhari < janani.cs...@gmail.com> wrote: > Hi Stephan, > > Yes, you are right. I will try writing a custom data source as you > mentioned. Also, I still need to check the possibility of our system to use > Apache Kafka as a broker. Could it be possible for you to point out here > the downsides of using UDP as a source for streaming data? > > Thanks, > Janani > > On Wed, Mar 25, 2015 at 9:44 PM, Stephan Ewen <se...@apache.org> wrote: > > > Hi Janani! > > > > Do I understand you correctly in that you want a Flink stream source that > > receives UDP datagrams and turns them into Flink DataStream? > > > > Such a thing is not in there, yet. The interface to define custom data > > sources is rather simple, though, it should be possible to add something > > like this. > > > > Two things are not clear to me however: > > > > 1) Where are the datagrams sent to? That would need to be the point where > > the data source runs. > > > > 2) You need a way of turning the datagrams (which are just bytes) into > > records. Would that be hard-wired in your case? > > > > So, while probably possible, I would guess a UDP source has quite a few > > downsides over a message queue as a source. Using something like Kafka to > > communicate the source data is easier and better recoverable. > > > > Does your setup allow to put the data into Kafka and having Flink read > the > > streams from Kafka? > > > > Greetings, > > Stephan > > Am 25.03.2015 19:20 schrieb "Janani Chakkaradhari" < > > janani.cs...@gmail.com > > >: > > > > > Hi, > > > > > > Does Flink's streaming api has support for > > > reading streams of data via Java UDP (DatagramSocket)? If so kindly > > > advise me in which release of Flink I can find it. > > > > > > Thanks, > > > Janani > > > > > >
