Wouldn't adding flume -> Kafka -> flink also introduce additional latency?
Georg Heiler <georg.kf.hei...@gmail.com> schrieb am So., 23. Apr. 2017 um 20:23 Uhr: > So you would suggest flume over a custom akka-source from bahir? > > Jörn Franke <jornfra...@gmail.com> schrieb am So., 23. Apr. 2017 um > 18:59 Uhr: > >> I would use flume to import these sources to HDFS and then use flink or >> Hadoop or whatever to process them. While it is possible to do it in flink, >> you do not want that your processing fails because the web service is not >> available etc. >> Via flume which is suitable for this kind of tasks it is more controlled >> and reliable. >> >> On 23. Apr 2017, at 18:02, Georg Heiler <georg.kf.hei...@gmail.com> >> wrote: >> >> New to flink I would like to do a small project to get a better feeling >> for flink. I am thinking of getting some stats from several REST api (i.e. >> Bitcoin course values from different exchanges) and comparing prices over >> different exchanges in real time. >> >> Are there already some REST api sources for flink as a sample to get >> started implementing a custom REST source? >> >> I was thinking about using https://github.com/timmolter/XChange to >> connect to several exchanges. E.g. to make a single api call by hand would >> look similar to >> >> val currencyPair = new CurrencyPair(Currency.XMR, Currency.BTC) >> CertHelper.trustAllCerts() >> val poloniex = >> ExchangeFactory.INSTANCE.createExchange(classOf[PoloniexExchange].getName) >> val dataService = poloniex.getMarketDataService >> >> generic(dataService) >> raw(dataService.asInstanceOf[PoloniexMarketDataServiceRaw]) >> System.out.println(dataService.getTicker(currencyPair)) >> >> How would be a proper way to make this available as a flink source? I >> have seen >> https://github.com/apache/flink/blob/master/flink-contrib/flink-connector-wikiedits/src/main/java/org/apache/flink/streaming/connectors/wikiedits/WikipediaEditsSource.java >> but >> new to flink am a bit unsure how to proceed. >> >> Regards, >> Georg >> >>
