I would use flume to import these sources to HDFS and then use flink or Hadoop or whatever to process them. While it is possible to do it in flink, you do not want that your processing fails because the web service is not available etc. Via flume which is suitable for this kind of tasks it is more controlled and reliable.
> On 23. Apr 2017, at 18:02, Georg Heiler <georg.kf.hei...@gmail.com> wrote: > > New to flink I would like to do a small project to get a better feeling for > flink. I am thinking of getting some stats from several REST api (i.e. > Bitcoin course values from different exchanges) and comparing prices over > different exchanges in real time. > > Are there already some REST api sources for flink as a sample to get started > implementing a custom REST source? > > I was thinking about using https://github.com/timmolter/XChange to connect to > several exchanges. E.g. to make a single api call by hand would look similar > to > > val currencyPair = new CurrencyPair(Currency.XMR, Currency.BTC) > CertHelper.trustAllCerts() > val poloniex = > ExchangeFactory.INSTANCE.createExchange(classOf[PoloniexExchange].getName) > val dataService = poloniex.getMarketDataService > > generic(dataService) > raw(dataService.asInstanceOf[PoloniexMarketDataServiceRaw]) > System.out.println(dataService.getTicker(currencyPair)) > > How would be a proper way to make this available as a flink source? I have > seen > https://github.com/apache/flink/blob/master/flink-contrib/flink-connector-wikiedits/src/main/java/org/apache/flink/streaming/connectors/wikiedits/WikipediaEditsSource.java > but new to flink am a bit unsure how to proceed. > > Regards, > Georg
