@sathich Here are my thoughts on your points -
1. Yes this should be able to handle any complex json structure returned by the target rest API. Essentially what it would be returning is Rows of that complex structure. Then one can use Spark SQL to further flatten it using the functions like inline, explode, etc. 2. In my current implementation I have kept an option as "callStrictlyOnce". This will ensure that the REST API is called only once for each set of parameter values and the result would be persisted/cached for next time use. 3. I'm not sure what exactly you have in mind regarding extending this to Spark Streaming. As such this cannot be used as a Spark Streaming receiver right now as this does not implement the necessary interfaces for a custom streaming receiver. But you can use this within your Spark Streaming application as a regular Data Source to merge the data you are receiving from streaming source. Regards, Sourav -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
