What sort of environment are you running Spark on - in the cloud, on premise ? Is its a real-time or batch oriented application? Please provide more details. Femi
On Thu, Oct 25, 2018 at 3:29 AM Affan Syed <as...@an10.io> wrote: > Spark users, > We really would want to get an input here about how the results from a > Spark Query will be accessible to a web-application. Given Spark is a well > used in the industry I would have thought that this part would have lots of > answers/tutorials about it, but I didnt find anything. > > Here are a few options that come to mind > > 1) Spark results are saved in another DB ( perhaps a traditional one) and > a request for query returns the new table name for access through a > paginated query. That seems doable, although a bit convoluted as we need to > handle the completion of the query. > > 2) Spark results are pumped into a messaging queue from which a socket > server like connection is made. > > What confuses me is that other connectors to spark, like those for > Tableau, using something like JDBC should have all the data (not the top > 500 that we typically can get via Livy or other REST interfaces to Spark). > How do those connectors get all the data through a single connection? > > > Can someone with expertise help in bringing clarity. > > Thank you. > > Affan > ᐧ > ᐧ > ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
