Re: Spark Streaming Design Suggestion

opics dynamically can be a >> nightmare to operate >> >> > On 13. Jun 2017, at 22:03, Shashi Vishwakarma >> wrote: >> > >> > Hi >> > >> > I have to design a spark streaming application with below use case. I >> am looking for best p

Spark Streaming Design Suggestion

Hi I have to design a spark streaming application with below use case. I am looking for best possible approach for this. I have application which pushing data into 1000+ different topics each has different purpose . Spark streaming will receive data from each topic and after processing it will wr

Spark Shuffle Encryption

Hi I was doing research on encrypting spark shuffle data and found that Spark 2.1 has got that feature. https://issues.apache.org/jira/browse/SPARK-5682 Does anyone has more documentation around it ? How do I aim to use this feature in real production environment keeping mind that I need to secu

Re: Securing Spark Job on Cluster

pr 2017, at 16:57, Shashi Vishwakarma > wrote: > > Agreed Jorn. Disk encryption is one option that will help to secure data > but how do I know at which location Spark is spilling temp file, shuffle > data and application data ? > > Thanks > Shashi > > On Fri, Apr 28

Re: Securing Spark Job on Cluster

the operating system. > Additionally, you may think about shredding disks after they are not used > anymore. > > > On 28. Apr 2017, at 14:45, Shashi Vishwakarma > wrote: > > > > Hi All > > > > I was dealing with one the spark requirement here where Client (

Re: Securing Spark Job on Cluster

ache kebros, or > we could use the security feature in Spark. > > > > On Apr 28, 2017, at 8:45 AM, Shashi Vishwakarma < > shashi.vish...@gmail.com> wrote: > > > > Hi All > > > > I was dealing with one the spark requirement here where Client (like &g

Securing Spark Job on Cluster

Hi All I was dealing with one the spark requirement here where Client (like Banking Client where security is major concern) needs all spark processing should happen securely. For example all communication happening between spark client and server ( driver & executor communication) should be on se

Re: Spark Job failing with exit status 15

ark were you using ? > > Can you post the command you used to run WordCount ? > > Cheers > > On Sat, Nov 7, 2015 at 7:59 AM, Shashi Vishwakarma < > shashi.vish...@gmail.com> wrote: > >> I am trying to run simple word count job in spark but I am getting >>

Spark Job failing with exit status 15

I am trying to run simple word count job in spark but I am getting exception while running job. For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1446699275562_0006/Then, click on links to logs of each attempt.Diagnostics: Exception from co

Realtime Data Visualization Tool for Spark

Hi I have got streaming data which needs to be processed and send for visualization. I am planning to use spark streaming for this but little bit confused in choosing visualization tool. I read somewhere that D3.js can be used but i wanted know which is best tool for visualization while dealing w