This article <http://www.virdata.com/tuning-spark/> gives you a pretty good start on the Spark streaming side. And this article <https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines> is for the kafka, it has nice explanation how message size and partitions effects the throughput. And this article <https://www.sigmoid.com/creating-sigview-a-real-time-analytics-dashboard/> has a use-case.
Thanks Best Regards On Tue, May 12, 2015 at 8:25 PM, dgoldenberg <dgoldenberg...@gmail.com> wrote: > Hi, > > I'm looking at a data ingestion implementation which streams data out of > Kafka with Spark Streaming, then uses a multi-threaded pipeline engine to > process the data in each partition. Have folks looked at ways of speeding > up this type of ingestion? > > Let's say the main part of the ingest process is fetching documents from > somewhere and performing text extraction on them. Is this type of > processing > best done by expressing the pipelining with Spark RDD transformations or by > just kicking off a multi-threaded pipeline? > > Or, is using a multi-threaded pipeliner per partition is a decent strategy > and the performance comes from running in a clustered mode? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-speed-up-data-ingestion-with-Spark-tp22859.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
