Local mode
-
__Vikash Pareek
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Number-Of-Partitions-in-RDD-tp28730p28786.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
CLuster mode with HDFS? or local mode?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Number-Of-Partitions-in-RDD-tp28730p28737.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
Spark 1.6.1
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Number-Of-Partitions-in-RDD-tp28730p28735.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To
What version of spark of spark are you using?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Number-Of-Partitions-in-RDD-tp28730p28732.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
>
> scala> people.getNumPartitions
> res45: Int = 11
>
> scala> val people = sc.textFile("file:///home/pvikash/data/test.txt", 11)
> people: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[70] at
> textFile
> at :27
>
> scala> people.getNumPartit
e we have small data (which can fit into one partition) then why spark
creates empty partitions?
Any explanation would be appreciated.
--Vikash
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Number-Of-Partitions-in-RDD-tp28730.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Hi list,
In an excelent blog post on Kafka and Spark Streaming integrartion (
http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/),
Michael Noll poses an assumption about the number of partitions of the RDDs
created by input DStreams. He says his hypothe
