Hey, We use a customize receiver to receive data from our MQ. We used to use def store(dataItem: T) to store data however I found the block size can be very different from 0.5K to 5M size. So that data partition processing time is very different. Shuffle is an option, but I want to avoid it.
I notice that def store(dataBuffer: ArrayBuffer[T]) can store the whole data into a block so I can control block size, however I also noticed that this method doesn't apply any rate limit on it, I have to do rate limit myself. So by now, I haven't have a good way to control block size, I am asking if spark can add rate limit on store(dataBuffer: ArrayBuffer[T]) method or have a way to control block size generated by BlockGenerator -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
