[SPARK STREAMING] Questions regarding foreachPartition

[Nipun Arora]

Hi,
I wanted to understand forEachPartition logic. In the code below, I am
assuming the iterator is executing in a distributed fashion.

1. Assuming I have a stream which has timestamp data which is sorted. Will
the stringiterator in foreachPartition process each line in order?

2. Assuming I have a static pool of Kafka connections, where should I get a
connection from a pool to be used to send data to Kafka?

addMTSUnmatched.foreachRDD(
        new Function<JavaRDD<String>, Void>() {
            @Override
            public Void call(JavaRDD<String> stringJavaRDD) throws Exception {
                stringJavaRDD.foreachPartition(

                        new VoidFunction<Iterator<String>>() {
                            @Override
                            public void call(Iterator<String>
stringIterator) throws Exception {
                                while(stringIterator.hasNext()){
                                    String str = stringIterator.next();
                                    if(OnlineUtils.ESFlag) {
                                        OnlineUtils.printToFile(str,
1, type1_outputFile, OnlineUtils.client);
                                    }else{
                                        OnlineUtils.printToFile(str,
1, type1_outputFile);
                                    }
                                }
                            }
                        }
                );
                return null;
            }
        }
);



Thanks

Nipun