Re: Performance tuning for standalone on one host

2016-07-25 Thread Mich Talebzadeh
Hi, >From your reference I can see that you are running in local mode with two cores. But that is not standalone. Can you please clarify whether you start master and slaves processes. Those are for standalone mode. sbin/start-master.sh sbin/start-slaves.sh HTH Dr Mich Talebzadeh LinkedIn *

Performance tuning for standalone on one host

2016-07-25 Thread on
Dear all, I am running spark on one host ("local[2]") doing calculations like this on a socket stream. mainStream = socketStream.filter(lambda msg: msg['header'].startswith('test')).map(lambda x: (x['host'], x) ) s1 = mainStream.updateStateByKey(updateFirst).map(lambda x: (1, x) ) s2 = mainStream.