Hi
Under Spark 1.0.0, standalone, client mode am trying to invoke a 3rd party udp
traffic generator, from the streaming thread. The excerpt is as follows:
...
do{
try {
p = Runtime.getRuntime().exec(Prog ");
socket.receive(packet);
output.clear();
kryo.writeObject(output, packet);
store(output);
...
Program has a test to check for existing instantiation, e.g. [ "$(pidof Prog)"
] && exit. This code runs fine, i.e., 3rd party application is invoked, data
is received, analyzed on driver, etc. Problem arises, when I test redundancy
and fault-tolerance. Specifically, when I manually terminate Prog, upon
recovery, multiple invocations are observed. This could be due to multiple
threads getting through [ "$(pidof Prog)" ] && exit. However, I was hoping by
adding semaphores, as follows, to avoid this problem:
...
do{
try {
sem.acquire();
p =
Runtime.getRuntime().exec("Prog");
sem.release();
}catch(IOException ioe) {
//ioe.printStackTrace();
break;
}
socket.receive(packet);
//InetAddress returnIPAddress = packet.getAddress();
returnPort = packet.getPort();
output.clear();
kryo.writeObject(output, packet);
store(output);
...
However, I am still seeing multiple invocations of Prog, upon recovery. I have
also experimented with the following configuration parameters, to no avail:
sparkConf.set("spark.cores.max", args[1]);
sparkConf.set("spark.task.cpus", args[2]);
sparkConf.set("spark.default.parallelism",args[2]);
with args={(1,1),(2,1), (1,2),...}
Any thoughts?
Cheers,
[http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]
Nastooh Avessta
ENGINEER.SOFTWARE ENGINEERING
nave...@cisco.com
Phone: +1 604 647 1527
Cisco Systems Limited
595 Burrard Street, Suite 2123 Three Bentall Centre, PO Box 49121
VANCOUVER
BRITISH COLUMBIA
V7X 1J1
CA
Cisco.com<http://www.cisco.com/>
[Think before you print.]Think before you print.
This email may contain confidential and privileged material for the sole use of
the intended recipient. Any review, use, distribution or disclosure by others
is strictly prohibited. If you are not the intended recipient (or authorized to
receive for the recipient), please contact the sender by reply email and delete
all copies of this message.
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
Cisco Systems Canada Co, 181 Bay St., Suite 3400, Toronto, ON, Canada, M5J 2T3.
Phone: 416-306-7000; Fax: 416-306-7099.
Preferences<http://www.cisco.com/offer/subscribe/?sid=000478326> -
Unsubscribe<http://www.cisco.com/offer/unsubscribe/?sid=000478327> -
Privacy<http://www.cisco.com/web/siteassets/legal/privacy.html>
