Hi, are you using Spark 2.0.*? Then it might be related to https://issues.apache.org/jira/browse/SPARK-18281 .
Best Eike 2016-12-18 6:21 GMT+01:00 Russell Jurney <russell.jur...@gmail.com>: > Anyone? This is for a book, so I need to figure this out. > > On Fri, Dec 16, 2016 at 12:53 AM Russell Jurney <russell.jur...@gmail.com> > wrote: > >> I have created a PySpark Streaming application that uses Spark ML to >> classify flight delays into three categories: on-time, slightly late, very >> late. After an hour or so something times out and the whole thing crashes. >> >> The code and error are on a gist here: https://gist.github.com/rjurney/ >> 17d471bc98fd1ec925c37d141017640d >> >> While I am interested in why I am getting an exception, I am more >> interested in understanding what the correct deployment model is... because >> long running processes will have new and varied errors and exceptions. >> Right now with what I've built, Spark is a highly dependable distributed >> system but in streaming mode the entire thing is dependent on one Python >> PID going down. This can't be how apps are deployed in the wild because it >> will never be very reliable, right? But I don't see anything about this in >> the docs, so I am confused. >> >> Note that I use this to run the app, maybe that is the problem? >> >> ssc.start() >> ssc.awaitTermination() >> >> >> What is the actual deployment model for Spark Streaming? All I know to do >> right now is to restart the PID. I'm new to Spark, and the docs don't >> really explain this (that I can see). >> >> Thanks! >> -- >> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com relato.io >> >> >> -- ------------------------------------------------ *Jan Eike von Seggern* Data Scientist ------------------------------------------------ *Sevenval Technologies GmbH * FRONT-END-EXPERTS SINCE 1999 Köpenicker Straße 154 | 10997 Berlin office +49 30 707 190 - 229 mail eike.segg...@sevenval.com www.sevenval.com Sitz: Köln, HRB 79823 Geschäftsführung: Jan Webering (CEO), Thorsten May, Sascha Langfus, Joern-Carlos Kuntze *Wir erhöhen den Return On Investment bei Ihren Mobile und Web-Projekten. Sprechen Sie uns an:*http://roi.sevenval.com/ ----------------------------------------------------------------------------------------------------------------------------------------------- FOLLOW US on [image: Sevenval blog] <http://sevenval.us11.list-manage1.com/track/click?u=5f2d34577b3182d6f029ebe63&id=ff955ef848&e=b789cc1a5f> [image: sevenval on twitter] <http://sevenval.us11.list-manage.com/track/click?u=5f2d34577b3182d6f029ebe63&id=998e8f655c&e=b789cc1a5f> [image: sevenval on linkedin] <http://sevenval.us11.list-manage.com/track/click?u=5f2d34577b3182d6f029ebe63&id=7ae7d93d42&e=b789cc1a5f>[image: sevenval on pinterest] <http://sevenval.us11.list-manage2.com/track/click?u=5f2d34577b3182d6f029ebe63&id=f8c66fb950&e=b789cc1a5f>
