hey igor! a few ways to work around this depending on the level of exception-handling granularity you're willing to accept: 1) use mapPartitions() to wrap the entire partition handling code in a try/catch -- this is fairly coarse-grained, however, and will fail the entire partition. 2) modify your transformation code to wrap a try-catch around the individual record handler -- return either "None" (or some other well-known empty value) for input records that fail and the actual value for records that succeed. use a filter() to filter out the "None" values. 3) same as #2, but use empty array for a failure and a single-element array for a success.
hope that helps! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/pyspark-exception-catch-tp20483p20730.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
