Persisting RDD: Low Percentage with a lot of memory available

2017-07-28 Thread pedroT
Hi, This problem is very annoying for me and I'm tired of surfing the network without any good advice to follow. I have a complex job. It has been worked fine until I needed to save partial results (RDDs) to files. So I tried to cache the RDDs and then call a saveAsText method and follow the workf

Broadcasting Non Serializable Objects

2016-10-18 Thread pedroT
Hi guys. I know this is a well known topic, but reading about (a lot) I'm not sure about the answer.. I need to broadcast a complex estructure with a lot of objects as fields, including some of external libraries which I can't happily turn in serializable ones. I tried making a static method r