State of support for dynamic allocation on K8s and possible CMs

2019-07-01 Thread Federico D'Ambrosio
Hello everyone, I wanted to ask what's the state of support of Spark dynamic allocation as of now, if there's any issue where I could track its advancement and missing features. We've just started evaluating possible alternatives for a production architectural setup for our use case, and dynamic

Re: Spark does not load all classes in fat jar

2019-03-18 Thread Federico D'Ambrosio
ieb Jörn Franke : > > Maybe that class is already loaded as part of a core library of Spark? > > Do you have concrete class names? > > In doubt create a fat jar and shade the dependencies in question > > Am 18.03.2019 um 12:34 schrieb Federico D'Ambrosio : > > Hel

Spark does not load all classes in fat jar

2019-03-18 Thread Federico D'Ambrosio
Hello everyone, We're having a serious issue, where we get ClassNotFoundException because, apparently the class is not found within the classpath of Spark, in both the Driver and Executors. First, I checked whether the class was actually within the jar with jar tf, and there actually is. Then, I

How to optimize iterative data processing in spark application

2019-01-23 Thread Federico D'Ambrosio
mended givent that I'm dealing with some sort of time constraint? Is there any way I can optimize the transformations flow I should look into? Thank you, Federico -- Federico D'Ambrosio