Re: Detect executor core count

2019-06-18 Thread Andrew Melo
On Tue, Jun 18, 2019 at 5:40 PM Steve Loughran wrote: > be aware that older java 8 versions count the #of cores in the host, not > those allocated for the container they run in > https://bugs.openjdk.java.net/browse/JDK-8140793 > > Ergh, that's good to know. I suppose, though, that in any case, t

Re: Detect executor core count

2019-06-18 Thread Steve Loughran
be aware that older java 8 versions count the #of cores in the host, not those allocated for the container they run in https://bugs.openjdk.java.net/browse/JDK-8140793 On Tue, Jun 18, 2019 at 8:13 PM Ilya Matiach wrote: > Hi Andrew, > > I tried to do something similar to that in the LightGBM > c

RE: Detect executor core count

2019-06-18 Thread Ilya Matiach
Hi Andrew, I tried to do something similar to that in the LightGBM classifier/regressor/ranker in mmlspark package, I try to use the spark conf and if not configured I get the processors from the JVM directly: https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/main/scala/LightGBMUtils