Re: Log4j 1.2.17 spark CVE

2021-12-13 Thread Martin Wunderlich
There is a discussion on Github on this topic and the recommendation is to upgrade from 1.x to 2.15.0, due to the vulnerability of 1.x: https://github.com/apache/logging-log4j2/pull/608 This discussion is also referenced by the German Federal Office for Information Security: https://www.bsi.bu

Re: [Spark] Does Spark support backward and forward compatibility?

2021-11-24 Thread Martin Wunderlich
Hi Amin, This might be only marginally relevant to your question, but in my project I also noticed the following: The trained and exported Spark models (i.e. pipelines saved to binary files) are also not compatible between versions, at least between major versions. I noticed this when trying

Re: Using MulticlassClassificationEvaluator for NER evaluation

2021-11-11 Thread Martin Wunderlich
Hi Gourav, Mostly correct. The output of SparNLP here is a trained pipeline/model/transformer. I am feeding this trained pipeline to the MulticlassClassificationEvaluator for evaluation and this MulticlassClassificationEvaluator only accepts floats or doubles are the labels (instead of NER la