Re: Using MulticlassClassificationEvaluator for NER evaluation

2021-11-10 Thread martin
Hi Sean, Apologies for the delayed reply. I've been away on vacation and then busy catching up afterwards. Regarding the evalution using MulticlassClassificationEvaluator: This is a about a sequence labeling task to identify specific non-standard named entities. The training and evaluation d

Spark How to group ciphertext data correctly

2021-11-10 Thread liuyehan
Hello! Thank you for taking time out of your busy schedule to read my email,My English is no good. The company data is ciphertext and needs to be grouped by calling the interface. Similar to two ciphertext data passed to the interface, the interface returns true or false to determine whether

Re: Spark types implicit conversion to string

2021-11-10 Thread Sean Owen
I don't think I'd expect an automatic conversion from a complex type like Map to anything - implicit casts for primitive types are on the other hand relatively common. On Wed, Nov 10, 2021 at 6:01 PM Dhiren wrote: > Hi Spark Users, > I am trying to understand Spark’s type system a little better.

Spark types implicit conversion to string

2021-11-10 Thread Dhiren
Hi Spark Users, I am trying to understand Spark’s type system a little better. I see that Spark implicitly casts some types to String and not for others. Here is an example. df = spark.createDataFrame([({"test": 1.0}, 2.0, 3.0, 4.0)], ("observed", "expected", "obs_wt

Re: apache-spark-how to run distributed spark jobs in Apache-spark-standalone-cluster

2021-11-10 Thread Dinakar Chennubotla
Hi Sean Owen, Thank you very much for your quick response. Could you please help us, with the below? coming to the point, yes I did check the link you sent. but I did not get whole clarity, what I am searching for. Our Prod problem statement is: = 1. Can we launch distri

Re: Log4J 2 Support

2021-11-10 Thread Yang,Jie(INF)
It may be more feasible to replace the current slf4j + log4j with log4j2-api, some projects that spark relies on may also use log4j at the code level, such as EventCounter and ContainerLogAppender in Hadoop, directly removing the dependency on log4j may lead to some code dependencies loss. 发