trouble implementing Transformer and calling DataFrame.withColumn()

[Andy Davidson]

I am trying to port the following python function to Java 8. I would like my
java implementation to implement Transformer so I can use it in a pipeline.

I am having a heck of a time trying to figure out how to create a Column
variable I can pass to DataFrame.withColumn(). As far as I know withColumn()
the only way to append a column to a data frame.

Any comments or suggestions would be greatly appreciated

Andy


def convertMultinomialLabelToBinary(dataFrame):
    newColName = "binomialLabel"
    binomial = udf(lambda labelStr: labelStr if (labelStr == "noise") else
³signal", StringType())
    ret = dataFrame.withColumn(newColName, binomial(dataFrame["label"]))
    return ret

trainingDF2 = convertMultinomialLabelToBinary(trainingDF1)


public class LabelToBinaryTransformer extends Transformer {

    private static final long serialVersionUID = 4202800448830968904L;

    private  final UUID uid = UUID.randomUUID();

    public String inputCol;

    public String outputCol;

    

    @Override

    public String uid() {

        return uid.toString();

    }



    @Override

    public Transformer copy(ParamMap pm) {

        Params xx = defaultCopy(pm);

        return ???;

    }



    @Override

    public DataFrame transform(DataFrame df) {

        MyUDF myUDF = new MyUDF(myUDF, null, null);

        Column c = df.col(inputCol);

??? UDF apply does not take a col????

        Column col = myUDF.apply(df.col(inputCol));

        DataFrame ret = df.withColumn(outputCol, col);

        return ret;

    }



    @Override

    public StructType transformSchema(StructType arg0) {

       ??? What is this function supposed to do???

      ???Is this the type of the new output column????

    }

    

    class MyUDF extends UserDefinedFunction {

        public MyUDF(Object f, DataType dataType, Seq<DataType> inputTypes)
{

            super(f, dataType, inputTypes);

            ??? Why do I have to implement this constructor ???

    ??? What are the arguments ???

        }

        

        @Override

        public

        Column apply(scala.collection.Seq<Column> exprs) {

    What do you do with a scala seq?

            return ???;

        }

    }

}