Hi Sid,
||finalDF = finalDF.repartition(finalDF.rdd.getNumPartitions())
.withColumn("status_for_batch", call_to_cust_bulk_api(policyUrl,
to_json(struct(*colsListToBePassed)))) | |
You are calling ||withColumn|| with the result of
||call_to_cust_bulk_api|| as the second argument. That result looks like
it is of type string. But ||withColumn|| expects type ||Column||. You
can turn that string into a ||Column|| using ||lit||:
||finalDF = finalDF.repartition(finalDF.rdd.getNumPartitions())
.withColumn("status_for_batch", lit(call_to_cust_bulk_api(policyUrl,
to_json(struct(*colsListToBePassed))))) ||
You are saying that gives you an error of column not iterable. I reckon
the ||struct(*colsListToBePassed))|| is wrong.
Method ||struct|| requires a single string followed by a list of
strings. Given your ||colsListToBePassed|| is a list of strings, this
does not work. Try:
|| struct(||||||colsListToBePassed.head,
||colsListToBePassed.tail|||||: _*|))||
Alternatively, ||struct|| requires a list of ||Column||, so try this:
|| struct(||||||colsListToBePassed.map(col)|||||||: _*|))||
The API is pretty clear about the types it expects.
If you are still having errors, you better please paste the code and error.
Enrico
Am 09.06.22 um 21:31 schrieb Sid:
Hi Experts,
I am facing one problem while passing a column to the method. The
problem is described in detail here:
https://stackoverflow.com/questions/72565095/how-to-pass-columns-as-a-json-record-to-the-api-method-using-pyspark
TIA,
Sid
