Issue while calling foreach in Pyspark

[rajat kumar]

Hi Team,

I am using Spark 2.4.4 with Python

While using below line:

dataframe.foreach(lambda record : process_logs(record))


My use case is , process logs will download the file from cloud storage
using Python code and then it will save the processed data.

I am getting the following error

  File "/opt/spark/python/lib/pyspark.zip/pyspark/java_gateway.py", line
46, in launch_gateway
    return _launch_gateway(conf)
  File "/opt/spark/python/lib/pyspark.zip/pyspark/java_gateway.py", line
108, in _launch_gateway
    raise Exception("Java gateway process exited before sending its port
number")
Exception: Java gateway process exited before sending its port number

Can anyone pls suggest what can be done?

Thanks
Rajat