DF.write.format(sourceType)
.option("timestampFormat", "/MM/dd HH:mm:ss ZZ")
.mode(SaveMode.Overwrite)
// .partitionBy("created")
.save("ftp://ftp:ftpu@127.0.0.1:21/sparkftp/write/";)
I get a error
java.io.IOException: Cannot rename source:
ftp://ftpuser:ftpuser@10.3.8
FTP is definitely not supported. Read the files to distributed storage
first then read from there.
On Sun, Aug 8, 2021, 10:18 PM igyu wrote:
> val ftpUrl = "ftp://ftpuser:ftpuser@10.3.87.51:21/sparkftp/";
>
> val schemas = StructType(List(
> new StructField("name", DataTypes.StringType,
val ftpUrl = "ftp://ftpuser:ftpuser@10.3.87.51:21/sparkftp/";
val schemas = StructType(List(
new StructField("name", DataTypes.StringType, true),
new StructField("age", DataTypes.IntegerType, true),
new StructField("remk", DataTypes.StringType, true))) val DF =
sparkSess
Sorted out :)
I removed all references to Jason Service Key file i.e.
spark.conf.set("GcpJsonKeyFile", config['GCPVariables']['jsonKeyFile'])
from the code and it worked.
writing to BigQuery table test.randomData
Populated BigQuery table test.randomData
rows written is 100
Reading from
Further on this to avoid the issue with Java 11 I went back and installed
Java 8 on the docker image. This was simply done by cp -R
/opt/jdk1.8.0_201 to $SPARK_HOME and amending the docker image to copy it
across to /opt inside the image and exporting JAVA HOME as below
RUN mkdir -p /opt/jdk1.8.
