Thanks Jungtaek!
It makes sense, we are currently changing to an HDFS-Compatible FS, I was
wondering how this change would impact the checkpoint, but after what you
said it is more clear now.
On Thu, 3 Dec 2020 at 00:23, Jungtaek Lim
wrote:
> In theory it would work, but works very inefficien
Thanks, I confused myself. I was looking at
org.apache.spark.ml.recommendation.ALS Javadoc. Not sure why it shows up. I
didn't notice the Developer API tag, so "fit" it is!
-S
From: Sean Owen
Sent: Wednesday, December 2, 2020 3:51 PM
To: Steve Pruitt
Cc: us
Hi folks,
I am trying to read the message headers from a Kafka structured
stream which should be stored in a column named ``headers``.
I try something like this:
val stream = sparkSession.readStream.format("kafka")..load()
stream.map(row => {
...
val headers = row.getAs[HeaderT]("headers
Hello,
see if this works, from the documentation:
// Subscribe to 1 topic, with headersval df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1")
.option("includeHeaders", "true")
.load()df.selectExpr("CAST
This is becoming serious pain.
using powershell I am using spark-submit as follows:
PS C:\Users\admin> spark-submit.cmd
C:\Users\admin\PycharmProjects\pythonProject\main.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe
You don't have to include all your config and log messages. The error
message would suffice. The java.lang.UnsatisfiedLinkError exception
indicates that the JVM can't find some OS-specific libraries (or
commonly referred as native libraries). On Windows, they would be some
dll files. Look i
