Hi,
Got a question here around! I am setting a setJobGroup to be cancelled within a
Future.
Thing is, such future is affected by several map, flatMap, for comprehensions
and even andThen side effects. I am being unable to cancel the job properly
within a group, and I am unsure whether this is
Hi,
I have figured this only happens in cluster mode. working properly in local[32]
From: saif.a.ell...@wellsfargo.com [mailto:saif.a.ell...@wellsfargo.com]
Sent: Thursday, October 08, 2015 10:23 AM
To: dev@spark.apache.org
Subject: RowNumber in HiveContext returns null, negative numbers or huge
Hi all, would this be a bug??
val ws = Window.
partitionBy("clrty_id").
orderBy("filemonth_dtt")
val nm = "repeatMe"
df.select(df.col("*"), rowNumber().over(ws).cast("int").as(nm))
stacked_data.filter(stacked_data("repeatMe").isNotNull).or
When running stand-alone cluster mode job, the process hangs up randomly during
a DataFrame flatMap or explode operation, in HiveContext:
-->> df.flatMap(r => for (n <- 1 to r.getInt(ind)) yield r)
This does not happen either with SQLContext in cluster, or Hive/SQL in local
mode, where it works
Hi all,
I have a process where local mode takes only 40 seconds. While the same on
stand-alone mode, being the same node used for local mode the only available
node, is taking up for ever. rdd actions hang up.
I could only "sort this out" by turning speculation on, so the same task
hanging is
Hi all,
wondering if any could make the new 1.5.0 stallSkinnyQR to work.
Follows my output, which is a big loop of the same errors until the shell dies.
I am curious since im failing to load any implementations from BLAS, LAPACK,
etc.
scala> mat.tallSkinnyQR(false)
15/09/22 10:18:11 WARN LAPACK:
Thank you, I will do as you suggested.
Ps: I read that in this random user archive I found:
http://mail-archives.us.apache.org/mod_mbox/spark-user/201506.mbox/%3c55709f7b.2090...@gmail.com%3E
Saif
From: Feynman Liang [mailto:fli...@databricks.com]
Sent: Monday, September 14, 2015 4:08 PM
To: Ell
Hi all, I'm very new to spark and looking forward to get deep into the topic.
Right now I am trying to inherit my own transformer, by what I am reading so
far, it is not very public that we can apply to this practice as "users".
I am defining my transformer based on the Binarizer, but simply fail
