On Mon, Oct 24, 2016 at 6:21 PM, Efe Selcuk wrote:
>
> I have an application that works in 2.0.0 but has been dying at runtime on
> the 2.0.1 distribution.
>
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$co
zuaki Ishizaki
From: Efe Selcuk
To:"user @spark"
Date:2016/10/25 10:23
Subject:[Spark 2.0.1] Error in generated code, possible regression?
--
I have an application that works in 2.0.0 but has been dying at runtime on
the 2.0.1 di
lity to zero will almost never work.
> >
> > Look at Goldberg's paper
> >
> https://ece.uwaterloo.ca/~dwharder/NumericalAnalysis/02Numerics/Double/paper.pdf
> > for a quick intro.
> >
> > Mike
> >
> > On Oct 24, 2016, at 10:36 PM, Efe Selcuk wrote:
ts the precision that Spark uses to store the decimal
>
> On Mon, Oct 24, 2016 at 7:32 PM, Jakob Odersky wrote:
> > An even smaller example that demonstrates the same behaviour:
> >
> > Seq(Data(BigDecimal(0))).toDS.head
> >
> > On Mon, Oct 24, 2016 at 7:03
I’m trying to track down what seems to be a very slight imprecision in our
Spark application; two of our columns, which should be netting out to
exactly zero, are coming up with very small fractions of non-zero value.
The only thing that I’ve found out of place is that a case class entry into
a Dat
I have an application that works in 2.0.0 but has been dying at runtime on
the 2.0.1 distribution.
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:893)
at
org.apache.spark.sql.catalys
m in a situation where I can't easily build from source.
On Mon, Oct 24, 2016 at 12:29 PM Cheng Lian wrote:
>
>
> On 10/22/16 1:42 PM, Efe Selcuk wrote:
>
> Ah, looks similar. Next opportunity I get, I'm going to do a printSchema
> on the two datasets and see if they
ou print the schema for data and for
> someCode.thatReturnsADataset() and see if there is any difference between
> the two ?
>
> On Fri, Oct 21, 2016 at 9:14 AM, Efe Selcuk wrote:
>
> Thanks for the response. What do you mean by "semantically" the same?
> They're bo
here
On Thu, Oct 20, 2016 at 8:34 PM Agraj Mangal wrote:
I believe this normally comes when Spark is unable to perform union due to
"difference" in schema of the operands. Can you check if the schema of both
the datasets are semantically same ?
On Tue, Oct 18, 2016 at 9:06 AM, Efe Selcuk
Bump!
On Thu, Oct 13, 2016 at 8:25 PM Efe Selcuk wrote:
> I have a use case where I want to build a dataset based off of
> conditionally available data. I thought I'd do something like this:
>
> case class SomeData( ... ) // parameters are basic encodable types like
> str
I have a use case where I want to build a dataset based off of
conditionally available data. I thought I'd do something like this:
case class SomeData( ... ) // parameters are basic encodable types like
strings and BigDecimals
var data = spark.emptyDataset[SomeData]
// loop, determining what dat
Hi Spark community,
This is a bit of a high level question as frankly I'm not well versed in
Spark or related tech.
We have a system in place that reads columnar data in through CSV and
represents the data in relational tables as it operates. It's essentially
schema-based ETL. This restricts our
t;1990-12-13")),
>> ("a", Date.valueOf("1990-12-13")),
>> ("a", Date.valueOf("1990-12-13"))
>> ).toDF("a", "b").as[ClassData]
>> ds.write.csv("/tmp/data.csv")
>> spark.read.csv("/tmp/data.csv&q
SV format can't represent the nested types in its
> own format.
>
> I guess supporting them in writing in external CSV is rather a bug.
>
> I think it'd be great if we can write and read back CSV in its own format
> but I guess we can't.
>
> Thanks!
>
>
We have an application working in Spark 1.6. It uses the databricks csv
library for the output format when writing out.
I'm attempting an upgrade to Spark 2. When writing with both the native
DataFrameWriter#csv() method and with first specifying the
"com.databricks.spark.csv" format (I suspect un
Bump!
On Wed, Aug 10, 2016 at 2:59 PM, Efe Selcuk wrote:
> Thanks for the replies, folks.
>
> My specific use case is maybe unusual. I'm working in the context of the
> build environment in my company. Spark was being used in such a way that
> the fat assembly jar that t
.
>
>
>
> On 10 August 2016 at 20:35, Holden Karau wrote:
>
>> What are you looking to use the assembly jar for - maybe we can think of
>> a workaround :)
>>
>>
>> On Wednesday, August 10, 2016, Efe Selcuk wrote:
>>
>>> Sorry, I should
ild now use "build/sbt package"
>
>
>
> On Wed, 10 Aug 2016 at 19:40, Efe Selcuk wrote:
>
>> Hi Spark folks,
>>
>> With Spark 1.6 the 'assembly' target for sbt would build a fat jar with
>> all of the main Spark dependencies for building an ap
Hi Spark folks,
With Spark 1.6 the 'assembly' target for sbt would build a fat jar with all
of the main Spark dependencies for building an application. Against Spark
2, that target is no longer building a spark assembly, just ones for e.g.
Flume and Kafka.
I'm not well versed with maven and sbt,
19 matches
Mail list logo
