Ha
CTO, Semantic Scholar
http://www.quora.com/What-is-Semantic-Scholar-and-how-will-it-work
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/renaming-SchemaRDD-DataFrame-tp10271p10612.html
Sent from the Apache Spark Developers List mail
;>>>>>>>>>>>
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> On Tue, Jan 27, 2015 at 6:28 AM, Dirceu Semighini Filho <
> >> > >>>>>>>
gt;
>> > >>>>>> removed
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> in
>> > >>>>>>>>>>
>> > >>>>>>>>>> the
vy
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> lifting
> > >>>>>>>>>>>>>> required for a full SQL support. It is amazing that even
> > >>>>>>>>>>>>
>>>>>
>>>>>>>>>>>> DataFrame?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With this, we don't impact in existing code
t;>>>>>>> I want to address the issue that Matei raised about the
> >>>
> >>> heavy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> lifting
> >>>>>>>>>&g
;>>>>> lifting
>>>>>>>>>>>>>> required for a full SQL support. It is amazing that even
>>>>>>>>>>>>>> after
>>>>>>
>>>>>> 30
>>>>>>>>>>
>>>>>>
w.youtube.com/watch?v=YWppYPWznSQ
From: Patrick Wendell
To: Reynold Xin
Cc: "dev@spark.apache.org"
Sent: Monday, January 26, 2015 4:01 PM
Subject: Re: renaming SchemaRDD -> DataFrame
One thing potentially not clear from this e-mail,
there
will
be
ak <
michaelma...@yahoo.com.invalid> wrote:
And in the off chance that anyone hasn't seen it yet,
the
Jan.
13
Bay
Area
Spark Meetup YouTube contained a wealth of background
information
on
this
idea (mostly from Patrick and Reynold :-).
https://www.youtube.com/watch?v=YWppYPWzn
more descriptive.
> >> > >> >> >>>>>
> >> > >> >> >>>>> Even if SchemaRDD's needs to rely on Spark SQL under the
> >> > covers,
> >> > >> >> >>>>> it
> >>
o at least
>> > choose a
>> > >> >> >>> package
>> > >> >> >>>>> name for it that omits "sql".
>> > >> >> >>>>>
>> > >> >> >>>>&
15 at 5:32 PM, Matei Zaharia <
> > >> >> >>> matei.zaha...@gmail.com>
> > >> >> >>>>> wrote:
> > >> >> >>>>>
> > >> >> >>>>>> (Actually when we designed Spark SQL we thought of giving i
t;>>>> be more clear from a user-facing perspective to at least
> >> >> >> >>>>> choose a
> >> >> >> >>> package
> >> >> >> >>>>> name for it that omits "sql".
> &
t;> >> >> this
>> >> >> >>>>> point?
>> >> >> >>>>>
>> >> >> >>>>> -Sandy
>> >> >> >>>>>
>> >> >> >>>>> On Mon, Jan 26, 2015 at 5
aria <
> >> >> >>> matei.zaha...@gmail.com>
> >> >> >>>>> wrote:
> >> >> >>>>>
> >> >> >>>>>> (Actually when we designed Spark SQL we thought of giving it
> >>
>>> name,
>> >> >>>>>> like Spark Schema, but we decided to stick with SQL since that
>> >> >>>>>> was
>> >> >>> the
>> >> >>>>> most
>> >> >>>>>&
; >> >>>>>>
> >> >>>>>>> On Jan 26, 2015, at 5:31 PM, Matei Zaharia <
> >> >>> matei.zaha...@gmail.com>
> >> >>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>> While it might be possible to move this concept to Sp
ly does require quite a bit of
>> >>> the
>> >>>>>> infrastructure in Spark SQL, such as query planning and columnar
>> >>>> storage.
>> >>>>>> The intent of Spark SQL though is to be more than a SQL server --
>> >>&
> Matei
> >>>>>>>
> >>>>>>>> On Jan 26, 2015, at 4:26 PM, Koert Kuipers
> >>>> wrote:
> >>>>>>>>
> >>>>>>>> "The context is that SchemaRDD is becoming a common data forma
is a library.
>>>>>>>
>>>>>>> Matei
>>>>>>>
>>>>>>>> On Jan 26, 2015, at 4:26 PM, Koert Kuipers
>>>> wrote:
>>>>>>>>
>>>>>>>> "The context is th
gt; > >> "The context is that SchemaRDD is becoming a common data format
>> used
>> > > for
>> > > > >> bringing data into Spark from external systems, and used for
>> various
>> > > > >> components of Spark, e.g. MLlib
It has been pretty evident for some time that's what it is, hasn't it?
Yes that's a better name IMO.
On Mon, Jan 26, 2015 at 2:18 PM, Reynold Xin wrote:
> Hi,
>
> We are considering renaming SchemaRDD -> DataFrame in 1.3, and wanted to
> get the community'
rk core, not sql
>>> >
>>> > On Mon, Jan 26, 2015 at 6:11 PM, Michael Malak <
>>> > michaelma...@yahoo.com.invalid> wrote:
>>> >
>>> >> And in the off chance that anyone hasn't seen it yet, the Jan. 13 Bay
>>> Area
>&
ground information on
>> this
>> >> idea (mostly from Patrick and Reynold :-).
>> >>
>> >> https://www.youtube.com/watch?v=YWppYPWznSQ
>> >>
>> >>
>> >> From: Patrick Wendell
>&g
t; And in the off chance that anyone hasn't seen it yet, the Jan. 13
> > Bay
> > > > Area
> > > > >>> Spark Meetup YouTube contained a wealth of background information
> > on
> > > > this
> > > > >>> idea (mostly fr
ose not immersed in data science
or AI and thus may have narrower appeal.
- Original Message -
From: Evan R. Sparks
To: Matei Zaharia
Cc: Koert Kuipers ; Michael Malak ;
Patrick Wendell ; Reynold Xin ;
"dev@spark.apache.org"
Sent: Tuesday, January 27, 2015 9:55 AM
Subject: Re: renaming
Bay
> > Area
> > >> Spark Meetup YouTube contained a wealth of background information on
> > this
> > >> idea (mostly from Patrick and Reynold :-).
> > >>
> > >> https://www.youtube.com/watch?v=YWppYPWznSQ
> > >>
> > >>
>
> >> From: Patrick Wendell
> >> To: Reynold Xin
> >> Cc: "dev@spark.apache.org"
> >> Sent: Monday, January 26, 2015 4:01 PM
> >> Subject: Re: renaming SchemaRDD -> DataFrame
> >>
> &g
gt; Spark Meetup YouTube contained a wealth of background information on
> this
> >>> idea (mostly from Patrick and Reynold :-).
> >>>
> >>> https://www.youtube.com/watch?v=YWppYPWznSQ
> >>>
> >>>
>
gt; > >>
> > > >> i agree. this to me also implies it belongs in spark core, not sql
> > > >>
> > > >> On Mon, Jan 26, 2015 at 6:11 PM, Michael Malak <
> > > >> michaelma...@yahoo.com.invalid> wrote:
> > > >>
> >
t; >>
> > >>> And in the off chance that anyone hasn't seen it yet, the Jan. 13 Bay
> > Area
> > >>> Spark Meetup YouTube contained a wealth of background information on
> > this
> > >>> idea (mostly from Patrick and Reynold
gt; Spark Meetup YouTube contained a wealth of background information on
> this
> >>> idea (mostly from Patrick and Reynold :-).
> >>>
> >>> https://www.youtube.com/watch?v=YWppYPWznSQ
> >>>
> >>>
>
mostly from Patrick and Reynold :-).
>>
>> https://www.youtube.com/watch?v=YWppYPWznSQ
>>
>>
>> From: Patrick Wendell
>> To: Reynold Xin
>> Cc: "dev@spark.apache.org"
>> Sent: Monday, January 26, 2015 4:01 PM
>> Subject: Re:
ube contained a wealth of background information on this
>>> idea (mostly from Patrick and Reynold :-).
>>>
>>> https://www.youtube.com/watch?v=YWppYPWznSQ
>>>
>>> ____
>>> From: Patrick Wendell
>>> To: Reynold Xin
>>> Cc: "dev@s
tps://www.youtube.com/watch?v=YWppYPWznSQ
>
>
> From: Patrick Wendell
> To: Reynold Xin
> Cc: "dev@spark.apache.org"
> Sent: Monday, January 26, 2015 4:01 PM
> Subject: Re: renaming SchemaRDD -> DataFrame
>
>
> One thin
t;> To: Reynold Xin
>> Cc: "dev@spark.apache.org"
>> Sent: Monday, January 26, 2015 4:01 PM
>> Subject: Re: renaming SchemaRDD -> DataFrame
>>
>>
>> One thing potentially not clear from this e-mail, there will be a 1:1
>> correspondence wh
l
To: Reynold Xin
Cc: "dev@spark.apache.org"
Sent: Monday, January 26, 2015 4:01 PM
Subject: Re: renaming SchemaRDD -> DataFrame
One thing potentially not clear from this e-mail, there will be a 1:1
correspondence where you can get an RDD to/from a DataFrame.
On Mon, Jan 2
One thing potentially not clear from this e-mail, there will be a 1:1
correspondence where you can get an RDD to/from a DataFrame.
On Mon, Jan 26, 2015 at 2:18 PM, Reynold Xin wrote:
> Hi,
>
> We are considering renaming SchemaRDD -> DataFrame in 1.3, and wanted to
> get the comm
Hi,
We are considering renaming SchemaRDD -> DataFrame in 1.3, and wanted to
get the community's opinion.
The context is that SchemaRDD is becoming a common data format used for
bringing data into Spark from external systems, and used for various
components of Spark, e.g. MLlib's
39 matches
Mail list logo
