Matei wrote (Jan 26, 2015; 5:31pm): "The intent of Spark SQL though is to be
more than a SQL server -- it's meant to be a library for manipulating
structured data."
I think this is an important but nuanced point. There are engineers who for
various reasons associate the term "SQL" with business an
;>>>>>>>>>>>
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> On Tue, Jan 27, 2015 at 6:28 AM, Dirceu Semighini Filho <
> >> > >>>>>>>
gt;
>> > >>>>>> removed
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> in
>> > >>>>>>>>>>
>> > >>>>>>>>>> the
t;>>>>>>> DataFrame?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> With this, we don't impact in existing code for the next
> few
> > >>>>>>>>>>>>> releases.
> > >>>>&g
>>>>>
>>>>>>>>>>>> DataFrame?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With this, we don't impact in existing code
t;>>>>>>> I want to address the issue that Matei raised about the
> >>>
> >>> heavy
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> lifting
> >>>>>>>>>&g
;>>>>> lifting
>>>>>>>>>>>>>> required for a full SQL support. It is amazing that even
>>>>>>>>>>>>>> after
>>>>>>
>>>>>> 30
>>>>>>>>>>
>>>>>>
2015 at 6:11 PM, Michael Malak <
michaelma...@yahoo.com.invalid> wrote:
And in the off chance that anyone hasn't seen it yet,
the
Jan.
13
Bay
Area
Spark Meetup YouTube contained a wealth of background
information
on
this
idea (mostly from Patrick and Reynold :-).
https://ww
ak <
michaelma...@yahoo.com.invalid> wrote:
And in the off chance that anyone hasn't seen it yet,
the
Jan.
13
Bay
Area
Spark Meetup YouTube contained a wealth of background
information
on
this
idea (mostly from Patrick and Reynold :-).
https://www.youtube.com/watch?v=YWppYPWzn
more descriptive.
> >> > >> >> >>>>>
> >> > >> >> >>>>> Even if SchemaRDD's needs to rely on Spark SQL under the
> >> > covers,
> >> > >> >> >>>>> it
> >>
o at least
>> > choose a
>> > >> >> >>> package
>> > >> >> >>>>> name for it that omits "sql".
>> > >> >> >>>>>
>> > >> >> >>>>&
adding a separate Spark Schema
> > module
> > >> >> >>>>> for
> > >> >> >>>> Spark
> > >> >> >>>>> SQL to rely on, but I imagine that might be too large a
> change
> > at
> > >> >> th
t;>>>> be more clear from a user-facing perspective to at least
> >> >> >> >>>>> choose a
> >> >> >> >>> package
> >> >> >> >>>>> name for it that omits "sql".
> &
t;> package
>> >> >> >>>>> name for it that omits "sql".
>> >> >> >>>>>
>> >> >> >>>>> I would also be in favor of adding a separate Spark Schema
>> >> >> >>>>>
aria <
> >> >> >>> matei.zaha...@gmail.com>
> >> >> >>>>> wrote:
> >> >> >>>>>
> >> >> >>>>>> (Actually when we designed Spark SQL we thought of giving it
> >>
>>> name,
>> >> >>>>>> like Spark Schema, but we decided to stick with SQL since that
>> >> >>>>>> was
>> >> >>> the
>> >> >>>>> most
>> >> >>>>>&
; >> >>>>>>
> >> >>>>>>> On Jan 26, 2015, at 5:31 PM, Matei Zaharia <
> >> >>> matei.zaha...@gmail.com>
> >> >>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>> While it might be possible to move this concept to Sp
ly does require quite a bit of
>> >>> the
>> >>>>>> infrastructure in Spark SQL, such as query planning and columnar
>> >>>> storage.
>> >>>>>> The intent of Spark SQL though is to be more than a SQL server --
>> >>&
>>>>> infrastructure in Spark SQL, such as query planning and columnar
> >>>> storage.
> >>>>>> The intent of Spark SQL though is to be more than a SQL server --
> >>> it's
> >>>>>> meant to be a library for manipulating structure
is a library.
>>>>>>>
>>>>>>> Matei
>>>>>>>
>>>>>>>> On Jan 26, 2015, at 4:26 PM, Koert Kuipers
>>>> wrote:
>>>>>>>>
>>>>>>>> "The context is th
gt; > >> "The context is that SchemaRDD is becoming a common data format
>> used
>> > > for
>> > > > >> bringing data into Spark from external systems, and used for
>> various
>> > > > >> components of Spark, e.g. MLlib
It has been pretty evident for some time that's what it is, hasn't it?
Yes that's a better name IMO.
On Mon, Jan 26, 2015 at 2:18 PM, Reynold Xin wrote:
> Hi,
>
> We are considering renaming SchemaRDD -> DataFrame in 1.3, and wanted to
> get the community's opinion.
>
> The context is that Sche
rk core, not sql
>>> >
>>> > On Mon, Jan 26, 2015 at 6:11 PM, Michael Malak <
>>> > michaelma...@yahoo.com.invalid> wrote:
>>> >
>>> >> And in the off chance that anyone hasn't seen it yet, the Jan. 13 Bay
>>> Area
>&
ground information on
>> this
>> >> idea (mostly from Patrick and Reynold :-).
>> >>
>> >> https://www.youtube.com/watch?v=YWppYPWznSQ
>> >>
>> >>
>> >> From: Patrick Wendell
>&g
hemaRDD is becoming a common data format
> used
> > > for
> > > > >> bringing data into Spark from external systems, and used for
> various
> > > > >> components of Spark, e.g. MLlib's new pipeline API."
> > > > >>
> > > > >> i agre
ose not immersed in data science
or AI and thus may have narrower appeal.
- Original Message -
From: Evan R. Sparks
To: Matei Zaharia
Cc: Koert Kuipers ; Michael Malak ;
Patrick Wendell ; Reynold Xin ;
"dev@spark.apache.org"
Sent: Tuesday, January 27, 2015 9:55 AM
Subject: Re: renaming
Bay
> > Area
> > >> Spark Meetup YouTube contained a wealth of background information on
> > this
> > >> idea (mostly from Patrick and Reynold :-).
> > >>
> > >> https://www.youtube.com/watch?v=YWppYPWznSQ
> > >>
> > >>
ne hasn't seen it yet, the Jan. 13 Bay
> Area
> >> Spark Meetup YouTube contained a wealth of background information on
> this
> >> idea (mostly from Patrick and Reynold :-).
> >>
> >> https://www.youtube.com/watch?v=YWppYPWznSQ
> >>
> >
gt; Spark Meetup YouTube contained a wealth of background information on
> this
> >>> idea (mostly from Patrick and Reynold :-).
> >>>
> >>> https://www.youtube.com/watch?v=YWppYPWznSQ
> >>>
> >>>
>
gt; > >>
> > > >> i agree. this to me also implies it belongs in spark core, not sql
> > > >>
> > > >> On Mon, Jan 26, 2015 at 6:11 PM, Michael Malak <
> > > >> michaelma...@yahoo.com.invalid> wrote:
> > > >>
> >
t; >>
> > >>> And in the off chance that anyone hasn't seen it yet, the Jan. 13 Bay
> > Area
> > >>> Spark Meetup YouTube contained a wealth of background information on
> > this
> > >>> idea (mostly from Patrick and Reynold
gt; Spark Meetup YouTube contained a wealth of background information on
> this
> >>> idea (mostly from Patrick and Reynold :-).
> >>>
> >>> https://www.youtube.com/watch?v=YWppYPWznSQ
> >>>
> >>>
>
mostly from Patrick and Reynold :-).
>>
>> https://www.youtube.com/watch?v=YWppYPWznSQ
>>
>>
>> From: Patrick Wendell
>> To: Reynold Xin
>> Cc: "dev@spark.apache.org"
>> Sent: Monday, January 26, 2015 4:01 PM
>> Subject: Re:
ube contained a wealth of background information on this
>>> idea (mostly from Patrick and Reynold :-).
>>>
>>> https://www.youtube.com/watch?v=YWppYPWznSQ
>>>
>>> ____
>>> From: Patrick Wendell
>>> To: Reynold Xin
>>> Cc: "dev@s
tps://www.youtube.com/watch?v=YWppYPWznSQ
>
>
> From: Patrick Wendell
> To: Reynold Xin
> Cc: "dev@spark.apache.org"
> Sent: Monday, January 26, 2015 4:01 PM
> Subject: Re: renaming SchemaRDD -> DataFrame
>
>
> One thin
t;> To: Reynold Xin
>> Cc: "dev@spark.apache.org"
>> Sent: Monday, January 26, 2015 4:01 PM
>> Subject: Re: renaming SchemaRDD -> DataFrame
>>
>>
>> One thing potentially not clear from this e-mail, there will be a 1:1
>> correspondence wh
l
To: Reynold Xin
Cc: "dev@spark.apache.org"
Sent: Monday, January 26, 2015 4:01 PM
Subject: Re: renaming SchemaRDD -> DataFrame
One thing potentially not clear from this e-mail, there will be a 1:1
correspondence where you can get an RDD to/from a DataFrame.
On Mon, Jan 2
One thing potentially not clear from this e-mail, there will be a 1:1
correspondence where you can get an RDD to/from a DataFrame.
On Mon, Jan 26, 2015 at 2:18 PM, Reynold Xin wrote:
> Hi,
>
> We are considering renaming SchemaRDD -> DataFrame in 1.3, and wanted to
> get the community's opinion.
38 matches
Mail list logo
