Hi Sean,
See http://www.oracle.com/technetwork/java/eol-135779.html
Java 7 hasn't EOLed yet. If you look at support you can get from Oracle,
it's actually goes to 2019. And you can even get more support after that.
Spark has always maintained great backward compatibility with other
systems, way
Following https://github.com/apache/spark/pull/12165#issuecomment-205791222
I'd like to make a point about process and then answer points below.
We have this funny system where anyone can propose a change, and any
of a few people can veto a change unilaterally. The latter rarely
comes up. 9 change
> On 4 Apr 2016, at 20:58, Ofir Manor wrote:
>
> I think that a backup plan could be to announce that JDK7 is deprecated in
> Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives
> admins enough warning to install JDK8 along side their "main" JDK (or fully
> migrate to
On 4 Apr 2016, at 16:41, Xuefeng Wu mailto:ben...@gmail.com>>
wrote:
Many open source projects are aggressive, such as Oracle JDK and Ubuntu, But
they provide stable commercial supporting.
supporting old versions of jdk is one of the key revenue streams for oracle's
sun group: there's a lot
Curveball: Is there a need to use lambdas quite yet?
On Mon, Apr 4, 2016 at 10:58 PM, Ofir Manor wrote:
> I think that a backup plan could be to announce that JDK7 is deprecated in
> Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives
> admins enough warning to install JDK
I think that a backup plan could be to announce that JDK7 is deprecated in
Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives
admins enough warning to install JDK8 along side their "main" JDK (or fully
migrate to it), while allowing the project to merge JDK8-specific change
Reynold,
Considering the performance improvements you mentioned in your original
e-mail and also considering that few other big data projects have already
or are in progress of abandoning JDK 7, I think it would benefit Spark if
we go with JDK 8.0 only.
Are there users that will be less aggressiv
Many open source projects are aggressive, such as Oracle JDK and Ubuntu, But
they provide stable commercial supporting.
In other words, the enterprises doesn't drop JDK7, might aslo do not drop Spark
1.x to adopt Spark 2.x early version.
On Sun, Apr 3, 2016 at 10:29 PM -0700, "Reynold Xi
Since my original email, I've talked to a lot more users and looked at what
various environments support. It is true that a lot of enterprises, and
even some technology companies, are still using Java 7. One thing is that
up until this date, users still can't install openjdk 8 on Ubuntu by
default.
t Kuipers ;
Kostas Sakellis ; Marcelo Vanzin ;
dev@spark.apache.org
Subject: Re: [discuss] ending support for Java 7 in Spark 2.0
Steve, those are good points, I had forgotten Hadoop had those issues.We
run with jdk 8, hadoop is built for jdk7 compatibility, we are running hadoop
2.7 on our
Steve, those are good points, I had forgotten Hadoop had those issues. We
run with jdk 8, hadoop is built for jdk7 compatibility, we are running hadoop
2.7 on our clusters and by the time Spark 2.0 is out I would expected a mix of
Hadoop 2.7 and 2.8. We also don't use spnego.
I didn't quite
Can I note that if Spark 2.0 is going to be Java 8+ only, then that means
Hadoop 2.6.x should be the minimum Hadoop version.
https://issues.apache.org/jira/browse/HADOOP-11090
Where things get complicated, is that situation of: Hadoop services on Java 7,
Spark on Java 8 in its own JVM
I'm not
+1.
Tom
On Tuesday, March 29, 2016 1:17 PM, Reynold Xin wrote:
They work.
On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers wrote:
if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd
party scala libraries compiled with a scala version < 2.10.4 might not work on
They work.
On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers wrote:
> if scala prior to sbt 2.10.4 didn't support java 8, does that mean that
> 3rd party scala libraries compiled with a scala version < 2.10.4 might not
> work on java 8?
>
>
> On Mon, Mar 28, 2016 at 7:06 PM, Kostas Sakellis
> wr
if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd
party scala libraries compiled with a scala version < 2.10.4 might not work
on java 8?
On Mon, Mar 28, 2016 at 7:06 PM, Kostas Sakellis
wrote:
> Also, +1 on dropping jdk7 in Spark 2.0.
>
> Kostas
>
> On Mon, Mar 28, 201
Also, +1 on dropping jdk7 in Spark 2.0.
Kostas
On Mon, Mar 28, 2016 at 2:01 PM, Marcelo Vanzin wrote:
> Finally got some internal feedback on this, and we're ok with
> requiring people to deploy jdk8 for 2.0, so +1 too.
>
> On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende
> wrote:
> > +1, I al
Finally got some internal feedback on this, and we're ok with
requiring people to deploy jdk8 for 2.0, so +1 too.
On Mon, Mar 28, 2016 at 1:15 PM, Luciano Resende wrote:
> +1, I also checked with few projects inside IBM that consume Spark and they
> seem to be ok with the direction of droping JDK
+1, I also checked with few projects inside IBM that consume Spark and they
seem to be ok with the direction of droping JDK 7.
On Mon, Mar 28, 2016 at 11:24 AM, Michael Gummelt
wrote:
> +1 from Mesosphere
>
> On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran
> wrote:
>
>>
>> > On 25 Mar 2016, at
+1 from Mesosphere
On Mon, Mar 28, 2016 at 5:12 AM, Steve Loughran
wrote:
>
> > On 25 Mar 2016, at 01:59, Mridul Muralidharan wrote:
> >
> > Removing compatibility (with jdk, etc) can be done with a major release-
> given that 7 has been EOLed a while back and is now unsupported, we have to
> d
> On 25 Mar 2016, at 01:59, Mridul Muralidharan wrote:
>
> Removing compatibility (with jdk, etc) can be done with a major release-
> given that 7 has been EOLed a while back and is now unsupported, we have to
> decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
>
> Given the
i asked around a little, and the general trend at our clients seems to be
that they plan to upgrade the clusters to java 8 within the year.
so with that in mind i wish this was a little later (i would have preferred
a java-8-only spark at the end of year). but since a major spark version
only come
+1 on removing Java 7 and Scala 2.10 support.
It looks to be entirely possible to support Java 8 containers in a YARN
cluster otherwise running Java 7 (example code for alt JAVA_HOME
https://issues.apache.org/jira/secure/attachment/12671739/YARN-1964.patch)
so really there should be no big problem
I do agree w.r.t scala 2.10 as well; similar arguments apply (though there
is a nuanced diff - source compatibility for scala vs binary compatibility
wrt Java)
Was there a proposal which did not go through ? Not sure if I missed it.
Regards
Mridul
On Thursday, March 24, 2016, Koert Kuipers wrote
I do agree w.r.t scala 2.10 as well; similar arguments apply (though there
is a nuanced diff - source compatibility for scala vs binary compatibility
wrt Java)
Was there a proposal which did not go through ? Not sure if I missed it.
Regards
Mridul
On Thursday, March 24, 2016, Koert Kuipers wrote
Arguments are really convincing; new Dataset API as well as performance
improvements is exiting, so I'm personally +1 on moving onto Java8.
However, I'm afraid Tencent is one of "the organizations stuck with Java7"
-- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and
wou
i think that logic is reasonable, but then the same should also apply to
scala 2.10, which is also unmaintained/unsupported at this point (basically
has been since march 2015 except for one hotfix due to a license
incompatibility)
who wants to support scala 2.10 three years after they did the last
Removing compatibility (with jdk, etc) can be done with a major release-
given that 7 has been EOLed a while back and is now unsupported, we have to
decide if we drop support for it in 2.0 or 3.0 (2+ years from now).
Given the functionality & performance benefits of going to jdk8, future
enhanceme
i think marcelo also pointed this out before. its very interesting to hear,
i was not aware of that until today. it would mean we would only have to
convince a group/client with a cluster to install jdk8 on the nodes,
without actually transitioning to it, if i understand it correctly. that
would de
Container Java version can be different from yarn Java version : we run
jobs with jdk8 on jdk7 cluster without issues.
Regards
Mridul
On Thursday, March 24, 2016, Koert Kuipers wrote:
> i guess what i am saying is that in a yarn world the only hard
> restrictions left are the the containers you
the good news is, that from an shared infrastructure perspective, most
places have zero scala, so the upgrade is actually very easy. i can see how
it would be different for say twitter
On Thu, Mar 24, 2016 at 7:50 PM, Reynold Xin wrote:
> If you want to go down that route, you should also as
On Thu, Mar 24, 2016 at 4:54 PM, Mark Hamstra
wrote:
> It's a pain in the ass. Especially if some of your transitive
> dependencies never upgraded from 2.10 to 2.11.
>
Yeah, I'm going to have to agree here. It is not as bad as it was in the
2.9 days, but its still non-trivial due to the eco-s
There aren't many such libraries, but there are a few. When faced with one
of those dependencies that still doesn't go beyond 2.10, you essentially
have the choice of taking on the maintenance burden to bring the library up
to date, or you do what is potentially a fairly larger refactoring to use
On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin wrote:
> If you want to go down that route, you should also ask somebody who has had
> experience managing a large organization's applications and try to update
> Scala version.
I understand both sides. But if you look at what I've been asking
since th
It's a pain in the ass. Especially if some of your transitive dependencies
never upgraded from 2.10 to 2.11.
On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin wrote:
> If you want to go down that route, you should also ask somebody who has
> had experience managing a large organization's application
In addition, with Spark 2.0, we are throwing away binary compatibility
anyways so user applications will have to be recompiled.
The only argument I can see is for libraries that have already been built
on Scala 2.10 that are no longer being maintained. How big of an issue do
we think that is?
Kos
If you want to go down that route, you should also ask somebody who has had
experience managing a large organization's applications and try to update
Scala version.
On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin wrote:
> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin wrote:
> > Actually it's *w
On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin wrote:
> Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
> upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not
> binary compatible, whereas JVM 7 and 8 are binary compatible except certain
> esoteric cases.
Tr
Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than
upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not
binary compatible, whereas JVM 7 and 8 are binary compatible except certain
esoteric cases.
On Thu, Mar 24, 2016 at 4:44 PM, Kostas Sakellis
wrote:
> If
If an argument here is the ongoing build/maintenance burden I think we
should seriously consider dropping scala 2.10 in Spark 2.0. Supporting
scala 2.10 is bigger build/infrastructure burden than supporting jdk7 since
you actually have to build different artifacts and test them whereas you
can targ
On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky wrote:
> You can, but since it's going to be a maintainability issue I would
> argue it is in fact a problem.
Every thing you choose to support generates a maintenance burden.
Support 3 versions of Scala would be a huge maintenance burden, for
exampl
I mean from the perspective of someone developing Spark, it makes
things more complicated. It's just my point of view, people that
actually support Spark deployments may have a different opinion ;)
On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky wrote:
> You can, but since it's going to be a maint
You can, but since it's going to be a maintainability issue I would
argue it is in fact a problem.
On Thu, Mar 24, 2016 at 2:34 PM, Marcelo Vanzin wrote:
> Hi Jakob,
>
> On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky wrote:
>> Reynold's 3rd point is particularly strong in my opinion. Supporting
Hi Jakob,
On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky wrote:
> Reynold's 3rd point is particularly strong in my opinion. Supporting
> Consider what would happen if Spark 2.0 doesn't require Java 8 and
> hence not support Scala 2.12. Will it be stuck on an older version
> until 3.0 is out?
Tha
+1 for Java 8 only
I think it will make it easier to make a unified API for Java and Scala,
instead of the wrappers of Java over Scala.
On Mar 24, 2016 11:46 AM, "Stephen Boesch" wrote:
> +1 for java8 only +1 for 2.11+ only .At this point scala libraries
> supporting only 2.10 are typicall
Reynold's 3rd point is particularly strong in my opinion. Supporting
Scala 2.12 will require Java 8 anyway, and introducing such a change
is probably best done in a major release.
Consider what would happen if Spark 2.0 doesn't require Java 8 and
hence not support Scala 2.12. Will it be stuck on an
Spark 2.x has to be the time for Java 8.
I'd rather increase JVM major version on a Spark major version than on a
Spark minor version, and I'd rather Spark do that upgrade for the 2.x
series than the 3.x series (~2yr from now based on the lifetime of Spark
1.x). If we wait until the next opportun
+1 for java8 only +1 for 2.11+ only .At this point scala libraries
supporting only 2.10 are typically less active and/or poorly maintained.
That trend will only continue when considering the lifespan of spark 2.X.
2016-03-24 11:32 GMT-07:00 Steve Loughran :
>
> On 24 Mar 2016, at 15:27, Koe
On 24 Mar 2016, at 15:27, Koert Kuipers
mailto:ko...@tresata.com>> wrote:
i think the arguments are convincing, but it also makes me wonder if i live in
some kind of alternate universe... we deploy on customers clusters, where the
OS, python version, java version and hadoop distro are not chos
On Thu, Mar 24, 2016 at 10:13 AM, Reynold Xin wrote:
> Yes
So is it safe to say the only hard requirements for Java 8 in your list is (4)?
(1) and (3) are infrastructure issues. Yes, it sucks to maintain more
testing infrastructure and potentially more complicated build scripts,
but does that re
Yes
On Thursday, March 24, 2016, Marcelo Vanzin wrote:
> On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin > wrote:
> > I actually talked quite a bit today with an engineer on the scala
> compiler
> > team tonight and the scala 2.10 + java 8 combo should be ok. The latest
> > Scala 2.10 release shou
On Thu, Mar 24, 2016 at 9:54 AM, Koert Kuipers wrote:
> i guess what i am saying is that in a yarn world the only hard restrictions
> left are the the containers you run in, which means the hadoop version, java
> version and python version (if you use python).
It is theoretically possible to run
On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin wrote:
> I actually talked quite a bit today with an engineer on the scala compiler
> team tonight and the scala 2.10 + java 8 combo should be ok. The latest
> Scala 2.10 release should have all the important fixes that are needed for
> Java 8.
So, do
i guess what i am saying is that in a yarn world the only hard restrictions
left are the the containers you run in, which means the hadoop version,
java version and python version (if you use python).
On Thu, Mar 24, 2016 at 12:39 PM, Koert Kuipers wrote:
> The group will not upgrade to spark 2
Thank you for the context Jean...
I appreciate it...
On Thu, Mar 24, 2016 at 12:40 PM, Jean-Baptiste Onofré
wrote:
> Hi Al,
>
> Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will go
> on Spark 2.0, but maintenance release can be performed on 1.x branch.
>
> Regards
> JB
>
>
Hi Al,
Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will
go on Spark 2.0, but maintenance release can be performed on 1.x branch.
Regards
JB
On 03/24/2016 05:38 PM, Al Pivonka wrote:
As an end user (developer) and Cluster Admin.
I would have to agree with Koet.
To me th
The group will not upgrade to spark 2.0 themselves, but they are mostly
fine with vendors like us deploying our application via yarn with whatever
spark version we choose (and bundle, so they do not install it separately,
they might not even be aware of what spark version we use). This all works
be
As an end user (developer) and Cluster Admin.
I would have to agree with Koet.
To me the real question is timing, current version is 1.6.1, the question
I have is how many more releases till 2.0 and what is the time frame?
If you give people six to twelve months to plan and make sure they know
(
(PS CDH5 runs fine with Java 8, but I understand your more general point.)
This is a familiar context indeed, but in that context, would a group
not wanting to update to Java 8 want to manually put Spark 2.0 into
the mix? That is, if this is a context where the cluster is
purposefully some stable
i think the arguments are convincing, but it also makes me wonder if i live
in some kind of alternate universe... we deploy on customers clusters,
where the OS, python version, java version and hadoop distro are not chosen
by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply
+1 to support Java 8 (and future) *only* in Spark 2.0, and end support
of Java 7. It makes sense.
Regards
JB
On 03/24/2016 08:27 AM, Reynold Xin wrote:
About a year ago we decided to drop Java 6 support in Spark 1.5. I am
wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
> On 24 Mar 2016, at 07:27, Reynold Xin wrote:
>
> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> wondering if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark
> 2.0 would require Java 8 to run).
>
> Oracle ended public updates for JDK 7 in one year ag
Maybe so; I think we have a ticket open to update to 2.10.6, which
maybe fixes it.
It brings up a different point: supporting multiple Scala versions is
much more painful than Java versions because of mutual
incompatibility. Right now I get the sense there's an intent to keep
supporting 2.10, and
I actually talked quite a bit today with an engineer on the scala compiler
team tonight and the scala 2.10 + java 8 combo should be ok. The latest
Scala 2.10 release should have all the important fixes that are needed for
Java 8.
On Thu, Mar 24, 2016 at 1:01 AM, Sean Owen wrote:
> I generally fa
I generally favor this for the simplification. I didn't realize there
were actually some performance wins and important bug fixes.
I've had lots of trouble with scalac 2.10 + Java 8. I don't know if
it's still a problem since 2.11 + 8 seems OK, but for a long time the
sql/ modules would never comp
Xin [mailto:r...@databricks.com]
> *Sent:* Thursday, March 24, 2016 9:37 AM
> *To:* dev@spark.apache.org
> *Subject:* Re: [discuss] ending support for Java 7 in Spark 2.0
>
>
>
> One other benefit that I didn't mention is that we'd be able to use Java
> 8's Opti
+1
Agree, dropping support for java 7 is long overdue - and 2.0 would be
a logical release to do this on.
Regards,
Mridul
On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin wrote:
> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> wondering if we should also just drop Java 7 s
From: Reynold Xin [mailto:r...@databricks.com]
Sent: Thursday, March 24, 2016 9:37 AM
To: dev@spark.apache.org
Subject: Re: [discuss] ending support for Java 7 in Spark 2.0
One other benefit that I didn't mention is that we'd be able to use Java 8's
Optional class to replace our built-
One other benefit that I didn't mention is that we'd be able to use Java
8's Optional class to replace our built-in Optional.
On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin wrote:
> About a year ago we decided to drop Java 6 support in Spark 1.5. I am
> wondering if we should also just drop Java
About a year ago we decided to drop Java 6 support in Spark 1.5. I am
wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.
Spark 2.0 would require Java 8 to run).
Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and
removed public downloads for JDK 7 in July 201
69 matches
Mail list logo
