Spark Compatibility with Java 11

2020-07-14 Thread Ankur Mittal
Hi, I am using Spark 2.X and need to execute Java 11 .Its not able to execute Java 11 using Spark 2.X. Is there any way we can use Java 11 with Spark2.X? Has this issue been resolved in Spark 3.0 ? -- Regards Ankur Mittal

Re: Spark Compatibility with Java 11

2020-07-14 Thread Prashant Sharma
Hi Ankur, Java 11 support was added in Spark 3.0. https://issues.apache.org/jira/browse/SPARK-24417 Thanks, On Tue, Jul 14, 2020 at 6:12 PM Ankur Mittal wrote: > Hi, > > I am using Spark 2.X and need to execute Java 11 .Its not able to execute > Java 11 using Spark 2.X. > > Is there any way w

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Patrick McCarthy
Please don't advocate for piracy, this book is not freely available. I own it and it's wonderful, Mr. Géron deserves to benefit from it. On Mon, Jul 13, 2020 at 9:59 PM Anwar AliKhan wrote: > link to a free book which may be useful. > > Hands-On Machine Learning with Scikit-Learn, Keras, and

Re: Spark Compatibility with Java 11

2020-07-14 Thread Ankur Mittal
Thanks a lot. On Tue, Jul 14, 2020 at 12:51 PM Prashant Sharma wrote: > Hi Ankur, > > Java 11 support was added in Spark 3.0. > https://issues.apache.org/jira/browse/SPARK-24417 > > Thanks, > > > On Tue, Jul 14, 2020 at 6:12 PM Ankur Mittal > wrote: > >> Hi, >> >> I am using Spark 2.X and need

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Anwar AliKhan
Please note It is freely available because it is an early unedited raw edition. It is not 100% complete , it is not entirely same as yours. So it is not piracy. I agree it is a good book. On Tue, 14 Jul 2020, 14:30 Patrick McCarthy, wrote: > Please don't advocate for piracy, this book is n

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Sean Owen
It is still copyrighted material, no matter its state of editing. Yes, you should not be sharing this on the internet. On Tue, Jul 14, 2020 at 9:46 AM Anwar AliKhan wrote: > > Please note It is freely available because it is an early unedited raw > edition. > It is not 100% complete , it is not

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Anwar AliKhan
Ok, thanks. You can buy it here https://www.amazon.com/s?k=hands+on+machine+learning+with+scikit-learn+and+tensorflow+2&crid=2U0P9XVIJ790T&sprefix=Hands+on+machine+%2Caps%2C246&ref=nb_sb_ss_i_1_17 This book is like an accompaniment to the Andrew Ng course on coursera. It uses exact same mathemati

Mock spark reads and writes

2020-07-14 Thread Dark Crusader
Sorry I wasn't very clear in my last email. I have a function like this: def main( read_file): df = spark.read.csv(read_file) ** Some other code ** df.write.csv(path) Which I need to write a unit test for. Would pythons unittest mock help me here? When I googled this, I most

Kotlin Spark API

2020-07-14 Thread Maria Khalusova
Hi folks, We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api. Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin langua

Re: Kotlin Spark API

2020-07-14 Thread Anwar AliKhan
Is kotlin another new language ? GRADY BOOCH; The United States Department of defence (DOD) is perhaps the largest user of computers in the world. By the mid-1970s, software development for its systems had reached crisis proportions: projects were often late, over budget and they often failed to

Re: Kotlin Spark API

2020-07-14 Thread Stephen Boesch
I started with scala/spark in 2012 and scala has been my go-to language for six years. But I heartily applaud this direction. Kotlin is more like a simplified Scala - with the benefits that brings - than a simplified java. I particularly like the simplified / streamlined collections classes. Reall

Re: Kotlin Spark API

2020-07-14 Thread Stephen Boesch
I just looked at the examples. https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples These look v nice! V concise yet flexible. I like the ability to do inline *side-effects. *E.g. caching or printing or showDs() package org.jetbrain

Re: Kotlin Spark API

2020-07-14 Thread Pasha Finkelshteyn
Hi Stephen, Thank you so much for finding time for looking at our examples! Yes, we've tried to implement as clean design of API as possible and are constantly looking for ways to make it even more readable, clear and friendly. And as Maria already stated we welcome any feedback! On 20/07/14 0