Re: Welcome Jose Torres as a Spark committer

Congrats, Jose! *Dean Wampler, Ph.D.* *VP, Fast Data Engineering at Lightbend* On Tue, Jan 29, 2019 at 12:52 PM Burak Yavuz wrote: > Congrats Jose! > > On Tue, Jan 29, 2019 at 10:50 AM Xiao Li wrote: > >> Congratulations! >> >> Xiao >> >> Shixio

Re: Make Scala 2.12 as default Scala version in Spark 3.0

lternative Scala version. dean *Dean Wampler, Ph.D.* *VP, Fast Data Engineering at Lightbend* Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do>, Fast Data Architectures for Streaming Applications <http://www.oreilly.com/data/free/fast-dat

Re: Scala 2.12 support

Do the tests expect a particular console output order? That would annoy them. ;) You could sort the expected and output lines, then diff... *Dean Wampler, Ph.D.* *VP, Fast Data Engineering at Lightbend* Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073

Re: Scala 2.12 support

I'll point the Scala team to this issue, but it's unlikely to get fixed any time soon. dean *Dean Wampler, Ph.D.* *VP, Fast Data Engineering at Lightbend* Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do>, Fast Data Architectur

Re: Scala 2.12 support

way. (They can provide more specific details). This would not address the closure cleaner changes required. However, the Scala team offered to provide suggestions or review changes. dean *Dean Wampler, Ph.D.* *VP, Fast Data Engineering at Lightbend* Author: Programming Scala, 2nd Edi

Re: time for Apache Spark 3.0?

closure cleaner would still need updating. Hence, it could be done for Spark 2.X. Let me if you want to pursue this approach. dean *Dean Wampler, Ph.D.* *VP, Fast Data Engineering at Lightbend* Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do>, Fast D

Re: welcoming Burak and Holden as committers

Congratulations to both of you! dean *Dean Wampler, Ph.D.* Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do>, Fast Data Architectures for Streaming Applications <http://www.oreilly.com/data/free/fast-data-architectures-for-streaming-applica

Re: Apache Spark chat channel

Okay, here is a Gitter room for this purpose: https://gitter.im/spark-scala/Lobby If you use the APIs, please join and help those who are learning. I can't answer every question. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/06369200

Re: Apache Spark chat channel

Since I'm a Scala Spark advocate, I'll try to get a Scala Spark Gitter channel created, one way or another. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Lightbend <http://lightbend.com> @dean

Re: Using Spark when data definitions are unknowable at compile time

specify the schema dynamically some how? Either way, once you know the types and indices of the fields you need for a given query, you can fetch them using the Row methods. HTH, dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073

Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle

you're ready to upgrade. So, I recommend that Spark 2.0 drop Scala 2.10 support from the beginning. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Lightbend <http://lightbend.com> @deanwampler

Re: Akka with Spark

As Reynold said, you can still use Akka with Spark, but now it's more like using any third-party library that isn't already a Spark dependency (at least once the current Akka dependency is fully removed). Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreill

Re: Akka with Spark

reams. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Sat, Dec 26, 2015 at 12:54 PM, T

Re: [ANNOUNCE] Spark 1.6.0 Release Preview

(3616 + 5) / 10] ... elided ... [Stage 1:=>(98393 + 4) / 10] [Stage 1:=>(99347 + 4) / 10] [Stage 1:=======

Re: Removing the Mesos fine-grained mode

Sounds like the right move. Simplifies things in important ways. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://poly

Re: [discuss] ending support for Java 6?

FWIW, another reason to start planning for deprecation of Java 7, too, is that Scala 2.12 will require Java 8. Scala 2.12 will be released early next year. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typ

Re: [sql] Dataframe how to check null values

your best choice is to use NaN as the placeholder for null, then create one DF using a filter that removes those values. Use that DF to compute the mean. Then apply a map step to the original DF to translate the NaN's to the mean. dean Dean Wampler, Ph.D. Author: Programming

Re: Need advice for Spark newbie

list would be slightly better.) dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Thu,

Re: Need advice for Spark newbie

rated as what you're used to. You're trading that polish for greater scalability and flexibility. HTH. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler

Re: Are there any plans to run Spark on top of Succinct

Interesting. I was wondering recently if anyone has explored working with compressed data directly. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twi

Re: How spark and hive integrate in long term?

s to be a fresh effort. dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Fri, Nov

Re: best IDE for scala + spark development?

ent versions of Scala, which is convenient for Spark's current 2.10.4 support and emerging 2.11 support. http://scala-ide.org/download/milestone.html Dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly

Re: SparkSubmit and --driver-java-options

ight way, so any quoted > >> strings get flattened. > >> > >> I think we'll need to figure out how to do this correctly in the bash > >> script so that quoted strings get passed in the right way. > > > > I tried a few different approaches but finally

Re: Spark 1.0.0 rc3

empty, but I > think in the end we decided to just go with the exact same semantics > as Hadoop (i.e. empty directories are a problem). > > - Patrick > > On Tue, Apr 29, 2014 at 9:43 AM, Dean Wampler > wrote: > > I'm observing one anomalous behavior. With the 1.0.

Re: Spark 1.0.0 rc3

now return Iterable[T] instead of Seq[T] > ==> Call toSeq on the result to restore the old behavior > > SparkContext.jarOfClass returns Option[String] instead of Seq[String] > ==> Call toSeq on the result to restore old behavior > > Streaming classes have been renamed: > NetworkReceiver -> Receiver > -- Dean Wampler, Ph.D. Typesafe @deanwampler http://typesafe.com http://polyglotprogramming.com