I think you can place the jar in lib/ in SPARK_HOME, and then compile without
any change to your class path. This could be a temporary way to include your
jar. You can also put them in your pom.xml.
Thanks,
Daoyuan
-Original Message-
From: flyson [mailto:m_...@msn.com]
Sent: Wednesday,
Hello everyone,
Could anybody tell me how to import and call the 3rd party java classes from
inside spark?
Here's my case:
I have a jar file (the directory layout is com.xxx.yyy.zzz) which contains
some java classes, and I need to call some of them in spark code.
I used the statement "import com.x
In Hive 13 (which is the default for Spark 1.2), parquet is included and
thus we no longer include the Hive parquet bundle. You can now use the
included
ParquetSerDe: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
If you want to compile Spark 1.2 with Hive 12 instead you can pass
-Ph
On Tue, Dec 2, 2014 at 4:40 PM, Ryan Williams
wrote:
>> But you only need to compile the others once.
>
> once... every time I rebase off master, or am obliged to `mvn clean` by some
> other build-correctness bug, as I said before. In my experience this works
> out to a few times per week.
No, yo
On Tue Dec 02 2014 at 4:46:20 PM Marcelo Vanzin wrote:
> On Tue, Dec 2, 2014 at 3:39 PM, Ryan Williams
> wrote:
> > Marcelo: by my count, there are 19 maven modules in the codebase. I am
> > typically only concerned with "core" (and therefore its two dependencies
> as
> > well, `network/{shuffle
On Tue, Dec 2, 2014 at 3:39 PM, Ryan Williams
wrote:
> Marcelo: by my count, there are 19 maven modules in the codebase. I am
> typically only concerned with "core" (and therefore its two dependencies as
> well, `network/{shuffle,common}`).
But you only need to compile the others once. Once you'v
Marcelo: by my count, there are 19 maven modules in the codebase. I am
typically only concerned with "core" (and therefore its two dependencies as
well, `network/{shuffle,common}`).
The `mvn package` workflow (and its sbt equivalent) that most people
apparently use involves (for me) compiling+pack
Hey Ryan,
What if you run a single "mvn install" to install all libraries
locally - then can you "mvn compile -pl core"? I think this may be the
only way to make it work.
- Patrick
On Tue, Dec 2, 2014 at 2:40 PM, Ryan Williams
wrote:
> Following on Mark's Maven examples, here is another related
On Tue, Dec 2, 2014 at 2:40 PM, Ryan Williams
wrote:
> Following on Mark's Maven examples, here is another related issue I'm
> having:
>
> I'd like to compile just the `core` module after a `mvn clean`, without
> building an assembly JAR first. Is this possible?
Out of curiosity, may I ask why? W
Following on Mark's Maven examples, here is another related issue I'm
having:
I'd like to compile just the `core` module after a `mvn clean`, without
building an assembly JAR first. Is this possible?
Attempting to do it myself, the steps I performed were:
- `mvn compile -pl core`: fails because
+1 tested on yarn.
Tom
On Friday, November 28, 2014 11:18 PM, Patrick Wendell
wrote:
Please vote on releasing the following candidate as Apache Spark version 1.2.0!
The tag to be voted on is v1.2.0-rc1 (commit 1056e9ec1):
https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;
I am happy to announce the availability of Spark 1.1.1! This is a
maintenance release with many bug fixes, most of which are concentrated in
the core. This list includes various fixes to sort-based shuffle, memory
leak, and spilling issues. Contributions from this release came from 55
developers.
Also a note on this for committers - it's possible to re-word the
title during merging, by just running "git commit -a --amend" before
you push the PR.
- Patrick
On Tue, Dec 2, 2014 at 12:50 PM, Mridul Muralidharan wrote:
> I second that !
> Would also be great if the JIRA was updated accordingl
+1. I also tested on Windows just in case, with jars referring other jars
and python files referring other python files. Path resolution still works.
2014-12-02 10:16 GMT-08:00 Jeremy Freeman :
> +1 (non-binding)
>
> Installed version pre-built for Hadoop on a private HPC
> ran PySpark shell w/ i
I second that !
Would also be great if the JIRA was updated accordingly too.
Regards,
Mridul
On Wed, Dec 3, 2014 at 1:53 AM, Kay Ousterhout wrote:
> Hi all,
>
> I've noticed a bunch of times lately where a pull request changes to be
> pretty different from the original pull request, and the tit
Hi all,
I've noticed a bunch of times lately where a pull request changes to be
pretty different from the original pull request, and the title /
description never get updated. Because the pull request title and
description are used as the commit message, the incorrect description lives
on forever
+1 (non-binding)
Installed version pre-built for Hadoop on a private HPC
ran PySpark shell w/ iPython
loaded data using custom Hadoop input formats
ran MLlib routines in PySpark
ran custom workflows in PySpark
browsed the web UI
Noticeable improvements in stability and performance during large sh
Apologies if people get this more than once -- I sent mail to dev@spark
last night and don't see it in the archives. Trying the incubator list
now...wanted to make sure it doesn't get lost in case it's a bug...
-- Forwarded message --
From: Yana Kadiyska
Date: Mon, Dec 1, 2014 at
Thanks Sean, I followed suit (brew install zinc) and that is working.
2014-12-01 22:39 GMT-08:00 Sean Owen :
> I'm having no problems with the build or zinc on my Mac. I use zinc
> from "brew install zinc".
>
> On Tue, Dec 2, 2014 at 3:02 AM, Stephen Boesch wrote:
> > Mac as well. Just found th
Thanks.
And @Reynold, sorry my bad, Guess I should have used something like
Stackoverflow!
On Tue, Dec 2, 2014 at 12:18 PM, Reynold Xin wrote:
> Oops my previous response wasn't sent properly to the dev list. Here you
> go for archiving.
>
>
> Yes you can. Scala classes are compiled down to cla
+1 (non-binding)
Verified on OSX 10.10.2, built from source,
spark-shell / spark-submit jobs
ran various simple Spark / Scala queries
ran various SparkSQL queries (including HiveContext)
ran ThriftServer service and connected via beeline
ran SparkSVD
On Mon Dec 01 2014 at 11:09:26 PM Patrick Wen
21 matches
Mail list logo
