Are you using Spark 2.3 or above?
See the documentation:
https://spark.apache.org/docs/latest/running-on-kubernetes.html
I looks like you do not need:
--conf spark.kubernetes.driver.podTemplateFile='/spark-pod-template.yaml' \
--conf spark.kubernetes.executor.podTemplateFile='/spark-pod-template.
To get a node local read from Spark to Cassandra, one has to use a read
consistency level of LOCAL_ONE. For some use cases, this is not an
option. For example, if you need to use a read consistency level
of LOCAL_QUORUM, as many use cases demand, then one is not going to get a
node local read.
A
Hi unk1102,
Try adding more memory to your nodes. Are you running Spark in the cloud?
If so, increase the memory on your servers.
Do you have default parallelism set (spark.default.parallelism)? If so,
unset it, and let Spark decided how many partitions to allocate.
You can also try refactoring
Hi Ajay,
Are you trying to save to your local file system or to HDFS?
// This would save to HDFS under "/user/hadoop/counter"
counter.saveAsTextFile("/user/hadoop/counter");
David
On Sun, Aug 30, 2015 at 11:21 AM, Ajay Chander wrote:
> Hi Everyone,
>
> Recently we have installed spark on yar
This is likely due to data skew. If you are using key-value pairs, one key
has a lot more records, than the other keys. Do you have any groupBy
operations?
David
On Tue, Jul 14, 2015 at 9:43 AM, shahid wrote:
> hi
>
> I have a 10 node cluster i loaded the data onto hdfs, so the no. of
> par
You can certainly query over 4 TB of data with Spark. However, you will
get an answer in minutes or hours, not in milliseconds or seconds. OLTP
databases are used for web applications, and typically return responses in
milliseconds. Analytic databases tend to operate on large data sets, and
retu
I am having the same problem reading JSON. There does not seem to be a way
of selecting a field that has a space, "Executor Info" from the Spark logs.
I suggest that we open a JIRA ticket to address this issue.
On Jun 2, 2015 10:08 AM, "ayan guha" wrote:
> I would think the easiest way would b
Does anyone know in which version of Spark will there be support for
ORCFiles via spark.sql.hive? Will it be in 1.4?
David
rame.
>> >
>> > So people.toDF.registerTempTable("people") should work
>> >
>> >
>> >
>> > —
>> > Sent from Mailbox
>> >
>> >
>> > On Sat, Mar 14, 2015 at 5:33 PM, David Mitchell <
>> jdavi
I am pleased with the release of the DataFrame API. However, I started
playing with it, and neither of the two main examples in the documentation
work: http://spark.apache.org/docs/1.3.0/sql-programming-guide.html
Specfically:
- Inferring the Schema Using Reflection
- Programmatically Spec
10 matches
Mail list logo
