Hello,
I'm using Spark streaming to process kafka message, and wants to use a prop
file as the input and broadcast the properties:
val props = new Properties()
props.load(new FileInputStream(args(0)))
val sc = initSparkContext()
val propsBC = sc.broadcast(props)
println(s"propFileBC 1: " + propsB
Hi,
I have a streaming application which uses a broadcast variable which I
populate from a database.
I would like every once in a while (or even every batch) to update/replace
the broadcast variable with the latest data from the database.
Only way I found online to do this is this "hackish" way (
Hi,
I am taking Broadcast value from file. I want to use it creating Rating
Object (ALS) .
But I am getting null. Here is my code
<https://gist.github.com/yaseminn/d6afd0263f6db6ea4ec5> :
At lines 17 & 18 is ok but 19 returns null so 21 returns me error. Why I
don't understand.
…
>
> private *Properties[] propertiesList*;
>
> private static final long serialVersionUID =
> -8139500301736028670L;
>
> }
>
>
>
> The broadcast value has no issue. But in my original form, if I broadcast
> it as array of my cust
can set this
>> spark.storage.memoryFraction to a very small number or even zero?
>>
>>
>>
>> I am writing a program which consume a lot of memory (broadcast value,
>> runtime, etc). But I have no cached RDD, so should I just turn off this
>> spark.storage.me
has no cached RDD at all (means that I have no
> .cache() or .persist() call on any RDD), then I can set this
> spark.storage.memoryFraction to a very small number or even zero?
>
>
>
> I am writing a program which consume a lot of memory (broadcast value,
> runtime, etc). But
spark.storage.memoryFraction to a very small number or even zero?
I am writing a program which consume a lot of memory (broadcast value,
runtime, etc). But I have no cached RDD, so should I just turn off this
spark.storage.memoryFraction to 0 (which will help me to improve the
performance)?
And I
static final long serialVersionUID =
-8139500301736028670L;
}
The broadcast value has no issue. But in my original form, if I broadcast it
as array of my custom subclass of Properties, after broadcast, the
propertiesList array will be an array of empty PropertiesUtils objects
there (empty
Hi All,
I have tested my code without problem on EMR yarn (spark 1.3.0) with default
serializer (java).
But when I switch to org.apache.spark.serializer.KryoSerializer, the
broadcast value doesn't give me right result (actually return me empty
custom class on inner object).
Basica
