Hi,
I am trying to submit a custom metrics.properties file to enable the
collection of spark metrics, but I am having a hard time even starting it
in local mode.
spark-submit \
...
--files "./metrics.properties"
--conf "spark.metrics.conf=metrics.properties"
...
However I am getti
andle this situation too ?
On Fri, Sep 25, 2015 at 12:20 PM, Cody Koeninger wrote:
> Storing passbacks transactionally with results in your own data store,
> with a schema that makes sense for you, is the optimal solution.
>
> On Fri, Sep 25, 2015 at 11:05 AM, Radu Brumariu wrote:
>
&
. Manual state management is also
> supported by the framework but it’s harder to control because:
>
>- you’re not guaranteed to shut down gracefully
>- You may have a bug that prevents the state to be saved and you can’t
>restart the app w/o upgrade
>
> Less
numerous times, TD's response has consistently
> been that it's unlikely to be possible
>
> On Thu, Sep 24, 2015 at 12:26 PM, Radu Brumariu > wrote:
>
>> It seems to me that this scenario that I'm facing, is quite common for
>> spark jobs using Kafka.
>
cant use checkpointing across code changes. Either store offsets
> yourself, or start up your new app code and let it catch up before killing
> the old one.
>
> On Thu, Sep 24, 2015 at 8:40 AM, Radu Brumariu > wrote:
>
>> Hi,
>> in my application I use Kafka di
Hi,
in my application I use Kafka direct streaming and I have also enabled
checkpointing.
This seems to work fine if the application is restarted. However if I
change the code and resubmit the application, it cannot start because of
the checkpointed data being of different class versions.
Is there
