[
https://issues.apache.org/jira/browse/SPARK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812025#comment-15812025
]
Paulo Cândido commented on SPARK-19125:
---------------------------------------
I understand when you say that it isn't a streaming problem 'cause I'm waiting
for a different trigger.
We're creating algorithms that works very well using wall time trigger.
However, in scientific environment, these same algorithms must be reproducible.
It means that other scientists must be able to obtain the same results given
the same parameters, regardles the infrastructure or environment. Today, it's
not possible, 'cause the throughput is variable. In real situation, it's ok,
but for scientific public, it doesn't work.
Maybe the 'Duration' class is being used in other places/situations that
doesn't make sense the element count, but maybe, create a constructor to
Context allowing element count as param is enough. I know it isn't easy as I
say, but it's very important for scientific comunity.
Regards.
> Streaming Duration by Count
> ---------------------------
>
> Key: SPARK-19125
> URL: https://issues.apache.org/jira/browse/SPARK-19125
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Environment: Java
> Reporter: Paulo Cândido
>
> I use the Spark Streaming in scientific way. In this cases, we have to run
> the same experiment many times using the same seed to obtain the same result.
> All randomic components have the seed as input, so I can controll it.
> However, there is a unique component that doesn't depend of seeds and we
> can't controll, it's the bach size. Regardless of the input way of stream,
> the metric to break the microbaches is wall time. It's a problem in
> scientific environment because if we run the same experiments with same param
> many times, each time we can get a diferent result, depending the quantity of
> elements read in each bach. The same stream source may generate diferent bach
> sizes on multiple executions because of wall time.
> My sugestion is provide a new Duration metric: Count of Elements.
> Regardless of time spent to fill a microbatch, they will be always the same
> size, and when the source has a seed to generate de same values, independent
> of throughput, we will can replicate the experiments with same result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
