Hi Aljoscha,
Thanks! I will look into this.
Best,
Aaron Levin
On Fri, Nov 9, 2018 at 5:01 AM, Aljoscha Krettek
wrote:
> Hi,
>
> I think for this case a model that is similar to how the Streaming File
> Source works should be good. You can have a look at
> ContinuousFileMonitoringFunction and
Hi,
I think for this case a model that is similar to how the Streaming File Source
works should be good. You can have a look at ContinuousFileMonitoringFunction
and ContinuousFileReaderOperator. The idea is that the first emits splits that
should be processed and the second is responsible for r
Hey,
Thanks for reaching out! I'd love to take a step back and find a better
solution, so I'll try to be succint in what I'm trying to accomplish:
We're trying to write a SourceFunction which:
* reads some Sequence files from S3 in a particular order (each task gets
files in a specific order).
*
Hi Aaron,
I'l like to take a step back and understand why you're trying to wrap an
InputFormatSourceFunction?
In my opinion, InputFormatSourceFunction should not be used because it has some
shortcomings, the most prominent among them that it does not support
checkpointing, i.e. in case of fail
Hey Friends! Last ping and I'll move this over to a ticket. If anyone can
provide any insight or advice, that would be helpful!
Thanks again.
Best,
Aaron Levin
On Fri, Oct 26, 2018 at 9:55 AM, Aaron Levin wrote:
> Hey,
>
> Not sure how convo threading works on this list, so in case the folks
Hey,
Not sure how convo threading works on this list, so in case the folks CC'd
missed my other response, here's some more info:
First, I appreciate everyone's help! Thank you!
I wrote several wrappers to try and debug this, including one which is an
exact copy of `InputFormatSourceFunction` whi
Hey,
First, I appreciate everyone's help! Thank you!
I wrote several wrappers to try and debug this, including one which is an
exact copy of `InputFormatSourceFunction` which also failed. They all
failed with the same error I detail above. I'll post two of them below.
They all extended `RichParal
Hi,
Since InputFormatSourceFunction is a subclass of
RichParallelSourceFunction, your wrapper should also extend this class.
In addition, remember to overwrite the methods defined in the
AbstractRichFunction interface and
proxy the call to the underlying InputFormatSourceFunction, in order
Hi Aaron,
Could you share the code of you custom function?
I am also adding Aljosha and Kostas to cc, who should be more helpful on
that topic.
Best,
Dawid
On 19/10/2018 20:06, Aaron Levin wrote:
> Hi,
>
> I'm writing a custom `SourceFunction` which wraps an underlying
> `InputFormatSourceFunc
Hi,
I'm writing a custom `SourceFunction` which wraps an underlying
`InputFormatSourceFunction`. When I try to use this `SourceFunction` in a
stream (via `env.addSource` and a subsequent sink) I get errors related to
the `InputSplitAssigner` not being initialized for a particular vertex ID.
Full e
10 matches
Mail list logo
