Hello to my squirrels,
I've been getting a NullPointerException for a DeltaIteration program I'm
trying to implement and I could really use your help :-)
It seems that some of the input Tuples of the Join operator that I'm using
to create the next workset / solution set delta are null.
It also seems that adding ForwardedFields annotations solves the issue.
I managed to reproduce the behavior using the ConnectedComponents example,
by removing the "@ForwardedFieldsFirst("*")" annotation from
the ComponentIdFilter join.
The exception message is the following:
Caused by: java.lang.NullPointerException
at
org.apache.flink.examples.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:186)
at
org.apache.flink.examples.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:1)
at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:198)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:139)
at
org.apache.flink.runtime.iterative.task.IterationIntermediatePactTask.run(IterationIntermediatePactTask.java:92)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217)
at java.lang.Thread.run(Thread.java:745)
I get this error locally with any sufficiently big dataset (~10000 nodes).
When the annotation is in place, it works without problem.
I also generated the optimizer plans for the two cases:
- with annotation (working):
https://gist.github.com/vasia/4f4dc6b0cc6c72b5b64b
- without annotation (failing):
https://gist.github.com/vasia/086faa45b980bf7f4c09
After visualizing the plans, the main difference I see is that in the
working case, the next workset node and the solution set delta nodes are
merged, while in the failing case they are separate.
Shouldn't this work with and without annotation (but be more efficient with
the annotation in place)? Or am I missing something here?
Thanks in advance for any help :))
Cheers,
- Vasia.
