Damian,
yes, that makes sense.
But I am still wondering: In your example, there's no prior knowledge "can
I map from t1->t2" that Streams can leverage for joining t1 and t2 other
than blindly relying on the user to provide an appropriate KeyValueMapper
for K1/V1 of t1 -> K2/V2 of t2. In other words, if we allow the user to
provide a KeyValueMapper from t1->t2 (Streams does not know at compile time
whether this mapping will actually work), then we can also allow the user
to provide a corresponding "reverse" mapper from t2->t1. That is, we could
say that an outer join between two global tables IS supported, but if and
only if the user provides two KeyValueMappers, one for t1->t2 and one for
t2->t1.
The left join t1->t2 (which is supported in the KIP), in general, works
only because of the existence of the user-provided KeyValueMapper from
t1->t2. The outer join, as you point out, cannot satisfied as easily
because Streams must know two mappers, t1->t2 plus t2->t1 -- otherwise the
outer join won't work.
On Wed, Dec 7, 2016 at 3:04 PM, Damian Guy <damian....@gmail.com> wrote:
> Hi Michael,
>
> Sure. Say we have 2 input topics t1 & t2 below:
> t1{
> int key;
> string t2_id;
> ...
> }
>
> t2 {
> string key;
> ..
> }
> If we create global tables out of these we'd get:
> GlobalKTable<Integer, ?> t1;
> GlobalKTable<String, ?> t2;
>
> So the join can only go in 1 direction, i.e, from t1 -> t2 as in order to
> perform the join we need to use a KeyValueMapper to extract the t2 key from
> the t1 value.
>
> Does that make sense?
>
> Thanks,
> Damian
>
> On Wed, 7 Dec 2016 at 10:44 Michael Noll <mich...@confluent.io> wrote:
>
> > > There is no outer-join for GlobalKTables as the tables may be keyed
> > > differently. So you need to use the key from the left side of the join
> > > along with the KeyValueMapper to resolve the right side of the join.
> This
> > > wont work the other way around.
> >
> > Care to elaborate why it won't work the other way around? If, for
> example,
> > we swapped the call from leftTable.join(rightTable) to
> > rightTable.join(leftTable), that join would work, too. Perhaps I am
> > missing something though. :-)
> >
> >
> >
> >
> > On Wed, Dec 7, 2016 at 10:39 AM, Damian Guy <damian....@gmail.com>
> wrote:
> >
> > > Hi Matthias,
> > >
> > > Thanks for the feedback.
> > >
> > > There is no outer-join for GlobalKTables as the tables may be keyed
> > > differently. So you need to use the key from the left side of the join
> > > along with the KeyValueMapper to resolve the right side of the join.
> This
> > > wont work the other way around.
> > >
> > > On the bootstrapping concern. If the application is failing before
> > > bootstrapping finishes, the problem is likely to be related to a
> terminal
> > > exception, i.e., running out of disk space, corrupt state stores etc.
> In
> > > these cases, we wouldn't want the application to continue. So i think
> > this
> > > is ok.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Tue, 6 Dec 2016 at 21:56 Matthias J. Sax <matth...@confluent.io>
> > wrote:
> > >
> > > > Thanks for the KIP Damian. Very nice motivating example!
> > > >
> > > > A few comments:
> > > >
> > > > - why is there no outer-join for GlobalKTables
> > > > - on bootstrapping GlobalKTable, could it happen that this never
> > > > finishes if the application fails before bootstrapping finishes and
> new
> > > > data gets written at the same time? Do we need to guard against this
> > > > (seems to be a very rare corner case, so maybe not required)?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > > > On 12/6/16 2:09 AM, Damian Guy wrote:
> > > > > Hi all,
> > > > >
> > > > > I would like to start the discussion on KIP-99:
> > > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=67633649
> > > > >
> > > > > Looking forward to your feedback.
> > > > >
> > > > > Thanks,
> > > > > Damian
> > > > >
> > > >
> > > >
> > >
> >
>
--
*Michael G. Noll*
Product Manager | Confluent
+1 650 453 5860 | @miguno <https://twitter.com/miguno>
Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
<http://www.confluent.io/blog>
