The key type returned by both KeyValueMappers (in the current trunk
version, that type is named `R`) would need to be the same for this to work.
On Wed, Dec 7, 2016 at 4:46 PM, Damian Guy <damian....@gmail.com> wrote:
> Michael,
>
> We can only support outerJoin if both tables are keyed the same way. Lets
> say for example you can map both ways, however, the key for each table is
> of a different type. So t1 is long and t2 is string - what is the key type
> of the resulting GlobalKTable? So when you subsequently join to this table,
> and do a lookup on it, which key are you using?
>
> Thanks,
> Damian
>
> On Wed, 7 Dec 2016 at 14:31 Michael Noll <mich...@confluent.io> wrote:
>
> > Damian,
> >
> > yes, that makes sense.
> >
> > But I am still wondering: In your example, there's no prior knowledge
> "can
> > I map from t1->t2" that Streams can leverage for joining t1 and t2 other
> > than blindly relying on the user to provide an appropriate KeyValueMapper
> > for K1/V1 of t1 -> K2/V2 of t2. In other words, if we allow the user to
> > provide a KeyValueMapper from t1->t2 (Streams does not know at compile
> time
> > whether this mapping will actually work), then we can also allow the user
> > to provide a corresponding "reverse" mapper from t2->t1. That is, we
> could
> > say that an outer join between two global tables IS supported, but if and
> > only if the user provides two KeyValueMappers, one for t1->t2 and one for
> > t2->t1.
> >
> > The left join t1->t2 (which is supported in the KIP), in general, works
> > only because of the existence of the user-provided KeyValueMapper from
> > t1->t2. The outer join, as you point out, cannot satisfied as easily
> > because Streams must know two mappers, t1->t2 plus t2->t1 -- otherwise
> the
> > outer join won't work.
> >
> >
> >
> >
> >
> > On Wed, Dec 7, 2016 at 3:04 PM, Damian Guy <damian....@gmail.com> wrote:
> >
> > > Hi Michael,
> > >
> > > Sure. Say we have 2 input topics t1 & t2 below:
> > > t1{
> > > int key;
> > > string t2_id;
> > > ...
> > > }
> > >
> > > t2 {
> > > string key;
> > > ..
> > > }
> > > If we create global tables out of these we'd get:
> > > GlobalKTable<Integer, ?> t1;
> > > GlobalKTable<String, ?> t2;
> > >
> > > So the join can only go in 1 direction, i.e, from t1 -> t2 as in order
> to
> > > perform the join we need to use a KeyValueMapper to extract the t2 key
> > from
> > > the t1 value.
> > >
> > > Does that make sense?
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Wed, 7 Dec 2016 at 10:44 Michael Noll <mich...@confluent.io> wrote:
> > >
> > > > > There is no outer-join for GlobalKTables as the tables may be keyed
> > > > > differently. So you need to use the key from the left side of the
> > join
> > > > > along with the KeyValueMapper to resolve the right side of the
> join.
> > > This
> > > > > wont work the other way around.
> > > >
> > > > Care to elaborate why it won't work the other way around? If, for
> > > example,
> > > > we swapped the call from leftTable.join(rightTable) to
> > > > rightTable.join(leftTable), that join would work, too. Perhaps I am
> > > > missing something though. :-)
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Dec 7, 2016 at 10:39 AM, Damian Guy <damian....@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Matthias,
> > > > >
> > > > > Thanks for the feedback.
> > > > >
> > > > > There is no outer-join for GlobalKTables as the tables may be keyed
> > > > > differently. So you need to use the key from the left side of the
> > join
> > > > > along with the KeyValueMapper to resolve the right side of the
> join.
> > > This
> > > > > wont work the other way around.
> > > > >
> > > > > On the bootstrapping concern. If the application is failing before
> > > > > bootstrapping finishes, the problem is likely to be related to a
> > > terminal
> > > > > exception, i.e., running out of disk space, corrupt state stores
> etc.
> > > In
> > > > > these cases, we wouldn't want the application to continue. So i
> think
> > > > this
> > > > > is ok.
> > > > >
> > > > > Thanks,
> > > > > Damian
> > > > >
> > > > > On Tue, 6 Dec 2016 at 21:56 Matthias J. Sax <matth...@confluent.io
> >
> > > > wrote:
> > > > >
> > > > > > Thanks for the KIP Damian. Very nice motivating example!
> > > > > >
> > > > > > A few comments:
> > > > > >
> > > > > > - why is there no outer-join for GlobalKTables
> > > > > > - on bootstrapping GlobalKTable, could it happen that this never
> > > > > > finishes if the application fails before bootstrapping finishes
> and
> > > new
> > > > > > data gets written at the same time? Do we need to guard against
> > this
> > > > > > (seems to be a very rare corner case, so maybe not required)?
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > >
> > > > > > On 12/6/16 2:09 AM, Damian Guy wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to start the discussion on KIP-99:
> > > > > > >
> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=67633649
> > > > > > >
> > > > > > > Looking forward to your feedback.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Damian
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Michael G. Noll*
> > Product Manager | Confluent
> > +1 650 453 5860 <(650)%20453-5860> | @miguno <https://twitter.com/miguno
> >
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | Blog
> > <http://www.confluent.io/blog>
> >
>
