;>
>>>>> I agree with Ankit that this problem could actually be solved quite
>>>>> elegantly with Flink's state. If you can ingest the product/account
>>>>> information changes as a stream, you can keep the latest version of it in
>>>>>
tor's state and join it with the
>>>> incoming event.
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/
>>>> [2]
>>>> https://ci.apache.org/projects/fl
>>> Cheers,
>>> Till
>>>
>>> On Tue, Jul 24, 2018 at 2:15 PM Harshvardhan Agrawal <
>>> harshvardhan.ag...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Thanks for your responses.
>>>
>>> There is no fixed interv
ew product or there are any mandates that change
>> will trigger the reference data to change.
>>
>> It’s not just the enrichment we are doing here. Once we have enriched the
>> data we will be performing a bunch of aggregations using the enriched data.
>>
>> Which a
;
>
>
> Also, just based on this feature, flink doesn’t seem to add a lot of value
> on top of Kafka. As Jorn said below, you can very well store all the events
> in an external store and then periodically run a cron to enrich later since
> your processing doesn’t seem to require
eal time.
Thanks
Ankit
From: Jörn Franke
Date: Monday, July 23, 2018 at 10:10 PM
To: Harshvardhan Agrawal
Cc:
Subject: Re: Implement Joins with Lookup Data
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not sca
elow, you can very well store all the events
> in an external store and then periodically run a cron to enrich later since
> your processing doesn’t seem to require absolute real time.
>
>
>
> Thanks
>
> Ankit
>
>
>
> *From: *Jörn Franke
> *Date: *Monday, July 2
hvardhan Agrawal
Cc:
Subject: Re: Implement Joins with Lookup Data
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not scale.
Depending on when you need to do the enrichment you could also first store the
data and enrich i
f value
> on top of Kafka. As Jorn said below, you can very well store all the events
> in an external store and then periodically run a cron to enrich later since
> your processing doesn’t seem to require absolute real time.
>
>
>
> Thanks
>
> Ankit
>
>
>
> *From: *J
e absolute real time.
Thanks
Ankit
From: Jörn Franke
Date: Monday, July 23, 2018 at 10:10 PM
To: Harshvardhan Agrawal
Cc:
Subject: Re: Implement Joins with Lookup Data
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational dat
gt;> How often is the product db updated? Based on that you can store
>>>>> product metadata as state in Flink, maybe setup the state on cluster
>>>>> startup and then update daily etc.
>>>>>
>>>>>
>>>>>
>>>>> Also, just based
gt;>> Also, just based on this feature, flink doesn’t seem to add a lot of
>>>> value on top of Kafka. As Jorn said below, you can very well store all the
>>>> events in an external store and then periodically run a cron to enrich
>>>> later since your process
nal store and then periodically run a cron to enrich
>>> later since your processing doesn’t seem to require absolute real time.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Ankit
>>>
>>>
>>>
>>> *From: *Jörn Franke
> Thanks
>>
>> Ankit
>>
>>
>>
>> *From: *Jörn Franke
>> *Date: *Monday, July 23, 2018 at 10:10 PM
>> *To: *Harshvardhan Agrawal
>> *Cc: *
>> *Subject: *Re: Implement Joins with Lookup Data
>>
>>
>>
>> For the firs
cron to enrich later since
> your processing doesn’t seem to require absolute real time.
>
>
>
> Thanks
>
> Ankit
>
>
>
> *From: *Jörn Franke
> *Date: *Monday, July 23, 2018 at 10:10 PM
> *To: *Harshvardhan Agrawal
> *Cc: *
> *Subject: *Re: Implement Jo
well store all the events in an
external store and then periodically run a cron to enrich later since your
processing doesn’t seem to require absolute real time.
Thanks
Ankit
From: Jörn Franke
Date: Monday, July 23, 2018 at 10:10 PM
To: Harshvardhan Agrawal
Cc:
Subject: Re: Implement Joins
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not scale.
Depending on when you need to do the enrichment you could also first store the
data and enrich it later as part of a batch process.
> On 24. Jul 2018, at 05:25, Ha
17 matches
Mail list logo
