I am able to maintain a list state in process function and aggregate the
values, how do i get a notification/event to remove the value from the
stored list state.
On Thu, May 6, 2021 at 8:47 PM Swagat Mishra wrote:
> I meant "Do you recommend the state to be maintained in* Value** S
I meant "Do you recommend the state to be maintained in* Value** State *or
external store like elastic?"
On Thu, May 6, 2021 at 8:46 PM Swagat Mishra wrote:
> I want to aggregate the user activity e.g number of products the user has
> purchased in the last 1 hour.
>
> so
> the same effect). Then you can use a process function to maintain the
> state. Just make sure that your data volume is low enough as this part is
> not parallelizable by definition.
>
> On Thu, May 6, 2021 at 10:09 AM Swagat Mishra wrote:
>
>> thank you
>>
>> i wi
o perform as many calculations as possible in the
> process function (for example create a customer with buy information
> record) and then have a DataStream#global() reshuffle to collect all
> aggregated information on one node.
>
> On Thu, May 6, 2021 at 9:20 AM Swagat Mishra wrote:
stamp. I'm
> assuming that nothing really moves from the event time's perspective.
>
> On Thu, May 6, 2021 at 8:58 AM Swagat Mishra wrote:
>
>> Yes customer generator is setting the event timestamp correctly like I
>> see below. I debugged and found that the events
y? Are your
> evictors evicting too early?
>
> You can try to add some debug output into the watermark assigner and see
> if it's indeed progressing as expected.
>
> On Thu, May 6, 2021 at 12:48 AM Swagat Mishra wrote:
>
>> This seems to be working fine in processing
ublic void onPeriodicEmit(WatermarkOutput watermarkOutput) {
> watermarkOutput.emitWatermark(new Watermark(currentMaxTimestamp));
>
> }
> .....
>
>
> On Thu, May 6, 2021 at 1:33 AM Swagat Mishra wrote:
>
>> Hi,
>>
>> Bit of background, I have a stream of
Mishra wrote:
> Hi Arvid,
>
> I sent a separate mail titled - Session Windows - not working as expected
>
> closing this thread.
>
> Please have a look when you have a few minutes, much appreciated.
>
> Regards,
> Swagat
>
>
> On Wed, May 5, 2021 at 7:24
Hi Arvid,
I sent a separate mail titled - Session Windows - not working as expected
closing this thread.
Please have a look when you have a few minutes, much appreciated.
Regards,
Swagat
On Wed, May 5, 2021 at 7:24 PM Swagat Mishra wrote:
> Hi Arvid,
>
> Tried a small POC to repr
Hi,
Bit of background, I have a stream of customers who have purchased some
product, reading these transactions on a KAFKA topic. I want to aggregate
the number of products the customer has purchased in a particular duration
( say 10 seconds ) and write to a sink.
I am using session windows to ac
dev/stream/state/queryable_state.html
> [3]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/stateful-stream-processing.html#state-persistence
> [4]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> On Mon, Ap
API is generating.
>
> [1] https://flink.apache.org/2021/01/18/rocksdb.html
> [2] https://flink.apache.org/news/2020/07/30/demo-fraud-detection-3.html
> [3]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/tableApi.html#session-session-windows
>
> On Su
1. What if there are a very high number of users, like a million customers
won't the service crash? Is it advisable to hold the data in memory.
2. What if state-functions are used to calculate the value ? How will this
approach differ from the one proposed below.
Regards,
Swagat
On Wed, Apr 21,
Why not use upserts? Wouldn't that solve the issue of duplicates and there
won't be a need to query database too?
On Sat, Apr 24, 2021, 8:12 PM David Anderson wrote:
> What are the other techniques for bootstrapping rocksdb state?
>
>
> Bootstrapping state involves somehow creating a snapshot (t
t; (also, adding back the user list)
>>
>> [1]:
>> https://prestodb.io/docs/0.187/connector/hive.html#amazon-s3-configuration
>> [2]:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/filesystems/s3.html#hadooppresto-s3-file-systems-plugins
>
m-roles-for-service-accounts-minimum-sdk.html
> [3]: https://github.com/aws/aws-sdk-java/releases/tag/1.11.704
> [4]: https://github.com/prestodb/presto/blob/master/pom.xml#L52
>
> On Sun, Apr 4, 2021 at 3:32 AM Swagat Mishra wrote:
>
>> Austin -
>>
>> In my case
12/deployment/config.html#kubernetes-service-account
>
> On Sat, Apr 3, 2021 at 10:18 PM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
>> Can you describe your setup a little bit more? And perhaps how you use
>> this setup to grant access to othe
ube2iam
>
> On Sat, Apr 3, 2021 at 10:40 AM Swagat Mishra wrote:
>
>> No we are running on aws. The mechanisms supported by flink to connect to
>> resources like S3, need us to make changes that will impact all services,
>> something that we don't want to do. So provi
n Azure Kubernetes Service.
>
> You should be able to do it because the identity can be mapped to the
> labels of the pods not necessary Flink.
>
> On Sat, Apr 3, 2021 at 6:31 AM Swagat Mishra wrote:
>
>> Hi,
>>
>> I think flink doesn't support pod ide
Hi,
I think flink doesn't support pod identity, any plans tk achieve it in any
subsequent release.
Regards,
Swagat
21 matches
Mail list logo
