Hi Christian,
I assume you're using the rocksdb state backend. You can enable some
metrics from Rocksdb, please refer to the doc[1].
Please note that the `State#clear` only removes the key/value for
specified key, rather than removing all keys. And the deletion will be
reflected in the checkpoint
Hi Banu,
Rocksdb is intelligently built to clear any un-useful state from its
purview. So you should be good and any required cleanup will be
automatically done by RocksDb itself.
>From the current documentation, it looks quite hard to relate Flink
Internal DS activity to RocksDB DS activity. In m
Dear Community,
Gentle reminder about my below email.
Thanks
Banu
On Sat, 20 Jul, 2024, 4:37 pm banu priya, wrote:
> Hi All,
>
> I have a flink job with RMQ Source, filters, tumbling window(uses
> processing time fires every 2s), aggregator, RMQ Sink.
>
> I am trying to understand about states
Thanks Martijn. Now it's more clear to me of the issue.
I'll try to see if the state processor API could help to convert the 1.9
savepoints to 1.13 compatible.
BR,
Le jeu. 14 avr. 2022 à 11:47, Martijn Visser a
écrit :
> Hi Qinghui,
>
> If you're using SQL, please be aware that there are unfor
Hi Qinghui,
If you're using SQL, please be aware that there are unfortunately no
application state compatibility guarantees. You can read more about this in
the documentation [1]. This is why the community has been working on
FLIP-190 to support this in future versions [2]
Best regards,
Martijn
Hello Yu'an,
Thanks for the reply.
I'm using the SQL api, not using the `DataStream` API in the job. So
there's no `keyby` call directly in our code, but we do have some `group
by` and joins in the SQL. (We are using deprecated table planners both
before and after migration)
Do you know what could
Hi Qinghui,
Did you used a difference keyby() for your KeyedCoProcesserOperator? For
example, did you use a fied name (keyBy(“id”)) in 1.9 and while use a lambda
(keyBy(e->e.getId()) in 1.13. This will make the key serializer incompatible.
You may reference this link for how to use Apache Flink
Hi Sandeep,
Could you provide the error stack trace and Flink version you are using?
Regards,
Roman
On Fri, Jul 2, 2021 at 6:42 PM Sandeep khanzode wrote:
>
> Hi Guowei,
>
> I followed the document, but somehow, I am unable to get a working Java
> example for Avro state.
>
> So, I tried to sim
Hi Guowei,
I followed the document, but somehow, I am unable to get a working Java example
for Avro state.
So, I tried to simply use the Java SpecificRecords created by Avro Maven Plugin
and inject. Now, that works correctly, but I use Avro 1.7.7 since it is the
last version that I saw which d
Hi Sandeep
What I understand is that you want to manipulate the state. So I think you
could use the old schema to read the state first, and then write it to a
new schema, instead of using a new schema to read an old schema format data.
In addition, I would like to ask, if you want to do "State Sch
Hi Aashutosh,
The queryable state feature is no longer actively maintained by the
community. What I would recommend is to output the aggregate counts via a
sink to some key value store which you query to obtain the results.
Looking at the implementation of CopyOnWriteStateMap, it does not look li
Hi,
Using the State Processor API, modifying the state in an existing savepoint
results in a new savepoint (new directory) with the new modified state.
The original savepoint remains intact.
The API allows you to only touch certain operators, without having to touch
any other state and have them r
That is doable via the state processor API, though Arvid's idea does sound
simpler :)
You could read the operator with the rules, change the data as necessary
and then rewrite it out as a new savepoint to start the job.
On Thu, Jul 30, 2020 at 5:24 AM Arvid Heise wrote:
> Another idea: since y
Another idea: since your handling on Flink is idempotent, would it make
sense to also periodically send the whole rule set anew?
Going further, depending on the number of rules, their size, and the update
frequency. Would it be possible to always transfer the complete rule set
and just discard the
Hi Kostas
Thanks for a possible help!
пт, 24 июл. 2020 г., 19:08 Kostas Kloudas :
> Hi Alex,
>
> Maybe Seth (cc'ed) may have an opinion on this.
>
> Cheers,
> Kostas
>
> On Thu, Jul 23, 2020 at 12:08 PM Александр Сергеенко
> wrote:
> >
> > Hi,
> >
> > We use so-called "control stream" pattern t
Hi Alex,
Maybe Seth (cc'ed) may have an opinion on this.
Cheers,
Kostas
On Thu, Jul 23, 2020 at 12:08 PM Александр Сергеенко
wrote:
>
> Hi,
>
> We use so-called "control stream" pattern to deliver settings to the Flink
> job using Apache Kafka topics. The settings are in fact an unlimited stre
I am not using any custom serialisation, but pojo is composite type, the pojo
I am trying to modify has variables which are other pojo defined by me, is
ther any example for TypeSerialization for this kind please share
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabbl
Hi,
Could you confirm that you’re using POJOSerializer before and after migration?
Best,
Paul Lam
> 在 2019年10月17日，21:34，ApoorvK 写道：
>
> It is throwing below error ,
> the class I am adding variables have other variable as an object of class
> which are also in state.
>
> Caused by: org.apach
It is throwing below error ,
the class I am adding variables have other variable as an object of class
which are also in state.
Caused by: org.apache.flink.util.StateMigrationException: The new state
typeSerializer for operator state must not be incompatible.
at
org.apache.flink.runtime.st
Yes, I have tried giving it as option, also the case class has default
constructor (this) still unable to migrate
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Can you try to add the new variables as option ?
On Wed, Oct 16, 2019, 17:17 ApoorvK wrote:
> I have been trying to alter the current state case class (scala) which has
> 250 variables, now when I add 10 more variables to the class, and when I
> run
> my flink application from the save point ta
Hi,
If you use RocksDBStateBackend, one member one state will get better
performance. Because RocksDBStateBackend needs to de/serialize the
key/value when put/get, with one POJO value, you need to de/serializer the
whole POJO value when put/get.
Best,
Congxian
Timothy Victor 于2019年6月17日周一 下午8:0
I would choose encapsulation if it the fields are indeed related and makes
sense for your model. In general, I feel it is not a good thing to let
Flink (or any other frameworks) internal mechanics dictate your data model.
Tim
On Mon, Jun 17, 2019, 4:59 AM Frank Wilson wrote:
> Hi,
>
> Is it be
stuff that we should have
>>>> at some point.
>>>>
>>>> One thing that comes to mind is checking the size of checkpoints, which
>>>> gives you an indirect way of figuring out how big state is but that's not
>>>> very exact, i.e. doesn't
;> Aljoscha
>>>
>>> > On 20. Dec 2017, at 08:09, Netzer, Liron
>>> wrote:
>>> >
>>> > Ufuk, Thanks for replying !
>>> >
>>> > Aljoscha, can you please assist with the questions below?
>>> >
>>> > Thanks,
;> > Ufuk, Thanks for replying !
>> >
>> > Aljoscha, can you please assist with the questions below?
>> >
>> > Thanks,
>> > Liron
>> >
>> > -Original Message-
>> > From: Ufuk Celebi [mailto:u...@apache.org <mailto:u..
>> Best,
>> Aljoscha
>>
>> > On 20. Dec 2017, at 08:09, Netzer, Liron wrote:
>> >
>> > Ufuk, Thanks for replying !
>> >
>> > Aljoscha, can you please assist with the questions below?
>> >
>> > Thanks,
>> > Liron
>> >
>
below?
> >
> > Thanks,
> > Liron
> >
> > -Original Message-
> > From: Ufuk Celebi [mailto:u...@apache.org <mailto:u...@apache.org>]
> > Sent: Friday, December 15, 2017 3:06 PM
> > To: Netzer, Liron [ICG-IT]
> > Cc: user@flink.apach
replying !
> >
> > Aljoscha, can you please assist with the questions below?
> >
> > Thanks,
> > Liron
> >
> > -Original Message-
> > From: Ufuk Celebi [mailto:u...@apache.org <mailto:u...@apache.org>]
> > Sent: Friday, December 1
gt;> Aljoscha
>>
>> > On 20. Dec 2017, at 08:09, Netzer, Liron wrote:
>> >
>> > Ufuk, Thanks for replying !
>> >
>> > Aljoscha, can you please assist with the questions below?
>> >
>> > Thanks,
>> > Liron
>
the questions below?
> >
> > Thanks,
> > Liron
> >
> > -Original Message-
> > From: Ufuk Celebi [mailto:u...@apache.org]
> > Sent: Friday, December 15, 2017 3:06 PM
> > To: Netzer, Liron [ICG-IT]
> > Cc: user@flink.apache.org
>
w?
>
> Thanks,
> Liron
>
> -Original Message-
> From: Ufuk Celebi [mailto:u...@apache.org]
> Sent: Friday, December 15, 2017 3:06 PM
> To: Netzer, Liron [ICG-IT]
> Cc: user@flink.apache.org
> Subject: Re: Flink State monitoring
>
> Hey Liron,
>
>
Ufuk, Thanks for replying !
Aljoscha, can you please assist with the questions below?
Thanks,
Liron
-Original Message-
From: Ufuk Celebi [mailto:u...@apache.org]
Sent: Friday, December 15, 2017 3:06 PM
To: Netzer, Liron [ICG-IT]
Cc: user@flink.apache.org
Subject: Re: Flink State
Hey Liron,
unfortunately, there are no built-in metrics related to state. In
general, exposing the actual values as metrics is problematic, but
exposing summary statistics would be a good idea. I'm not aware of a
good work around at the moment that would
work in the general case (taking into accou
34 matches
Mail list logo
