Re: Queryable state feature in latest version!!

Hi, Puneet. Queryable State has been deprecated in the latest version which will be removed in Flink 2.0. The Interface/Usage is freezed in the 1.x, so you still could reference the documents of previous versions to use it. BTW, Could you also share something about your scenarios using it ? That wi

Re: Queryable state feature in latest version!!

Hi, Puneet Thank you for reaching out. In the latest release of Flink (version 1.18), we have marked Queryable State as @Deprecated and removed the related content from the stable documentation. This means that Queryable State is no longer actively supported or recommended for use. More details c

Re: Queryable State Deprecation

Hi Dawid - I’m pretty keen on keeping it alive. Do we have a sense of what it would take to get it “to a production ready state?” Thanks! Ron > On Feb 4, 2022, at 5:06 AM, Dawid Wysakowicz wrote: > > Hi Karthik, > > The reason we deprecated it is because we lacked committers who could spend

Re: Queryable State Deprecation

*From: *Jatti, Karthik *Date: *Friday, February 11, 2022 at 6:00 PM *To: *Frank Dekervel , user@flink.apache.org , dwysakow...@apache.org *Subject: *Re: Queryable State Deprecation Thank you Frank and Dawid for providing the context here. *From: *Frank Dekervel *Date: *Friday, February

Re: Queryable State Deprecation

anything that would look like an obvious case. Thanks, Karthik From: Jatti, Karthik Date: Friday, February 11, 2022 at 6:00 PM To: Frank Dekervel , user@flink.apache.org , dwysakow...@apache.org Subject: Re: Queryable State Deprecation Thank you Frank and Dawid for providing the context here

Re: Queryable State Deprecation

Thank you Frank and Dawid for providing the context here. From: Frank Dekervel Date: Friday, February 4, 2022 at 9:56 AM To: user@flink.apache.org Subject: Re: Queryable State Deprecation EXTERNAL SENDER Hello, To give an extra datapoint: after a not so

Re: Queryable State Deprecation

Hello, To give an extra datapoint: after a not so successful experiment with faust-streaming we moved our application to flink. Since flinks queryable state was apparently  stagnant, we implemented what was needed to sink the state to an external data store for querying. However, if queryabl

Re: Queryable State Deprecation

Hi Karthik, The reason we deprecated it is because we lacked committers who could spend time on getting the Queryable state to a production ready state. I might be speaking for myself here, but I think the main use case for the queryable state is to have an insight into the current state of th

Re: Queryable State Lookup Failure

Hello, Could you check that TMs didn't fail and therefore unregistered KV states and are still running at the time of the query? Probably after changing the memory settings there is another error that is reported later than the state is unregistered. Regards, Roman On Sat, Jul 24, 2021 at 12:50

Re: Queryable State unavailable after Kubernetes HA State cleanup

Hi Sandeep, I don't fully understand the problematic scenario yet. What exactly is the HA state maintained by Kubernetes in S3? Queryable state works by asking for the current state of an operator. If you use asQueryableState, then you create a reducing state which appends all stream elements. Th

Re: Queryable state on task managers that are not running the job

Hi Martin, What kind of deploy mode you choose? If you use per-job mode [1] to launch jobs, there might exist only idle slots instead of idle taskmanagers. Currently, queryable state is bounded to specific job and if the idle taskmanager is not registered in the target's resource manager, no qu

Re: Queryable state and TTL

Thanks, I'll check it out. On Sun, Jul 7, 2019 at 5:40 AM Eron Wright wrote: > *This Message originated outside your organization.* > -- > Here's a PR for queryable state TLS that I closed because I didn't have > time, and because I get the impression that the queryab

Re: Queryable state and TTL

Here's a PR for queryable state TLS that I closed because I didn't have time, and because I get the impression that the queryable state feature is used very often.Feel free to take it up, if you like. https://github.com/apache/flink/pull/6626 -Eron On Wed, Jul 3, 2019 at 11:21 PM Avi Levi wr

Re: Queryable state and TTL

Hi Yu, Our sink is actually Kafka hence we cannot query it properly, from there we distribute it to different consumers. We keep info in our state such as entry time, some accumulated data etc' , this data is not kept elsewhere hence we need to query our state. Best regards Avi On Thu, Jul 4, 20

Re: Queryable state and TTL

Thanks for the ping Andrey. For me the general answer is yes, but TBH it will probably not be added in the foreseeable future due to lack of committer bandwidth (not only QueryableState with TTL but all about QueryableState module) as per Aljoscha pointed out in another thread [1]. Although we co

Re: Queryable state and TTL

Hi Avi, It is on the road map but I am not aware about plans of any contributor to work on it for the next releases. I think the community will firstly work on the event time support for TTL. I will loop Yu in, maybe he has some plans to work on TTL for the queryable state. Best, Andrey On Wed,

Re: Queryable State race condition or serialization errors?

Hi Burgess, Would you be able to provide a minimal project that can reproduce your error? That would help a lot with figuring out the issue. If you prefer to share that only privately, please feel free to send me a private email with the details. Another thing you can do is set logging level to "D

Re: Queryable State race condition or serialization errors?

Hi Gary. Thanks for the reply. I am using RocksDBStateBackend though. Best, Chen -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Queryable State race condition or serialization errors?

Hi Burgess Chen, If you are using MemoryStateBackend or FsStateBackend, you can observe race conditions on the state objects. However, the RocksDBStateBackend should be safe from these issues [1]. Best, Gary [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/queryab

Re: Queryable state formal release plan

Hi, Currently no committers (or PMC members) are focusing on the queryable state feature. This will probably mean that not much is going to happen there in the near future. However, there is some discussion on the development by the larger community about QS: https://lists.apache.org/thread.ht

Re: Queryable state support in Flink 1.9

Thanks thats it. Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Apr 16, 2019, at 8:31 AM, Guowei Ma wrote: > > AbstractQueryableStateTestBase

Re: Queryable state support in Flink 1.9

Hi, 1. I think Mini cluster supports queryable state. 2. You could set queryable-state.enable to true and try again. You could check AbstractQueryableStateTestBase and there are some tests. :) Best, Guowei Boris Lublinsky 于2019年4月16日周二 下午9:09写道： > Thanks Guowei > The questions that I am askin

Re: Queryable state support in Flink 1.9

Thanks Guowei The questions that I am asking is slightly different: 1. Does Mini cluster support queryable state? 2. If the answer is yes, how to set it up? Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Apr 15, 2019, at 12:07 AM, Guowei Ma wrote: >

Re: Queryable state support in Flink 1.9

Hi, I think you should check TM log first and check if there are some info like: 1430 [main] INFO org.apache.flink.queryablestate.server.KvStateServerImpl - Started Queryable State Server @ /127.0.0.1:9069. 1436 [main] INFO org.apache.flink.queryablestate.client.proxy.KvStateClientProxyImpl - Sta

Re: Queryable state when key is UUID - getting Kyro Exception

er, which is usually >>> used in POJO class, but from the error messages, I think you can add >>> flink-avro as a dependency and try again. >>> >>> >>> Best, >>> >>> Jiayi Liao >>> >>> Original Message >>> *S

Re: Queryable state when key is UUID - getting Kyro Exception

add flink-avro >> as a dependency and try again. >> >> >> Best, >> >> Jiayi Liao >> >> Original Message >> *Sender:* Jayant Ameta >> *Recipient:* bupt_ljy >> *Cc:* trohrmann; Tzu-Li (Gordon) Tai< >> tzuli...@apache.org>; user &

Re: Queryable state when key is UUID - getting Kyro Exception

hink you can add flink-avro > as a dependency and try again. > > > Best, > > Jiayi Liao > > Original Message > *Sender:* Jayant Ameta > *Recipient:* bupt_ljy > *Cc:* trohrmann; Tzu-Li (Gordon) Tai< > tzuli...@apache.org>; user > *Date:* Tuesday, Nov 13

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jayant, I don’t know why flink uses the Avro serializer, which is usually used in POJO class, but from the error messages, I think you can add flink-avro as a dependency and try again. Best, Jiayi Liao Original Message Sender:Jayant ametawittyam...@gmail.com Recipient:bupt_ljybupt_...@163

Re: Queryable state when key is UUID - getting Kyro Exception

pports managed keyed state. > >By the way, I think we should optimize the error messages with which > what Jayant met. > > Best, > Jiayi Liao > > Original Message > *Sender:* Jayant Ameta > *Recipient:* trohrmann > *Cc:* bupt_ljy; Tzu-Li (Gordon) Tai; > user &g

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, Jayant The key you specified in getKvState function should be the key of the keyed stream instead of the key of the map. From what I’ve seen onhttps://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html, this feature only supports managed keyed state. By th

Re: Queryable state when key is UUID - getting Kyro Exception

ll looks okay. >>>> >>>> I’m not very sure about this problem. Maybe you can write a very >>>> simple demo to try if it works. >>>> >>>> >>>> Jiayi Liao, Best >>>> >>>> Original Message >>>> *

Re: Queryable state when key is UUID - getting Kyro Exception

rite a very >>> simple demo to try if it works. >>> >>> >>> Jiayi Liao, Best >>> >>> Original Message >>> *Sender:* Jayant Ameta >>> *Recipient:* bupt_ljy >>> *Cc:* Tzu-Li (Gordon) Tai; user< >>> user@flink.a

Re: Queryable state when key is UUID - getting Kyro Exception

’m not very sure about this problem. Maybe you can write a very >> simple demo to try if it works. >> >> >> Jiayi Liao, Best >> >> Original Message >> *Sender:* Jayant Ameta >> *Recipient:* bupt_ljy >> *Cc:* Tzu-Li (Gordon) Tai; user< >>

Re: Queryable state when key is UUID - getting Kyro Exception

:* Jayant Ameta > *Recipient:* bupt_ljy > *Cc:* Tzu-Li (Gordon) Tai; user > > *Date:* Monday, Oct 29, 2018 11:53 > *Subject:* Re: Queryable state when key is UUID - getting Kyro Exception > > Hi Jiayi, > Any further help on this? > > Jayant Ameta > > > On Fri, O

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, Jayant Your code looks good to me. And I’ve tried the serialize/deserialize of Kryo on UUID class, it all looks okay. I’m not very sure about this problem. Maybe you can write a very simple demo to try if it works. Jiayi Liao, Best Original Message Sender:Jayant ametawittyam...@gmail.

Re: Queryable state when key is UUID - getting Kyro Exception

omString("3b3f17a0-d81a-11e8-bb91-7fd1412de84d"), >>> TypeInformation.of(new TypeHint() {}), descriptor); >>> >>> >> Jiayi Liao, Best >> >> >> Original Message >> *Sender:* Jayant Ameta >> *Recipient:* bupt_ljy >> *Cc:

Re: Queryable state when key is UUID - getting Kyro Exception

Original Message > *Sender:* Jayant Ameta > *Recipient:* bupt_ljy > *Cc:* Tzu-Li (Gordon) Tai; user > > *Date:* Friday, Oct 26, 2018 02:26 > *Subject:* Re: Queryable state when key is UUID - getting Kyro Exception > > Also, I haven't provided any custom serializer in my flin

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, Can you show us the descriptor in the codes below? client.getKvState(JobID.fromHexString("c7b8af14b8afacf4fac16cdd0da7e997"), "rule", UUID.fromString("3b3f17a0-d81a-11e8-bb91-7fd1412de84d"), TypeInformation.of(new TypeHintUUID() {}), descriptor); Jiayi Liao, Best Original Message Sen

Re: Queryable state when key is UUID - getting Kyro Exception

ID]).createSerializer(env.getConfig) >> >> >> By the way, can you show us your kryo exception like what Gordon said? >> >> >> Jiayi Liao, Best >> >> >> >> Original Message >> *Sender:* Tzu-Li (Gordon) Tai >> *Recipient:* Jayant Am

Re: Queryable state when key is UUID - getting Kyro Exception

essage > *Sender:* Tzu-Li (Gordon) Tai > *Recipient:* Jayant Ameta; bupt_ljy > > *Cc:* user > *Date:* Thursday, Oct 25, 2018 17:18 > *Subject:* Re: Queryable state when key is UUID - getting Kyro Exception > > Hi Jayant, > > What is the Kryo exception message that you

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jayant, There should be a Serializer parameter in the constructor of the StateDescriptor, you should create a new serializer like this: new GenericTypeInfo(classOf[UUID]).createSerializer(env.getConfig) By the way, can you show us your kryo exception like what Gordon said? Jiayi Liao, B

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jayant, What is the Kryo exception message that you are getting? Cheers, Gordon On 25 October 2018 at 5:17:13 PM, Jayant Ameta (wittyam...@gmail.com) wrote: Hi, I've not configured any serializer in the descriptor. (Neither in flink job, nor in state query client). Which serializer should

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, I've not configured any serializer in the descriptor. (Neither in flink job, nor in state query client). Which serializer should I use? Jayant Ameta On Thu, Oct 25, 2018 at 2:13 PM bupt_ljy wrote: > Hi, > >It seems that your codes are right. Are you sure that you’re using the > same Se

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, It seems that your codes are right. Are you sure that you’re using the same Serializer as the Flink program do? Could you show the serializer in descriptor? Jiayi Liao, Best Original Message Sender:Jayant ametawittyam...@gmail.com Recipient:useru...@flink.apache.org Date:Thursday, Oct 2

Re: Queryable State

I have used connected streams where one part of the connected stream maintains state and the other part consumes it. However it was not queryable externally. For state that is queryable externally you are right you probably need another operator to store state and support query-ability. Sent

Re: Queryable state and state TTL

Hi, Fabian is right, support of TTL for queryable state needs an extra effort because of some specifics of its interaction with state objects, but there is no fundamental problem. It is on the roadmap for the future realises. Best, Andrey > On 29 Aug 2018, at 09:30, Fabian Hueske wrote: > >

Re: Queryable state and state TTL

Hi, I guess that this is not a fundamental problem but just a limitation in the current implementation. Andrey (in CC) who implemented the TTL support should be able to give more insight on this issue. Best, Fabian Am Mi., 29. Aug. 2018 um 04:06 Uhr schrieb vino yang : > Hi Elias, > > From the

Re: Queryable state and state TTL

Hi Elias, >From the source code, the reason for throwing this exception is because StateTtlConfig is set to StateTtlConfig.DISABLED. Please refer to the usage and description of the official Flink documentation for details.[1] And there is a note you should pay attention : Only TTLs in reference

Re: Queryable State

Thank you for the clarification. On Wed, Mar 21, 2018, 4:28 AM Kostas Kloudas wrote: > Hi Vishal, > > As Fabian said, queryable state is just a feature that exposes the state > kept within Flink, and it is not made to > replace functionality that would otherwise be made by a sink. In the > futur

Re: Queryable State

Hi Vishal, As Fabian said, queryable state is just a feature that exposes the state kept within Flink, and it is not made to replace functionality that would otherwise be made by a sink. In the future the functionality will definitely evolve but for there are no discussions currently, for keepi

Re: Queryable State

Thank you. These do look like show stoppers for us. But again thank you. On Mon, Mar 19, 2018 at 10:31 AM, Fabian Hueske wrote: > AFAIK, there have been discussions to replicate state among TMs to speed > up recovery (and improve availability). > However, I'm not aware of plans to implement tha

Re: Queryable State

AFAIK, there have been discussions to replicate state among TMs to speed up recovery (and improve availability). However, I'm not aware of plans to implement that. I don't think serving state while a job is down has been considered yet. 2018-03-19 15:17 GMT+01:00 Vishal Santoshi : > Are there pl

Re: Queryable State

Are there plans to address all or few of the above apart from the "JM LB not possible" which seems understandable ? On Mon, Mar 19, 2018 at 9:58 AM, Fabian Hueske wrote: > Queryable state is "just" an additional feature on regular keyed state. > i.e., the only difference is that you can read th

Re: Queryable State

Queryable state is "just" an additional feature on regular keyed state. i.e., the only difference is that you can read the state from an outside application. Besides that it behaves exactly like regular application state Queryable state is (at the moment) designed to be accessible if a job runs. I

Re: Queryable State

Those are understandable. I am more interested in a few things ( and may be more that could be added ) * As far as I can understand JM is the SPOF. Does HA become a necessity ? * If there are 2 or more JM could we theoretically have a LB fronting them ? Thus it is a peer to peer access ( Cassandra

Re: Queryable State

Hi Vishal, In general, Queryable State should be ready to use. There are a few things to consider though: - State queries are not synchronized with the application code, i.e., they can happen at the same time. Therefore, the Flink application should not modify objects that have been put into or r

Re: Queryable State

To be more precise, is anything thing similar to https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin . done in Samza, can be replicated with production level guarantees with Flink Queryable state ( as it stands currently version 1.5 ) ? On

Re: queryable state API

Hi Maciek, AFAIK, there is some ongoing work to integrate queryable state with the new FLIP-6 mode. Maybe Kostas (in CC) who has worked on the queryable state API can help to answer your questions. Best, Fabian 2018-02-01 9:19 GMT+01:00 Maciek Próchniak : > Hello, > > Currently (1.4) to be able

Re: Queryable State - Count within Time Window

Thank you, Fabian, for the references. This is helpful. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Queryable State in Flink 1.4

Thanks Boris. I've filed FLINK-8391 [1] to extend the documentation. Best, Fabian [1] https://issues.apache.org/jira/browse/FLINK-8391 2018-01-05 19:52 GMT+01:00 Boris Lublinsky : > Thanks > This was it. > It would help to have this in documentation along with > `flink-queryable-state-client` >

Re: Queryable State - Count within Time Window

Hi, you can implement that with a ProcessFunction [1]. The ProcessFunction would have to compute counts at some granularity (either event-time or processing-time) of records that are processed by the ProcessFunction in processElement(). I would do this with a MapState that has a timestamp as key

Re: Queryable State in Flink 1.4

Thanks This was it. It would help to have this in documentation along with `flink-queryable-state-client` Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Jan 5, 2018, at 11:46 AM, Till Rohrmann wrote: > > Did you add the `flink-queryable-state-runti

Re: Queryable State in Flink 1.4

Did you add the `flink-queryable-state-runtime` jar as a dependency to your project? You can check the log whether a queryable state proxy and server have been started. Cheers, Till On Fri, Jan 5, 2018 at 5:19 PM, Boris Lublinsky < boris.lublin...@lightbend.com> wrote: > I also tried to comment

Re: Queryable State in Flink 1.4

I also tried to comment out //config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 2); Still no luck. Do you guys have a working example for queryable state for 1.4 somewhere? Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > On Jan 5, 2018, at

Re: Queryable State in Flink 1.4

Thanks Till, I am probably slow. I changed the code to the following: // In a non MiniCluster setup queryable state is enabled by default. config.setString(QueryableStateOptions.PROXY_PORT_RANGE, "50100-50101") config.setInteger(QueryableStateOptions.PROXY_NETWORK_THREADS, 2) config.setInteger(Que

Re: Queryable State in Flink 1.4

Hi Boris, if you start 2 TaskManagers on the same host, then you have to define a port range for the KvState server and the proxy. Otherwise the Flink cluster should not be able to start. Cheers, Till On Thu, Jan 4, 2018 at 11:19 PM, Boris Lublinsky < boris.lublin...@lightbend.com> wrote: > It

Re: Queryable State Client - Actor Not found Exception

Thank you Aljoscha. I am able to Query state when I use the hostname of Job Manager instead of its IP Address. But I couldn't understand why it is not working if I give IP address. On Thu, Jan 4, 2018 at 6:42 PM, Aljoscha Krettek wrote: > Hi, > > Is my-machine:52650 the correct address for the

Re: Queryable State Client - Actor Not found Exception

Hi, Is my-machine:52650 the correct address for the JobManager running in YARN? Also, Queryable State in Flink 1.3.2 is not easy to get to work when you use YARN with HA mode. Best, Aljoscha > On 3. Jan 2018, at 16:02, Velu Mitwa wrote: > > Hi, > I am running a Flink Job which uses the Quer

Re: Queryable State Python

Hi Martin, I will try to reply to your questions inline: > On Nov 10, 2017, at 1:59 PM, Martin Eden wrote: > > Hi, > > Our team is looking at replacing Redis with Flink's own queryable state > mechanism. However our clients are using python. > > 1. Is there a python integration with the Flin

Re: Queryable State

No, it doesn't work even if I increase the timeout. The state being fetched is a Map of data and has around 100 entries in it. I have a single job manager and 3 task managers with 16 slots each running on AWS EC2. final TypeSerializer keySerializer = TypeInformation.of(new TypeHint() {}).

Re: Queryable State

Hi Navneeth, If you increase the timeout, everything works ok? I suppose from your config that you are running in standalone mode, right? Any other information about the job (e.g. code and/or size of state being fetched) and the cluster setup that can help us pin down the problem, would be appr

Re: Queryable State

Hi, Any idea on how to solve this issue? Thanks On Wed, Sep 13, 2017 at 10:12 AM, Navneeth Krishnan < reachnavnee...@gmail.com> wrote: > Hi, > > I am sure I have provided the right job manager details because the > connection timeout ip is the task manager where the state is kept. I guess > the

Re: Queryable State

Hi, I am sure I have provided the right job manager details because the connection timeout ip is the task manager where the state is kept. I guess the client is able to reach the job manager and figure out where the state is. Also if I provide a wrong state name, I'm receiving unknown state except

RE: Queryable State

StateDescriptor. What exactly does it mean when the KvStateLocation can't be found? -Original Message- From: Biplob Biswas [mailto:revolutioni...@gmail.com] Sent: Wednesday, September 13, 2017 2:20 PM To: user@flink.apache.org Subject: Re: Queryable State Hi, are you sure your jobmanager is ru

Re: Queryable State

Hi, are you sure your jobmanager is running and is accessible from the supplied hostname and port? If you can start up the FLink UI of the job which creates your queryable state, it should have the details of the job manager and the port to be used in this queryable client job. -- Sent from:

Re: Queryable State

Hi All, Any suggestions would really be helpful. Thanks On Sun, Sep 10, 2017 at 12:04 AM, Navneeth Krishnan < reachnavnee...@gmail.com> wrote: > Hi All, > > I'm running a streaming job on flink 1.3.2 with few queryable states. > There are 3 task managers and a job manager. I'm getting timeout ex

Re: Queryable State max number of clients

You don't have to worry about this as Flink takes care of all of this. I was just trying to explain what the query threads are there for. If we would only use the network threads for the queries, these might be blocked during queries and hence not answer network events. But since we use these two

Re: Queryable State max number of clients

Hi Aljoscha, Ufuk Thank you for the replies. I'm using RocksDB state backend. Could you please explain the blocking I/O calls mentioned? when will it happen? And what will be the effect? A timeout exception? Best Ziyad On Mon, Aug 14, 2017 at 10:17 PM, Ufuk Celebi wrote: > This is as Aljosch

Re: Queryable State max number of clients

This is as Aljoscha describes. Each thread can handle many different clients at the same time. You shouldn't need to change the defaults in most cases. The network threads handle the TCP connections and dispatch query tasks to the query threads which do the actual querying of the state backend. In

Re: Queryable State max number of clients

Hi, I think the number of network treads and number of query threads only roughly correlate with the number of clients that can query in parallel since this is using asynchronous communication via Akka/Netty. Of course, increasing that number means there can be more connections but I think even

Re: queryable state vs. writing result back to Kafka

One advantage is that you don't need an external system for storing results if they are readily available in Flink. There are also downsides, however. For example, queryable state is not available if the Job is suspended or restarting. Generally, I think this depends on the use case. Best, Aljo

Re: Queryable State

Hi Chet, you should not see a org.apache.flink.runtime.query.UnknownKvStateKeyGroupLocation when querying an existing(!) key. However, if you query a key the non-registered TaskManager is responsible for, I suppose this is the exception you will get. Unfortunately, the queryable state API still

Re: Queryable State Client with 1.3.0-rc0

Sorry for yet another update but this is the complete settings for making it work on Flink 1.3: Configuration configuration = new Configuration(); configuration.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 8); configuration.setString(JobManagerOptions.ADDRESS, "localhost"); configurat

Re: Queryable State Client with 1.3.0-rc0

Hi Claudio, Quick question: what exactly was your call for getting the local environment with web UI? Did you also have a custom Configuration where you specified, for example, that the queryable state server should be enabled? I can make an example work where I start a local cluster in one pro

Re: Queryable State Client with 1.3.0-rc0

Hi Claudio, Quick follow up: querying a locally started cluster does not work out-of-box anymore in Flink 1.3. You can manually start a mini cluster that has the required settings, though. You would do something like this: Configuration configuration = new Configuration(); configuration.addAll(

Re: Queryable State Client with 1.3.0-rc0

Hi Claudio, The documentation for this was recently updated: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/queryable_state.html#querying-state . Could you see if

Re: Queryable state in a keyed stream not querying properly

Hi Philip, The JobManager should figure out the correct TaskManager for your query based on the key. You mentioned that you get the result 1/3 of the time, is this 1/3 of the time for queries with exactly the same key or for queries with different keys? Also, could it be that the state you’re

Re: Queryable State

I found the issue. When parallelism = 3, my test data set was skewed such that data was only going to two of the three task managers (kafka partition = 3, number of flink nodes = 3, parallelism = 3). As soon as I created a test data set with enough keys that spread across all three task managers, q

Re: Queryable State

Could you try KvStateRegistry#registerKvState please? In the JM logs you should see something about the number of connected task managers and in the task manager logs that each one connects to a JM. – Ufuk On Tue, May 2, 2017 at 2:53 PM, Chet Masterson wrote: > Can do. Any advice on where the

Re: Queryable State

Can do. Any advice on where the trace prints should go in the task manager source code? BTW - How do I know I have a correctly configured cluster? Is there a set of messages in the job / task manager logs that indicate all required connectivity is present? I know I use the UI to make sure all the t

Re: Queryable State

 Any insight here? I've got a situation where a key value state on a task manager is being registered with the job manager, but when I try to query it, the job manager responds it doesn't know the location of the key value state...  26.04.2017, 12:11, "Chet Masterson" :After setting the logging to

Re: Queryable State

After setting the logging to DEBUG on the job manager, I learned four things: (On the message formatting below, I have the Flink logs formatted into JSON so I can import them into Kibana) 1. The appropriate key value state is registered in both parallelism = 1 and parallelism = 3 environments. In p

Re: Queryable State

Ok...more information. 1. Built a fresh cluster from the ground up. Started testing queryable state at each step.2. When running under any configuration of task managers and job managers were parallelism = 1, the queries execute as expected.3. As soon as I cross over to parallelism = 3 with 3 task

Re: Queryable State

Ufuk - thank you for your help. My flink-conf.yaml is now configured cluster-wide (with restart) as: # my flink-conf.yaml, on all flink nodes:jobmanager.rpc.address: x.x.x.xjobmanager.rpc.port: 6123query.server.port: 6123query.server.enable: true When I try to issue my query now with the above sett

Re: Queryable State

You should be able to use queryable state w/o any changes to the default config. The `query.server.port` option defines the port of the queryable state server that serves the state on the task managers and it is enabled by default. The important thing is to configure the client to discover the Job

Re: Queryable State

Hi Chet, the following thins may create the error you mentioned: * the job ID of the query must match the ID of the running job * the job is not running anymore * the queryableStateName does not match the string given to setQueryable("query-name") * the queried key does not exist (note that you ne

Re: Queryable State

Hi Chet, Nico or Ufuk (in CC) should be able to help you. Thanks, Fabian 2017-03-13 11:23 GMT+01:00 Chet Masterson : > Any guidance on troubleshooting error > > Error: No KvStateLocation found for KvState instance with name X > > when trying to make a queryable state call in Flink 1.2.0?

Re: Queryable State and Windows

I just want to add another workaround, which does not need a self-compiled version. You can use TimeWindow with a CountTriger.of(1) combined with a FoldFunction for pre-aggregration and a RichWindowFunction to update the queryable state. Additionally, you need a TimeWindow for the final results. So

Re: Queryable State

Hi Nico, No problem at all, I've already presented my showcase with ValueStateDescriptor. Anyway, if I could help you somehow with the Queryablestate let me know. I will be happy to contribute some code. 2017-01-25 14:47 GMT+01:00 Nico Kruber : > Hi Dawid, > sorry for the late reply, I was fixi

Re: Queryable State

Hi Dawid, sorry for the late reply, I was fixing some issues for queryable state and may now have gotten to the point of your error: you may be seeing a race condition with the MemoryStateBackend state backend (the default) as described here: https://issues.apache.org/jira/browse/FLINK-5642 I'm c

  1   2   >