Re: Regarding java.lang.IllegalStateException

We are also seeing something similar: 2024-04-26 16:30:44,401 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Power Consumption:power_consumption -> Ingest Power Consumption -> PopSysFields -> WindowingWatermarkPreCheck (1/1) (cb8c425b6463b1ade9b8359c0514386b_28

Re: Regarding Changes in Flink Operator 1.3.1

Thanks a lot. On Thu, Jan 19, 2023, 1:08 PM Gyula Fóra wrote: > Please see the release announcements: > > https://flink.apache.org/news/2022/10/07/release-kubernetes-operator-1.2.0.html > > https://flink.apache.org/news/2022/12/14/release-kubernetes-operator-1.3.0.html > > https://flink.apache.o

Re: Regarding Changes in Flink Operator 1.3.1

Please see the release announcements: https://flink.apache.org/news/2022/10/07/release-kubernetes-operator-1.2.0.html https://flink.apache.org/news/2022/12/14/release-kubernetes-operator-1.3.0.html https://flink.apache.org/news/2023/01/10/release-kubernetes-operator-1.3.1.html https://nightlies.ap

Re: Regarding Changes in Flink Operator 1.3.1

also are the changes in operator version 1.3.1 backward compatible ? On Thu, Jan 19, 2023 at 12:38 PM Sumit Aich wrote: > Hi Team, > > Can you please share what has changed in Flink Kubernetes Operator version > 1.3.1 from the 1.1.0 version. > > Thanks, > Sumit >

Re: Regarding Flink Upgrades

Hi, What you linked to is what the community agreed to support. So far we've been able to support three versions at all times (e.g. currently we merge bugfixes to 1.17.x, 1.16.x, 1.15.x), which is one extra version than what is described in the docs. I don't think this will ever decrease. As

Re: Regarding Flink Upgrades

Hello Prasanna, 1) Of course we would always recommend you keep up to date. To receive support and fixes from the Flink community you should try to stick to the current/previous minor version, as per the policy. Releases for older versions are rare and typically only performed under exceptional ci

Re: regarding flink metrics

Your best bet is to create a custom reporter that does this calculation. You could either wrap the reporter, subclass is, or fork it. In any case, https://github.com/apache/flink/tree/master/flink-metrics/flink-metrics-datadog should be a good starting point. On 01/02/2022 13:26, Jessy Ping wr

Re: Regarding Queryable state in Flink

Hi Jessy, Queryable State is considered approaching end of life [1] per the Flink Roadmap. There are currently no development activities planned for it. Best regards, Martijn [1] https://flink.apache.org/roadmap.html Op di 25 jan. 2022 om 18:00 schreef Jessy Ping > Hi Matthias, > > I want t

Re: Regarding Queryable state in Flink

Hi Matthias, I want to query the current state of the application at real-time. Hence, state processor API won't fit here. I have the following questions, * Is the queryable state stable enough to use in production systems ?. Are there any improvements or development activities planned or going

RE: Regarding Queryable state in Flink

Hi Jessy, Have you considered using the state processor api [1] for offline analysis of checkpoints and savepoints? [1] https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/libs/state_processor_api/ Sincere greetings Thias From: Jessy Ping Sent: Montag, 24. Januar 2022 16:47 To:

Re: Regarding Connector Options - value.deserializer

gt; > Ronak Beejawat > > > > > > > > *From:* Hang Ruan > *Sent:* Monday, January 10, 2022 3:06 PM > *To:* d...@flink.apache.org; Ronak Beejawat (rbeejawa) > *Cc:* commun...@flink.apache.org; user@flink.apache.org > *Subject:* Re: Regarding Connector Opti

Re: Regarding Connector Options - value.deserializer

Hi, Ronak, I think you should implement a custom format by yourself instead of overriding. The 'value.format' is a required table option. Best, Hang Ronak Beejawat (rbeejawa) 于2022年1月10日周一 17:09写道： > Hi Team, > > Is there any way we use value.deserializer in Connector Options from kafka > via

Re: Regarding the size of Flink cluster

Hi Jessy, let me try to answer some of your questions. > 16 Task Managers with 1 task slot and 1 CPU each Every additional task manager also involves management overhead. So I would suggest option 1. But in the end you need to perform some benchmarks yourself. I could also imagine that a mixt

Re: Regarding state access in UDF

Hi Ingo, Thank you for your advice, we've not tried it yet, we just thought it may work that way, but now it seems not then. We'll see how it could match our use case with the AggregateFunction interface. On Thu, Jul 1, 2021 at 1:57 PM Ingo Bürk wrote: > Hi Kai, > > CheckpointedFunction is not

Re: Regarding state access in UDF

Hi Kai, CheckpointedFunction is not an interface meant to be used with UDFs (in the Table API / SQL sense[1]), but is rather an interface for DataStream API[2]; the term "user-defined function" has a different meaning there. Did you actually try it to see if it works? I'd be surprised it it did.

Re: Regarding state access in UDF

Hi Ingo, Thank you for the reply, we actually need more fine-grained control on the states in UDF. Per investigation, we found that the states can be simply created/accessed via implementing `CheckpointedFunction` interface, please advise if there is any side-effect by doing that. On Wed, Jun 30,

Re: Regarding state access in UDF

Hi Kai, AggregateFunction and TableAggregateFunction are both stateful UDF interfaces. This should cover most scenarios given where they would be used. If you need more fine-grained control you can also always drop down into the DataStream API (using #toDataStream) and work there. Table API / SQL

Re: Regarding FLIP-91's status

Friday, May 28, 2021 5:18 AM To: Sonam Mandal Cc: user@flink.apache.org ; Jark Wu Subject: Re: Regarding FLIP-91's status Hi Sonam, It looks like it has been stale for some time. You might be able to restart the discussion replying to the respective thread in the dev mailing list [1]. You

Re: Regarding FLIP-91's status

Hi Sonam, It looks like it has been stale for some time. You might be able to restart the discussion replying to the respective thread in the dev mailing list [1]. You seem to be right about the repository based on Jark's reply in the related ticket FLINK-15472 [2]. I'm adding Jark to the thread. M

Re: Regarding Stateful Functions

Hi Austin, Thanks for your insights. We are currently following a microservice architecture for accomplishing our data processing requirements. We are planning to use Flink as our unified platform for all data processing tasks. Although most of our use cases are a suitable fit for Flink, there

Re: Regarding Stateful Functions

Hey Jessy, I'm not a Statefun expert but, hopefully, I can point you in the right direction for some of your questions. I'll also cc Gordan, who helps to maintain Statefun. *1. Is the stateful function a good candidate for a system(as above) that > should process incoming requests at the rate of

Re: Regarding json/xml/csv file splitting

Normally parallel processing of text input files is handled via Hadoop TextInputFormat, which support splitting of files on line boundaries at (roughly) HDFS block boundaries. There are various XML Hadoop InputFormats available, which try to sync up with splittable locations. The one I’ve used

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Fabian, Thanks for explaining in detail. But we know and you also mentioned the issues in 1) and 2). So, I am continuing with point 3). Thanks & Regards Gaurav Luthra Mob:- +91-9901945206 On Mon, Oct 1, 2018 at 3:11 PM Fabian Hueske wrote: > Hi, > > There are basically three options: > 1)

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi, There are basically three options: 1) Use an AggregateFunction and store everything that you would put into state into the Accumulator. This can become quite expensive because the Accumulator is de/serialized for every function call if you use RocksDB. The advantage is that you don't have to s

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi ken, Mine is very generic use case. Means I am building an aggregation function using flink, which can be configured according to any use case. Actually, It will not be for a specific use case and every user can enter their business logic and use this aggregator to get result. And about windowi

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Gaurav, I’m curious - for your use case, what are the windowing & aggregation requirements? E.g. is it a 10 second sliding window? And what’s the aggregation you’re trying to do? Thanks, — Ken > On Sep 28, 2018, at 4:00 AM, Gaurav Luthra wrote: > > Hi Chesnay, > > I know it is an issu

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Chesnay, I know it is an issue, And won't be fixed because of window merging feature in case of session window. But I am looking if someone has implemented aggregation function using ProcessFunction and process() method instead of AggregationFunction and aggregate() method. I hope you got my po

Re: Regarding implementation of aggregate function using a ProcessFunction

Please see: https://issues.apache.org/jira/browse/FLINK-10250 On 28.09.2018 11:27, vino yang wrote: Hi Gaurav, Yes, you are right. It is really not allowed to use RichFunction. I will Ping Timo, he may give you a more professional answer. Thanks, vino. Gaurav Luthra

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Gaurav, Yes, you are right. It is really not allowed to use RichFunction. I will Ping Timo, he may give you a more professional answer. Thanks, vino. Gaurav Luthra 于2018年9月28日周五 下午4:27写道： > Hi Vino, > > Kindly check below flink code. > > package org.apache.flink.streaming.api.datastream.Win

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Vino, Kindly check below flink code. package org.apache.flink.streaming.api.datastream.WindowedStream @PublicEvolving public SingleOutputStreamOperator aggregate(AggregateFunction function) { checkNotNull(function, "function"); if (*function instanceof RichFunction*) { throw new *Unsupporte

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Gaurav, This is very strange, can you share your code and specific exceptions? Under normal circumstances, it should not throw an exception. Thanks, vino. Gaurav Luthra 于2018年9月28日周五 下午3:27写道： > Hi Vino, > > RichAggregateFunction can surely access the state. But the problem is, In > aggrega

Re: Regarding implementation of aggregate function using a ProcessFunction

Hi Gaurav, Why do you think the RichAggregateFunction cannot access the State API? RichAggregateFunction inherits from AbstractRichFunction (it provides a RuntimeContext that allows you to access the state API). Thanks, vino. Gaurav Luthra 于2018年9月28日周五 下午1:38写道： > Hi, > > As we are aware, Cur

RE: Regarding the use of RichAsyncFunction as a FlatMap

Regarding the ‘partiality’ you mention in option one, wouldn’t you have to give that up anyway to maintain exactly once processing? Suppose input message A results in asynchronous queries/futures B and C, and imagine the following series of events: 1. Your function receives A 2. As

Re: Regarding external metastore like HIVE

+1 on this feature, there have been a lot of pains for us trying to connect to external catalog / metastore as well. @shivam can you comment on the tickets regarding the specific use case and the type of external catalogs you are interested ? Thanks, Rong On Tue, Jul 3, 2018 at 3:16 AM Shivam Sh

Re: Regarding external metastore like HIVE

Thanks Timo, Fabian I will follow this. Best On Tue, Jul 3, 2018 at 3:01 PM Timo Walther wrote: > Hi, > > you can follow the progress here: > https://issues.apache.org/jira/browse/FLINK-9171 > > Regards, > Timo > > > Am 03.07.18 um 10:32 schrieb Fabian Hueske: > > Hi, > > The docs explain that

Re: Regarding external metastore like HIVE

Hi, you can follow the progress here: https://issues.apache.org/jira/browse/FLINK-9171 Regards, Timo Am 03.07.18 um 10:32 schrieb Fabian Hueske: Hi, The docs explain that the ExternalCatalog interface *can* be used to implement a catalog for HCatalog or Metastore. However, there is no suc

Re: Regarding external metastore like HIVE

Hi, The docs explain that the ExternalCatalog interface *can* be used to implement a catalog for HCatalog or Metastore. However, there is no such implementation in Flink yet. You would need to implement such as catalog connector yourself. I think there would be quite a few people interested in su

Re: Regarding external metastore like HIVE

Hi, Please find the documentation link here: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog . Currently, Flink provides an InMemoryExternalCatalog for demo and testing > purposes. However, the ExternalCatalog interface can also be us

Re: Regarding external metastore like HIVE

@ashwin I cannot find the documention about `metastore` Could you give the reference? Ashwin Sinha 于2018年7月3日周二 下午4:08写道： > By metastore we mean catalog where table information is stored. > > On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, wrote: > >> What do you mean with "Metastore"? Are you re

Re: Regarding external metastore like HIVE

By metastore we mean catalog where table information is stored. On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, wrote: > What do you mean with "Metastore"? Are you referring to state backends > > ? > > On 02.07.20

Re: Regarding external metastore like HIVE

What do you mean with "Metastore"? Are you referring to state backends ? On 02.07.2018 18:41, Shivam Sharma wrote: Hi, I have read Flink documentation that Flink supports Metastore which is currently InMemor

Re: Regarding Keyed State of Connected Stream

Hi Garvit, If you don't mind to take a look at the source code, I would suggest you to have a look at the "StreamTwoInputProcessor#processInput()" in StreamTwoInputProcessor.java. Your customer function will be called via here, the "CoFlatMapFunction#flatMap1()" will be called via streamOpera

Re: Regarding Keyed State of Connected Stream

Hi Sihua, Thanks for the quick response. Could you please let me know, where can I find more details about it. Thanks, On Fri, May 25, 2018 at 11:04 AM, sihua zhou wrote: > Hi Garvit, > > I think you don't need to lock it, they are executed in the same thread > sync. > > Best, Sihua > > > >

Re: Regarding Keyed State of Connected Stream

Hi Garvit, I think you don't need to lock it, they are executed in the same thread sync. Best, Sihua 在2018年05月25日 10:26，Garvit Sharma 写道： Hi, Let's consider, I have two keyed streams one for rules and another for data and I have created a connected stream. I am maintaining a managed keyed

Re: Regarding BucketingSink

I would expect that to be possible as well, yes. > On 21. Apr 2018, at 17:33, Vishal Santoshi wrote: > > >> After the savepoint state has been written, the sink might start new > >> .in-progress files. These files are not part of the savepoint but renamed > >> to .pending in close(). > >> On r

Re: Regarding BucketingSink

>> After the savepoint state has been written, the sink might start new .in-progress files. These files are not part of the savepoint but renamed to .pending in close(). >> On restore all pending files that are part of the savepoint are moved into final state (and possibly truncated). See handlePen

Re: regarding the use of colocation groups

Hello Chesnay, Thank you for your prompt reply and helpful feedback on the matter, For the fist case-study a partitioner seems like a good idea, we have managed to get it working on the default local minicluster but unfortunately are running into issues deploying it on AWS. When we pinpoint the i

Re: regarding the use of colocation groups

Hello, your first use-case should be achievable by using a custom partitioner , probably with a KeySelector that returns the word. As for the second use-case, typically this would be achieved b

Re: Regarding BucketingSink

Thank you Fabian, What is more important ( and I think you might have addressed it in your post so sorry for being a little obtuse ) is that deleting them does not violate "at-least-once" delivery. And if that is a definite than we are fine with it, though we will test it further. Thanks and

Re: Regarding BucketingSink

Hi Vishal, hi Mu, After the savepoint state has been written, the sink might start new .in-progress files. These files are not part of the savepoint but renamed to .pending in close(). On restore all pending files that are part of the savepoint are moved into final state (and possibly truncated).

Re: Regarding BucketingSink

Hi Aljoscha, Thanks for confirming that fact that Flink doesn't clean up pending files. Is that safe to clean(remove) all the pending files after cancel(w/ or w/o savepointing) or failover? If we do that, will we lose some data? Thanks! Best, Mu On Wed, Feb 21, 2018 at 5:27 AM, Vishal Santos

Re: Regarding BucketingSink

Sorry, but just wanted to confirm that the assertion "at-least-once" delivery true if there is a dangling pending file ? On Mon, Feb 19, 2018 at 2:21 PM, Vishal Santoshi wrote: > That is fine, till flink assure at-least-once semantics ? > > If the contents of a .pending file, through the turbu

Re: Regarding BucketingSink

That is fine, till flink assure at-least-once semantics ? If the contents of a .pending file, through the turbulence ( restarts etc ) are assured to be in another file than anything starting with "_" underscore will by default ignored by hadoop ( hive or MR etc ). On Mon, Feb 19, 2018 at 11:03

Re: Regarding BucketingSink

Hi, Sorry for the confusion. The framework (Flink) does currently not do any cleanup of pending files, yes. Best, Aljoscha > On 19. Feb 2018, at 17:01, Vishal Santoshi wrote: > > >> You should only have these dangling pending files after a failure-recovery > >> cycle, as you noticed. My sugg

Re: Regarding BucketingSink

>> You should only have these dangling pending files after a failure-recovery cycle, as you noticed. My suggestion would be to periodically clean up older pending files. A little confused. Is that what the framework should do, or us as part of some cleanup job ? On Mon, Feb 19, 2018 at 10:47

Re: Regarding Task Slots allocation

Hi Vinay, try to set the parallelism to 2 for the job you are executing via the RemoteExecutionEnvironment. Where have you specified the number of TaskManager slots? In the flink-conf.yaml file which you used to deploy the remote Flink cluster? Cheers, Till On Fri, Feb 16, 2018 at 7:14 PM, Vina

Re: Regarding BucketingSink

Hi, The BucketingSink does not clean up pending files on purpose. In a distributed setting, and especially with rescaling of Flink operators, it is sufficiently hard to figure out which of the pending files you actually can delete and which of them you have to leave because they will get moved

Re: Regarding BucketingSink

Hi Vishal, what pending files should indeed get eventually finalized. This happens on a checkpoint complete notification. Thus, what you report seems not right. Maybe Aljoscha can shed a bit more light into the problem. In order to further debug the problem, it would be really helpful to get acce

Re: Regarding BucketingSink

Hi Vishal, I have the same concern about save pointing in BucketingSink. As for your question, I think before the pending files get cleared in handleRestoredBucketState . They are finalized in notifyCheckpointComplete https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-fi

Re: Regarding BucketingSink

-rw-r--r-- 3 root hadoop 11 2018-02-14 18:48 /kraken/sessions_uuid_csid_v3/dt=2018-02-14/_part-0-18.valid-length -rw-r--r-- 3 root hadoop 54053518 2018-02-14 19:15 /kraken/sessions_uuid_csid_v3/dt=2018-02-14/_part-0-19.pending -rw-r--r-- 3 root hadoop 11 2018-02-14 21:17 /

Re: Regarding BucketingSink

without --allowNonRestoredState, on a suspend/resume we do see the length file along with the finalized file ( finalized during resume ) -rw-r--r-- 3 root hadoop 10 2018-02-09 13:57 /vishal/sessionscid/dt=2018-02-09/_part-0-28.valid-length that does makes much more sense. I guess we sh

Re: Regarding BucketingSink

This is 1.4 BTW. I am not sure that I am reading this correctly but the lifecycle of cancel/resume is 2 steps 1. Cancel job with SP closeCurrentPartFile https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors

Re: Regarding flink-cassandra-connectors

Sure, I will create a Jira for that. In addition to that, I would like to confirm, would it be possible to reuse the connection builder object across queries and across jobs. i.e if I create a Singleton class which would create a connection builder instance and could I use across the queries. I h

Re: Regarding flink-cassandra-connectors

Hi Jagadish, Yes, that indeed is something missing. If that is something you’re interested in, could you perhaps open a JIRA for that (AFAIK there isn’t one for the feature yet). Gordon On 26 September 2017 at 2:09:37 PM, Jagadish Gangulli (jagadi...@gmail.com) wrote: Thanks Gordon, Have f

Re: Regarding flink-cassandra-connectors

Thanks Gordon, Have few more queries on the same lines, if I have to perform fetch i.e. select queries, I have to go for the batch queries, no streaming support is available. Regards, Jagadisha G On Tue, Sep 26, 2017 at 3:40 PM, Tzu-Li (Gordon) Tai wrote: > Hi Jagadish, > > Yes, you are right

Re: Regarding flink-cassandra-connectors

Ah, sorry I just realized Till also answered your question on your cross-post at dev@. It’s usually fine to post questions to just a single mailing list :) On 26 September 2017 at 12:10:55 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Hi Jagadish, Yes, you are right that the Flink Cass

Re: Regarding flink-cassandra-connectors

Hi Jagadish, Yes, you are right that the Flink Cassandra connector uses the Datastax drivers internally, which is also the case for all the other Flink connectors; e.g., the Kafka connector uses the Kafka Java client, Elasticearch connector uses the ES Java client, etc. The main advantage when

Re: Regarding exception relating to FlinkKafkaConsumer09

Hi! The `PropertiesUtil.getBoolean` currently only exists in `1.3-SNAPSHOT`. The method was added along with one of the Kafka consumer changes recently. Generally, you should always use matching versions of the Flink installation and the library, otherwise these kind of errors can always be exp

Re: Regarding Flink as a web service

There's also a nice documentation page on the feature now: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/queryable_state.html On Tue, Jan 31, 2017 at 6:18 PM, Aljoscha Krettek wrote: > +u...@apache.org Because he implemented queryable state. > > There is also queryable

Re: Regarding Flink as a web service

+u...@apache.org Because he implemented queryable state. There is also queryable state, which allows you to query the internal keyed state of Flink user functions. On Mon, 30 Jan 2017 at 00:46 Jonas wrote: > You could write your data back to Kafka using the FlinkKafkaProducer and > then > use

Re: Regarding Flink as a web service

You could write your data back to Kafka using the FlinkKafkaProducer and then use websockets to read from kafka using NodeJS or other. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Flink-as-a-web-service-tp11364p11365.html Sent fr

Re: Regarding caching the evicted elements and re-emitting them to the next window

Hi Abdul, You may want to check out FLIP13 "side output" https://goo.gl/6KSYd0 . Once we have this feature, you should be able to collect the data to the external distributed storage, and use these data later on demand. BTW, can you explain your use case in more details, such that people here may h

Re: Regarding caching the evicted elements and re-emitting them to the next window

Hi, I'm afraid there is no functionality for this in Flink. What you can do, however, is to not evict these elements from the window buffer but instead ignore them when processing your elements in the WindowFunction. This way they will be preserved for the next firing. You have to make sure to even

Re: Regarding ordering of events

Thanks a lot Aljoshca, this was a perfect answer to my vague question. On 09-Jan-2017 4:52 pm, "Aljoscha Krettek" wrote: Hi, to clarify what Kostas said. A "single window" in this case is a window for a given key and time period so the window for "key1" in time t1 to t2 can be processed on a dif

Re: Regarding ordering of events

Hi, to clarify what Kostas said. A "single window" in this case is a window for a given key and time period so the window for "key1" in time t1 to t2 can be processed on a different machine from the window for "key2" in time t1 to t2. Cheers, Aljoscha On Thu, 5 Jan 2017 at 21:56 Kostas Kloudas w

Re: Regarding ordering of events

Hi Abdul, Every window is handled by a single machine, if this is what you mean by “partition”. Kostas > On Jan 5, 2017, at 9:21 PM, Abdul Salam Shaikh > wrote: > > Thanks Fabian and Kostas, > > How can I put to use the power of flink as a distributed system ? > > In cases where we have

Re: Regarding ordering of events

Thanks Fabian and Kostas, How can I put to use the power of flink as a distributed system ? In cases where we have multiple windows, is one single window handled by one partition entirely or is it spread across several partitions ? On Thu, Jan 5, 2017 at 12:25 PM, Fabian Hueske wrote: > Flink

Re: Regarding ordering of events

Flink is a distributed system and does not preserve order across partitions. The number prefix (e.g., 1>, 2>, ...) tells you the parallel instance of the printing operator. You can set the parallelism to 1 to have the stream in order. Fabian 2017-01-05 12:16 GMT+01:00 Kostas Kloudas : > Hi Abdu

Re: Regarding ordering of events

Hi Abdul, Flink provides no ordering guarantees on the elements within a window. The only “order” it guarantees is that the results referring to window-1 are going to be emitted before those of window-2 (assuming that window-1 precedes window-2). Thanks, Kostas > On Jan 5, 2017, at 11:57 AM, Ab

Re: Regarding windows and custom trigger

Hi Abdul, Probably the new enhanced evictors can help you do what you want. You can have a look here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor and also in the related J

Re: Regarding time window based on the values received in the stream

Hi, sorry for the late reply. There is a repository [1] with an example application that uses a custom trigger [2] (though together with a TimeWIndow and not with a GlobalWindow). I'm not aware of a repo with an example of a GlobalWIndow. Regarding the question about timestamps and watermarks: I

Re: Regarding dividing the streams using keyby

Hi, the result of a window operation on a KeyedStream is a regular DataStream. So, you would need to call keyBy() on the result again if you'd like to have a KeyedStream. You can also key a stream by two or more attributes: DataStream> windowedStream = jsonToTuple.keyBy(0,1,3) /

Re: Regarding time window based on the values received in the stream

Hello Mr Hueske, Thank you for reaching out to my query. The example stated in the documentation is the same use case for me where I am trying to build a prototype regarding a traffic metric in Germany as a part of my thesis. The data is received from multiple detectors and there is a field whic

Re: Regarding time window based on the values received in the stream

Hi, that does not sound like a time window problem because there is not time-related condition to split the windows. I think you can implement that with a GlobalWindow and a custom trigger. The documentation about global windows, triggers, and evictors [1] and this blogpost [2] might be helpful O

Re: Regarding

Hi Rashmi, as Marton said, you do not need to start a local Flink instance (start-lcoal.bat) if you want to run programs from your IDE. Maybe running a local instance causes a conflict when starting an instance from IDE. Developing and running Flink programs on Windows should work, both from the I

Re: Regarding

I think Flink supports running on Windows, but developing against Flink on Windows is something that we do not test. I have to admit that my knowledge in that area is very limited, Fabian (ccd) is the goto person with that topic. Could you download the Flink binaries and try running the examples [

Re: Regarding

Hi Rashmi, The issue is that although you have a JobManager running (the master component of a Flink cluster scheduling the jobs) there are no TaskManagers running (the components doing the actual work). Hence you got the log line "Resources available to scheduler: Number of instances=0, total num

Re: Regarding Late Elements

e-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers

Re: Regarding Late Elements

g List archive., click here > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNames

Re: Regarding Late Elements

Hi Kostas, Yes you are right , I am always doing FIRE_AND_PURGE , if we don't do this and only use FIRE , the window function will get the elements in incremental fashion (1, 2,3..so on) I had observed this while testing. Can you please explain me the importance of canMerge and onMerge functions

Re: Regarding Late Elements

Hi Vinay, By setting the allowed_lateness to LongMax you are ok. Sorry I forgot that this was the default value. Just a note (although you have it right in your code), in this case you should always FIRE_AND_PURGE and not just FIRE. In other case your state will keep growing as it is never ga

Re: Regarding Late Elements

am with only these elements. >> >> Thanks, >> Kostas >> >> On Oct 3, 2016, at 5:28 PM, [hidden email] >> <http://user/SendEmail.jtp?type=node&node=9307&i=0> wrote: >> >> Not yet. >> I'm hoping a Flink export on this mailing list wil

Re: Regarding Late Elements

=9307&i=0> wrote: >> >> Not yet. >> I'm hoping a Flink export on this mailing list will reply. >> >> >> - LF >> >> >> >> From: vinay patil <[hidden email] >> <http://user/SendEmail.jtp?type=node&node=9307&am

Re: Regarding Late Elements

-- > *From:* vinay patil <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=9307&i=1>> > *To:* [hidden email] <http:///user/SendEmail.jtp?type=node&node=9307&i=2> > *Sent:* Monday, October 3, 2016 8:09 AM > *Subject:* Re: Regarding

Re: Regarding Late Elements

et. > I'm hoping a Flink export on this mailing list will reply. > > > - LF > > > > From: vinay patil > To: user@flink.apache.org > Sent: Monday, October 3, 2016 8:09 AM > Subject: Re: Regarding Late Elements > > Hi LF, > > So did you manage to g

Re: Regarding Late Elements

Not yet. I'm hoping a Flink export on this mailing list will reply. - LF From: vinay patil To: user@flink.apache.org Sent: Monday, October 3, 2016 8:09 AM Subject: Re: Regarding Late Elements Hi LF, So did you manage to get the workaround for it ? I am using a Custom Tr

Re: Regarding Late Elements

Hi LF, So did you manage to get the workaround for it ? I am using a Custom Trigger which is similar to 1.0.3 with few changes Regards, Vinay Patil On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.] wrote: > We have the same requirement - we cannot discard an

Re: Regarding Late Elements

We have the same requirement - we cannot discard any data even if it arrives late.  - LF   From: Vinay Patil To: user@flink.apache.org Sent: Sunday, October 2, 2016 8:21 PM Subject: Regarding Late Elements Hi Guys, Just wanted to get an idea on Why Flink decided to completely dis

Re: Regarding Global Configuration in Flink

Hi! Are you referring to the GlobalConfiguration class? That used to be a singleton class in Flink version < 1.2.x which would load the configuration only once per VM, if it found a config file. It allowed operations that could change that config after it had been loaded. It has since then been re

Re: Regarding QueryableState

Thanks Ufuk, will take a look at it. Regards, Vinay Patil On Thu, Aug 4, 2016 at 4:26 PM, Ufuk Celebi [via Apache Flink User Mailing List archive.] wrote: > You can expect it to be merged by the end of this week. > > Note that the APIs are very low level at the moment though. In the PR > branch

  1   2   >