Re: Streaming - memory management

Hi Fabian/Stephan, This makes things clear. This is the use case I have : I am performing a outer join operation on the two streams (in window) after which I get matchingAndNonMatchingStream, now I want to make sure that the matching rate is high (matching cannot happen if one of the source is no

Wikiedit QuickStart with Kinesis

Hi: I am using the following WikiEdit example: https://ci.apache.org/projects/flink/flink-docs-master/quickstart/run_example_quickstart.html It works when printing the contents to a file or stdout. But I wanted to modify it to use Kinesis instead of Kafka. So instead of the Kafka part, I put: P

Re: Not able to query : Queryable State

Hi, Are you sure the job id matches with running job? The error clearly says JobID not found. How are you setting the JobID, If I remember correctly, I used something like val jobId = JobID.fromHexString("myjob_id_here") to get the JobID object and used it to call client.getKvState() Thanks, Vis

Re: Setting EventTime window width using stream data

Thanks Aljoscha, The elements can certainly be configured to have the window size and slide as part of them and that will not change for the same elements. There may a different class of elements with a different window or slide but those values will be essentially final for that class of el

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

I don't know whether my usual error is related to this one but is very similar and it happens randomly...I still have to figure out the root cause of the error: java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at createResult(IndexMappingExecutor.java:43)) -> Map

Re: Streaming - memory management

Hi Stephan, Just wanted to jump into this discussion regarding state. So do you mean that if we maintain user-defined state (for non-window operators), then if we do not clear it explicitly will the data for that key remains in RocksDB. What happens in case of checkpoint ? I read in the documen

Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Hi, could you provide the log outputs for your job (ideally with debug logging enabled)? Best, Stefan > Am 31.08.2016 um 14:40 schrieb ANDREA SPINA <74...@studenti.unimore.it>: > > Hi everyone. > I'm running the FlinkML ALS matrix factorization and I bumped into the > following exception: >

Re: Setting EventTime window width using stream data

Just checking, all the elements that would fall into a window of length X also have X as a property? In that case you should be able to do something like this: public Collection assignWindows(PojoType element, long timestamp, WindowAssignerContext context) { long size = element.windowSize;

Re: Gracefully Stopping Streaming Job Programmatically in LocalStreamEnvironment

Hi Aljoscha, thanks for the answer. executeDetached() sounds super helpful for testing. You could basically return a Future for stopping, cancelling and so on. In my current IT I dont have a special sink, I am checking the resulting files directly, but live access to accumulators sounds very help

Re: Not able to query : Queryable State

Hi, can you provide some more code from your job and a full stack trace for your problem? That would help us to figure out the reason. Best, Stefan > Am 31.08.2016 um 13:00 schrieb Pushpendra Jaiswal > : > > > -- Forwarded message -- > From: Pushpendra Jaiswal

RE: "select as" in Flink SQL

Thank you Timo. From: Timo Walther [mailto:twal...@apache.org] Sent: Wednesday, August 31, 2016 4:28 AM To: user@flink.apache.org Subject: Re: "select as" in Flink SQL Hi Davran, Flink SQL uses a lexical policy similar to Java: The case of identifiers is preserved whether or not they are

FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null

Hi everyone. I'm running the FlinkML ALS matrix factorization and I bumped into the following exception: org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed. at org.apache.flink.client.program.Client.runBlocking(Client.java:381) at org.apa

Fwd: Not able to query : Queryable State

-- Forwarded message -- From: Pushpendra Jaiswal Date: Wed, Aug 31, 2016 at 3:52 PM Subject: Not able to query : Queryable State To: user@flink.apache.org Hi I have been trying to use QueryableState with my project. It's always failing the query saying. Job 461acde52fae479a6e02c

Not able to query : Queryable State

Hi Flink community, I have been trying to use QueryableState with my project. It's always failing the query saying. Job 461acde52fae479a6e02c6b2922cd9c1 not found Both keys(one in queryablestate and one in query) are of same type containing same data. Still I am not able to fetch the result. T

Re: Accessing state in connected streams

Worked like a charm. I realise I tried to do something stupid. The state created by EventA was handled by a different operator and I was trying to find a way to access it downstream. As I understand, the state is operator-scoped which means that only events passing through it can interact with

Re: Gracefully Stopping Streaming Job Programmatically in LocalStreamEnvironment

Hi Konstantin, I think this is not possible with the current API but I've been thinking about similar stuff this week. Let me quickly outline what I was thinking and then you can tell me whether that would also be helpful for you. The basic problem is this: I want to be able to write ITCases that

Re: "select as" in Flink SQL

Hi Davran, Flink SQL uses a lexical policy similar to Java: The case of identifiers is preserved whether or not they are quoted; after which, identifiers are matched case-sensitively. Unlike Java, back-ticks allow identifiers to contain non-alphanumeric characters. So "select assetClass as `a