Re: could I chain two timed window?

Hi, sliding windows replicate their records for each window. If you have use an incrementally aggregating function (ReduceFunction, AggregateFunction) with a sliding, the space requirement should not be an issue because each window stores a single value. However, this also means that each window p

Re: when does the timed window ends?

Hi, this depends on the window type. Tumbling and Sliding Windows are (by default) aligned with the epoch time (1970-01-01 00:00:00). For example a tumbling window of 2 hour starts and ends every two hours, i.e., from 12:00:00 to 13:59:59.999, from 14:00:00 to 15:59:59.999, etc. The documentation

Re: How to deal with dynamic types

Hi, For truly dynamic class you would need a custom TypeInformation or TypeDeserializationSchema and store the fields on some kind of Map. Maybe something could be done with inheritance if records that always share the same fields could be deserialized to some specific class with fixed/predefin

Re: Custom Metrics

Hi, Reporting once per 10 seconds shouldn’t create problems. Best to try it out. Let us know if you get into some troubles :) Piotrek > On 11 Dec 2017, at 18:23, Navneeth Krishnan wrote: > > Thanks Piotr. > > Yes, passing the metric group should be sufficient. The subcomponents will > not

could I chain two timed window?

Hi All, Given one stream source which generates 20k events/sec, and I need to aggregate the element count using sliding window of 1 hour size. The problem is, the window may buffer too many elements (which may cause a lot of block I/O because of checkpointing?), and in fact it does not necessary

RE: ProgramInvocationException: Could not upload the jar files to the job manager / No space left on device

And if it helps, I'm running on flink 1.2.1. I saw this ticket: https://issues.apache.org/jira/browse/FLINK-5828 It only started happening when I was running all 50 flows at the same time. However, it looks like it's not an issue with creating the cache directory but with running out of space th

when does the timed window ends?

Hi All, The document said "a window is created as soon as the first element that should belong to this window arrives, and the window is completely removed when the time (event or processing time) passes its end timestamp plus the user-specified allowed lateness (see Allowed Lateness).". I am sti

ProgramInvocationException: Could not upload the jar files to the job manager / No space left on device

Hi, I'm currently submitting 50 separate jobs to a 50TM, 1 slot set up. Each job has 1 parallelism. There's plenty of space left in my cluster and on that node. It's not clear to me what's happening. Any pointers? On the client side, when I try to execute, I see the following: org.apache.flink.

Re: The timing operation is similar to storm’s tick

thanks. 2017-12-11 17:51 GMT+08:00 Fabian Hueske : > Hi, > > I think you are looking for a ProcessFunction with timers [1]. > > Best, > Fabian > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.3/dev/stream/process_function.html > > 2017-12-11 9:03 GMT+01:00 Marvin777 : > >> h

Re: save points through REST API not supported ?

One last question.. Can you conform that the "This will be available in 1.5 where we rework the client-cluster communication to go entirely through the REST API." comment , ches...@apache.org ? On Mon, Dec 11, 2017 at 4:54 AM, Chesnay Schepler wrote: > This doesn't sound like proper behavior, c

Re: Flink-Kafka connector - partition offsets for a given timestamp?

Hi Connie, We do have a pull request for the feature, that should almost be ready after rebasing: https://github.com/apache/flink/pull/3915, JIRA: https://issues.apache.org/jira/browse/FLINK-6352. This means, of course, that the feature isn't part of any release yet. We can try to make sure this h

Flink-Kafka connector - partition offsets for a given timestamp?

Hi, Does Flink-Kafka connector allow job graph to consume topoics/partitions from a specific timestamp? https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java#L469 seems to sug

Re: Custom Metrics

Thanks Piotr. Yes, passing the metric group should be sufficient. The subcomponents will not be able to provide the list of metrics to register since the metrics are created based on incoming data by tenant. Also I am planning to have the metrics reported every 10 seconds and hope it shouldn't be

Re: Hardware Reference Architecture

Hi Hayden, This is a talk from Flink Forward that may be of help to you: https://www.youtube.com/watch?v=8l8dCKMMWkw and here are the slides: www.slideshare.net/FlinkForward/flink-forward-berlin-2017-robert-metzger-keep-it-going-how-to-reliably-and-e

How to deal with dynamic types

Hi, I am trying some initial samples with flink. I have one doubt regarding data types. Flink support data types Tuple(max 25 fields), Java POJOs, Primitive types, Regular classes etc., In my case I do not have fixed type. I have meta data with filed names & its types. For ex., (Id:int, Name:Strin

Re: aggregate does not allow RichAggregateFunction ?

Perfect, f in our use case, the kafka partition key and the keyBy use the same exact field and thus the order will be preserved. On Mon, Dec 11, 2017 at 4:34 AM, Fabian Hueske wrote: > Hi, > > the order or records that are sent from one task to another task is > preserved (task refers to the pa

Re: REST api: how to upload jar?

Hi, Have you tried this https://stackoverflow.com/questions/41724269/apache-flink-rest-client-jar-upload-not-working ? Piotrek > On 11 Dec 2017, at 14:22, Edward wrote: > > Let me try that again

Re: REST api: how to upload jar?

Let me try that again -- it didn't seem to render my commands correctly: Thanks for the response, Shailesh. However, when I try with python, I get the same error as when I attempted this with cURL: $ python uploadJar.py java.io.FileNotFoundException: /tmp/flink-web-4bed7801-fa5e-4e5e-abf1-3fa13

Re: REST api: how to upload jar?

Thanks for the response, Shailesh. However, when I try with python, I get the same error as when I attempted this with cURL: That is, if I tell python (or cURL) that my jar file is at /path/to/jar/file.jar, the file path it uses on the server side includes that entire path. And if I try the scrip

Re: Exception when using the time attribute in table API

Hi, NoSuchMethodError probably comes from some mismatched compile/runtime versions of the Flink. Do you have to use 1.4-SNAPSHOT version? It can change on daily basis, so you have to be more careful about what Flink jar's you are using at runtime and what on compile time. If you really need som

Re: Parallelizing a tumbling group window

Hi Colin, unfortunately, selecting the parallelism for parts of a SQL query is not supported yet. By default, tumbling window operators use the default parallelism of the environment. Simple project and select operations have the same parallelism as the inputs they are applied on. I think th

Re: ayncIO & TM akka response

Hi, Please search the task manager logs for the potential reason of failure/disconnecting around the time when you got this error on the job manager. There should be some clearly visible exception. Thanks, Piotrek > On 9 Dec 2017, at 20:35, Chen Qin wrote: > > Hi there, > > In recent, our

Re: Custom Metrics

Hi, I’m not sure if I completely understand your issue. 1. - You don’t have to pass RuntimeContext, you can always pass just the MetricGroup or ask your components/subclasses “what metrics do you want to register” and register them at the top level. - Reporting tens/hundreds/thousands of metric

Re: Exception when using the time attribute in table API

Hi Sendoh, at a first glance this looks like a Maven issue to me. Are you sure you are using a consistent version for both core Flink and flink-table (also consistent Scala version 2.11)? Maybe you can share your pom.xml with us. It seems that flink-table is a newer version that your Flink c

Re: save points through REST API not supported ?

This doesn't sound like proper behavior, could you provide instructions on how to reproduce this? On 07.12.2017 13:42, Lasse Nedergaard wrote: I hope it can be put in 1.4.1. I have one concern about the rest api. We running 1.3.1 on dc/os and if we apply parameters as arguments and in our cod

Re: The timing operation is similar to storm’s tick

Hi, I think you are looking for a ProcessFunction with timers [1]. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html 2017-12-11 9:03 GMT+01:00 Marvin777 : > hi, > > I'm new to apache Flink. I want to update the property value per minu

Re: aggregate does not allow RichAggregateFunction ?

Hi, the order or records that are sent from one task to another task is preserved (task refers to the parallel instance of an operator). However, a task that receives records from multiple input tasks, consumes records from its inputs in arbitrary order. If a job reads from a partitioned Kafka to

Re: Problem with runGatherSumApplyIteration

Dear Stefan, thanks for your answer. Here is the flink version: org.apache.flink flink-java 1.3.2 org.apache.flink flink-gelly_2.11 1.3.2 org.apache.flink

RE: slot group indication per operator

Hi. Any update or suggestion on this? Best regards, Tovi From: Timo Walther [mailto:twal...@apache.org] Sent: יום ג 05 דצמבר 2017 18:55 To: user@flink.apache.org Cc: ches...@apache.org Subject: Re: slot group indication per operator Hi Tovi, you are right, it is difficult to check the correct b

The timing operation is similar to storm’s tick

hi, I'm new to apache Flink. I want to update the property value per minute via an HTTP request.I did this in storm using tick tuple. Does Flink have something similar which i can use in a flink operator? thanks.