Re: [DISCUSS] solve unstable build capacity problem on TravisCI

echo "And then make sure 'Build branch updates' option is enabled in the settings https://travis-ci.org/${AUTHOR}/zeppelin/settings."; echo "" echo "To trigger CI after setup, you will need ammend your last commit with" echo "git commit

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)

@RecordAPI: Yes, i was curious where we are at regarding the removal of the Record API. If there are still tests left to port (or other related things) I'd be more than happy to do it (got a /lot/ of free time on my hands). The related JIRA issues weren't particularly helpful though in figuring

Re: Record API removal

On 22.10.2015 17:24, Fabian Hueske wrote: Hmm, it took IntelliJ some time to figure out all the consequences of removing the Record API. Seems to be more than I initially expected. @Chesnay, do you want to help? I would push my current version to my repository and you could take over some packag

Re: job failed while initiating Transport client for Elasticsearch

could this be related? https://github.com/elastic/elasticsearch/issues/13052 On 02.11.2015 10:02, santosh_rajaguru wrote: java.lang.NoSuchFieldError: JRE_IS_64BIT

Re: Function input type validation

On 08.11.2015 21:28, Gyula Fóra wrote: Let's say I want to implement my own TupleTypeinfo that handles null values, and I pass this typeinfo in the returns call of an operation. This will most likely fail when the next operation validates the input although I think it shouldn't. So i just tried t

Re: Function input type validation

On 09.11.2015 08:49, Aljoscha Krettek wrote: In the case of the TupleTypeInfo subclass it only works because the equals method of TypleTypeInfo is used, IMHO. I've overridden the equals method to check specifically for my implementation and not TupleTypeInfo, implemented a different serializer

Re: withParameters() for Streaming API

should we do the same for IOFormats to be consistent? after FLINK-2351 they aren't used by our formats. On 24.11.2015 15:17, Suneel Marthi wrote: Agree with @till +1 to change this now On Tue, Nov 24, 2015 at 9:15 AM, Till Rohrmann wrote: If not API breaking before 1.0, then probably never?

Python start scripts broken

Hello everyone, I just noticed an odd behavior that occurs when using the python api start scripts: i suddenly get another argument i didn't get before; the flink-python jar path. which inadvertently breaks everything. An example as to how the execution of an example should go: bin\pyflink2.

Re: Python start scripts broken

at 6:05 PM, Chesnay Schepler wrote: Hello everyone, I just noticed an odd behavior that occurs when using the python api start scripts: i suddenly get another argument i didn't get before; the flink-python jar path. which inadvertently breaks everything. An example as to how the execution

Re: Python start scripts broken

Fabian is very experienced fixing windows path issues before every release ... and usually the issue are paths containing spaces. Maybe that's the case here as well? On Wed, Jan 27, 2016 at 6:51 PM, Chesnay Schepler wrote: I was trending more towards changes to the CLI. On 27.01.2016 1

Re: Python start scripts broken

i went back 30 commits and the issue is gone, as such its not a system problem. gonna narrow it down now. On 27.01.2016 19:29, Chesnay Schepler wrote: Yes i am on windows. no, ain't no path containing spaces. if it were due to spaces i would assume that either a) the python jar wouldn

Re: Python start scripts broken

Running mvn clean package resolved the issue. still the weirdest thing that happened this week. :/ On 27.01.2016 20:19, Chesnay Schepler wrote: i went back 30 commits and the issue is gone, as such its not a system problem. gonna narrow it down now. On 27.01.2016 19:29, Chesnay Schepler

Remove final flag from ResultPartitionWriter class

rTest({ResultPartitionWriter.class}). But it breaks my TemporaryFolder annotated with @ClassRule. (apart from that there also was a classloader issue, but i could resolve that) To me these seem like unnecessary problems; could we remove the flag, or is there a good reason for it being there? Regards, Chesnay Schepler

Re: [VOTE] Release Apache Flink 1.0.0 (RC1)

tested the RC on Windows: - source compiles - some tests categorically fail: see FLINK-3491 / FLINK-3496 - start/stop scripts work in both cygwin and windows CMD - ran several examples from batch/streaming/python - scripts also work on paths containing spaces On 25.02.2016 12:41, Robert Metzger

Re: JDBC Streaming Connector

I'm actually working on a JDBC streaming connector with transactions. On 26.02.2016 10:33, Stephan Ewen wrote: Hi Timothy! I would first just write a stream data sink based on the JDBCOutputFormat, and then let's look into making it exactly-once. For JDBC with support for transactions, the bas

Re: zipWithIndex in Python API

The subtaskIndex is not currently exposed to the python operator. Fortunately this can be changed very easily: On the java side, within PythonStreamer.startPython() the python process is started and several parameters are transferred (L.129++) using stdin/-out. These parameters are received on

Re: [streaming, scala] Scala DataStream#addSink returns Java DataStreamSink

On 13.03.2016 12:14, Robert Metzger wrote: I think its too early to fork off a 2.0 branch. I have absolutely no idea when a 2.0 release becomes relevant, could be easily a year from now. at first i was going to agree with Robert, but then...I mean the issue with not allowing breaking changes is

Re: Native iterations in PyFlink

Hello Shannon, you've picked yourself quite a feature there. The following classes will be relevant: * Python o DataSet o OperationInfo o Environment (_send_operation method) o Constants._Identifier * Java o PythonPlanBinder o PythonOperationInfo An (Python)Ope

Re: RichMapPartitionFunction - problems with collect

Haven't looked to deeply into this, but this sounds like object reuse is enabled, at which point buffering values effectively causes you to store the same value multiple times. can you try disabling objectReuse using env.getConfig().disableObjectReuse() ? On 22.03.2016 16:53, Sergio Ramírez

Re: Issue deploying a topology to flink with a java api

http://stackoverflow.com/questions/36584784/issues-while-submitting-a-topology-to-apache-flink-using-the-flink-api On 13.04.2016 14:28, Till Rohrmann wrote: Hi jstar, what's exactly the problem you're observing? Cheers, Till On Wed, Apr 13, 2016 at 2:23 PM, star jlong wrote: Hi there, I'm

Re: Issue deploying a topology to flink with a java api

bject."); } } sadly another case where we just swallow the exception cause. On 13.04.2016 14:35, Chesnay Schepler wrote: http://stackoverflow.com/questions/36584784/issues-while-submitting-a-topology-to-apache-flink-using-the-flink-api On 13.04.2016 14:28, Till Rohrmann wrote:

Re: Issue deploying a topology to flink with a java api

erned. Yes I'm actaully having the same issue as indicated on that post because I'm the one that posted that issue. Le Mercredi 13 avril 2016 13h35, Chesnay Schepler a écrit : http://stackoverflow.com/questions/36584784/issues-while-submitting-a-topology-to-apache-flink-usi

Re: FLINK-3750 (JDBCInputFormat)

On 14.04.2016 17:22, Fabian Hueske wrote: Hi Flavio, that are good questions. 1) Replacing null values by default values and simply forwarding records is very dangerous, in my opinion. I see two alternatives: A) we use a data type that tolerates null values. This could be a POJO that the user h

Re: FLINK-3750 (JDBCInputFormat)

tablishConnection? On 14 Apr 2016 17:28, "Chesnay Schepler" wrote: On 14.04.2016 17:22, Fabian Hueske wrote: Hi Flavio, that are good questions. 1) Replacing null values by default values and simply forwarding records is very dangerous, in my opinion. I see two alternatives: A) we

Re: Adding custom monitoring to Flink

I'm currently working on a metric system that a) exposes several TaskManger metrics b) allows gathering metrics in various parts of a task, most notably user-defined functions. The first version makes these metrics available via JMX on each TaskManager. While a mechanism to make that pluggable

Re: Master test stability poor

If this doesn't work we may want to think about disabling the problematic profile temporarily. On 23.05.2016 09:53, Ufuk Celebi wrote: Caches have been cleared again (see https://issues.apache.org/jira/browse/INFRA-11773) The first time did not help. This second request was more an act of desp

Re: Master test stability poor

make the builds slower on travis because we need to download more. On Mon, May 23, 2016 at 10:18 AM, Chesnay Schepler wrote: If this doesn't work we may want to think about disabling the problematic profile temporarily. On 23.05.2016 09:53, Ufuk Celebi wrote: Caches have been cleared

Re: Hotfixes on the master

I fully agree with Max, it just enables people to be sloppy and/or lazy without providing any good means of quality control. On 27.05.2016 12:10, Maximilian Michels wrote: Hi Flinksters, I'd like to address an issue that has been concerning me for a while. In the Flink community we like to pus

Re: [PROPOSAL] Structure the Flink Open Source Development

so are we discarding the other "shepherd" role then? On 31.05.2016 19:47, Robert Metzger wrote: Hi, to keep this discussion going, I pasted Stephan's Component proposal into the Wiki: https://cwiki.apache.org/confluence/display/FLINK/Components+and+Shepherds Also, I suggest to rename the "main

Re: [PROPOSAL] Structure the Flink Open Source Development

I'd also go with maintainer. On 01.06.2016 10:32, Aljoscha Krettek wrote: Hi, I think maintainer is also fine if we clearly specify that they are not meant as dictators or gatekeepers of the component that they are responsible for. -Aljoscha On Wed, 1 Jun 2016 at 09:48 Vasiliki Kalavri wrote:

Re: [PROPOSAL] Structure the Flink Open Source Development

ts we can also set a component Lead that would get notified of issues for that component. On Wed, 1 Jun 2016 at 10:43 Chesnay Schepler wrote: I'd also go with maintainer. On 01.06.2016 10:32, Aljoscha Krettek wrote: Hi, I think maintainer is also fine if we clearly specify that they are not

Re: Future of Python support

Hello Julius, I don't think there is any real roadmap for the Python API, regardless of batch or streaming. Of the top of my head i can think of the following issue: The batch Python API makes heavy use of MapPartitions to transfer data in batches, I'm not sure how well this could be done f

Re: Scheduling task slots in round-robin

FLINK-1003 may be related. On 13.06.2016 12:46, Gyula Fóra wrote: Hey, The Flink scheduling mechanism has become quite a bit of a pain lately for us when trying to schedule IO heavy streaming jobs. And by IO heavy I mean it has a fairly large state that is being continuously updated/read. The

Re: Testing for FLINK-3839

1: https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/building.html 2: Generally none, provided the test cases are good. Certain features require a test run on a cluster. 3: The code review will be done as part of the PR review. On 15.06.2016 19:50, THORMAN, ROBERT D wrote: I

Re: Testing for FLINK-3839

ons in IntelliJ. I keep failing the build on one format violation. On 6/15/16, 1:21 PM, "Chesnay Schepler" wrote: 1: https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/building.html 2: Generally none, provided the test cases are good. Certain features require a test

Re: Adding a Histogram Metric

That is currently not possible. We would have expose the internal addMetric(String name, Metric metric) method. Regards, Chesnay On 18.06.2016 04:48, Steve Cosenza wrote: Hi Till, How would I plugin a custom counter so that I could use the existing MetricGroup and AbstractReporter functional

Re: Adding a Histogram Metric

? -Steve On Friday, June 17, 2016, Chesnay Schepler <mailto:ches...@apache.org>> wrote: That is currently not possible. We would have expose the internal addMetric(String name, Metric metric) method. Regards, Chesnay On 18.06.2016 04:48, Steve Cosenza wrote:

Re: Python API design doc?

Hello Daniel, there is neither Design Doc for the Python API nor for how it interacts the the DataSet API. If you have any questions feel free to contact me, I'll provide as much information as i can. Regards, Chesnay On 29.06.2016 22:53, Daniel Li wrote: I am evaluating the feasibility o

Re: [DISCUSS] Releasing Flink 1.1.0

There are also 2 fixes for Cassandra that should be included: https://github.com/apache/flink/pull/2167 https://github.com/apache/flink/pull/2183 We should also include the documentation for the metrics stuff (hopefully merged today) https://github.com/apache/flink/pull/2158 In regards to met

Re: [DISCUSS] FLIP-1 : Fine grained recovery from task failures

shouldn't the proposal be contained in the wiki instead of GoogleDocs? On 12.07.2016 19:55, Stephan Ewen wrote: Hi all! Here is the very first FLIP (FLink Improvement Proposal): Fine grained recovery from task failures It describes a proposed enhancement for reducing the work done during recov

[DISCUSS] FLIP-7 Expose metrics to WebInterface

Hello, I just created a new FLIP which aims at exposing our metrics to the WebInterface. https://cwiki.apache.org/confluence/display/FLINK/FLIP-7%3A+Expose+metrics+to+WebInterface Looking forward to feedback :) Regards, Chesnay Schepler

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

s from the subtasks, for example in case you want a metric across subtasks. In general, not transferring objects (only strings / numbers) would be preferable, because the WebMonitor may run in an environment where no user-code classloader can be used. It may run in the dispatcher (which must be truste

Re: [DISCUSS] FLIP-7 Expose metrics to WebInterface

the Dumper be simply a special reporter that understands the component metric groups and does not use scope formats? On Tue, Aug 2, 2016 at 3:50 PM, Chesnay Schepler wrote: Thank you for your feedback :) Regarding names: The Dumper does not create a MetricSnapshot. The Dumper creates a

Re: Flink contributor list

Kalavri Michael Hopstock Sebastian Schelter Sebastian Kruse Gabor Herrmann Matthias Sax Moritz Kaufmann Philipp Richter Timo Walther Maximilian Michels Andra Lungu Henry Saputra Markus Holzemer Yingjun Wu Rico Bergmann Chesnay Schepler Minglian Qi Judit Feher Jonas Traub Mathias Peters Bjorn Lohrmann

Re: Types in the Python API

because it still goes through the Java API that requires some kind of type information. imagine a java api program where you omit all generic types, it just wouldn't work as of now. On 30.07.2015 21:17, Gyula Fóra wrote: Hey! Could anyone briefly tell me what exactly is the reason why we forc

Re: Types in the Python API

plications though. Gyula Chesnay Schepler ezt írta (időpont: 2015. júl. 30., Cs, 22:04): because it still goes through the Java API that requires some kind of type information. imagine a java api program where you omit all generic types, it just wouldn't work as of now. On 30.07.2015 21:17,

Re: Types in the Python API

). If proper sort order would be required this would have to be done on the python side. On Thu, 30 Jul 2015 at 22:21 Chesnay Schepler wrote: To be perfectly honest i never really managed to work my way through Spark's python API, it's a whole bunch of magic to me; not even the gener

Re: Types in the Python API

whole "key blob". Maybe it's overly simplistic, but it might work. :D On Thu, 30 Jul 2015 at 23:35 Chesnay Schepler wrote: I can see this working for basic types, but am unsure how it would work with Tuples. Wouldn't the java API still need to know the arity to setup serializer

Re: Types in the Python API

suggestion is a good one: have a PythonTypeInfo that is dynamic. Till' also found a pretty nice way to connect Python and Java in his Zeppelin-based demo at the meetup. On Fri, Jul 31, 2015 at 10:30 AM, Chesnay Schepler wrote: if its just a single array, how would you define group/sort ke

Re: Tuple

there's no specific reason. it was added fairly recently by me (mid of april), and you're most likely the second person to use it. i didn't integrate into all our tuple related stuff because, well, i never thought anyone would actually need it, so i saved myself the trouble. Hi, is there an

Re: Tuple

also, I'm not sure if I ever sent a Tuple0 through a program, it could be that the system freaks out. On 31.07.2015 22:40, Chesnay Schepler wrote: there's no specific reason. it was added fairly recently by me (mid of april), and you're most likely the second person to use

Re: Tuple

y IMHO. I will open a JIRA for it. -Matthias On 07/31/2015 10:44 PM, Chesnay Schepler wrote: also, I'm not sure if I ever sent a Tuple0 through a program, it could be that the system freaks out. On 31.07.2015 22:40, Chesnay Schepler wrote: there's no specific reason. it was added fair

Re: Tuple

Flink cannot deal with Tuple0 it should be removed completely for consistency IMHO. I will open a JIRA for it. -Matthias On 07/31/2015 10:44 PM, Chesnay Schepler wrote: also, I'm not sure if I ever sent a Tuple0 through a program, it could be that the system freaks out. On 31.07.2015 22:40, Che

Re: Failing Test again

Aren't we dropping java 6 support? On 04.08.2015 12:21, Stephan Ewen wrote: The "StateCheckpointedITCase" has not failed so far, which also test these guarantees thoroughly. But we need to first rule out the BarrierBuffer. The problem is that the bug occur only on Java 6 and cannot be reproduce

Re: Tuple

e discussion, I vote for keeping Tuple0 in Flink core, and fix the serialization problem. Should we have another JIRA for this? Or should I extend the existing JIRA? (https://issues.apache.org/jira/browse/FLINK-2457) -Matthias On 08/03/2015 12:22 AM, Chesnay Schepler wrote: First of all, it was a

Re: Tuple

should have happened. I will try a batch job, later today. On 08/04/2015 01:27 PM, Chesnay Schepler wrote: so I'm not to much into the streaming API, but as i see it this program creates an infinite number of tuples and then counts them, right? The problem with serialization as i understa

Re: Improve performance of call system.currentTimeMillis()

saying "no I'm right.", and not much else. And now we're here starting at square one. SO, my suggestion is this: You said you could test the performance, so go ahead. Then we can talk. Regards, a slightly disgruntled Chesnay Schepler On 12.08.2015 11:33, Fangfengbin wrote: Hel

Re: 答复: 答复: Some problems about Flink applications

How about something like this: DataStream> sink = user .join(area) .onWindow(15, TimeUnit.MINUTES) .where(0) .equalTo(0) .flatMap(new FlatMapFunction,Tuple2>, Tuple2>() { @Override public void flatMap(Tuple2,Tuple2> value, Co

Encountered "TypeComparator creation has a bug"

While running the following job i run into an exception: Tuple comparator creation has a bug java.lang.IllegalArgumentException: Tuple comparator creation has a bug at org.apache.flink.api.java.typeutils.TupleTypeInfo.getNewComparator(TupleTypeInfo.java:131) ... job: ExecutionEnvironme

Re: Pulling Streaming out of staging and project restructure

I like it in general. But while we're at it, what is the purpose of the flink-tests project, or rather which tests belong there instead of the individual projects? Where would new projects reside in, that previously would have been put into flink-staging? Lastly, I'd like to merge flink-lang

Re: Pulling Streaming out of staging and project restructure

etc). This would break maven dependencies (temporary hell :-) so it's not a decision to take lightly. I'm not strongly advocating for it. On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler wrote: I like it in general. But while we're at it, what is the purpose of the flink-tests

Re: [DISCUSS] Introducing a review process for pull requests

I think we can't do that since we don't have direct control over the github repository, similar to how we have to rely on the bot to close PR's. On 17.10.2015 12:32, Alexander Alexandrov wrote: One suggestion from me: in GitHub you can make clear who the current sheppard is through the "Assigne

Re: [DISCUSS] Python API for Fllink libraries

Hello, I would say no, as the general reception of the Python API was *tumbleweed* so far. In my opinion this would just lead to a massive increase in code to maintain; we would need at least 2-3 active long-term python contributors. Especially so since ML, CEP and Table are afaik still in h

Re: [DISCUSS] Python API for Fllink libraries

/FLINK-1926?jql= project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python% 20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER% 20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC Best regards, Ivan. On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler wrote: Hello, I would say no, as the gener

Re: [DISCUSS] Python API for Fllink libraries

ache.org/jira/browse/FLINK-1926?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python%20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC Best regards, Ivan. On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler wrote: Hello, I

Re: Performance and Latency Chart for Flink

It is normal that you don't see it in the WebInterface. FLINK-4389 was only about exposing metrics *to* the WebInterface, not exposing them *from* it. Essentially, a metric travels from TaskManager -> WebInterface -> User. FLINK-4389 was about the first arrow, which is a prerequisite step for

Re: Exception from in-progress implementation of Python API bulk iterations

Hello, I'll try to take a look this week. Regards, Chesnay On 20.09.2016 02:38, Geoffrey Mon wrote: Hello all, I have recently been working on adding bulk iterations to the Python API of Flink in order to facilitate a research project I am working on. The current changes can be seen in this G

Re: Get Flink ExecutionGraph Programmatically

Hello, this is a rather subtle issue you stumbled upon here. The ExecutionGraph is not serializable. The only reason why the WebInterface can access it is because it runs in the same JVM as the JobManager. I'm not sure if there is a way for what you are trying to do. Regards, Chesnay On 21

Re: Flink Accumulators vs Metrics

Hello, you are correct that metrics are not aggregated on the Task- or JobManager. Their primary purpose is to expose fine-grained information to an external metric system, which could then do aggregations if desired. There are plans to aggregate metrics in the WebInterface though, like the

Re: Flink JDBCOutputFormat - Flush last batch enhancement

The JDBCOutputFormat writes records in batches, that's what he is referring to. On 26.09.2016 11:48, Stephan Ewen wrote: Hi! I am not sure I understand what you want to do, but here are some comments: - There is no "batching" in Flink's streaming API, not sure what you are referring to in

Re: Exception from in-progress implementation of Python API bulk iterations

error? Also, which OS/python version are you using? Regards, Chesnay On 20.09.2016 11:13, Chesnay Schepler wrote: Hello, I'll try to take a look this week. Regards, Chesnay On 20.09.2016 02:38, Geoffrey Mon wrote: Hello all, I have recently been working on adding bulk iterations to the P

Re: Flink JDBCOutputFormat - Flush last batch enhancement

find jdbc connector from streaming connectors. If it is not there, may we have jdbc connector for streaming use cases? Thanks, Swapnil On Mon, Sep 26, 2016 at 3:32 PM, Chesnay Schepler wrote: The JDBCOutputFormat writes records in batches, that's what he is referring to. On 26.09.

Re: Flink JDBCOutputFormat - Flush last batch enhancement

* setting the batch interval _to 1_ On 26.09.2016 15:25, Chesnay Schepler wrote: Hello Swapnil, setting the batch interval should be pretty much equivalent to having a streaming jdbc connector. Regards, Chesnay On 26.09.2016 13:21, Swapnil Chougule wrote: Hi Stephen/Chesnay, I have used

Re: Exception from in-progress implementation of Python API bulk iterations

d to a new branch to make it easier to test and hopefully review soon: https://github.com/GEOFBOT/flink/tree/new-iterations Cheers, Geoffrey On Mon, Sep 26, 2016 at 6:32 AM Chesnay Schepler wrote: Hello Geoffrey, i could not reproduce this issue with the commits and plan you provided. I trie

Re: currentLowWatermark metric not reported for all tasks?

Hello Dan, the technical reason is that this metric is only collected in the *InputProcessor classes, which aren't used for source tasks. I do recall that there were discussions about source watermarks, but frankly i don't remember why we didn't add them. In order to add them one would only

Re: KafkaProducer can not be instantiated

if you were to move the partition list fetching out of the constructor int open(), is there any guarantee that for each fetching subtask the partition list is identical? On 05.10.2016 17:17, Tzu-Li (Gordon) Tai wrote: Hi Timo, I haven’t had the chance to look at the producer side too much yet

Re: Type problem in RichFlatMapFunction when using GenericArray type

Hello Martin, Could you include the error you are getting? Regards, Chesnay On 10.10.2016 13:31, Martin Junghanns wrote: Hi, I ran into a problem when using generic arrays in a tuple. I wrote a minimal program to reproduce the error [1]. The problem seems to be related to the order of tupl

Re: Type problem in RichFlatMapFunction when using GenericArray type

it once the issue is there. Cheers, Martin On 10.10.2016 13:39, Chesnay Schepler wrote: Hello Martin, Could you include the error you are getting? Regards, Chesnay On 10.10.2016 13:31, Martin Junghanns wrote: Hi, I ran into a problem when using generic arrays in a tuple. I wrote a

Re: Exception from in-progress implementation of Python API bulk iterations

n Wed, Sep 28, 2016 at 9:16 AM Chesnay Schepler wrote: Hello Geoffrey, this one works for me as well :D Regards, Chesnay On 28.09.2016 05:38, Geoffrey Mon wrote: Hello Chesnay, Thank you for your help. After receiving your message I recompiled my version of Flink completely, and bot

Re: Exception from in-progress implementation of Python API bulk iterations

A temporary work around appears to be disabling chaining, which you can do by commenting out L215 "self._find_chains()" in Environment.py. Note that you then run into a division by zero error, but i can't tell whether that is a problem of the job or not. On 13.10.2016 13:41, C

Re: Exception from in-progress implementation of Python API bulk iterations

d it is an important optimization that would be important for the performance comparison I wish to make in my project. Cheers, Geoffrey On Thu, Oct 13, 2016 at 9:11 AM Chesnay Schepler wrote: A temporary work around appears to be disabling chaining, which you can do by commenting out L215 "se

Re: Exception from in-progress implementation of Python API bulk iterations

In this branch: https://github.com/zentol/flink/tree/new-iterations you can find a more fine-grained fix for chaining with iterations. relevant commit: ac2305d9589a5c6ab9e94d04c870fba52716d695 On 13.10.2016 23:11, Chesnay Schepler wrote: The chaining code is definitely related, I also have a

Re: Efficient Batch Operator in Streaming

Could you not do the same thing today with a FlatMap function that stores incoming elements and only computes and collects a result when a certain threshold is reached? On 20.10.2016 09:50, Xiaowei Jiang wrote: Very often, it's more efficient to process a batch of records at once instead of pro

Re: Python API iteration issues

Hey Geoffrey, I believe what we got here is that multiple operators are sending data over the same channel. The data transfer between the java and python processes uses memory-mapped files. The file name is based on the ID of the set and the subtask index of the operator and thus determinist

Re: [DISCUSS] Releasing Flink 1.1.4

I've added the following fix to the 1.1 branch * [FLINK-4875] [metrics] Use correct operator name It is a crucial fix for streaming topologies that involve multi-chains. 2 users already ran into this. On 25.10.2016 14:43, Robert Metzger wrote: +1 for a bugfix release soon. On Tue, Oct 25, 20

Re: Python API iteration issues

down Flink because it relies on more cryptographically strong number generators. Would there be a significant risk of collision if we used a Random.nextInt() or the current UNIX timestamp? Cheers, Geoffrey On Sun, Oct 23, 2016 at 4:14 PM Chesnay Schepler wrote: Hey Geoffrey, I believe what we g

Re: Monitoring backpressure

Hello Alan, the backpressure information can be retrieved from the web ui's REST API . |/jobs//vertices//backpressure This will give you a JSON object that looks something like this: { | |status:"ok"|"d

Re: [Dev] Dependencies issue related to implementing InputFormat Interface

Hello, Did you write something like this? public class MyInputFormat implements InputFormat { } Regards, Chesnay On 17.01.2017 04:18, Pawan Manishka Gunarathna wrote: Hi, I'm currently working on Flink InputFormat Interface implementation. I'm writing a java program to rea

Re: [Dev] Dependencies issue related to implementing InputFormat Interface

ReadFromFile implements InputFormat{ } Is that a problem with that declaration or dependencies ? Thanks, Pawan On Tue, Jan 17, 2017 at 7:56 PM, Chesnay Schepler wrote: Hello, Did you write something like this? public class MyInputFormat implements InputFormat { } Regards

Re: GlobFilePathFilter NotSerializableException

Hello, this appears to be a bug in Flink. The problem is that the PathMatcher objects in the GlobFilePathFilter all contain a reference to the encapsulating class. The easiest solution to this would be to build the PathMatcher after they were shipped within the filterPath method. Since this

Re: clean up jira

fixed link: https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%201.2.0%20AND%20project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC Previous link lead to FLINK-5048. On 24.01.2017 16:33, Robert Metzger wrote: Hey friends of a clean JIRA, I've unm

Re: [VOTE] Release Apache Flink 1.2.0 (RC1)

Hello, FLINK-5612 might be a blocker as well; basically if you use the GlobFilePathFilter in a job, chances are it will fail. Also, the lastCheckpointSize metric that we collect for each task was broken when the key-groups were introduced. Regards, Chesnay On 25.01.2017 10:52, Stefan Richt

Re: [VOTE] Release Apache Flink 1.2.0 (RC1)

- One was added to the license file - One is ASL 2.0 (no license update needed) => One is MIT and reference in the License is missing I think we have a release blocker here :-( On Wed, Jan 25, 2017 at 11:12 AM, Chesnay Schepler wrote: Hello, FLINK-5612 might be a blocker as well; b

Re: [VOTE] Release Apache Flink 1.2.0 (RC1)

ommit/34e106f63c9dcd2673d66b47fda1555b7dced770 On Wed, Jan 25, 2017 at 1:48 PM, Chesnay Schepler wrote: Which dependency is MIT? On 25.01.2017 13:24, Stephan Ewen wrote: Did a License cross-check: All Maven Dependencies are okay. Added a small note on the RabbitMQ dependency'

Re: [DISCUSS] Code style / checkstyle

I agree with Till. I would propose enforcing checkstyle on a subset of the modules, basically those that are not flink-runtime, flink-java, flink-streaming-java. These are the ones imo where messing with the history can be detrimental; for the others it isn't really important imo. (Note that i

Re: [DISCUSS] Code style / checkstyle

nt style" I have some doubts if this is possible, as it changes file to file. Chesnay's suggestion as to were enforce the checkstyle seems reasonable to me, but I am quite new to the community :). Enabling checkstyle for particular packages is possible. 2017-02-22 12:07 GMT+01:00 Chesnay

Re: How to register and send custom metrics from Flink?

Hello, yes, Flink can handle custom metrics. It is quite odd that you can see the system metrics but not your own; I don't see a problem with your code. How long is the job running? Since you create a metric for the flatMap operation the metric will only be exposed as long as the flatMap oper

Re: [VOTE] Release Apache Flink 1.2.1 (RC1)

If possible I would like to include FLINK-6183 & FLINK-6184 as well. They fix 2 metric-related issues that could arise when a Task is cancelled very early. (like, right away) FLINK-6183 fixes a memory leak where the TaskMetricGroup was never closed FLINK-6184 fixes a NullPointerExceptions in t

Re: [VOTE] Release Apache Flink 1.2.1 (RC1)

allelism of 1, potentially due to FLINK-5808. Filed FLINK-6209 to track it. ~Haohui On Mon, Mar 27, 2017 at 2:59 AM Chesnay Schepler < ches...@apache.org> wrote: If possible I would like to include FLINK-6183 & FLINK-6184 as well. They fix 2 metric-related issues that could arise w

Re: [VOTE] Release Apache Flink 1.2.1 (RC1)

hat all jobs submitted via UI will have a parallelism of 1, potentially due to FLINK-5808. Filed FLINK-6209 to track it. ~Haohui On Mon, Mar 27, 2017 at 2:59 AM Chesnay Schepler < ches...@apache.org> wrote: If possible I would like to include FLINK-6183 & FLINK-6184 as well. They f

<    2   3   4   5   6   7   8   9   10   11   >