Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi Shaoxuan, I really appreciate your prompt reply. What you explained makes sense to me. There is only one point that I got some different ideas about "we have to buffer all the delta data between watermarks of two inputs". Consider the following SQL on joining two streams l and r: SELECT * FR

[jira] [Created] (FLINK-7330) Unable to read file on windows - InvalidFilePathException

Mohit Anchlia created FLINK-7330: Summary: Unable to read file on windows - InvalidFilePathException Key: FLINK-7330 URL: https://issues.apache.org/jira/browse/FLINK-7330 Project: Flink Issue

[jira] [Created] (FLINK-7329) Unable to use FileInputStream with StreamExecutionEnvironment.readFile with interval

Mohit Anchlia created FLINK-7329: Summary: Unable to use FileInputStream with StreamExecutionEnvironment.readFile with interval Key: FLINK-7329 URL: https://issues.apache.org/jira/browse/FLINK-7329 Pr

Re: Class Cache

We have also noticed such behaviour when running on Yarn, had to restart the session for the changes in the jar to be picked up. On Mon, 31 Jul 2017 at 17:13, Ufuk Celebi wrote: > Hey Mike! > > Thanks for the detailed information about your setup. I'm also puzzled > by this... > > (1) Which vers

Re: Class Cache

Hey Mike! Thanks for the detailed information about your setup. I'm also puzzled by this... (1) Which version of Flink are you using? We recently merged some changes to the JAR distribution components, which might cause this (although I think that's unlikely). (2) As a temporary work around you

Re: Class Cache

Hi Mike! Flink does in fact cache jar files in the "blob server". But these are cached subject to the following conditions: - No caching across "sessions", meaning start/stop of the cluster/jobmanager. If you run the per-job-yarn setup, the job does not cache anything. - Files are cached und

[jira] [Created] (FLINK-7328) Remove Flink's futures from SlotManager

Till Rohrmann created FLINK-7328: Summary: Remove Flink's futures from SlotManager Key: FLINK-7328 URL: https://issues.apache.org/jira/browse/FLINK-7328 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-7327) Remove Flink's futures from StreamRecordQueueEntry

Till Rohrmann created FLINK-7327: Summary: Remove Flink's futures from StreamRecordQueueEntry Key: FLINK-7327 URL: https://issues.apache.org/jira/browse/FLINK-7327 Project: Flink Issue Type:

Re: Class Cache

No, I did not explicitly create an uber-jar. The mylib1.jar is very light. It only contains my main application class (including ProcessFunction). I have been specifying --classpath option on my flink run command to pull in the mylib2.jar . Plus, I have been rebuilding mylib1.jar frequentl

Re: Class Cache

A Flink program is typically packaged as an 'uber-jar' containing its dependencies. The Flink quickstart project illustrates this (see the use of the shading plugin in pom.xml). Based on your description, the classes of mylib2.jar were copied into mylib1.jar when the latter was built. Try rebui

Class Cache

Are classes cached somewhere in flink? I am running in a very basic, local environment on Linux (start_local.sh). I've somehow gotten my environment into a strange state that I don't understand. I feel like I am overlooking something simple, but I've checked everything I can think of. My ma

[jira] [Created] (FLINK-7326) Remove Flink's futures from RegisteredRpcConnection

Till Rohrmann created FLINK-7326: Summary: Remove Flink's futures from RegisteredRpcConnection Key: FLINK-7326 URL: https://issues.apache.org/jira/browse/FLINK-7326 Project: Flink Issue Type:

[jira] [Created] (FLINK-7325) Remove Flink's futures from MiniCluster

Till Rohrmann created FLINK-7325: Summary: Remove Flink's futures from MiniCluster Key: FLINK-7325 URL: https://issues.apache.org/jira/browse/FLINK-7325 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-7324) Remove Flink's futures from SlotPool

Till Rohrmann created FLINK-7324: Summary: Remove Flink's futures from SlotPool Key: FLINK-7324 URL: https://issues.apache.org/jira/browse/FLINK-7324 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-7323) Remove Flink's futures from MasterHooks

Till Rohrmann created FLINK-7323: Summary: Remove Flink's futures from MasterHooks Key: FLINK-7323 URL: https://issues.apache.org/jira/browse/FLINK-7323 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-7322) Remove Flink's futures from CheckpointCoordinator

Till Rohrmann created FLINK-7322: Summary: Remove Flink's futures from CheckpointCoordinator Key: FLINK-7322 URL: https://issues.apache.org/jira/browse/FLINK-7322 Project: Flink Issue Type: S

[jira] [Created] (FLINK-7321) Remove Flink's futures from HeartbeatManager

Till Rohrmann created FLINK-7321: Summary: Remove Flink's futures from HeartbeatManager Key: FLINK-7321 URL: https://issues.apache.org/jira/browse/FLINK-7321 Project: Flink Issue Type: Sub-ta

[jira] [Created] (FLINK-7320) Remove Flink's futures from Scheduler

Till Rohrmann created FLINK-7320: Summary: Remove Flink's futures from Scheduler Key: FLINK-7320 URL: https://issues.apache.org/jira/browse/FLINK-7320 Project: Flink Issue Type: Sub-task

dev@flink.apache.org

+1 to support this change, as it makes the sql API more accurate and elegant. I hope this will not introduce too much troubles in the release upgrading for the existing flink SQL users. On Mon, Jul 31, 2017 at 3:42 PM, Fabian Hueske wrote: > Hi Lincoln, > > thank you for this proposal and discu

[jira] [Created] (FLINK-7319) Remove Flink's futures from MesosResourceManager

Till Rohrmann created FLINK-7319: Summary: Remove Flink's futures from MesosResourceManager Key: FLINK-7319 URL: https://issues.apache.org/jira/browse/FLINK-7319 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-7318) Remove Flink's futures from StackTraceSampleCoordinator

Till Rohrmann created FLINK-7318: Summary: Remove Flink's futures from StackTraceSampleCoordinator Key: FLINK-7318 URL: https://issues.apache.org/jira/browse/FLINK-7318 Project: Flink Issue T

[jira] [Created] (FLINK-7317) Remove Flink's futures from ExecutionGraph

Till Rohrmann created FLINK-7317: Summary: Remove Flink's futures from ExecutionGraph Key: FLINK-7317 URL: https://issues.apache.org/jira/browse/FLINK-7317 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Table API / SQL internal timestamp handling

Xingcan, Watermark is the “estimate of completion”. User defines the waterMark based on the best estimation per each input of when it pretty much sees all the data. It is usually calculated by the event timestamp. When we do a windowed join, we have to make sure the watermark for both inputs are re

Re: KeyBy fields do not support java.lang.Array?

Hi, Which version of Flink are you using? This issue should have been resolved at least by 1.3.0: https://issues.apache.org/jira/browse/FLINK-5874 . Currently such keys should be rejected. There is also this issue, that aims to re-introduce prop

[jira] [Created] (FLINK-7316) always use off-heap network buffers

Nico Kruber created FLINK-7316: -- Summary: always use off-heap network buffers Key: FLINK-7316 URL: https://issues.apache.org/jira/browse/FLINK-7316 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-7315) use flink's buffers in netty

Nico Kruber created FLINK-7315: -- Summary: use flink's buffers in netty Key: FLINK-7315 URL: https://issues.apache.org/jira/browse/FLINK-7315 Project: Flink Issue Type: Improvement Comp

Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi Fabian, I got a similar question with Jark. Theoretically, the row times of two streams could be quite difference, e.g., one for today and the other one for yesterday. How can we align them? Best, Xingcan On Mon, Jul 31, 2017 at 9:04 PM, Fabian Hueske wrote: > Hi Jark, > > yes, the handling

[jira] [Created] (FLINK-7313) Add conversion utilities (Scala and old Flink futures)

Till Rohrmann created FLINK-7313: Summary: Add conversion utilities (Scala and old Flink futures) Key: FLINK-7313 URL: https://issues.apache.org/jira/browse/FLINK-7313 Project: Flink Issue Ty

[jira] [Created] (FLINK-7314) Remove Flink's futures from TaskManagerLogHandler

Till Rohrmann created FLINK-7314: Summary: Remove Flink's futures from TaskManagerLogHandler Key: FLINK-7314 URL: https://issues.apache.org/jira/browse/FLINK-7314 Project: Flink Issue Type: S

[jira] [Created] (FLINK-7312) activate checkstyle for flink/core/memory/*

Nico Kruber created FLINK-7312: -- Summary: activate checkstyle for flink/core/memory/* Key: FLINK-7312 URL: https://issues.apache.org/jira/browse/FLINK-7312 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-7311) refrain from using fail(Exception#getMessage()) in core memory tests

Nico Kruber created FLINK-7311: -- Summary: refrain from using fail(Exception#getMessage()) in core memory tests Key: FLINK-7311 URL: https://issues.apache.org/jira/browse/FLINK-7311 Project: Flink

[jira] [Created] (FLINK-7310) always use HybridMemorySegment

Nico Kruber created FLINK-7310: -- Summary: always use HybridMemorySegment Key: FLINK-7310 URL: https://issues.apache.org/jira/browse/FLINK-7310 Project: Flink Issue Type: Improvement Co

[jira] [Created] (FLINK-7309) NullPointerException in CodeGenUtils.timePointToInternalCode() generated code

Liangliang Chen created FLINK-7309: -- Summary: NullPointerException in CodeGenUtils.timePointToInternalCode() generated code Key: FLINK-7309 URL: https://issues.apache.org/jira/browse/FLINK-7309 Proje

KeyBy fields do not support java.lang.Array?

Hi Aljoscha： The java.lang.Array hashCode depends on the reference instead of the content. If the keyBy field contains an array, Two records are hash-partitioned to different stream although their keys are equal. int a1[] = new int[]{1, 2}; // hashcode is : 55

Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi Jark, yes, the handling of watermarks is very tricky. It is not directly related to the proposal which is only about the representation of timestamps but becomes important for event-time joins. We have a JIRA about an operator that is able to hold back watermarks [1]. Roughly the idea is to tr

Re: [DISCUSS] Table API / SQL internal timestamp handling

Hi, @Fabian, I read your proposal carefully again, and I'm big +1 to do it. The proposal can address the problem of that how to forward both input tables' rowtime of dual stream join (windowed/non-windowed). The additional payload drawback is acceptable. You mentioned that: > The query operators

[jira] [Created] (FLINK-7308) Kafka end to end test unstable

Till Rohrmann created FLINK-7308: Summary: Kafka end to end test unstable Key: FLINK-7308 URL: https://issues.apache.org/jira/browse/FLINK-7308 Project: Flink Issue Type: Bug Compon

[jira] [Created] (FLINK-7307) Add proper command line parsing tool to ClusterEntrypoint

Till Rohrmann created FLINK-7307: Summary: Add proper command line parsing tool to ClusterEntrypoint Key: FLINK-7307 URL: https://issues.apache.org/jira/browse/FLINK-7307 Project: Flink Issue

[jira] [Created] (FLINK-7306) function notFollowedBy in CEP dont return a Pattern object

Hanmiao Li created FLINK-7306: - Summary: function notFollowedBy in CEP dont return a Pattern object Key: FLINK-7306 URL: https://issues.apache.org/jira/browse/FLINK-7306 Project: Flink Issu

Re: [ANNOUNCE] New Flink PMC member: Chesnay Schepler

Congratulations Chesnay! 2017-07-31 17:44 GMT+08:00 Dawid Wysakowicz : > Congrats! > > > On 31 Jul 2017, at 11:43, Till Rohrmann wrote: > > > > Congrats Chesnay :-) > > > > On Sun, Jul 30, 2017 at 1:34 AM, jincheng sun > > wrote: > > > >> Congrats Chesnay! > >> > >> > >> > >> 2017-07-30 1:43 GM

[jira] [Created] (FLINK-7305) Add new import block for shaded dependencies

Chesnay Schepler created FLINK-7305: --- Summary: Add new import block for shaded dependencies Key: FLINK-7305 URL: https://issues.apache.org/jira/browse/FLINK-7305 Project: Flink Issue Type:

[jira] [Created] (FLINK-7304) Simnplify garbage collector configuration in taskmanager.sh

Chesnay Schepler created FLINK-7304: --- Summary: Simnplify garbage collector configuration in taskmanager.sh Key: FLINK-7304 URL: https://issues.apache.org/jira/browse/FLINK-7304 Project: Flink

[jira] [Created] (FLINK-7303) Build elasticsearch5 by default

Chesnay Schepler created FLINK-7303: --- Summary: Build elasticsearch5 by default Key: FLINK-7303 URL: https://issues.apache.org/jira/browse/FLINK-7303 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Flink PMC member: Chesnay Schepler

Congrats! > On 31 Jul 2017, at 11:43, Till Rohrmann wrote: > > Congrats Chesnay :-) > > On Sun, Jul 30, 2017 at 1:34 AM, jincheng sun > wrote: > >> Congrats Chesnay! >> >> >> >> 2017-07-30 1:43 GMT+08:00 Shaoxuan Wang : >> >>> Congratulations Chesnay! >>> >>> On Fri, Jul 28, 2017 at 10:0

Re: [ANNOUNCE] New Flink PMC member: Chesnay Schepler

Congrats Chesnay :-) On Sun, Jul 30, 2017 at 1:34 AM, jincheng sun wrote: > Congrats Chesnay! > > > > 2017-07-30 1:43 GMT+08:00 Shaoxuan Wang : > > > Congratulations Chesnay! > > > > On Fri, Jul 28, 2017 at 10:05 PM, Greg Hogan wrote: > > > > > Developers, > > > > > > On behalf of the Flink PMC

[jira] [Created] (FLINK-7302) Failed to run CorrelateITCase class under windows environment

mingleizhang created FLINK-7302: --- Summary: Failed to run CorrelateITCase class under windows environment Key: FLINK-7302 URL: https://issues.apache.org/jira/browse/FLINK-7302 Project: Flink Is

[jira] [Created] (FLINK-7301) Rework state documentation

Timo Walther created FLINK-7301: --- Summary: Rework state documentation Key: FLINK-7301 URL: https://issues.apache.org/jira/browse/FLINK-7301 Project: Flink Issue Type: Improvement Comp

dev@flink.apache.org

Hi Lincoln, thank you for this proposal and discussing the motivation for this change. I think this makes a lot of sense (as you said, we discussed this before). I'd like to highlight the breaking change (among a several non-breaking changes) proposed here: We propose to deprecate TableEnvironme