Re: [Discuss]: Adding Metrics to StreamingFileSink

Hello Till, Thanks a lot for the information. It makes a lot of sense to have generic sink based metrics. Some things which maybe useful for file system sinks is the number of files written as well. I am assuming that will be abstracted under number of records for something like a bulk writer ( mu

[jira] [Created] (FLINK-12542) Kafka011ITCase#testAutoOffsetRetrievalAndCommitToKafka test failed

vinoyang created FLINK-12542: Summary: Kafka011ITCase#testAutoOffsetRetrievalAndCommitToKafka test failed Key: FLINK-12542 URL: https://issues.apache.org/jira/browse/FLINK-12542 Project: Flink I

contributor permission

Hi, I want to contribute to Apache Flink. Would you please give me the contributor permission? My JIRA ID is wanghongbin thanks

[jira] [Created] (FLINK-12541) Add deploy a Python Flink job and session cluster on Kubernetes support.

sunjincheng created FLINK-12541: --- Summary: Add deploy a Python Flink job and session cluster on Kubernetes support. Key: FLINK-12541 URL: https://issues.apache.org/jira/browse/FLINK-12541 Project: Flink

[jira] [Created] (FLINK-12540) Kafka011ProducerExactlyOnceITCase#testExactlyOnceCustomOperator

vinoyang created FLINK-12540: Summary: Kafka011ProducerExactlyOnceITCase#testExactlyOnceCustomOperator Key: FLINK-12540 URL: https://issues.apache.org/jira/browse/FLINK-12540 Project: Flink Issu

Re: Job recovery with task manager restart

Hi Thomas, I have a sort of question regarding the class loader issue, as it seems interesting. My understanding is that at least user class loader is unregistered and re-registered (from/to library cache on TM) across task restart. If I understand it correctly, unregistered one should be GCed

[jira] [Created] (FLINK-12539) StreamingFileSink: Make the class extendable to customize for different usecases

Kailash Hassan Dayanand created FLINK-12539: --- Summary: StreamingFileSink: Make the class extendable to customize for different usecases Key: FLINK-12539 URL: https://issues.apache.org/jira/browse/FLINK-1

Job recovery with task manager restart

Hi, When a job fails and is recovered by Flink, task manager JVMs are reused. That can cause problems when the failed job wasn't cleaned up properly, for example leaving behind the user class loader. This would manifest in rising base for memory usage, leading to a death spiral. It would be good

[jira] [Created] (FLINK-12538) Network notifyDataAvailable() only called after getting a new buffer

Nico Kruber created FLINK-12538: --- Summary: Network notifyDataAvailable() only called after getting a new buffer Key: FLINK-12538 URL: https://issues.apache.org/jira/browse/FLINK-12538 Project: Flink

[jira] [Created] (FLINK-12537) Improve Documentation Build Time

Seth Wiesman created FLINK-12537: Summary: Improve Documentation Build Time Key: FLINK-12537 URL: https://issues.apache.org/jira/browse/FLINK-12537 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-12536) Make BufferOrEventSequence#getNext() non-blocking

Piotr Nowojski created FLINK-12536: -- Summary: Make BufferOrEventSequence#getNext() non-blocking Key: FLINK-12536 URL: https://issues.apache.org/jira/browse/FLINK-12536 Project: Flink Issue T

[jira] [Created] (FLINK-12535) Make CheckpointBarrierHandler non-blocking

Piotr Nowojski created FLINK-12535: -- Summary: Make CheckpointBarrierHandler non-blocking Key: FLINK-12535 URL: https://issues.apache.org/jira/browse/FLINK-12535 Project: Flink Issue Type: Su

Re: [DISCUSS] FLIP-39: Flink ML pipeline and ML libs

Hi, I had a look at the document mostly from a module structure/dependency structure perspective. We should make the expected dependency structure explicit in the document. From the discussion in the doc it seems that the intention is that flink-ml-lib should depend on flink-table-planner (the

[jira] [Created] (FLINK-12534) Reduce the test cost for Python API

sunjincheng created FLINK-12534: --- Summary: Reduce the test cost for Python API Key: FLINK-12534 URL: https://issues.apache.org/jira/browse/FLINK-12534 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Proposal for Flink job execution/availability metrics impovement

On 16/05/2019 11:34, Piotr Nowojski wrote: Luckily it seems like those four issues/proposals could be implemented/discussed independently or in stages. I fully agree, and believe we should split this thread. We will end up discussing too many issues at once. Nevertheless, On 16/05/2019 11:34

[jira] [Created] (FLINK-12533) Introduce TABLE_AGGREGATE_FUNCTION FunctionDefinition.Type

Hequn Cheng created FLINK-12533: --- Summary: Introduce TABLE_AGGREGATE_FUNCTION FunctionDefinition.Type Key: FLINK-12533 URL: https://issues.apache.org/jira/browse/FLINK-12533 Project: Flink Issu

Re: [Discuss]: Adding Metrics to StreamingFileSink

Hi Kailash, have you seen FLIP-33 [1] and the corresponding ML thread [2]. The scope of this improvement proposal is to extend the set of standard metrics a connector should offer. Maybe this can already solve your problem. Concerning your second proposal for the StreamingFileSink, I think this s

[jira] [Created] (FLINK-12532) Upgrade Avro to version 1.9.0

Ismaël Mejía created FLINK-12532: Summary: Upgrade Avro to version 1.9.0 Key: FLINK-12532 URL: https://issues.apache.org/jira/browse/FLINK-12532 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Proposal for Flink job execution/availability metrics impovement

Hi Hwanju, Thanks for starting the discussion. Definitely any improvement in this area would be very helpful and valuable. Generally speaking +1 from my side, as long as we make sure that either such changes do not add performance overhead (which I think they shouldn’t) or they are optional.

[jira] [Created] (FLINK-12531) flink sql-client throw NoMatchingTableFactoryException

leishuiyu created FLINK-12531: - Summary: flink sql-client throw NoMatchingTableFactoryException Key: FLINK-12531 URL: https://issues.apache.org/jira/browse/FLINK-12531 Project: Flink Issue Type:

[jira] [Created] (FLINK-12530) Move Task.inputGatesById to NetworkEnvironment

Andrey Zagrebin created FLINK-12530: --- Summary: Move Task.inputGatesById to NetworkEnvironment Key: FLINK-12530 URL: https://issues.apache.org/jira/browse/FLINK-12530 Project: Flink Issue Ty

Re: [VOTE] Release flink-shaded 7.0, release candidate 1

Hi All, Currently, https://issues.apache.org/jira/browse/FLINK-11580 has merged by @Chesnay Schepler ! Great thanks! And I have opened the PR(https://github.com/apache/flink-shaded/pull/62 ) for FLINK-12515 . I appreciate if you can review the c

[jira] [Created] (FLINK-12529) Release buffers of the record deserializer timely to improve the efficiency of heap memory usage on taskmanager

Haibo Sun created FLINK-12529: - Summary: Release buffers of the record deserializer timely to improve the efficiency of heap memory usage on taskmanager Key: FLINK-12529 URL: https://issues.apache.org/jira/browse/FLIN

[jira] [Created] (FLINK-12528) Remove progressLock in ExecutionGraph

vinoyang created FLINK-12528: Summary: Remove progressLock in ExecutionGraph Key: FLINK-12528 URL: https://issues.apache.org/jira/browse/FLINK-12528 Project: Flink Issue Type: Sub-task

Re: Applying for permission as a contributor

Done. On 14/05/2019 08:16, Wei Sun wrote: Hi Guys, I want to contribute to Apache Flink. Would you please give me the permission as a contributor? My JIRA ID is *Andrew Sun* Best Regards, Andrew

[jira] [Created] (FLINK-12527) Remove GLOBAL_VERSION_UPDATER in ExecutionGraph

vinoyang created FLINK-12527: Summary: Remove GLOBAL_VERSION_UPDATER in ExecutionGraph Key: FLINK-12527 URL: https://issues.apache.org/jira/browse/FLINK-12527 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12526) Remove STATE_UPDATER in ExecutionGraph

vinoyang created FLINK-12526: Summary: Remove STATE_UPDATER in ExecutionGraph Key: FLINK-12526 URL: https://issues.apache.org/jira/browse/FLINK-12526 Project: Flink Issue Type: Sub-task

Re: User Interface not showing the actual count received and produced

Hi Shakir, This is a frequently reported issue in Flink's metrics collection / UI. Send and received records and bytes only include data that is shared between Flink tasks but not between a source system (Kafka) and Flink or Flink and a sink system (Kinesis). IIRC, there is an effort to fix this p

Re: FW: Apply for a Contributor permission

Done. On 16/05/2019 08:51, Zhou, Brian wrote: My JIRA ID: Brian Zhou Best Regards, Brian From: Zhou, Brian Sent: Thursday, May 16, 2019 13:39 To: dev@flink.apache.org Subject: Apply for a Contributor permission Hi, I want to contribute to Apache Flink, starting from Chinese translation. Woul

Re: grant jira permission

Done. On 16/05/2019 08:15, Charoes wrote: Hello, I want to contribute to flink project Please grant me the jira permission. My Jira ID: charoes Thanks Charoes

Re: [DISCUSS] Clean up and reorganize the JIRA components

Just to clarify, by adding benchmark component I meant just admitting that we have some benchmarks both in the flink and flink-benchmarks repositories, and additional support infrastructure (machine executing benchmarks + Jenkins and Codespeed service) and to assign ownership of those components