Is Flink 1.15.3 planned in foreseeable future?

Hello, I suppose that committers are heavily concentrated on 1.16, but are there plans to have 1.15.3 out? We've been affected by https://issues.apache.org/jira/browse/FLINK-28488 and it's preventing us from using 1.15.x at this moment. thanks, maciek

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

Hi Martin, Could you please remind what was the conclusion of discussion on upgrading Scala to 2.12.15/16? https://lists.apache.org/thread/hwksnsqyg7n3djymo7m1s7loymxxbc3t - I couldn't find any follow-up vote? If it's acceptable to break binary compatibility by such an upgrade, then upgradi

Re: Slow Tests in Flink 1.15

Hi, we also had similar problems in Nussknacker recently (tests on fake sources), my colleague found out it's due to ENABLE_CHECKPOINTS_AFTER_FINISH flag (https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/#waiting-for-the-final-checkpoint-be

Kafka connector depending on Table API

Hello, we are testing 1.14 RC0 and we discovered that we need to include table-api as dependency when using kafka connector - e.g. due to this change: https://github.com/apache/flink/blame/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/

Flink 1.11.4?

Hello, I'd like to ask if there are any plans to release 1.11.4 - I understand it will be last bugfix release for 1.11.x branch, as 1.13.0 is "just round the corner"? There are a few fixes we'd like to use - e.g. https://issues.apache.org/jira/browse/FLINK-9844, https://issues.apache.org/ji

Re: 回复： period batch job lead to OutOfMemoryError: Metaspace problem

uot; ' to flink-config, but problem still exist. Is there lightweight way to put clickhouse JDBC driver on Flink lib/ folder? ------ 原始邮件 -- *发件人:* "Maciek Próchniak" mailto:m...@touk.pl>>; *发送时间:* 2021年4月9日(星期五) 凌晨3:24 *收件人:

Re: 回复： period batch job lead to OutOfMemoryError: Metaspace problem

Hi, Did you put the clickhouse JDBC driver on Flink main classpath (in lib folder) and not in user-jar - as described here: https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/debugging/debugging_classloading.html#unloading-of-dynamically-loaded-classes-in-user-code? When we enco

Re: Flink does not cleanup some disk memory after submitting jar over rest

Hi, don't know if this is the problem you're facing, but some time ago we encountered two issues connected to REST API and increased disk usage after each submission: https://issues.apache.org/jira/browse/FLINK-21164 https://issues.apache.org/jira/browse/FLINK-9844 - they're closed ATM, but

Re: Future of QueryableState

end of life" in the sense that there is no active development on that component anymore. Best, Konstantin On Tue, Mar 9, 2021 at 7:08 AM Maciek Próchniak <mailto:m...@touk.pl>> wrote: Hello, We are using QueryableState in some of Nussknacker deployments as a

Future of QueryableState

Hello, We are using QueryableState in some of Nussknacker deployments as a nice addition, allowing end users to peek inside job state for a given key (we mostly use custom operators). Judging by mailing list and feature radar proposition by Stephan: https://github.com/StephanEwen/flink-web

Re: JobManager seems to be leaking temporary jar files

place to put is probably JarRunHandler#handleRequest, within handle after the job was run. A similar issue also exists in the JarPlanHandler. I've opened https://issues.apache.org/jira/browse/FLINK-21164 to fix this issue. On 1/26/2021 12:21 PM, Maciek Próchniak wrote: Hi Matthias, I think

Re: JobManager seems to be leaking temporary jar files

untime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L797 <https://github.com/apache/flink/blob/2c4e0ab921ccfaf003073ee50faeae4d4e4f4c93/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L797> On Mon, Jan 25, 2021 at 8:37 PM Maciek Pró

JobManager seems to be leaking temporary jar files

Hello, in our setup we have: - Flink 1.11.2 - job submission via REST API (first we upload jar, then we submit multiple jobs with it) - additional jars embedded in lib directory of main jar (this is crucial part) When we submit jobs this way, Flink creates new temp jar files via Packaged

Re: Partitioned tables in SQL client configuration.

elp you with this problem. Cheers, Till On Tue, Dec 1, 2020 at 6:51 PM Maciek Próchniak mailto:m...@touk.pl>> wrote: Hello, I try to configure SQL Client to query partitioned ORC data on local filesystem. I have directory structure like that:

Partitioned tables in SQL client configuration.

Hello, I try to configure SQL Client to query partitioned ORC data on local filesystem. I have directory structure like that: /tmp/table1/startdate=2020-11-28 /tmp/table1/startdate=2020-11-27 etc. If I run SQL Client session and create table by hand: create table tst (column1 string, star

Re: Conversion of Table (Blink/batch) to DataStream

a hack). Best, Jark [1]: https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L419 On Sat, 4 Apr 2020 at 15:42, Maciek Próchniak <mailto:m...@touk.pl>> wrote: Hello, I&#

Re: Dynamic Flink SQL

Hi Krzysiek, the idea is quite interesting - although maintaining some coordination to be able to handle checkpoints would probably pretty tricky. Did you figure out how to handle proper distribution of tasks between TMs? As far as I understand you have to guarantee that all sources reading fr

Conversion of Table (Blink/batch) to DataStream

Hello, I'm playing around with Table/SQL API (Flink 1.9/1.10) and I was wondering how I can do the following: 1. read batch data (e.g. from files) 2. sort them using Table/SQL SortOperator 3. perform further operations using "normal" DataStream API (treating my batch as finite stream) - to

queryable state API

Hello, Currently (1.4) to be able to use queryable state client has to know ip of (working) task manager and port. This is a bit awkward - as it forces external services to know details of flink cluster. Event more complex when we define port range for queryable state proxy and we're not sure

Re: flowable <-> flink integration

Hi Martin, I did some activiti development so your mail caught my attention :) I don't think I understand what are you trying to achieve - where is process you're simulating, where is simulation running and where is place for Flink. Do you want to invoke Flink (batch job I suppose?) from Flo

Re: Checkpoint was declined (tasks not ready)

gt;(Task.java:702) atjava.lang.Thread.run <http://java.lang.thread.run/>(Thread.java:745) On 23/10/2017 13:54, Maciek Próchniak wrote: we also have similar problem - it happens really often when we invoke async operators (ordered one). But we also observe that job is not starting properly

Re: Checkpoint was declined (tasks not ready)

we also have similar problem - it happens really often when we invoke async operators (ordered one). But we also observe that job is not starting properly - we don't process any data when such problems appear we'll keep you posted if we manage to find exact cause... thanks, maciek On 09/10/2

Re: ANN: TouK Nussknacker - designing Flink processes with GUI

aring this nice tool maciek. Does it handle both batch and Streaming? Is it able to visualize also an existing Flink program? Best, Flavio On Mon, Sep 4, 2017 at 3:03 PM, Maciek Próchniak <mailto:m...@touk.pl>> wrote: Hello, we would like to announce availability of TouK

ANN: TouK Nussknacker - designing Flink processes with GUI

flink-made-easier-for-analysts-and-business/ I'll also be talking about Nussknacker next week at Flink Forward - https://berlin.flink-forward.org/kb_sessions/touk-nussknacker-creating-flink-jobs-with-gui/ - hope to see you there :) thanks, maciek próchniak TouK

Re: Running job in "dry mode"?

On 07/06/2017 10:27, Maciek Próchniak wrote: On 07/06/2017 10:07, Tzu-Li (Gordon) Tai wrote: Hi Maciek, Is there any particular reason why you do not wish to start running the Kafka sources on the test run? Otherwise, it would be perfectly fine to start the test job for testing to see

Running job in "dry mode"?

Hello, I'd like to be able to see if new version of my job is compatible with the old one. I can make a savepoint and run new version from that, but I'd like to be able to do it without actually starting sources and so on - so that e.g. it won't start to read from my kafka topics. Of cours

Re: Graphite reporter recover from broken pipe

Starting with flink 1.2 it's possible to use UDP transport for graphite - I think it can be good workaround if you can listen on UDP port on your graphite installation thanks, maciek On 01/02/2017 13:22, Philipp Bussche wrote: Hi there, after moving my graphite service to another host my ta

accessing flink HA cluster with scala shell/zeppelin notebook

Hi, I have standalone Flink cluster configured with HA setting (i.e. with zookeeper recovery). How should I access it remotely, e.g. with Zeppelin notebook or scala shell? There are settings for host/port, but with HA setting they are not fixed - if I check which is *current leader* host and

Re: ContinuousFileMonitoringFunction - deleting file after processing

ckpoint was successfully performed and then we can purge the already processed files. This can be a good solution. Thanks, Kostas On Oct 18, 2016, at 9:40 AM, Maciek Próchniak wrote: Hi, we want to monitor hdfs (or local) directory, read csv files that appear and after successful processing

Re: window-like use case

4/06/cep-monitoring.html [2]: https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/cep.html -Ursprüngliche Nachricht- Von: Maciek Próchniak [mailto:m...@touk.pl <mailto:m...@touk.pl>] Gesendet: Freitag, 23. September 2016 10:36 An: user@flink.apache.org

Re: ContinuousFileMonitoringFunction - deleting file after processing

nd then we can purge the already processed files. This can be a good solution. Thanks, Kostas On Oct 18, 2016, at 9:40 AM, Maciek Próchniak wrote: Hi, we want to monitor hdfs (or local) directory, read csv files that appear and after successful processing - delete them (mainly not to run o

ContinuousFileMonitoringFunction - deleting file after processing

Hi, we want to monitor hdfs (or local) directory, read csv files that appear and after successful processing - delete them (mainly not to run out of disk space...) I'm not quite sure how to achieve it with current implementation. Previously, when we read binary data (unsplittable files) we m

window-like use case

Hi, in our project we're dealing with a stream of billing events. Each has customerId and charge amount We want to have a process that will trigger event (alarm) when sum of charges for customer during last 4 hours exceeds certain threshold, say - 10. The triggered event should contain data fr

Re: Enriching events with data from external http resources

Hi Ufuk, thanks for info - this is good news :) maciek On 16/08/2016 12:16, Ufuk Celebi wrote: On Mon, Aug 15, 2016 at 8:52 PM, Maciek Próchniak wrote: I know it's not really desired way of using flink and that it would be better to keep data as state inside stream and have it updat

Enriching events with data from external http resources

Hi, Our data streams do some filtering based on data from external http resources (not maintained by us, they're really fast with redis as storage). So far we did that by just invoking synchronously some http client in map/flatMap operations. It works without errors but it seems somehow inef

Re: checkpoints not being removed from HDFS

automatically? Sorry for confustion and thanks for help thanks, maciek On 12/05/2016 21:28, Maciek Próchniak wrote: thanks, I'll try to reproduce it in some test by myself... maciek On 12/05/2016 18:39, Ufuk Celebi wrote: The issue is here: https://issues.apache.org/jira/browse/FLINK-

Re: checkpoints not being removed from HDFS

ingle state handle, which works well. I will open an issue for this and try to reproduce it reliably and then fix it. – Ufuk On Thu, May 12, 2016 at 10:28 AM, Maciek Próchniak wrote: Hi, we have stream job with quite large state (few GB), we're using FSStateBackend and we're s

checkpoints not being removed from HDFS

Hi, we have stream job with quite large state (few GB), we're using FSStateBackend and we're storing checkpoints in hdfs. What we observe is that v. often old checkpoints are not discarded properly. In hadoop logs I can see: 2016-05-10 12:21:06,559 INFO BlockStateChange: BLOCK* addToInvalidat

Re: Streaming job software update

Hi, in our more-or-less development environment we're doing sth like that in our main method: val processName = name_of_our_stream val configuration = GlobalConfiguration.getConfiguration val system = JobClient.startJobClientActorSystem(configuration) val timeout = FiniteDura

Re: multi-application correlated savepoints

hmm... quite interesting question. But I think I don't fully understand your use case - how are your applications coupled? Through kafka topics? E.g. output of one is input for other? Or do they consume from same input? And why exactly do you want to get back to specific point in all of them?

Re: Threads waiting on LocalBufferPool

rs, Aljoscha On Thu, 21 Apr 2016 at 16:41 Maciek Próchniak <mailto:m...@touk.pl>> wrote: Well... I found some time to look at rocksDB performance. It takes around 0.4ms to lookup value state and 0.12ms to update - these are means, 95th percentile was > 1ms for get... W

Re: Threads waiting on LocalBufferPool

, maciek On 21/04/2016 08:41, Maciek Próchniak wrote: Hi Ufuk, thanks for quick reply. Actually I had a little time to try both things. 1) helped only temporarily - it just took a bit longer to saturate the pool. After few minutes, periodically all kafka threads were waiting for bufferPool.

Re: Threads waiting on LocalBufferPool

you could also work with the FileSystemStateBackend, which keeps state in memory (on-heap) and writes checkpoints to files. This would help in checking how much RocksDB is slowing things down. I'm curious about the results. Do you think you will have time to try this? – Ufuk On Wed, Apr 20,

Threads waiting on LocalBufferPool

Hi, I'm running my flink job on one rather large machine (20 cores with hyperthreading, 120GB RAM). Task manager has 20GB heap allocated. It does more or less: read csv from kafka -> keyBy one of the fields -> some custom state processing. Kafka topic has 24 partitions, so my parallelism is al

Re: streaming job reading from kafka stuck while cancelling

Mar 9, 2016 at 2:06 PM, Stephan Ewen <mailto:se...@apache.org>> wrote: Hi! Thanks for the debugging this, I think there is in fact an issue in the 0.9 consumer. I'll open a ticket for it, will try to fix that as soon as possible... Stephan On Wed, M

streaming job reading from kafka stuck while cancelling

Hi, from time to time when we cancel streaming jobs (or they are failing for some reason) we encounter: 2016-03-09 10:25:29,799 [Canceler for Source: read objects from topic: (...) ' did not react to cancelling signal, but is stuck in method: java.lang.Object.wait(Native Method) java.lang.T

Re: rebalance of streaming job after taskManager restart

k cancel bin/flink run -s … The last command is your usual run command but with the additional “-s” parameter to continue from a savepoint. I hope that helps. Cheers, Aljoscha On 08 Mar 2016, at 15:48, Maciek Próchniak wrote: Hi, we have streaming job with paralelism 2 and two

rebalance of streaming job after taskManager restart

Hi, we have streaming job with paralelism 2 and two task managers. The job is occupying one slot on each task manager. When I stop manager2 the job is restarted and it runs on manager1 - occupying two of it's slots. How can I trigger restart (or other similar process) that will cause the job t