Multi-stream SQL-like processing

Hi community, I would like to confront one idea with you. I was thinking that Flink SQL could be a Flink's answer for Kafka Connect (more powerful, with advantages like being decoupled from Kafka). Flink SQL would be the configuration language for Flink "connectors", sounds great!. But one thing d

Re: unknown process kdevtmpfsi is taking more cpu

No, not immediately. After running the container for some minutes. I am aware of the link which is provided by you. But we don't have redis configure it on that particular server itself. On Mon, Nov 2, 2020 at 6:48 PM Chesnay Schepler wrote: > I could not find any traces of *kdevtmpfsi* in our

Re: Increase in parallelism has very bad impact on performance

Hi, Sidney, What is the data generation rate of your Kafka topic? Is it a lot bigger than 6000? Best, Yangze Guo Best, Yangze Guo On Tue, Nov 3, 2020 at 8:45 AM Sidney Feiner wrote: > > Hey, > I'm writing a Flink app that does some transformation on an event consumed > from Kafka and then cr

Error while retrieving the leader gateway after making Flink config changes

Hello, I have Flink 1.10.2 installed in a Kubernetes cluster. Anytime I make a change to the flink.conf, the Flink jobmanager pod fails to restart. For example, I modified the following memory setting in the flink.conf: jobmanager.memory.flink.size. After I deploy the change, the pod fails to rest

Re: What does Kafka Error sending fetch request mean for the Kafka source?

Sorry, got confused with your reply... Does the message "Error sending fetch request" cause retries/duplicates down stream or it doesn't? I'm guessing it's even before the source can even send anything downstream... On Sat, 31 Oct 2020 at 09:10, John Smith wrote: > Hi my flow is Kafka Source -

Questions on Table to Datastream conversion and onTimer method in KeyedProcessFunction

Hi Flink Community, I am doing some research work on Flink Datastream and Table API and I meet two major problems. I am using Flink 1.11.2, scala version 2.11, java 8. My use case looks like this. I plan to write a data processing pipeline with two stages. My goal is to construct a business object

Increase in parallelism has very bad impact on performance

Hey, I'm writing a Flink app that does some transformation on an event consumed from Kafka and then creates time windows keyed by some field, and apply an aggregation on all those events. When I run it with parallelism 1, I get a throughput of around 1.6K events per second (so also 1.6K events p

Native kubernetes setup

Hi, I was trying to follow instructions https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/native_kubernetes.html but non e of them really worked. For session I tried: /Users/bori

Dependency injection and flink.

Hi, I'm trying to integrate a dependency injection framework with flink within my company. When I searched the user-mailing list, I found the following thread in flink which discussed about this in the past: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Dependency-Injection-a

I have some interesting result with my test code

Hi guys, I've been recently experimenting with end-to-end testing environment with Kafka and Flink (1.11) I've setup an infrastructure with Docker Compose composed of single Kafka broker / Flink (1.11) / MinIO for checkpoint saves Here's the test scenario 1. Send 1000 messages with manual timest

Good tutorial troubleshoot and reading logs

Hi there, I see there is a good page tutorial out there that explains flink's logging. Specifically I am seeing hung jobs and would like to understand more about what is causing the jobs to hang. Also more details about the checkpoint logging. Cheers

Re: LEGACY('STRUCTURED_TYPE' to pojo

My jobs normally use the blink planner, I noticed with this test that may not be the case. On Mon, Nov 2, 2020 at 12:38 PM Rex Fenley wrote: > Flink 1.11.2 with Scala 2.12 > > Error: > [info] JobScalaTest: > [info] - dummy *** FAILED *** > [info] org.apache.flink.table.api.ValidationException:

Re: LEGACY('STRUCTURED_TYPE' to pojo

Flink 1.11.2 with Scala 2.12 Error: [info] JobScalaTest: [info] - dummy *** FAILED *** [info] org.apache.flink.table.api.ValidationException: Field types of query result and registered TableSink do not match. [info] Query schema: [user: BIGINT, product: ROW<`name` VARCHAR(2147483647), `id` BIGI

Wrong results for join post tumble grouping

Hello, I have a table T0 with the following schema - root |-- amount: BIGINT}} |-- timestamp: TIMESTAMP(3) The table T0 has two rows - amount timestamp 0 0 1 8640 The following query with tumble grouping returns the wrong result - WITH CTE AS (SELECT SUM(amount) AS _output

Configure vvp 2.3 with file blob storage

Hello, I need to try out the SQL editor in Ververica platform 2.3. It requires to configure the persistence stack as local, which I did. It also requires a blob storage. To keep things simple, I'd like to configure a blob storage as local file system, or k8s volume. Configuring it as: blobStor

Manage multiple jobs in Flink

Hi, I have some questions: 1. How can you manage multiple jars (jobs) easily using Flink? 2. All jobs should run on the same task manager or do we need to use one for each job? 3. Can we store the jars in some persistent storage (such as S3) and start a job for each jar from that storage? 4. Also

Re: Flink 1.11 not showing logs

Sure. I will check that and get back to you. could you please share how to check java dynamic options? Best, Diwakar On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote: > If you have already updated the log4j.properties, and it still could not > work, then I > suggest to log in the Yarn NodeManager

Re: Filter By Value in List

I believe this is happening because the type system does not recognize that list of Strings as anything special but treats it as a black-box type. @Timo: Would this work with the new type system? Best, Aljoscha On 02.11.20 06:47, Rex Fenley wrote: Hello, I'm trying to filter the rows of a ta

Re: Error parsing annotations in flink-table-planner_blink_2.12-1.11.2

But you're not using apiguardian yourself or have it as a dependency before this, right? Best, Aljoscha On 02.11.20 14:59, Yuval Itzchakov wrote: Yes, I'm using SBT. I managed to resolve this by adding: "org.apiguardian" % "apiguardian-api" % "1.1.0" To the dependency list. Perhaps this dep

Re: Error parsing annotations in flink-table-planner_blink_2.12-1.11.2

Yes, I'm using SBT. I managed to resolve this by adding: "org.apiguardian" % "apiguardian-api" % "1.1.0" To the dependency list. Perhaps this depedency needs to be shaded as well in flink-core? My SBT looks roughly like this: lazy val flinkVersion = "1.11.2" libraryDependencies ++= Seq(

Re: LEGACY('STRUCTURED_TYPE' to pojo

@Timo: Is this sth that would work when using the new type stack? From the message I'm assuming it's using the older type stack. @Rex: Which Flink version are you using and could you maybe post the code snipped that you use to do conversions? Best, Aljoscha On 02.11.20 06:50, Rex Fenley wrot

Re: Error parsing annotations in flink-table-planner_blink_2.12-1.11.2

@Timo and/or @Jark, have you seen this problem before? @Yuval, I'm assuming you're using sbt as a build system, is that correct? Could you maybe also post a snippet of your build file that shows the dependency setup or maybe the whole file(s). Best, Aljoscha On 01.11.20 13:34, Yuval Itzchako

Re: unknown process kdevtmpfsi is taking more cpu

I could not find any traces of *kdevtmpfsi* in our official docker image. The only references I could find on the web for *kdevtmpfsi* are that it is some form of bitcoin malware, commonly associated with redis. I do not believe that this was introduced by Flink, and would currently conclude th

Re: unknown process kdevtmpfsi is taking more cpu

No, I didn't modify anything into the docker file. Also we are not using redis. On Mon, Nov 2, 2020 at 6:09 PM Chesnay Schepler wrote: > What modifications have you made to the docker file? > Are you using redis by any chance? > > On 11/2/2020 1:20 PM, Avinash Tripathy wrote: > > Hi, > > Based o

Re: unknown process kdevtmpfsi is taking more cpu

What modifications have you made to the docker file? Are you using redis by any chance? On 11/2/2020 1:20 PM, Avinash Tripathy wrote: Hi, Based on the official image we have made the docker setup in our server. But after starting docker-compose.yml, there is one unknown process called "*kdevt

unknown process kdevtmpfsi is taking more cpu

Hi, Based on the official image we have made the docker setup in our server. But after starting docker-compose.yml, there is one unknown process called "*kdevtmpfsi *" is taking more cpu. https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/docker.html flink version: 1.11.2 Any

Re: Feature request: Removing state from operators

Renaming operators and migrating the state we still need manually is what we have done in the past. I was just hoping for a more convenient solution. Peter From: David Anderson Date: Friday, October 30, 2020 at 5:55 PM To: Peter Westermann , "user@flink.apache.org" Subject: Re: Feature reques

Re: Native kubernetes setup failed to start job

Sorry, I overlooked the logs for detection-engine-dev-taskmanager-1-1. Could you start a busybox to check the connectivity for the K8s service "detection-engine-dev"? It seems that the TaskManager try to connect and get a response "Connection reset by peer". Best, Yang Yang Wang 于2020年11月2日周一 下

Re: Native kubernetes setup failed to start job

Hi Liangde Chen, Thanks for providing the logs. After checking the logs, I am afraid that there is something wrong with your K8s cluster. Since detection-engine-dev-taskmanager-1-2 has been started and registered to JobManager successfully. I suggest finding which K8s node detection-engine-dev-ta

Re: Flink 1.11 not showing logs

If you have already updated the log4j.properties, and it still could not work, then I suggest to log in the Yarn NodeManager machine and check the log4j.properties in the container workdir is correct. Also you could have a look at the java dynamic options are correctly set. I think it should work

Re:Re: How to use both of SQL and DataStream in 1.11

Hi, Danny: Thanks for your help. As in the question, some result was saved using DataStream API: ``` table.toAppendStream[Row].addSink(new MyStreamSink) class MyStreamSink extends RichSinkFunction[Row] { override def invoke(r: Row): Unit = { // save result } } ``` So if us

Re: CoProcessFunction processElement1 and processElement2

Hi Min, all methods in your user-defined function are always run in the same thread - they cannot occur simultaneously. That includes all functions related to state management (snapshotState, initializeState), life-cycle (open, close), and timers. To support multiple slots per machine, Flink main

Re: How to use both of SQL and DataStream in 1.11

You can still convert the datastream to table and register it with method void TableEnvironment.createTemporaryView(String path, Table view) Then create a StatementSet with StatementSet TableEnvironment.createStatementSet(), With the StatementSet, you can execute multiple insert statements alto

Re: Flink 1.11 not showing logs

Hi Yang, Thank you so much for taking a look at the log files. I changed my log4j.properties. Below is the actual file that I got from EMR 6.1.0 distribution of flink 1.11. I observed that it is different from Flink 1.11 that i downloaded so i changed it. Still I didn't see any logs. *Actual* log