Re: Rocksdb to filesystem state migration errors

Thanks for pointing me to the JIRA, Congxian. On Thu, Mar 14, 2019 at 6:14 PM Congxian Qiu wrote: > Hi Lakshmi > > Currently, we can’t switch between rocksdb and filesystem backend using > savepoint, there is an issue to fix this[1]. > > > [1] https://issues.apache.org/jira/browse/FLINK-11254 >

Where does the logs in Flink GUI's Exception tab come from?

Hi everyone, I am running Flink in EMR YARN cluster, and when the job failed and restarted, I could see some logs in the Exception tab of Flink GUI. I could not find this piece of l

Re: What should I take care if I enable object reuse

Keep one thing in mind: if you want the element remains legal after the function call ends (maybe map(), flatmap(), depends on what you are using), you should copy the elements. Typical scenarios includes: 1. Save the elements into some collection like array, list, map for later usage, you should c

Re: Set partition number of Flink DataSet

Hi Ken, That looks awesome! I’ve implemented something similar to your bucketing sink, but using multiple internal writers rather than multiple internal output. Besides this, I’m also curious whether Flink can achieve this like Spark: allow user to specify partition number in partitionBy() meth

Re: Re: Re: Flink 1.7.2: Task Manager not able to connect to Job Manager

Hi Gary, An update. I noticed the line “–host cluster” in the program arguments section of the job manager logs. So, I commented the following section in jobmanager.sh, the task manager is now able to connect to job manager without issues. if [ ! -z $HOST ]; then args+=("--host")

Re: Re: Re: Flink 1.7.2: Task Manager not able to connect to Job Manager

Hi Gary, Here are the full job manager and task manager logs. In the job manager logs, I see it says “starting StandaloneSessionClusterEntrypoint”, whereas in Flink 1.4.2, it used to say “starting JobManager”. Is this correct? Job manager logs: https://paste.ubuntu.com/p/DCVzsQdpHq/ (https://p

Re: S3 parquet sink - failed with S3 connection exception

Hi Kostas and everyone, I tried to change setFailOnCheckpointingErrors from True to False, and got the following trace in Flink GUI when the checkpoint/uploading failed. Not sure whether it would be of any help in identifying the issue. BTW, could you please help tell where to find the log file t

Re: Rocksdb to filesystem state migration errors

Hi Lakshmi Currently, we can’t switch between rocksdb and filesystem backend using savepoint, there is an issue to fix this[1]. [1] https://issues.apache.org/jira/browse/FLINK-11254 Best, Congxian Lakshmi Gururaja Rao 于2019年3月15日周五 上午8:07写道： > Hey all, > > I'm trying to do a state migratio

Re: What should I take care if I enable object reuse

Hi Elias, Thanks. Would it be good enough as long as we use always use different object when call the Collector.collect() method in the operator? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Rocksdb to filesystem state migration errors

Hey all, I'm trying to do a state migration from rocksdb --> filesystem backend. The approach I'm taking here is: 1) Cancel job with savepoint while its running on rocksdb 2) Update the job/cluster with filesystem as the state backend 3) Submit a job with the previous rocksdb savepoint >From what

Re: What should I take care if I enable object reuse

Avoid mutating the object if you keep a reference to it within an operator. On Wed, Mar 13, 2019 at 11:51 PM yinhua.dai wrote: > Hi Community, > > I saw from the document that we need to be careful about enable the object > reuse feature. > So which part should I check to avoid any issues? Can a

Re: Set partition number of Flink DataSet

Hi Qi, See https://github.com/ScaleUnlimited/flink-utils/ , for a rough but working version of a bucketing sink. — Ken > On Mar 13, 2019, at 7:46 PM, qi luo wrote: > > Hi Ken, > > Agree. I will try partitonBy() to reducer the number of parall

Re: Technical consulting resources

Here is a better link from a service buyer perspective: https://touk.pl/ --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Thu, Mar 14, 2019 at 1:02 PM Oytun Tez wrote: > Hi Ron, > > I've been experimenting with Nussknacke

Re: Technical consulting resources

Hi Ron, I've been experimenting with Nussknacker, a Flink application, which is built by TouK: https://touk.pl/esp/ It looks like they have experience with large deployments (telecom), so they may be helpful. I am CC'ing 2 engineers from their team. --- Oytun Tez *M O T A W O R D* The World's

Technical consulting resources

I have a team that I’m working with that’s looking for some Flink consulting. They’re running on an older version (1.4) and are looking for help with upgrading to a later version and even a path for staying current). There are some complications of our deployment model that make this harder than

Re: Re: Flink 1.7.2: Task Manager not able to connect to Job Manager

Hi Harshith, The truncated log is not enough. Can you share the complete logs? If that's not possible, I'd like to see the beginning of the log files where the cluster configuration is logged. The TaskManager tries to connect to the leader that is advertised in ZooKeeper. In your case the "cluste

Re: Re: Flink 1.7.2: Task Manager not able to connect to Job Manager

Hi Gary, I’ve attached the relevant portions of the JM and TM logs. Job Manager Logs: 2019-03-14 11:38:28,257 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2019-03-14 11:38:28,309 INFO org.apache.flink.runtime.webmo

Re: Flink 1.7.2: Task Manager not able to connect to Job Manager

Hi Harshith, Can you share JM and TM logs? Best, Gary On Thu, Mar 14, 2019 at 3:42 PM Kumar Bolar, Harshith wrote: > Hi all, > > > > I'm trying to upgrade our Flink cluster from 1.4.2 to 1.7.2 > > > > When I bring up the cluster, the task managers refuse to connect to the > job managers with t

Flink 1.7.2: Task Manager not able to connect to Job Manager

Hi all, I'm trying to upgrade our Flink cluster from 1.4.2 to 1.7.2 When I bring up the cluster, the task managers refuse to connect to the job managers with the following error. 2019-03-14 10:34:41,551 WARN akka.remote.ReliableDeliverySupervisor - Association with remote syste

Re: [DISCUSS] Create a Flink ecosystem website

Thanks for writing up the specifications. Regarding the website source code, Austin found a website[1] whose frontend code[2] is available publicly. It lacks some support (e.g login), but it is still a good starting point. One thing is that I did not find a License statement for that source code.

RE: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

It seems that Flink cluster does not retrieve program arguments correctly. For reference, I sent the following request: Method Type: POST URL: http://dbtpa05p.ch3.dev.i.com:9171/v1/jars/321febd8-a5e8-4255-858b-c221b49aef18_aa-lmx-etl-1.0.0.1-SNAPSHOT.jar/run Body: {"programArgs" : "--job=mediaSp

Re: ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

Please enable debug logging, re-submit the job, check the JobManager logs and report back the logged exception. On 14.03.2019 13:16, Papadopoulos, Konstantinos wrote: Hi all, As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading a

ProgramInvocationException when trying to submit a job by running a jar using Monitoring REST API

Hi all, As part of our projects, I experiment with Flink Monitoring REST API and, especially, its capabilities of uploading and running jar files. When I am trying to submit one of our jobs by running a jar previously uploaded via '/jars/upload', I am getting an 500 Internal Server Error respons

Re: How to join stream and dimension data in Flink?

Hi Henry, 1. Also take a look at the regular joins limitations : > However, this operation has an important implication: it requires to keep > both sides of the join input in Flink’s state forev

Re: flink-io FileNotFoundException

Is there some periodic cleanup job running on your system for /tmp? On 11.03.2019 10:38, Alexander Smirnov wrote: Hi everybody, I am using Flink 1.4.2 and periodically my job goes down with the following exception in logs. Relaunching the job does not help, only restarting the whole cluster.

How to sorted by flink sql?

hi all, Results are not sorted package com.opensourceteams.module.bigdata.flink.example.sql.test import org.apache.flink.api.scala.ExecutionEnvironment import org.apache.flink.table.api.scala._ import org.apache.flink.api.scala._ import org.apache.flink.table.api.TableEnvironment ob

Re: Custom Partitioner and Graph Algorithms

I have added a working code example to the stackoverflow question that is representative of what I am using. The github repo can be found here: https://github.com/MBtech/graphtest On 2019/03/13 09:46:52, MBilal wrote: > Hi, > > I am observing a behaviour in the task statistics that I don't fu