Dear Friend:
Now ,I am a learn flink for 20 days.I would to trouble friends
to help solve two problems.
Questions are as follows：
1. If I have some jobs,How can I merge the some jobs to One
that convenient for me to manage?
I have look for some restful api in
"https://c
Hi Abdul,
in Flink 1.4 we use Akka's death watch to detect no longer reachable hosts.
The downside of the death watch mechanism is that hosts which were detected
to be dead are being quarantined. Once in this state you need to restart
the ActorSystem in order to receive messages again. The idea be
Hi,
Re Q1: The main purpose of the Trigger.clean() method is to remove all
custom state of the Trigger. State must be explicitly removed, otherwise
the program leaks memory.
Re Q3: If you are using a keyed stream, you need to manually clean up the
state by calling State.clear(). If you are using a
This means that the Dispatcher has not set its leader session id which it
gets once gaining the leadership. This can also happen if the Dispatcher
just lost its leadership after you've sent the message. This problem should
resolve itself once the new leadership information has been propagated.
Che
Thank you Fabian.
All my doubts are cleared now.
Best regards,
Averell
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi,
1) you currently cannot merge multiple jobs into one after they have been
submitted. What you can do though, is to combine multiple jobs in your
Flink program before you submit it.
2) you can pass program arguments when you submit your job. After it
has been submitted, it is no longer possibl
Hi,
2) You may also want to look into ParameterTool[1] class to parse and read
passed properties file [2].
--
Thanks,
Amit
[1]
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/java/utils/ParameterTool.html
[2]
https://ci.apache.org/projects/flink/flink-docs-re
Hi Seye,
Thanks for digging into the problem.
As Vino and Jorn suggested, this looks like a bug and please file a JIRA issue.
It would be also nice if you could post it here so that we know the related
discussion.
Cheers,
Kostas
> On Oct 14, 2018, at 9:46 AM, Jörn Franke wrote:
>
> You have
Hi Averell,
This could be the root cause of your problem!
Thanks for digging into it.
Would it be possible for you to verify that this is your problem by manually
setting
the UUID and seeing if the problem disappears? In addition, please file a JIRA.
Thanks a lot,
Kostas
> On Oct 15, 2018, a
Hi,
I think it is rather unlikely that this is the problem because it should give
a different kind of exception. Would it be possible to provide a minimal and
self-contained example code for a problematic job?
Best,
Stefan
> On 15. Oct 2018, at 08:29, Averell wrote:
>
> Hi everyone,
>
> In
Hi Kostas, Stefan,
The problem doesn't come on all of my builds, so it is a little bit
difficult to track. Are there any specific classes that I can turn DEBUG on
to help in finding the problem? (Turning DEBUG on globally seems too much).
Will try to minimize the code and post it.
One more point
Hi all,
We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT,
LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see
that GET, PUT and LIST requests are successful but it throws an AWS access
denied error for DELETE request.
Here’s a snippet of
Hi,
I see, then the important question for me is if the problem exists on the
release/master code or just on your branches. Of course we can hardly give any
advice for custom builds and without any code. In general, you should debug in
HeapKeyedStateBackend lines lines 774-776 (the write part)
Hi,
As mentioned in the title, my Flink job will not check point if there are
any finished source operator.
Is this a bug or is it working as intended ?
Regards,
Kien
Known issue: https://issues.apache.org/jira/browse/FLINK-2491
On 15.10.2018 12:23, Kien Truong wrote:
Hi,
As mentioned in the title, my Flink job will not check point if there
are any finished source operator.
Is this a bug or is it working as intended ?
Regards,
Kien
Thank you Stefan, I'll try to follow your guide to debug.
And sorry for being confusing in the previous email. When I said "different
builds", I meant different versions of my application, not different builds
of Flink.
Between versions of my application, I do add/remove some operators. However,
Hi Harshith,
Did you enable delete permission on S3 for running machines? Are you using
IAM roles or access key id and secret access key combo?
--
Thanks,
Amit
On Mon, Oct 15, 2018 at 3:15 PM Kumar Bolar, Harshith
wrote:
> Hi all,
>
>
>
> We store Flink checkpoints in Amazon S3. Flink periodic
Hi Stefan,
Sorry but it doesn't seem immediately clear to me what's a good way to use
https://github.com/king/bravo.
How are people using it? Would you for example modify build.gradle somehow
to publish the bravo as a library locally/internally? Or add code directly
in the bravo project (locally)
Hi,
I'm looking for the property file where IP and Port of TaskManagers are
stored in Flink.
Does anyone know where it is located and when it's updated?
(And for the case that there should not be such a file, where does the
JobManager and TaskManagers take this information from?)
Thanks.
Thanks Amit,
I’m now in the process of checking our IAM roles to see if the user has been
given DeleteObject permission to S3. I’m guessing that’s the most likely cause
for this error.
- Harshith
From: Amit Jain
Date: Monday, 15 October 2018 at 4:46 PM
To: Harshith Kumar Bolar
Cc: "user@flin
Hi,
Below is the In/Out metrics as they appear in the Flink UI.
I was wondering what are the possible reasons that the "Records sent" of
one operator is different than the "Records received" of the next one. I
would expect to see the same number...
[image: image.png]
* We're using Flink 1.5.0 v
Hello
I have two question, would you please answer to me?
Please tell me "what is "pyflink.bat" for?" Does Flink support python or
not?
Moreover, I want to have multiprocessing with Flink on windows. Is it
possible?
Thank you in advance
--
Sent from: http://apache-flink-user-mailing-list-ar
Hello
I have two question, would you please answer to me?
Please tell me "what is "pyflink.bat" for?" Does Flink support python or
not?
Moreover, I want to have multiprocessing with Flink on windows. Is it
possible?
Thank you in advance
--
Sent from: http://apache-flink-user-mailing-list-ar
I'm going to work on effect of parallelism for different operators on
heterogeneous machines.
I need to know the processing time of each operator instance as well as
overall processing time of all instances of each specific operator.
I think there are different ways for this purpose.
However, what
There is a known issue in 1.5.0 with the numRecordsIn/Out metrics for
chained tasks: https://issues.apache.org/jira/browse/FLINK-9530
This has been fixed in 1.5.1.
On 15.10.2018 14:37, Rafi Aroch wrote:
Hi,
Below is the In/Out metrics as they appear in the Flink UI.
I was wondering what are t
Hi mates, during the migration to Flink 1.6.1, I've faced with the following issue - flink application hangs infinitely, when application is running from IDE, at the same time, everything works fine on cluster.Seems that problem is occurred during termination of Flink cluster, I’ve checked in 1.6.0
Awesome, thanks!
On Mon, Oct 15, 2018, 17:36 Chesnay Schepler wrote:
> There is a known issue in 1.5.0 with the numRecordsIn/Out metrics for
> chained tasks: https://issues.apache.org/jira/browse/FLINK-9530
>
> This has been fixed in 1.5.1.
>
> On 15.10.2018 14:37, Rafi Aroch wrote:
>
> Hi,
>
>
Hi Bowen,
Thank you for your feedback and interest in the project. Your contribution is
certainly welcome. Per your suggestion, I have created an Uber JIRA
(https://issues.apache.org/jira/browse/FLINK-10556) to track our overall effort
on this. For each subtask, we'd like to see a short descrip
Hi Shuyi,
Thank you for your input. Yes, I agreed with a phased approach and like to move
forward fast. :) We did some work internally on DDL utilizing babel parser in
Calcite. While babel makes Calcite's grammar extensible, at first impression it
still seems too cumbersome for a project when t
Hi Folani,
I see one option that we can achieve this through metrics[1].
Each operator can report it's processing time as a metric. These metrics
can be gathered and queried later. For example, you can get a metric for a
specified TaskManager or get max/min/avg value of all TaskManagers.
Best, He
Hi，
Thanks for you reply!I will read this [1] and [2] carefully！If I also
have question ,Hope to hear from you again.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi,
1.I think I have some questions.When I have not submitted my jobs,I need
to read a job to contol jobs.It is mean that If I want to cancel a job which
is running , I can contol it. Maybe my jobs it is not running,I also can
contol them.
2.When I run a job in terminal and pass program arg
Hi Mar_zieh,
For questions about Python, you can ask Chesnay, I try to answer.
Pyflink.bat is a script for running Flink programs for flink-python*.jar.
You can see the implementation of it. Flink supports Jython.
Also, if Python is your main programming language, you can try using the
Apache Bea
Hi Chris,
Please refer to official documentation [1][2].
The tm's RPC port is even dynamic, and Flink does not persist it.
You can view them from the "Task Managers" menu on the left side of the
Flink web UI.
Thanks, vino.
[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/con
34 matches
Mail list logo
