Hi Cliff,
Thanks for bringing this up again.
I think it would make sense to at least move this forward be only
exclusively checking the schema for user keys in MapState, and allow value
schema evolution.
I'll comment on the JIRA about this, and also make it a blocker for 1.9.0
to make sure it wil
Hello,
Trying to link Zeppelin 0.9 with Flink 1.8. It`s a small dev cluster deployed
in standalone manner.
Got the same error as described here
https://stackoverflow.com/questions/54257671/runnning-a-job-in-apache-flink-standalone-mode-on-zeppelin-i-have-this-error-to
Would appreciate for any su
Hi,
I am trying to authenticate Flink using NGINX. In the document it is mentioned
to deploy a "side car proxy", here is the link for the section of the document
which I have referred to
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/security-ssl.html#external--rest-connectivit
Thanks for the update, Robert. Looking forward to the website. If there is
already a list of software we need to run the website, we can ask Apache
infra team to prepare the VM for us, as that may also take some time.
On Wed, Apr 24, 2019 at 11:57 PM Robert Metzger wrote:
> Hey all,
>
> quick up
You could try to set queryable-state.enable to true. And check again.
Vishal Santoshi 于2019年4月25日 周四上午1:40写道：
> Any one ?
>
> On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi <
> vishal.santo...@gmail.com> wrote:
>
>> Hello folks,
>>
>> Following
>> https://ci.apache.org/projects/flink/f
Hello!
Is there a way (via the REST API) to see the parameters used to start a job?
-Steve
I found the issue was a hard-coded timeout value in MiniCluster class, which is
used for stand alone execution:
public MiniCluster(MiniClusterConfiguration miniClusterConfiguration) {
this.miniClusterConfiguration =
checkNotNull(miniClusterConfiguration, "config may not
Thanks Till
What about this workaround.
If I after the watermark assignment split the stream in elements that fits in
the watermark (s1) and those that don’t (s2). The s1 I process with the table
api with a window aggregate using watermark and s2 I handle with an unbounded
non-windows aggregat
Any one ?
On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi
wrote:
> Hello folks,
>
> Following
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state
> .
> for setting up the Queryable Server and proxy, I have my classpath ( the
>
Thanks Abdul for the help. So I added this:
cfg.setString(AkkaOptions.LOOKUP_TIMEOUT, "2 min");
But I am still I am getting the same error:
Caused by: akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher62bab021-4a79-4d10-8d45-7a33c493a925#-199361569]
Hi Mans,
Sameer is correct. if you would like to control window triggering based on
other elements that does not belong to this window (in a keyed stream
context) then this is probably the best way to approach.
I think you've also posted in another thread that describes what will be
left after fi
Hi Till,
Thanks for the info!
It's good to know.
Regards,
Sergey
On Wed, Apr 24, 2019, 13:08 Till Rohrmann wrote:
> Hi Sergey,
>
> at the moment neither local nor incremental savepoints are supported in
> Flink afaik. There were some ideas wrt incremental savepoints floating
> around in the c
Hi Lasse,
at the moment this is not supported out of the box by Flink. The community
thought about this feature but so far did not implement it. Unfortunately,
I'm also not aware of an easy workaround one could do in the user code
space.
Cheers,
Till
On Wed, Apr 24, 2019 at 3:26 PM Lasse Nederga
Hello folks,
Following
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state
.
for setting up the Queryable Server and proxy, I have my classpath ( the
lib directory ) that has the required jar, But I do not see the mentioned
log and
Hey all,
quick update on this project: The frontend and backend code have been put
together into this repository: https://github.com/sorahn/flink-ecosystem
We also just agreed on an API specification, and will now work on finishing
the backend.
It will probably take a few more weeks for this to f
Hi Steven,
As much as I am aware,
1) no update call. our build flow feels a little weird to us as well.
definitely requires scripting.
2) we are using Flink management API remotely in our build flow to 1) get
jobs, 2) savepoint them, 3) cancel them etc. I am going to release a Python
script for th
Hi Steve,
I recently solved this problem using the REST api and some python scripts. The
script has a function "upgrade_job" which will cancel with savepoint,
optionally upload a new jar from the local filestystem or S3, and start the job
from the savepoint including any changes in parallelism.
Good to hear. Could you create a documentation JIRA issue for this problem?
Thanks a lot.
Cheers,
Till
On Wed, Apr 24, 2019 at 4:58 PM Vishal Santoshi
wrote:
> Verified, I think we just need to make sure that it is documented :)
>
> On Wed, Apr 24, 2019 at 9:47 AM Vishal Santoshi
> wrote:
>
>>
Hello!
I am working on automating our deployments to our Flink cluster. I had a
couple questions about the flink cli.
1) I thought there was an "update" command that would internally manage the
cancel with savepoint, upload new jar, restart from savepoint process.
2) Is there a way to get the Fl
Verified, I think we just need to make sure that it is documented :)
On Wed, Apr 24, 2019 at 9:47 AM Vishal Santoshi
wrote:
> This makes total sense and actually is smart ( defensive ). Will test and
> report. I think though that this needs to be documented :)
>
> On Wed, Apr 24, 2019 at 6:03 AM
This makes total sense and actually is smart ( defensive ). Will test and
report. I think though that this needs to be documented :)
On Wed, Apr 24, 2019 at 6:03 AM Till Rohrmann wrote:
> Hi Vishal,
>
> it seems that the following is happening: You triggered the cancel with
> savepoint command f
The idea is to also remove the rescaling code in the JobMaster. This will
make
it easier to remove the ExecutionGraph reference from the JobMaster which is
needed for the scheduling rework [1].
[1] https://issues.apache.org/jira/browse/FLINK-12231
On Wed, Apr 24, 2019 at 12:14 PM Shuai Xu wrote:
Hi.
We work with IoT data and we have cases where the IoT-device delay data
transfer if it can't get network access. We would like to use table windows
aggregate function over each device to calculate some statistics, but for
windows aggregate functions to work we need to assign a watermark. This
I think you also need to specify a path for the checkpoint directory. Try
to set
state.checkpoints.dir: swift://spout-checkpoints.magellan/flink/checkpoints
Cheers,
Till
On Wed, Apr 24, 2019 at 2:58 PM PoolakkalMukkath, Shakir <
shakir_poolakkalmukk...@comcast.com> wrote:
> Hi Till, Thanks for
Hi Till, Thanks for the response. Yes, I looks at the document. But still
trying to figure out
Let me summaries my config and what I did
1. Copied flink-swift-fs-hadoop-1.6.2.jar to lib
2. flink-conf.yaml
#==
Thank you all!
@David and @Fabian can guide me (or Deepak as well) to maintain this
document if they'd like. I can export HTML from this that we can easily
play with and put in docs.
---
Oytun Tez
*M O T A W O R D*
The World's Fastest Human Translation Platform.
oy...@motaword.com — www.motaword
Till,
Thank you for escalating this to blocker. I agree that data loss is always a
serious issue.
For reference, the workaround is to unchain the stateful operators. To make the
new job be able to recover from previous checkpoint, we also had to change the
UID of the operator that was missing
Global Windows is fine for this use case. I have used the same strategy.
You just define custom evictors and triggers and you are all good. Windows
are managed by keys, so as such as long as events are evicted from the
window, that counts towards reclaiming memory for the key+window
combination. Pl
Hi Dominik,
I think it is not possible to use Flink's AsyncFunction together with a
ConnectedStream or when you use BroadcastState. Therefore, it would be
necessary that you inject the control messages into your normal stream and
then filter them out in the AsyncFunction#asyncInvoke call.
Cheers,
Cross linking the dev ML thread [1]. Let us continue the discussion there.
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/HA-lock-nodes-Checkpoints-and-JobGraphs-after-failure-td28432.html
Cheers,
Till
On Tue, Apr 23, 2019 at 9:52 AM dyana.rose wrote:
> originally posted to
Hi Rong:
Thanks for your answer.
>From what I understand the dynamic gap session windows are also created when
>the event is encountered. I need to be able to change the window end time at
>a later time based on what other events are in that window. One way to do
>this is to use GlobalWindows
Hi Gordon,
I noticed there has been no movement on this issue and I'm wondering if I
can find some way to work around this.
My MapState value is a case class container of Avro-generated
SpecificRecords. If one SpecificRecord changes I am stuck.
>From the issue It seems like the blocker is around
I want to volunteer for maintaining or adding to this kind of document.
Please do let me know if i can.
Thanks
Deepak
On Wed, Apr 24, 2019 at 6:33 AM Deepak Sharma wrote:
>
>
> On Wed, Apr 24, 2019 at 5:14 AM Till Rohrmann
> wrote:
>
>> Thanks for sharing this resource with the community Oytun
On Wed, Apr 24, 2019 at 5:14 AM Till Rohrmann wrote:
> Thanks for sharing this resource with the community Oytun. It looks really
> helpful.
>
> I'm pulling in David and Fabian who work a lot on documentation. Maybe
> it's interesting for them to take a look at. The community had once the
> idea
Thanks for sharing this resource with the community Oytun. It looks really
helpful.
I'm pulling in David and Fabian who work a lot on documentation. Maybe it's
interesting for them to take a look at. The community had once the idea to
set up a cook book with common Flink recipes but we never manag
Hi Sergey,
at the moment neither local nor incremental savepoints are supported in
Flink afaik. There were some ideas wrt incremental savepoints floating
around in the community but nothing concrete yet.
Cheers,
Till
On Tue, Apr 23, 2019 at 6:58 PM Sergey Zhemzhitsky
wrote:
> Hi Stefan, Paul,
Hi Shakir,
have you checked out Flink's documentation for Filesystems [1]? What is the
problem you are observing?
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html
Cheers,
Till
On Tue, Apr 23, 2019 at 9:30 PM PoolakkalMukkath, Shakir <
shakir_poolakkalmukk...@comca
Hi Vishal,
it seems that the following is happening: You triggered the cancel with
savepoint command from via the REST call. This command is an asynchronous
operation which produces a result (the savepoint path). In order to deliver
asynchronous results to the caller, Flink waits before shutting d
"No metrics available" indicates that either
a) metrics have not been queried yet or
b) metrics can not be transferred from the TaskManagers to the JobManager.
Regarding the first option, how long have you waited for metrics to show
up? It may take a bit for metrics to be available (around 10 se
Hey,
I wanted to use the control stream to dynamically adjust parameters of the
tasks. I know that it is possible to use *connect()* and *BroadcastState *to
obtain such a thing. But I would like to have the possibility to control
the parameters inside the *AsyncFunction. *Like specific timeout for
For future reference here is a cross link to the referred ML thread
discussion [1].
[1]
http://mail-archives.apache.org/mod_mbox/flink-user/201904.mbox/%3cm2ef5tpfwy.wl-nings...@gmail.com%3E
Cheers,
Till
On Wed, Apr 24, 2019 at 4:00 AM Ning Shi wrote:
> Hi Congxian,
>
> I think I have figured
You should be able to get the value of your custom metric. You'll have
to set the log level to DEBUG and scan the logs for metric-related errors.
On 23/04/2019 22:25, Georgi Stoyanov wrote:
I've got custom metric ->
||
And I'm using them as suggested in the documentation ->
||
|
|getRuntime
Thanks for reporting this issue Ning. I think this is actually a blocker
for the next release and should be fixed right away. For future reference
here is the issue [1].
I've also pulled in Stefan who knows these components very well.
[1] https://issues.apache.org/jira/browse/FLINK-12296
Cheers,
Hi Rong,
thanks for your reply. I guess I already did something regarding what you
have told to me. I have one example on this application [1], which uses
this state [2] and computes a CountMinSketch [3].
I am seeking how to implement my own operator over a window in order to
have more fine-grai
+1 for temporarily removing support for the modify command.
Eventually, we have to add it again in order to support auto scaling. The
next time we add it, we should address the known limitations.
Cheers,
Till
On Wed, Apr 24, 2019 at 9:06 AM Paul Lam wrote:
> Hi Gary,
>
> + 1 to remove it for n
Hi Gary,
+ 1 to remove it for now. Actually some users are not aware of that it’s still
experimental, and ask quite a lot about the problem it causes.
Best,
Paul Lam
> 在 2019年4月24日，14:49，Stephan Ewen 写道：
>
> Sounds reasonable to me. If it is a broken feature, then there is not much
> value i
46 matches
Mail list logo
