Hi Experts,
I recently tried to run yarn-application mode on my yarn cluster, and I had
a problem related to configuring `execution.target`.
After reading the source code and doing some experiments, I found that
there should be some room of improvement for `FlinkYarnSessionCli` or
`AbstractYarnCli
t;> > Cheers,
>> > Till
>> >
>> > On Mon, Apr 26, 2021 at 8:37 AM Yangze Guo wrote:
>> >>
>> >> Hi, Tony.
>> >>
>> >> What is the version of your flink-dist. AFAIK, this issue should be
>> >> addressed in F
Hi Till,
I have created the ticket to extend the description of `execution.targe`.
https://issues.apache.org/jira/browse/FLINK-22476
best regards,
Tony Wei 於 2021年4月26日 週一 下午5:24寫道：
> Hi Till, Yangze,
>
> I think FLINK-15852 should solve my problem.
> It is my fault that my flin
Hi Expert,
this document [1] said `TO_TIMESTAMP` will use the session time zone to
convert date time string into a timestamp.
If I understand correctly, when I set session time zone to `Asia/Shanghai`
and query `SELECT TO_TIMESTAMP('1970-01-01 08:00:00');`,
the result should be epoch timestamp `0`
Hi Leonard,
Thanks for confirmation. I have created the jira ticket [1]. The pull
request will be submitted later.
best regards,
[1] https://issues.apache.org/jira/browse/FLINK-22970
Leonard Xu 於 2021年6月10日 週四 下午8:58寫道：
> Hi，Tony
>
> > I found this code snippet [2] might be related to `TO_TIM
ka. A
workaround might be registering the same kafka topic twice with different
name, group_id for
two SQLs. But I would still like to know if there is any way to make two
SQLs just read from the
same KafkaTableSource? Thanks in advance.
Best,
Tony Wei
forgot to send to user mailing list.
Tony Wei 於 2019年8月9日 週五 下午12:36寫道：
> Hi Zhenghua,
>
> I didn't get your point. It seems that `isEagerOperationTranslation` is
> always return false. Is that
> means even I used Blink planner, the sql translation is still in a lazy
>
is fine with me. My
original question just focused on reused nodes in SQL api, and seems Blink
planner is what I need. Thanks
for your help again.
Best,
Tony Wei
Zhenghua Gao 於 2019年8月9日 週五 下午1:54寫道：
> Blink planner support lazy translation for multiple SQLs, and the common
> nodes will
Hi,
I had the same exception recently. I want to confirm that if it is due to
transaction timeout,
then I will lose those data. Am I right? Can I make it fall back to at
least once semantic in
this situation?
Best,
Tony Wei
Piotr Nowojski 於 2018年3月21日 週三 下午10:28寫道：
> Hi,
>
> B
Hi Piotr,
Thanks a lot. I need exactly once in my use case, but instead of having the
risk of losing data, at least once is more acceptable when error occurred.
Best,
Tony Wei
Piotr Nowojski 於 2019年8月12日 週一 下午3:27寫道：
> Hi,
>
> Yes, if it’s due to transaction timeout you will lose
nt checkpoint: 1
max checkpoint duration before and after the exception occurred: < 2
minutes
Best,
Tony Wei
to find out
what's wrong about my kafka producer. Could someone help me please?
Best,
Tony Wei
Fabian Hueske 於 2019年8月16日 週五 下午4:10寫道：
> Hi Tony,
>
> I'm sorry I cannot help you with this issue, but Becket (in CC) might have
> an idea what went wrong here.
>
> Best
8 07:13:42,779] DEBUG [TransactionCoordinator id=2] Returning
> NOT_COORDINATOR error code to client for blacklist -> Sink:
> kafka-sink--eba862242e60de7e4744f3307058f865-7's AddPartitions request
> (kafka.coordinator.transaction.TransactionCoordinator)
> [2019-09-18 07:13:43,633] DEBUG
Hi Becket,
One more thing, I have tried to restart other brokers without active
controller, but
this exception might happen as well. So it should be independent of the
active
controller like you said.
Best,
Tony Wei
Tony Wei 於 2019年9月18日 週三 下午6:14寫道：
> Hi Becket,
>
> I have reprod
r's behavior. I tried to use kafka
java producer
to reproduce the exception, but it worked well. Maybe my observation is not
correct,
but the experiment result seems like that. Do you have any thoughts on this?
Best,
Tony Wei
Tony Wei 於 2019年9月19日 週四 上午11:08寫道：
> Hi Becket,
>
> On
? Is there any expert who is
familiar
with both kafka and flink's kafka connector could help me solve this?
Thanks very much.
The attachment is my code to reproduce this problem.
The cluster's versions are the same as I mentioned in my first email.
Best,
Tony Wei
*flink taskmanager:*
x27;s coordinator, since
the empty transaction won't
make any request to server.
The attachments are my simple producer code. Please help to verify what I
thought is correct. Thanks.
Best,
Tony Wei
[1]
https://github.com/apache/kafka/blob/c0019e653891182d7a95464175c9b4ef63f8bae1/clients/src/mai
action` list is empty before executing
`enqueueNewPartitions`
function. Am I right?
If it can be confirmed as a bug, I would like to submit my patch to fix it.
Thanks for your help.
Best,
Tony Wei
[1]
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L20
Thanks in advance.
Best,
Tony Wei
ier and led to this exception happened, because
there was no pending transaction in queue.
Does anyone know the details about subsumed notifications mechanism and how
checkpoint coordinator handle this situation? Please correct me if I'm
wrong. Thanks.
Best,
Tony Wei
Stefan Richter 於 2018年10
Hi,
As the follow up, it seem that savepoint can't be subsumed, so that its
notification could still be send to each TMs.
Is this a bug that need to be fixed in TwoPhaseCommitSinkFunction?
Best,
Tony Wei
Tony Wei 於 2019年11月27日 週三 下午3:43寫道：
> Hi,
>
> I want to raise this questio
esney said.
The following checkpoint succeeded before the previous savepoint, handling
both of their pending transaction, but savepoint still succeeded and sent
the notification to each TaskManager. That led to this exception. Could you
double check if this is the case? Thank you.
Best,
Tony Wei
Hi Piotrek,
There was already an issue [1] and PR for this thread. Should we mark it as
duplicated or related issue?
Best,
Tony Wei
[1] https://issues.apache.org/jira/browse/FLINK-10377
Piotr Nowojski 於 2019年11月28日 週四 上午12:17寫道：
> Hi Tony,
>
> Thanks for the explanation. Assumi
generated by another state descriptor.
Please correct me if I misunderstood. Thank you.
Best Regards,
Tony Wei
2018-06-09 9:45 GMT+08:00 TechnoMage :
> Thank you all. This discussion is very helpful. It sounds like I can
> wait for 1.6 though given our development status.
>
>
`records_lag_max` is the maximum lag
in terms of number of records for any partition in this "window". I'm not
sure
what this "window" means and if it is configurable. If it is configurable,
then
you can directly pass the config argument to Flink-Kafka-Connector to set
kafka consumer.
ator?
And could this behavior differ between different state backend (Memory, FS,
RocksDB) ?
Many thanks,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology
Hi Chesnay,
Thanks for your quick reply. I have another question. Will the state, which
is ignored, be transported
to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and
only those states reuired
by operators be transported to each TM?
Best,
Tony Wei
2018-08-17 14:38 G
Hi Stefan,
Thanks for your detailed explanation.
Best,
Tony Wei
2018-08-17 15:56 GMT+08:00 Stefan Richter :
> Hi,
>
> it will not be transported. The JM does the state assignment to create the
> deployment information for all tasks. If will just exclude the state for
> operato
filesystem and I thought it might have a simple way to support this setting
like other s3.xxx config.
Very much appreciate for your answer and help.
Best,
Tony Wei
2018-08-29 11:51 GMT+08:00 vino yang :
> Hi Tony,
>
> A while ago, I have answered a similar question.[1]
>
> You
Hi Vino,
I thought this config is for aws s3 client, but this client is inner
flink-s3-fs-presto.
So, I guessed I should find a way to pass this config to this library.
Best,
Tony Wei
2018-08-29 14:13 GMT+08:00 vino yang :
> Hi Tony,
>
> Sorry, I just saw the timeout, I thought
Hi Andrey,
Cool! I will add it in my flink-conf.yaml. However, I'm still wondering if
anyone is familiar with this
problem or has any idea to find the root cause. Thanks.
Best,
Tony Wei
2018-08-29 16:20 GMT+08:00 Andrey Zagrebin :
> Hi,
>
> the current Flink 1.6.0 version uses
that FLIP-6
tried to get rid of akka and use its own rpc interface. Please correct me
if I misunderstood. Thanks.
akka.watch.heartbeat.interval
akka.watch.heartbeat.pause
taskmanager.exit-on-fatal-akka-error
Best Regards,
Tony Wei
these metrics?
Or did I miss something?
Thank you.
Best Regards,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#cluster
Hi Gary,
Thanks for your information.
Best,
Tony Wei
2018-09-11 20:26 GMT+08:00 Gary Yao :
> Hi Tony,
>
> You are right that these metrics are missing. There is already a ticket for
> that [1]. At the moment you can obtain these information from the REST API
> (/overview) [2].
&
he.flink.runtime.blob.TransientBlobCache - Shutting
> down BLOB cache
> 2018-11-05 13:28:04,761 INFO org.apache.flink.runtime.blob.BlobServer
> - Stopped BLOB server at 0.0.0.0:42075
Best,
Tony Wei.
[1] https://github.com/scallop/scallop
store checkpoints. So, we
want to know if is it
feasible to let users provide their checkpoint path and corresponding aws
key to access their own
s3 bucket?
If not, could you show me why it doesn't work currently? And, is it
possible to become a new
feature?
Thanks in advance for your help.
Best,
Tony Wei
the way that Rong provided to set the initial start
position.
cc. Gordon who know more about the details of kafka source.
Best,
Tony Wei
Rong Rong 於 2018年11月22日 週四 上午8:23寫道：
> Hi Vishal,
>
> You can probably try using similar offset configuration as a service
> consumer.
> May
Hi,
Is there anyone can answer me?
Thanks,
Tony Wei
Tony Wei 於 2018年11月20日 週二 下午7:39寫道：
> Hi,
>
> Is there any way to provide s3.access-key and s3.secret-key in flink
> application, instead of setting
> them in flink-conf.yaml?
>
> In our use case, my team provide a fli
write
your own implementation with kafka client and always seek to the latest
position when the job begin to run.
Best,
Tony Wei
Vishal Santoshi 於 2018年11月25日 週日 上午4:51寫道：
> I think I can set . a new uuid but it seems `allowNonRestoreState` is a
> CLI hint. I need the "automatic" restart
to submit different flink applications with different s3 key for flink
presto s3 filesystem.
Any other suggestions are also welcome. Thank you.
Best,
Tony Wei
yinhua.dai 於 2018年11月27日 週二 上午11:37寫道：
> Did you try "-Dkey=value"?
>
>
>
> --
> Sent from:
> http
Hi yinhua,
Our flink version is 1.6.0.
Best,
Tony Wei
yinhua.dai 於 2018年11月27日 週二 下午2:32寫道：
> Which flink version are you using.
> I know how it works in yarn, but not very clear with standalone mode.
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing
Hi Andrey,
Thanks for your detailed answer, and I have created a JIRA issue to discuss
it [1].
Please check the description and help me to fill the details, like
component/s, since
I'm not sure where it should be put. Thank you very much.
Best,
Tony Wei
[1] https://issues.apache.org/jira/b
ecuting TTL mechanism.
Could you give me more references to learn about it? A simple example
to illustrate it is more
appreciated. Thank you.
Best,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink-1.8.html#state
red states should be
clean up as well based
on Flink 1.6's implementation. Am I right?
Best,
Tony Wei
Konstantin Knauf 於 2019年3月9日 週六 上午7:00寫道：
> Hi Tony,
>
> before Flink 1.8 expired state is only cleaned up, when you try to access
> it after expiration, i.e. when user code tries
n't work here in
streaming mode, is there
any other optimization can apply in this case? It is not necessary to focus
on SQL only. Any
improvement on DataStream is also welcome. Thank you.
Best Regards,
Tony Wei
Hi,
Is there any update for this issue? I have had the same problem just like
Karl's.
After I remove query like "select collect(data) ..." from one of the joined
tables,
the sql can be executed correctly without throwing any NPE.
Best regards,
Tony Wei
Xingcan Cui 於 2019年2月2
Hi,
I also found the similar issue here [1].
Best,
Tony Wei
[1] https://issues.apache.org/jira/browse/FLINK-11433
Tony Wei 於 2019年7月19日 週五 下午5:38寫道：
> Hi,
>
> Is there any update for this issue? I have had the same problem just like
> Karl's.
> After I remove query like
`!user.is_robot` first then executing
UDF, it will reduce the number of database access. Those records with
`true` in `user.is_robot` will
be dropped earlier and don't need to access database.
select *
from users
where !user.is_robot and UDF_NEED_TO_QUERY_DB(user)
Thanks,
Tony Wei
operator of physical plan have any meaning to
represent the
execution order of `where` conditions?
Best,
Tony Wei
sri hari kali charan Tummala 於 2019年7月27日 週六
上午3:02寫道：
> try cte common table expressions if it supports or sql subquery.
>
> On Fri, Jul 26, 2019 at 1:00 PM Fanbin Bu w
true
> END
Best regards,
Tony Wei
Hequn Cheng 於 2019年7月28日 週日 下午3:30寫道：
> Hi Tony,
>
> There is no order guarantee for filter conditions. The conditions would be
> pushed down or merged during query optimization.
>
> However, you can use the case when[1] to achieve what you want
much between each
subtask.
I'm wondering if this is a good practice, because based on the
official document it is not recommended actually.
If possible, I would like to know the detail about this side-effect. Which
state backend will have this issue? and Why?
Please give me an advice. Thanks in
Hi Hangxiang, David,
Thank you for your replies. Your responses are very helpful.
Best regards,
Tony Wei
David Anderson 於 2023年3月14日 週二 下午12:12寫道：
> I believe there is some noticeable overhead if you are using the
> heap-based state backend, but with RocksDB I think the differe
the
parallelism as the number of TMs.
Hope this will help you.
Best,
Tony Wei
[1] https://issues.apache.org/jira/browse/FLINK-7851
2017-10-22 10:35 GMT+08:00 Le Xu :
> Hello!
>
> Just wondering if there is a way to load-balancing all executors on one
> operator across the machi
Hi,
Did you enable externalized checkpoints? [1]
Best,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/checkpoints.html#externalized-checkpoints
2017-10-24 13:07 GMT+08:00 vipul singh :
> Thanks Aljoscha for the answer above.
>
> I am experimen
Hi Max,
Good to know. Thanks very much.
Best Regards,
Tony Wei
2017-10-24 13:52 GMT+08:00 Maximilian Bode :
> Hi Tony,
>
> thanks for troubleshooting this. I have added a commit to
> https://github.com/apache/flink/pull/4586 that should enable you to use
> the reporter wit
`SingleOutputStreamOperator` [1], I think it is easier to just change the
return type of that function.
And you can leverage the functionality of the `SingleOutputStreamOperator`.
Hope this will help you.
Best Regards,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink
.select(new InferenceSelectFunction>() {...};
((SingleOutputStreamOperator) LCxAccResult).slotSharingGroup("...");
Best Regards,
Tony Wei
[1]
https://github.com/htm-community/flink-htm/blob/master/flink-htm-streaming-java/src/main/java/org/numenta/nupic/flink/streaming/api/HT
Hi Andrea,
In this way, you will only set a slotSharing group on select operator and
learn operator will remain in the default group.
If you want to set lean operator as well, I am afraid that you need to
refactor `HTMStream` to expose `InferenceStreamBuilder.build()`.
Best Regards,
Tony Wei
Hi Andrea,
The `learn` operator is defined in this method [1]. If you need to set its
slotSharing group, you should add `slotSharingGroup(...)` behind line 97
[2] or a new API to get the result from `inferenceStreamBuilder.build()`.
Best Regards,
Tony Wei
[1]
https://github.com/htm-community
other slotSharing group.
Best Regards,
Tony Wei
2017-10-27 17:18 GMT+08:00 AndreaKinn :
> I'm trying to create an API to get results from build() but anyway I'm
> stuck
> on the StreamTransformation which is essential to implement
> slotSharingGroup(...). I have to pro
Hi,
I think ProcessFunction[1] is what you want. You can add it after keyBy and
emit the result to sink after timeout or buffer filled.
The reference has a good example that show you how to use it.
Best Regards,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream
which subtask that operator will go to.
I'm not sure if this is a good solution and I am also wondering if it can
be fulfilled.
Best Regards,
Tony Wei
[1]
https://github.com/apache/flink/blob/0182141d41be52ae0cf4a3563d4b8c6f3daca02e/flink-runtime/src/main/java/org/apache/flink/runtime/
.flatMap(...);
dataWithRule.keyBy(...).process(...);
Hope this will make sense to you.
Best Regards,
Tony Wei
2017-11-09 6:25 GMT+08:00 Ladhari Sadok :
> Hello,
>
> I'm working on Rules Engine project with Flink 1.3, in this project I want
> to update some keyed operator state whe
. update some rules or 2. refetch all rules from
Redis.
Is that what you want?
Best Regards,
Tony Wei
2017-11-09 15:52 GMT+08:00 Ladhari Sadok :
> Thank you for the answer, I know that solution, but I don't want to stream
> the rules all time.
> In my case I have the rules in Redis
and be calculated by the
models.
I think this approach might be close to the solution in that article, but
it could have some difference according to different use cases.
Best Regards,
Tony Wei
2017-11-09 17:27 GMT+08:00 Ladhari Sadok :
>
> -- Forwarded message --
> From
.
My Kafka cluster version is 0.10.2.1.
My Zookeeper version for Kafka is 3.4.6-1569965, built on 02/20/2014 09:09
GMT.
My Kafka connector library is "org.apache.flink" %
"flink-connector-kafka-0.10_2.10" % "1.3.2"
Thanks for your help in advance.
Best Regards,
Tony Wei
Hi KZ,
https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-flink
This article seems to be a good example to trigger a new calculation on a
running job. Maybe you can get some help from it.
Best Regards,
Tony Wei
2017-11-29 4:53 GMT+08:00 zanqing zhang :
> Hi All,
>
Hi,
I think FLINK-7692 won't solve this problem. It is just a new feature to
provide more functionality for user to use metric system.
Regards,
Tony Wei
2017-12-26 18:23 GMT+08:00 Marvin777 :
> Hi,
>
> I just want to say we're having the same issues. Using latency marke
transaction id pool, which has
multiple ids to be reused by producer, and it aborts all ids in this pool
in the `initializeState`. Is this pool designed for the situation in the
first problem or something I haven't noticed?
Thank you.
Best Regards,
Tony Wei
2018-01-04 22:15 GMT+08:00 Stefan Ri
Hi Stefan,
Your reply really helps me a lot. Thank you.
2018-01-08 19:38 GMT+08:00 Stefan Richter :
> Hi,
>
> 1. If `snapshotState` failed at the first checkpoint, does it mean there
> is no state and no transaction can be aborted by default?
>
>
> This is a general problem and not only limited
HA's
responsibility.
One possible way might be using supervisord [1] to launch your JM instance,
it can help you monitor your process and automatically restart when the
process accidentally failed. Or you can use YARN cluster, the YARN cluster
will be responsible for recovering the dead JM.
Best,
state because I didn't
override Object#equals.
Is it necessary that POJO key type overrides Object#equals? Or PojoTypeInfo
didn't rely on MyClass#equals? Or keyBy() didn't rely on equals?
Thank you very much.
Best Regards,
Tony Wei
ValueState as key state.
BTW, could you please give more explanations about what heap-based state
is? Since I'm not familiar with the details below the state
implementations, it will be great if you can share more technical details
or some references to me. Thank you!
Best Regards,
Tony Wei
20
Hi Timo,
Thanks a lot. I will try it out.
Best Regards,
Tony Wei
2018-02-06 17:25 GMT+08:00 Timo Walther :
> With heap-based state I meant state that is stored using the
> MemoryStateBackend or FsStateBackend [1]. In general, even if you are just
> using ValueState, the key migh
Hi Stefan,
Here is my checkpointing configuration.
Checkpointing Mode Exactly Once
Interval 20m 0s
Timeout 10m 0s
Minimum Pause Between Checkpoints 0ms
Maximum Concurrent Checkpoints 1
Persist Checkpoints Externally Enabled (delete on cancellation)
Best Regards,
Tony Wei
2018-03-05 21:30 GMT+08
another s3 filesystem package, because it is the last
thing I want to see that the checkpoint timeout happened very often.
Thank you very much for all your advices.
Best Regards,
Tony Wei
2018-03-06 1:07 GMT+08:00 Stefan Richter :
> Hi,
>
> thanks for all the info. I had a look into the problem
and it is far
from my timeout configuration. I believe "incremental checkpoint" will
reduce the latency and make this issue might occur seldom, but I can't
promise it won't happen again if I have bigger states growth in the future.
Am I right?
Best Regards,
Tony Wei
2018-03-
Sent to the wrong mailing list. Forward it to the correct one.
-- Forwarded message --
From: Tony Wei
Date: 2018-03-06 14:43 GMT+08:00
Subject: Re: checkpoint stuck with rocksdb statebackend and s3 filesystem
To: 周思华 , Stefan Richter
Cc: "user-subscr...@flink.apache.org&q
root cause or stop the investigation and make the
conclusion in this mailing thread. What do you think?
Best Regards,
Tony Wei
2018-03-06 15:13 GMT+08:00 周思华 :
> Hi Tony,
>
> Sorry for missing the factor of cpu, I found that the "bad tm"'s cpu load
> is so much high
Hi Stefan,
We prepared to run it on local SSDs yesterday. However, as I said, the
problem just disappeared. Of course we will replace it to local SSDs, but
I'm afraid that I might be able to reproduce the same situation for both
environments to compare the difference.
Best Regards,
Ton
and enabled incremental checkpoint mechanism
as well. Our job has run healthily for more than two weeks.
Thank you all for helping me to investigate and solve this issue.
Best Regards,
Tony Wei
[1] EBS: I/O Credits and Burst Performance
<https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVo
ny times during the last weekend and made my
kafka log delay grew up. Please help me. Thank you very much!
Best Regards,
Tony Wei
st ZK connection.
So, as Piotr said, it looks like an error in Kafka producer and I will pay
more attention on it to see if there is something unexpected happens again.
Best Regards,
Tony Wei
2018-05-15 19:56 GMT+08:00 Piotr Nowojski :
> Hi,
>
> It looks like there was an error in asynchronous
was implemented state operator by `ProcessFunction` API, is it
possible to port it to `KeyedProcessOperator` and do the state migration
that you mentioned?
And are there something concerned and difficulties that will leads to
restored state failed or other problems? Thank you!
Best Regards,
Tony Wei
7;t look up all keys and migrate the entire previous
states to the new states in `ProcessFunction#open()`?
As I said, do I need to port `ProcessFunction` to `KeyedProcessOperator` to
migration state like the manner showed in `WindowOperator`?
Best Regards,
Tony Wei
2018-06-07 20:28 GMT+08:00
ut getKeyes you still can implement lazy conversion) you would have
> to write your own operator.
>
> Piotrek
>
>
> On 7 Jun 2018, at 15:26, Tony Wei wrote:
>
> Hi Piotrek,
>
> I used `ProcessFunction` to implement it, but it seems that I can't call
> `getKey
Hi Tomasz,
In my opinion, I would move .window() function down to these two
DataStream. (rawEvent.window().reduce().map(), and so does metrics)
It makes sure that they won't share the same constructor.
Regards,
Tony Wei
2017-08-23 17:51 GMT+08:00 Tomasz Dobrzycki :
> Hi Tony,
>
>
operators
and they created their own window state by themselves.
For more details, you can refer to this blog (https://flink.apache.org/
news/2015/12/04/Introducing-windows.html)
Therefore, the modified version of mine is not different from yours.
Regards,
Tony Wei
2017-08-23 18:11 GMT+08:00 Tomasz
ions are all correct. If not, is there any
mechanisms about watermark and window in Flink that I missed.
Thank for your help.
Best Regards,
Tony Wei
Hi Alijoscha,
It is very helpful to me to understand the behavior on such scenario. Thank
you very much!!!
Best Regards,
Tony Wei
2017-08-28 20:00 GMT+08:00 Aljoscha Krettek :
> Hi Tony,
>
> I think your analyses are correct. Especially, yes, if you re-read the
> data the (ts=3)
one to one
mapping.
You can check your topic's setup or create another topic to try this out.
Hope this will help you.
Best Regards,
Tony Wei
2017-08-29 12:26 GMT+08:00 Isuru Suriarachchi :
> Hi all,
>
> I'm trying to implement a Flink consumer which consumes a Kafka topic
is any solution to solve this without
SideInput.
Best,
Tony Wei
2017-08-31 16:10 GMT+08:00 Martin Eden :
> Hi all,
>
> I am trying to implement the following using Flink:
>
> I have 2 input message streams:
>
> 1. Data Stream:
> KEY VALUE TIME
> .
> .
> .
> C
Hi Marchant,
HDFS is not a must for storing checkpoints. S3 or NFS are all acceptable,
as long as it is accessible from job manager and task manager.
For AWS S3 configuration, you can refer to this page (
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html).
Best,
Tony Wei
wondering if it is possible to generate a completely new
function in the runtime.
Best,
Tony Wei
2017-08-31 18:33 GMT+08:00 Martin Eden :
> Thanks for your reply Tony.
>
> So there are actually 2 problems to solve:
>
> 1. All control stream msgs need to be broadcasted to all tasks.
.
After that, keyBy function could be used to group those arguments belonging
to the same lambda function. Joining this stream with Control Stream by
function id could make arguments and function be in the same instance.
What do you think? Could this solution solve your problem?
Best,
Tony Wei
2017
Hi Martin,
Yes, that is exactly what I thought.
But the first step also needs to be fulfilled by SideInput. I'm not sure
how to achieve this in the current release.
Best,
Tony Wei
Martin Eden 於 2017年8月31日 週四，下午11:32寫道：
> Hi Aljoscha, Tony,
>
> Aljoscha:
> Yes it'
es' state in a global view, so that no matter what
time a new lambda comes it can always get its variables' state immediately. In
that case, I have the same opinion as Aljoscha.
Best,
Tony Wei
2017-09-01 23:59 GMT+08:00 Aljoscha Krettek :
> Hi Martin,
>
> I think with those r
l. See the reference
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/checkpoints.html#directory-structure
.
Best,
Tony Wei
2017-09-03 7:09 GMT+08:00 Jose Miguel Tejedor Fernandez <
jose.fernan...@rovio.com>:
> Hi,
>
> After enabling checkpoints and set the propert
. Thank you.
Best Regards,
Tony Wei
o. Can this assumption be acceptable in your case?
What do you think?
Best,
Tony Wei
2017-09-06 22:41 GMT+08:00 Martin Eden :
> Hi Aljoscha, Tony,
>
> We actually do not need all the keys to be on all nodes where lambdas are.
> We just need the keys that represent the data for the l
1 - 100 of 125 matches
Mail list logo
