Hi all,
The Flink community is going to integrate some popular Change Data Capture
(CDC) tools. We would like to support reading and processing some common
binlog/changelog data in Flink SQL in the next release. We hope this survey
can help identify the most common cases and prioritize our roadmap
Hi Lei,
Yes. If you are creating a Kafka table, then the kafka connector jar and
some format jars are required.
That's weird. If DDL is failed, the yaml way should fail in the same
exception, unless some connector properties value is not the same.
Could you share the detailed exception stack?
B
I am using flink-1.10. But I add flink-json-1.9.1.jar and
flink-sql-connector-kafka_2.11-1.9.1.jar to lib directory.
After change to flink-json-1.10.0.jar,
flink-sql-connector-kafka_2.12-1.10.0.jar, it works.
But I have no idea why the yaml way works when i use flink-json-1.9.1.jar and
flink
Hi Kurt,
What you said is the 1st reason.
The second reason is this query need to scan the whole table. I think we
can do better :-)
Best,
Jiawei
On Wed, Mar 11, 2020 at 10:52 AM Kurt Young wrote:
> Hi Jiawai,
>
> Sorry I still didn't fully get your question. What's wrong with your
> proposed
>
> About the problem, we have 2 choices. The first one is using Flink as
> described in this email thread. The second one is using AWS Lambda
> triggered by CDC stream and compute the latest 15 days record, which is a
> walk-around solution and looks not as elegant as Flink to me.
>
>
Currently we
@Eyal:
The image you are using is for 1.9.2, but the logging configuration you
fetched was from master.
In 1.9.2 we use Log4j1, but on master we switched to Log4j2 instead,
which uses a different configuration syntax. Log4j1 pretty much ignores
the entire file, causing the error.
Please us
Please open a JIRA; we may have to split the datatog report into several
chunks.
On 09/03/2020 07:47, Fanbin Bu wrote:
quote from the following link:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-Query-named-operator-exceeds-80-characters-td24807.html#a24818
"This is
Have you tried to use existing operators such as
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator or
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator?
On Wed, Mar 11, 2020 at 4:46 AM Xintong Song wrote:
> Hi Eleanore,
>
> That does't sound like a scaling issue. It's proba
Sorry I wanted to mention https://github.com/lyft/flinkk8soperator (I don't
know which one of the 2 is better)
On Wed, Mar 11, 2020 at 10:19 AM Flavio Pompermaier
wrote:
> Have you tried to use existing operators such as
> https://github.com/GoogleCloudPlatform/flink-on-k8s-operator or
> https:/
> The second reason is this query need to scan the whole table. I think we
can do better :-)
Not necessarily, you said all the changes will trigger a DDB stream, you
can use Flink to consume such
stream incrementally.
For the 1st problem, I think you can use DataStream API and register a
timer on
用 sql-client create 了一个 kafka table：
CREATE TABLE order_status (
out_order_code VARCHAR,
intput_date TIMESTAMP(3),
owner_code VARCHAR,
status INT
) WITH (
'connector.type' = 'kafka',.
'format.type' = 'json', 'format.derive-schema' = 'true' )
然后往
Sorry i sent the Chinese written email to user@
Let me translate it to English.
I create a table using sql-client from kafka topic:
CREATE TABLE order_status (
out_order_code VARCHAR,
intput_date TIMESTAMP(3),
owner_code VARCHAR,
status INT
) WITH (
'connector.ty
Hi Lei,
The "2020-03-11T13:00:00.123Z" format is correct, but you defined the wrong
field name in the DDL.
It should be "input_date", not "intput_date".
Best,
Jark
On Wed, 11 Mar 2020 at 17:52, wangl...@geekplus.com.cn <
wangl...@geekplus.com.cn> wrote:
>
> Sorry i sent the Chinese written emai
Thanks Jark,
No word to express my '囧'.
wangl...@geekplus.com.cn
Sender: Jark Wu
Send Time: 2020-03-11 18:32
Receiver: wangl...@geekplus.com.cn
cc: user; user-zh
Subject: Re: json 中 timestamp 类型在json中怎样写才能被 flink sql 识别
Hi Lei,
The "2020-03-11T13:00:00.123Z" format is correct, but you defin
Thank you all.
I will change the configuration file to and deploy next week.
Best regards
Eyal Peer
From: Chesnay Schepler
Sent: Wednesday, March 11, 2020 11:10 AM
To: Yang Wang ; miki haiat
Cc: Eyal Pe'er ; Rafi Aroch ;
user ; StartApp R&D Data Platform
Subject: Re: Setting app Flink logger
Hi Flavio,
We have implemented our own flink operator, the operator will start a flink
job cluster (the application jar is already packaged together with flink in
the docker image). I believe Google's flink operator will start a session
cluster, and user can submit the flink job via REST. Not look
Hi Morgan,
> I am interested in knowing more about the failure detection mechanism
used by Flink, unfortunately information is a little thin on the ground and
I was hoping someone could shed a little light on the topic.
It is probably best to look into the implementation (see my answers below).
>
Hi,
How can I specify a yarn queue when I start a new job programmatically?
Regards,
Vitaliy
Hi.
Did someone encountered problem with sending metrics with datadog http
reporter?
My setup is flink version 1.8.2 deployed on k8s with 1 job manager and 10
task managers.
Every version deploy I see metrics on my dashboard but after a few minutes
its stopped being sent from all task managers whi
Hi Fabian,
We ran into the same issue. We modified the reporter to emit the metrics in
chunks and it worked fine after. Would be interested in seeing a ticket on
this as well.
- Steve
On Wed, Mar 11, 2020 at 5:13 AM Chesnay Schepler wrote:
> Please open a JIRA; we may have to split the datatog
Hi Vitaliy,
You can specify a yarn queue by either setting the configuration option
'yarn.application.queue' [1], or using the command line option '-qu' (or
'--queue') [2].
Thank you~
Xintong Song
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#yarn-application
EnvironmentSettings settings =
EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
TableEnvironment tEnv = TableEnvironment.create(settings);
tEnv.sqlUpdate()...
Is there a way i can set stateBackEnd like normal streaming program as
folloing:StreamExecutionEnvironment
Hi wanglei,
If you are using Flink 1.10, you can set "state.backend=rocksdb" to
"TableConfig.getConfiguration".
And you can find related config options here[1].
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html
Jingsong Lee
On Thu, Mar 12, 2020 at 11:15 AM wangl..
Hi Jingsong,
So i can write the code as following?
EnvironmentSettings settings =
EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
TableEnvironment tEnv = TableEnvironment.create(settings);
tEnv.getConfig().getConfiguration().setString("state.backend","rocksdb");eE
Yes,
You should take a look to [1]. Others config you need too.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html
Best,
Jingsong Lee
On Thu, Mar 12, 2020 at 12:26 PM wangl...@geekplus.com.cn <
wangl...@geekplus.com.cn> wrote:
> Hi Jingsong,
>
> So i can write th
Hi folks,
I have a question related configuration for new memory introduced in flink
1.10. Has anyone encountered similar problem?
I'm trying to make use of *taskmanager.memory.process.size* configuration
key in combination with mesos session cluster, but I get an error like this:
2020-03-11 11:4
Hi, Alexander
I could not reproduce it in my local environment. Normally, Mesos RM
will calculate all the mem config and add it to the launch command.
Unfortunately, all the log I could found for this command is at the
DEBUG level. Would you mind changing the log level to DEBUG or sharing
anything
Hi Alex,
Could you try to check and post your TM launch command? I suspect that
there might be some unrecognized arguments that prevent the rest of
arguments being parsed.
The TM memory configuration process works as follow:
1. The resource manager will parse the configurations, checking whic
28 matches
Mail list logo
