This is a general thing; see
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/connector/
The python documentation isn't particularly clear on how to use Java
connectors. The easiest thing would be to use the "sql-*" connector jars
I guess.
On 08/11/2022 11:49
Ok thanks, will give that a try. Is that something that should be added to
the Kinesis connector docs page? There are existing instructions there for
adding the flink-connector-kinesis jar as a dependency, but no instructions
for adding commons-logging
Or if this is something more general, it migh
Said dependency (on commons-logging) is not meant to be provided by the
docker image, but bundled in your user-jar (along with the connector).
On 08/11/2022 02:14, Matt Fysh wrote:
Hi, I'm following the kinesis connector instructions as documented
here:
https://nightlies.apache.org/flink/flink
Hi congxian,
Thank you for your reply. As I shared details in my previous mail, in my
case, last successful checkpoint is missing details for some of the shards.
I am not doing any up scale or down scale of kinesis shard. I always run
with fix number of shards, so there is no possibility of new sh
Hi
For idle shards, I think restore from the previous not consumed data is ok,
because Flink did not consume any data before, but for not idle shards this
is problematic. From my experience of other connectors, could you check
whether the "error" shards are newly split? maybe the newly split shard
In my situation I believe it's because we have idle shards (it's a testing
environment). I dug into the connector code and it looks like it only
updates the shard state when a record is processed or when the shard hits
shard_end. So, for an idle shard it would never get a checkpointed state. I
gues
Do you know step by step process to reproduce this problem?
-Ravi
On Wed 16 Oct, 2019, 17:40 Steven Nelson, wrote:
> I have verified this behavior in 1.9.0, 1.8.1 and 1.7.2.
>
> About half my shards start over at trim horizon. Why would some shard
> statuses appear to not exist in a savepoints
I have verified this behavior in 1.9.0, 1.8.1 and 1.7.2.
About half my shards start over at trim horizon. Why would some shard
statuses appear to not exist in a savepoints? This seems like a big problem.
-Steve
On Wed, Oct 16, 2019 at 12:08 AM Ravi Bhushan Ratnakar <
ravibhushanratna...@gmail.co
Hi,
I am also facing the same problem. I am using Flink 1.9.0 and consuming
from Kinesis source with retention of 1 day. I am observing that when the
job is submitted with "latest" initial stream position, the job starts well
and keep on processing data from all the shards for very long period of
Hi Steven
If you restore savepoint/checkpoint successfully, I think this might due to the
shard wasn't discovered in the previous run, therefore it would be consumed
from the beginning. Please refer to the implementation here: [1]
[1]
https://github.com/apache/flink/blob/2c411686d23f456cdc502a
Hey,
Have you updated the versions both on the environment and the dependency on
the job?
>From my personal experience, 95 % of such issues is due to the mismatch
between Flink versions on the cluster you are using and Your job.
Best Regards,
Dom.
wt., 20 lis 2018 o 07:41 Steve Bistline
napisał
Hi Steve,
I think we can check the following things step by step:
1. Confirm if the data source has data.
2. Take a look at the log of Taskmanager or Jobmanager and check if there
are exceptions.
3. Take a thread dump to see what was doing in the TaskManager.
Best, Hequn
On Fri, Nov 2, 2018 at
nevermind :) PEBKAC error.
using the CLI and specifying the switches in the wrong location.
Hopefully this will not be followed by an 'actually... there's still this other
issue' post
On 2018/02/28 15:38:29, dyana.r...@salecycle.com
wrote:
> 2018-02-28 15:16:36,675 INFO
> org.apache.flink
should note this is under 1.4.1
On 2018/02/28 15:38:29, dyana.r...@salecycle.com
wrote:
> 2018-02-28 15:16:36,675 INFO
> org.apache.flink.api.java.typeutils.TypeExtractor - class
> org.apache.flink.streaming.connectors.kinesis.model.SequenceNumber does not
> contain a setter for
Hi guys,
I was able to solve the issue. I deleted all my Flink distributions
and followed these steps:
1) Clone Flink source (because I'm building Flink with Kinesis connector)
2) Checkout to release-1.3.1 (that's the version of Flink on EMR)
3) mvn clean install -Pinclude-kinesis -DskipTests (us
Yes I am using quickstart template. I have removed the exclusions for
jackson: core, databind and annotations.
On 26 September 2017 at 16:36, Tzu-Li (Gordon) Tai wrote:
> Ah, I see.
>
> Are you using the Flink quickstart template to build your application?
> I think exclusion is defined in the po
Ah, I see.
Are you using the Flink quickstart template to build your application?
I think exclusion is defined in the pom.xml of that archetype.
Just above the exclusion I do see this message:
“WARNING: You have to remove these excludes if your code relies on other
version of these dependencies.
Hi Gordon,
Thanks for your answer.
- I've built it with Maven 3.2.5
- I am using Jackson in my application (version 2.7.4)
Something that I have noticed when building Kinesis connector is that
it excludes jackson:
[INFO] Excluding
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.7.3
Hi Tomasz,
Yes, dependency clashes may surface when executing actual job runs on clusters.
A few things to probably check first:
- Have you built Flink or the Kinesis connector with Maven version 3.3 or
above? If yes, try using a lower version, as 3.3+ results in some shading
issues when used t
Hi Steffen,
Thanks for bringing up the discussion!
I think the reason why SHARD_GETRECORDS_INTERVAL_MILLIS was defaulted to 0 in
the first place was because we didn’t want false impressions that the there was
some latency introduced in Flink with the default settings.
To this end, I’m leaning t
Thanks for filing the JIRA!
Would you also be up to open a PR to for the change? That would be very very
helpful :)
Cheers,
Gordon
On 24 April 2017 at 3:27:48 AM, Steffen Hausmann (stef...@hausmann-family.de)
wrote:
Hi Gordon,
thanks for looking into this and sorry it took me so long to fi
Hi Gordon,
thanks for looking into this and sorry it took me so long to file the
issue: https://issues.apache.org/jira/browse/FLINK-6365.
Really appreciate your contributions for the Kinesis connector!
Cheers,
Steffen
On 22/03/2017 20:21, Tzu-Li (Gordon) Tai wrote:
Hi Steffan,
I have to ad
Hi Steffan,
I have to admit that I didn’t put too much thoughts in the default values for
the Kinesis consumer.
I’d say it would be reasonable to change the default values to follow KCL’s
settings. Could you file a JIRA for this?
In general, we might want to reconsider all the default values f
t; .operators.StreamSource.run(StreamSource.java:80)
> >>>
> >>> at org.apache.flink.streaming.api
> >>> .operators.StreamSource.run(StreamSource.java:53)
> >>>
> >>>
mSource.java:80)
>>>
>>> at org.apache.flink.streaming.api
>>> .operators.StreamSource.run(StreamSource.java:53)
>>>
>>> at org.apache.flink.streaming.run
>>> time.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
>>&
>>> .operators.StreamSource.run(StreamSource.java:80)
>>>
>>> at org.apache.flink.streaming.api
>>> .operators.StreamSource.run(StreamSource.java:53)
>>>
>>> at org.apache.flink.streaming.run
>>> time.tasks.SourceS
at org.apache.flink.streaming.run
>> time.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
>>
>> at org.apache.flink.streaming.run
>> time.tasks.StreamTask.invoke(StreamTask.java:266)
>>
>> at org.apache.flink.runti
at org.apache.flink.runtime.taskmanager.Task.run(Task.
> java:585)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
>
>
> *From: *Robert Metzger
> *Reply-To: *"user@flink.apache.org"
> *Date: *Friday, November 4,
Thank you for helping to investigate the issue. I've filed an issue in our
bugtracker: https://issues.apache.org/jira/browse/FLINK-5013
On Wed, Nov 2, 2016 at 10:09 PM, Justin Yan wrote:
> Sorry it took me a little while, but I'm happy to report back that it
> seems to be working properly with E
Sorry it took me a little while, but I'm happy to report back that it seems
to be working properly with EMR 4.8. It seems so obvious in retrospect...
thanks again for the assistance!
Cheers,
Justin
On Tue, Nov 1, 2016 at 11:44 AM, Robert Metzger wrote:
> Hi Justin,
>
> thank you for sharing t
Hi Justin,
thank you for sharing the classpath of the Flink container with us. It
contains what Till was already expecting: An older version of the AWS SDK.
If you have some spare time, could you quickly try to run your program with
a newer EMR version, just to validate our suspicion?
If the erro
Hi there,
We're using EMR 4.4.0 -> I suppose this is a bit old, and I can migrate
forward if you think that would be best.
I've appended the classpath that the Flink cluster was started with at the
end of this email (with a slight improvement to the formatting to make it
readable).
Willing to po
Hi Justin,
I think this might be a problem in Flink's Kinesis consumer. The Flink
Kinesis consumer uses the aws-java-sdk version 1.10.71 which indeed
contains the afore mentioned methods. However, already version 1.10.46 no
longer contains this method. Thus, I suspect, that Yarn puts some older
ve
Hi Josh,
Thanks for the description. From your description and a check into the code,
I’m suspecting what could be happening is that before the consumer caught up to
the head of the stream, Kinesis was somehow returning the same shard iterator
on consecutive fetch calls, and the consumer kept o
Hi Gordon,
My job only went down for around 2-3 hours, and I'm using the default
Kinesis retention of 24 hours. When I restored the job, it got this
exception after around 15 minutes (and then restarted again, and got the
same exception 15 minutes later etc) - but actually I found that after this
Hi Josh,
Thank you for reporting this, I’m looking into it. There was some major changes
to the Kinesis connector after mid June, but the changes don’t seem to be
related to the iterator timeout, so it may be a bug that had always been there.
I’m not sure yet if it may be related, but may I ask
Hi Aljoscha,
I opened an issue here https://issues.apache.org/jira/browse/FLINK-4115 and
submitted a pull request.
I'm not sure if my fix is the best way to resolve this, or if it's better
to just remove the verification checks completely.
Thanks,
Josh
On Thu, Jun 23, 2016 at 9:41 AM, Aljoscha Kr
Hi Josh,
do you maybe want to open an issue for that and contribute your fix for
that?
Cheers,
Aljoscha
On Fri, 17 Jun 2016 at 17:49 Josh wrote:
> Hi Aljoscha,
>
> Thanks! It looks like you're right. I've ran it with the FsStateBackend
> and everything works fine.
>
> I've also got it working w
Hi Aljoscha,
Thanks! It looks like you're right. I've ran it with the FsStateBackend and
everything works fine.
I've also got it working with RocksDBStateBackend now, by rebuilding Flink
master with:
- the verify step in FsStateBackend skipped for URIs with s3 schemes.
- the initialisation of fil
Hi,
I think the problem with the missing Class
com.amazon.ws.emr.hadoop.fs.EmrFileSystem is not specific to RocksDB. The
exception is thrown in the FsStateBackend, which is internally used by the
RocksDB backend to do snapshotting of non-partitioned state. The problem is
that the FsStateBackend tri
I found that I can still write to s3, using my Flink build of 1.1-SNAPSHOT,
for example if I run the word count example:
./bin/flink run ./examples/batch/WordCount.jar --input hdfs:///tmp/LICENSE
--output s3://permutive-flink/wordcount-result.txt
This works fine - it's just the RocksDBStateBackend
Hi Gordon/Fabian,
Thanks for helping with this! Downgrading the Maven version I was using to
build Flink appears to have fixed that problem - I was using Maven 3.3.3
before and have downgraded to 3.2.5.
Just for reference, I printed the loaded class at runtime and found that
when I was using Flin
Hi Josh,
I assume that you build the SNAPSHOT version yourself. I had similar
version conflicts for Apache HttpCore with Flink SNAPSHOT versions on EMR.
The problem is cause by a changed behavior in Maven 3.3 and following
versions.
Due to these changes, the dependency shading is not working corre
Hi Josh,
I’m looking into the problem. Seems like the connector is somehow using
older versions of httpclient.
Can you print the loaded class path at runtime, and check the path &
version of the loaded httpclient / httpcore dependency?
i.e. `classOf[HttpConnectionParams].getResource("HttpConnectio
Thank you for picking up this issue. I am excited to have a Kinesis
connector in the code base.
I added you as a contributor in JIRA, you can assign the issue to yourself
now, to let others know you are working on this.
Thanks!
Stephan
On Sat, Jan 9, 2016 at 8:18 AM, Tzu-Li (Gordon) Tai
wrote:
Hi Giancarlo,
Since it has been a while since the last post and there hasn't been a JIRA
ticket opened for Kinesis connector yet, I'm wondering how you are doing on
the Kinesis connector and hope to help out with this feature :)
I've opened a JIRA (https://issues.apache.org/jira/browse/FLINK-3211
Hi Giancarlo!
Cool that you are working on a Kinesis connector, very exciting :-)
To have a look at the Kafka fault tolerance, you can check out this blog
post, it explains it in one of the later sections:
http://data-artisans.com/kafka-flink-a-practical-how-to/
A general overview of checkpointi
Hi Stephan,
I’m not a lot familiar with Kafka on the other hand, but I think they offer a
very similar abstraction. Kinesis has a low-level api and an high level
consumer, the Kinesis Client Library (KCL).
I‘ve implemented a first version of the connector using the KCL, that I’ve been
using for
Hi Giancarlo!
I am not aware of any existing Kinesis connector. Would be definitely
something to put onto the roadmap for the near future. This is a stream
source we should support similarly to Kafka.
I am not super familiar with Kinesis, but it looks a bit like offering a
similar abstraction as
Hi Giancarlo,
I have no knowledge of someone working on such a project. However it would
be a valuable contribution, if you were to start the effort please keep us
notified, I would also suggest to file a JIRA ticket for it.
Best,
Marton
On Thu, Sep 17, 2015 at 12:54 PM, Giancarlo Pagano
wrote
50 matches
Mail list logo
