Re: Clarification on checkpoint/savepoint usage with Kafka (and RabbitMQ)

2025-03-19 Thread Ahmed Hamdy
while if you use savepoints you can redeploy from the last correct version savepoint and reprocess the data that was processed by the buggy job. Best Regards Ahmed Hamdy On Wed, 19 Mar 2025 at 00:54, mejri houssem wrote: > Hello Ahmed, > > Thanks for the response. > > Does that

Re: Clarification on checkpoint/savepoint usage with Kafka (and RabbitMQ)

2025-03-18 Thread Ahmed Hamdy
1- https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#starting-offset Best Regards Ahmed Hamdy On Tue, 18 Mar 2025 at 22:20, mejri houssem wrote: > > Hello everyone, > > We have a stateless Flink job that uses a Kafka source with at-least-once >

Re: Flink custom source

2024-10-06 Thread Ahmed Hamdy
ljdbc/DatabaseSplitReader.java Best Regards Ahmed Hamdy On Sun, 6 Oct 2024 at 16:48, Anil Dasari wrote: > Hi Ahmed, > Thanks for the response. > This is the part that I find unclear in the documentation and FLIP-27. The > actual split assignment happens in the SplitEnumerator#handleSpli

Re: Flink custom source

2024-10-06 Thread Ahmed Hamdy
://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java#L286 Best Regards Ahmed Hamdy On Sun, 6 Oct 2024 at 06:41, Anil Dasari wrote: > Hello, > I have implemented a custom source that

Re: PyFlink on EMR on EKS

2024-09-03 Thread Ahmed Hamdy
Java's maven/gradle dependency management systems. 1- https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/filesystems/overview/#local-file-system 2- https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/dependency_management Best Regards Ahmed Hamdy On M

Re: flink kafka sink batch mode delivery guaranties limitations

2024-08-23 Thread Ahmed Hamdy
RollingPolicy <https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/filesystem/#rolling-policy> won’t work. Best Regards Ahmed Hamdy On Fri, 23 Aug 2024 at 11:48, Nicolas Paris wrote: > hi > > From my tests kafka sink in exactly-once and ba

Re: Getting Direct buffer memory. Errors with Kafka.

2024-08-23 Thread Ahmed Hamdy
Why do you believe it is an SSL issue? The error trace seems like a memory issue. you could refer to taskmanager memory setup guide[1]. 1- https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_tm/ Best Regards Ahmed Hamdy On Fri, 23 Aug 2024 at 13:47, John Smith

Re: S3 schema for jar location?

2024-08-02 Thread Ahmed Hamdy
/opt/flink/plugins/flink-s3-fs-hadoop/ ``` 1- https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-hadoop 2- https://mvnrepository.com/artifact/org.apache.flink/flink-s3-fs-presto Best Regards Ahmed Hamdy On Fri, 2 Aug 2024 at 00:05, Maxim Senin via user wrote: > When will Fl

Re: Elasticsearch 8.x Connector in Maven

2024-07-31 Thread Ahmed Hamdy
release manager to pick up the 3.1 release. 1-https://issues.apache.org/jira/browse/FLINK-26088 2-https://lists.apache.org/list.html?d...@flink.apache.org Best Regards Ahmed Hamdy On Wed, 31 Jul 2024 at 15:06, Rion Williams wrote: > Hi again all, > > Just following up on this as I’v

Re: Event de duplication in flink with rabbitmq connector

2024-07-18 Thread Ahmed Hamdy
periment with different configurations for rocksdb compaction. Best Regards Ahmed Hamdy On Thu, 18 Jul 2024 at 13:52, banu priya wrote: > Hi Ahmed, > > Thanks for the clarification. I see from flink documentation that Kafka > sinks are transactional and de duplication happ

Re: Event de duplication in flink with rabbitmq connector

2024-07-18 Thread Ahmed Hamdy
, reprocessing is bound by the checkpoint interval which is 5 minutes, you can make it tighter if it suits your case better. Best Regards Ahmed Hamdy On Thu, 18 Jul 2024 at 11:37, banu priya wrote: > Hi All, > > Gentle reminder about bow query. > > Thanks > Banu > > On T

Re: DynamoDB Table API Connector Failing On Row Deletion - "The provided key element does not match the schema"

2024-05-31 Thread Ahmed Hamdy
https://github.com/apache/flink-connector-aws repo. One of the committers should be able to review it then. Best Regards Ahmed Hamdy On Fri, 31 May 2024 at 00:55, Rob Goretsky wrote: > Hello! > > I am looking to use the DynamoDB Table API connector to write rows to AWS > DynamoDB.

Re: Flink kafka connector for v 1.19.0

2024-05-10 Thread Ahmed Hamdy
Hi Aniket The community is currently working on releasing a new version for all the connectors that is compatible with 1.19. Please follow the announcements in Flink website[1] to get notified when it is available. 1-https://flink.apache.org/posts/ Best Regards Ahmed Hamdy On Fri, 10 May 2024

Re: Apache Flink-Redis Connector Depreciated In New Version | Adsolut Media

2024-05-09 Thread Ahmed Hamdy
Hi Kush Unfortunately there is currently no real Redis connector maintained by the Flink community. I am aware that Bahir's version might be outdated but we are currently working on a community supported connector[1] 1-https://github.com/apache/flink-connector-redis-streams Best Regards

Re: Suggestions for aggregating records to a Kinesis Sink, (or generic Async Sink)?

2024-04-29 Thread Ahmed Hamdy
enting a custom process function to hold batches of records in state and emit an aggregated record but this might not be consistent with KPL aggregation of course and de-aggregated records could be not retrieved so I would advise not to take this approach. Best Regards Ahmed Hamdy On Mon, 29 Apr

Re: Discussion thread : Proposal to add Conditions in Flink CRD's Status field

2024-03-27 Thread Ahmed Hamdy
I am sorry it should be "d...@flink.apache.org" Best Regards Ahmed Hamdy On Wed, 27 Mar 2024 at 13:00, Ahmed Hamdy wrote: > Hi Lajith, > Could you please open the discussion thread against "d...@apache.flink.org", > I believe it is better suited there. > Best Re

Re: Discussion thread : Proposal to add Conditions in Flink CRD's Status field

2024-03-27 Thread Ahmed Hamdy
Hi Lajith, Could you please open the discussion thread against "d...@apache.flink.org", I believe it is better suited there. Best Regards Ahmed Hamdy On Wed, 27 Mar 2024 at 05:33, Lajith Koova wrote: >  > > Hello, > > > Starting discussion thread here to discuss