Re: [DISCUSS] camel-iceberg component

seems reasonable to keep camel-iceberg inside camel project, which already has many integration components. +1 for that. On Wed, May 22, 2024 at 8:58 AM Ajantha Bhat wrote: > +1, > > It is always good to have new ways to ingest data as an Iceberg table. > > - Ajantha > > On Wed, May 22, 2024 at

Re: [Discuss] Heap pressure with RewriteFiles APIs

I'd think chunking the work as much as possible, and disabling metrics for columns where they're not helpful probably goes far but perhaps may be insufficient for extreme cases. I've also been thinking about if there are better space-efficient data structures for maintaining file paths which exploi

Re: [DISCUSS] camel-iceberg component

+1, It is always good to have new ways to ingest data as an Iceberg table. - Ajantha On Wed, May 22, 2024 at 7:32 PM Jean-Baptiste Onofré wrote: > Hi Omar, > > That's the plan (see the last section in my previous email). Just > wanted to bring some attention in the Iceberg community :) > > Reg

Re: [DISCUSS] camel-iceberg component

Hi Omar, That's the plan (see the last section in my previous email). Just wanted to bring some attention in the Iceberg community :) Regards JB On Wed, May 22, 2024 at 10:01 AM Omar Al-Safi wrote: > > IMO the Camel iceberg component should live in the camel repo. it can be part > of the camel

Re: [DISCUSS] camel-iceberg component

IMO the Camel iceberg component should live in the camel repo. it can be part of the camel components registry in camel On Wed, May 22, 2024 at 9:58 AM Jean-Baptiste Onofré wrote: > Hi Manish > > No, Camel is not an alternative to Spark or Flink: Camel is not a > query engine. It's more a "compl

Re: [DISCUSS] camel-iceberg component

Hi Ryan Yes, I agree: I started on Iceberg repo to facilitate the discussion and inform the Iceberg community. As said in my previous email, I will move the camel-iceberg component directly in Camel. "Classic" use cases for Apache Camel is IoT, system integration and event streaming. For instance

Re: [DISCUSS] camel-iceberg component

Hi Manish No, Camel is not an alternative to Spark or Flink: Camel is not a query engine. It's more a "complement" to Kafka Connect. Regards JB On Wed, May 22, 2024 at 7:09 AM Manish Malhotra wrote: > > Is Camel can be used as an alternate to Flink? > > > On Tue, May 21, 2024 at 10:17 AM Ryan B

Re: [DISCUSS] camel-iceberg component

Hi Yufei 1. No Camel has its own routing engine, fully self-contained, message oriented 2. The potential limitations is the size of the Exchange (depending of the route, the Exchange can be offload to a store). And yes, potentially, depending of the route, the number of commits can be important (e

Re: [Discuss] Heap pressure with RewriteFiles APIs

Hi Szehon, Thanks for your email. I agree configuring metadata metrics per column will create a smaller manifest file with lower and upper bounds per content entry. Assuming your patch is merged, it will works as following: 1. A user should identif