Hello community,
Any help here please?
Thanks,
Megh
On Mon, May 19, 2025, 18:48 megh vidani wrote:
> I'm aware that Spark does not rely on the kafka committed offsets. It is
> purely for monitoring purposes.
>
> Thanks,
> Megh
>
> On Mon, May 19, 2025, 18:46 m
Hello Spark Dev Community,
Reaching out for the below problem statement.
Thanks,
Megh
On Mon, May 19, 2025, 13:16 megh vidani wrote:
> Hello Spark Community, I have a structured streaming job in which I'm
> consuming a topic with the same name in two different kafka clusters
I'm aware that Spark does not rely on the kafka committed offsets. It is
purely for monitoring purposes.
Thanks,
Megh
On Mon, May 19, 2025, 18:46 megh vidani wrote:
> Hi Prashant,
>
> I would like to do it so that I can monitor the consumer group along with
> my othe
ffsets (e.g. from last read points). Why do
> you want to commit?
>
> On Mon, May 19, 2025 at 5:58 PM megh vidani
> wrote:
>
>> Hello Spark Dev Community,
>>
>> Reaching out for the below problem statement.
>>
>> Thanks,
>> Megh
>>
>> On
Hello Spark Community, I have a structured streaming job in which I'm
consuming a topic with the same name in two different kafka clusters and
then creating a union of these two streams. I've developed a custom query
listener to commit the offsets back to the kafka clusters once every batch
is comp
Hello Karan,
Apart from Celeborn, there is Apache Uniffle (Incubating) as well. We also
have similar setup as yours and we're trying out a PoC with Uniffle right
now.
What I've gathered so far is, with Uniffle:
1. Storing data in PVCs is not well supported
2. Storing data in host path is possible
