Hello,
I don't have an answer for this. Ideally, the bigger the better. For our
use case: working with Web data, hundreds of megabytes should be enough.
But there could be other use cases, such as transferring of binary or media
files.
There is another dimension to consider: amount of large messa
See my answers below,
On Thu, Sep 10, 2020 at 5:03 PM Gwen Shapira wrote:
> There is another option of doing the splitting on the server and hiding
> this from the clients. My personal (and highly controversial) take is that
> Kafka clients could use less complexity rather than more. They are
>
On 2020/09/04 07:17:36, Alexander Sibiryakov
wrote:
> Hello,
>
> I would like to get your opinions on this KIP idea.
>
> In short it will allow to transfer messages of bigger size than allowed by
> the broker.
>
> https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2Q
Thanks for the KIP. I think the main question is what's the upper bound for
message size you are looking to support. Is it hundreds of MBs, GBs, tens
of GBs, something else? That would inform the options.
Ismael
On Thu, Sep 10, 2020 at 8:03 AM Gwen Shapira wrote:
> There is another option of do
There is another option of doing the splitting on the server and hiding
this from the clients. My personal (and highly controversial) take is that
Kafka clients could use less complexity rather than more. They are
incredibly difficult to reason about as is. But maybe this
splitting/merging won't b
There is also another approach (harder to design, but may be easier to use
and maintain), which is to make Kafka handle large messages better and
allow users to set higher limits - for example, can Kafka provide really
high throughput with 1GB messages? Some systems do it well.
I don't know where
Hey Ben, thanks for the link. My proposal is partially based on Becket's
ideas, but I haven't reached out to him directly.
+Becket
Hi Becket, would you mind to have a look at my proposal (link is in the
first message) ?
A.
On Tue, Sep 8, 2020 at 12:35 PM Ben Stopford wrote:
> LinkedIn had some
LinkedIn had something like this. Becket did a talk on it a few years ago.
It would be interesting to know what became of it and if there were lessons
learned.
https://www.youtube.com/watch?v=ZrbaXDYUZY8
On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov <
sibirya...@scrapinghub.com> wrote:
> Hell
Hello,
I would like to get your opinions on this KIP idea.
In short it will allow to transfer messages of bigger size than allowed by
the broker.
https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing
If all that makes sense, I'll create a full fledged
