Re: [DISCUSSION] Add DataStreamer's default per-node-batch-setting for PDS.

2022-11-02 Thread Vladimir Steshin
    Hi, Stan.     Thank you for the answer. >>>  "your data streamer queue size is something like" You are right about writes queue on primary node. It has just some fixed size. But based on number of the CPUs. (x8). Even for my laptop I get 16x8=128 batches. I wonder why so much by default f

Re: [DISCUSSION] Add DataStreamer's default per-node-batch-setting for PDS.

2022-10-30 Thread Stanislav Lukyanov
Hi Vladimir, I think this is potentially an issue but I don't think this is about PDS at all. The description is a bit vague, I have to say. AFAIU what you see is that when the caches are persistent the streamer writes data faster than the nodes (especially, backup nodes) process the writes. Th

[DISCUSSION] Add DataStreamer's default per-node-batch-setting for PDS.

2022-10-28 Thread Vladimir Steshin
    Hi Folks,     I found that Datastreamer may consume heap or use increased heap amount when loading into a persistent cache. This may happen with streamer's 'allowOverwite'==true and the cache is in PRIMARY_SYNC mode.     What I don't like here is that the case looks simple. Not the defau