GitHub user mcpawali created a discussion: Best practices for sending millions of events to a Pekko actor without running out of memory(actor as Nats Consumer of 10 secnds interval)
Hi all, I have a scenario with Pekko actors where I am implementing a NATS consumer actor that fetches batches of events from a NATS Object Store every 10 seconds. Each batch can contain millions of events, with a total size of around 100 MB. After fetching, the consumer actor needs to send the events to a processor actor for handling. I see two approaches: 1. Sending each event individually `(processorActor ! event)` 2. Sending the entire batch as a single message `(processorActor ! eventsArray)` I’m concerned about: - Actor mailbox memory usage and backpressure - Serialization limits (e.g., maximum-frame-size) - Processing throughput - Hot shards / distribution if using cluster sharding **My questions:** 1. What is the recommended pattern for safely sending millions of events from a NATS consumer actor to a processor actor in Pekko? 2. Should I batch messages (e.g., 1k–10k events per message), or is sending the whole batch (~100 MB) ever acceptable? 3. Any best practices for cluster sharding or distributing load for high-volume message processing? Thanks in advance for your guidance! GitHub link: https://github.com/apache/pekko/discussions/2600 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
