500 is a bit extreme unless you're planning on running the job on some 200 machines and try to exploit their full power. I personally run 4 in production for our system processing 100 messages/s and there's plenty of room to grow.
Lukas On Thursday, May 21, 2015, Michael Ravits <michaelr...@gmail.com> wrote: > Hi, > > I wonder what are the considerations I need to account for in regard to the > number of partitions in input topics for Samza. > When testing with a 500 partitions topic with one Samza job I noticed the > start up time to be very long. > Are there any problems that might occur when dealing with this number of > partitions? > > Thanks, > Michael >
