Thanks for all the advice. I just thought that 1000 partitions are not that big, but it turns out it is not :)
Best, On Sun, Oct 27, 2019 at 6:37 AM Todd Palino <tpal...@gmail.com> wrote: > Going to pile on here, and also say take that blog post with a grain of > salt :) > > Look at your data size, and your desired degree of parallelism for > consumers, and guide to that. Yes, 1000 partitions takes a while to create. > But unless your data size is something like 1 TB per day or more, you > probably don't need that. > > -Todd > > > On Sat, Oct 26, 2019, 1:19 PM Edward Capriolo <edlinuxg...@gmail.com> > wrote: > > > On Saturday, October 26, 2019, M. Manna <manme...@gmail.com> wrote: > > > > > You should also check out Becket Qin’s presentation on producer > > performance > > > tuning on YouTube. Both these items should give you all positives and > > > negatives of having many/less portions. > > > > > > Thanks, > > > > > > On Sat, 26 Oct 2019 at 09:19, Manasvi Gupta <fatbab...@gmail.com> > wrote: > > > > > > > > > > > https://www.confluent.io/blog/how-choose-number-topics- > > > partitions-kafka-cluster > > > > > > > > This might be useful for you. > > > > > > > > On Sat 26 Oct, 2019, 12:40 PM Jungyeol Lee, < > jungy...@skelterlabs.com> > > > > wrote: > > > > > > > > > I'm running a kafka cluster running only one broker with GCP > > > > n1-standard-2 > > > > > instance. > > > > > I configured the number of partitions to 1000, but it is suffering > > from > > > > > running a broker on startup. It seems that it needs much more time > > for > > > > > topic creation. So, what I did is just set the smaller number of > > > > partitions > > > > > which is now 32. Everything works okay for now. Is this normal > > behavior > > > > or > > > > > performance? Or, are there any guidelines on setting the number of > > > > > partitions? > > > > > > > > > > Best, > > > > > -- > > > > > > > > > > > > > > > > To many partitions is too many files and folders and file systems and > > longer failovers. > > > > > > -- > > Sorry this was sent from mobile. Will do less grammar and spell check > than > > usual. > > >
