Perhaps I wasn't clear, but I am not suggesting to use one goroutine per table. You can limit the no. of goroutines running (say 25 at a time) by just spawning them and wait for the entire task to complete, without creating a worker pool of 25 goroutines. It is basically a counting semaphore with a buffered channel.
On Sun, Dec 29, 2019 at 12:31 AM Chris Burkert <burkert.ch...@gmail.com> wrote: > I needed a way to let the database complete a few tables (max 25) at a > time without the need for the database to handle other tables in between. A > worker pool which reads one table after the other does that perfectly. That > also means that if all workers wait for the database then my whole program > waits - which is intented. I didn't test it but I guess that one goroutine > per table would burden the database to handle all tables at once. I asume > that a goroutine which triggered the database and waits for it to complete > is put aside by the runtime letting other goroutines also trigger the > database. And I think then I will have the same out of memory (on the > database side, not on the Go side). However I try to test the "one > goroutine per table" approach ... maybe it improves my code and I learn > something new. > > Anyway, all I wanted to say was that there are situations where one > approach fits more while it may be opposite in another situation. And I > don't like the term "anti-pattern" as it lets you put ideas aside which may > be worth thinking about. > > Am Sa., 28. Dez. 2019 um 14:01 Uhr schrieb Agniva De Sarker < > agniva.quicksil...@gmail.com>: > >> > (the task was to limit the whole thing to about 10% of cores) >> >> I still don't think you needed a worker pool here. Like OP mentioned >> above, you could just limit the number of goroutines executed to 10% of >> total cores. >> >> >> On Saturday, 28 December 2019 18:02:08 UTC+5:30, Chris Burkert wrote: >>> >>> There are Pros and Cons for everything in life. Some time ago I wrote a >>> database tool which does something per table where the runtime largely >>> depends on the table size. I started with one goroutine per table because >>> it was easy. But that put a high load on the database (the task was to >>> limit the whole thing to about 10% of cores) with out of memory situations. >>> So I switched to a worker pool which solved that. However now the overall >>> runtime was unpredictable. When the large tables were handled at the end >>> their runtime defined the overall runtime. So I to feed the pool with large >>> tables first. This again lead to out of memory situations so I reordered >>> the tables such that large tables are mixed with smaller tables. >>> >>> Brian Candler <b.ca...@pobox.com> schrieb am Sa. 28. Dez. 2019 um 11:09: >>> >>>> On Friday, 27 December 2019 16:30:48 UTC, Bruno Albuquerque wrote: >>>>> >>>>> This might be useful too you, in any case: >>>>> >>>>> https://git.bug-br.org.br/bga/workerpool >>>>> >>>>> >>>> I think the point from Bryan Mills' video is, "worker pool" is >>>> something of an anti-pattern in go. goroutines are so cheap that you might >>>> as well start a goroutine for each piece of work you have to do, and let it >>>> terminate when that piece of work is done. >>>> >>>> Apart from the startup cost, the other reason for having a "worker >>>> pool" is to limit the number of concurrent tasks being executed, and there >>>> are better ways of doing that in go (also shown in the video). >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golan...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/f840beee-748f-42b6-809f-4c7505208aee%40googlegroups.com >>>> <https://groups.google.com/d/msgid/golang-nuts/f840beee-748f-42b6-809f-4c7505208aee%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/ecddafbc-eb73-45e1-8a5a-f738e88c6821%40googlegroups.com >> <https://groups.google.com/d/msgid/golang-nuts/ecddafbc-eb73-45e1-8a5a-f738e88c6821%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOpCn4iFxrNV1PGKKsZ%2BfAPKiX%3DN_As4BniLajRZT%2BGR5yTKcA%40mail.gmail.com.
