On Mon, Jul 22, 2024 at 1:05 PM Brendan Goldacker <brendan...@gmail.com> wrote:
> I've been using the 1 billion row challenge > <https://github.com/gunnarmorling/1brc/discussions/67> as a way to learn > go over the past few days. With my limited go experience, I've seem to hit > a wall. > > I have my submission here <https://github.com/Goldabj/1brc-go>. On my M1 > macbook pro with 16GB of memory, it runs in ~39s. I can get it a few > seconds faster if I play around with some parameters (numWorkers and > chunkSizes). I'm trying to make it a goal to get <20s. However, I'm stuck. > > I came across another submission from this blog post > <https://www.bytesizego.com/blog/one-billion-row-challenge-go>. I've > copied over the submission to my repo (otherSubmission.go) and it runs in > ~19s. I was looking over their solution and it appears very similar to mine > in apprach. The only difference is that they use os.Read to read the file > sequentially, and my submission uses MMap. Plus a few other minor > differences. Overall, both approaches look to be about the same. However, > theirs runs in about half the time. I can't find anything that stands out > to why there would be such a large difference in runtime. > > I've been trying to use the profile and execution traces to figure out > why. Below are my flameGraphs and execution traces > [image: Screenshot 2024-07-22 at 1 19 40 PM] > <https://private-user-images.githubusercontent.com/11671931/351074916-985123e4-966c-4e17-9706-4275b4143e7a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2NzQ2NDcsIm5iZiI6MTcyMTY3NDM0NywicGF0aCI6Ii8xMTY3MTkzMS8zNTEwNzQ5MTYtOTg1MTIzZTQtOTY2Yy00ZTE3LTk3MDYtNDI3NWI0MTQzZTdhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIyVDE4NTIyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFlY2E3ZGRlOGQ0Njc5NGJlZDAzMTFjYjk2ZmQwMDhlOTk3MjM4NzNmZjNkNzI3N2RmZjAzYTIwNTRhM2NmNGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.A930je5Bc_a8pcF2TD_2zvLY5ZINVL_FH5bnq9l_Pig> > [image: Screenshot 2024-07-22 at 1 21 47 PM] > <https://private-user-images.githubusercontent.com/11671931/351074943-9340b735-5858-4de8-a1b2-e5d07ea3762b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2NzQ2NDcsIm5iZiI6MTcyMTY3NDM0NywicGF0aCI6Ii8xMTY3MTkzMS8zNTEwNzQ5NDMtOTM0MGI3MzUtNTg1OC00ZGU4LWExYjItZTVkMDdlYTM3NjJiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIyVDE4NTIyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI4ODBiYTcyOGM2OTk5ZmJlMTU2ZGViMTg0N2VhYjRmNmUxOTZlZjcwYWY2YWU1OTY3MzcxNTc0MjZhNzZiMDUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FhWgcWxT992KkMV6m21LoQnCJU5-JaoCP2TcTfidSNE> > > Below are the otherSubmission profile and execution traces: > [image: Screenshot 2024-07-22 at 1 23 39 PM] > <https://private-user-images.githubusercontent.com/11671931/351075058-d46a2447-2e9e-49c2-97f4-2d48b9451490.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2NzQ2NDcsIm5iZiI6MTcyMTY3NDM0NywicGF0aCI6Ii8xMTY3MTkzMS8zNTEwNzUwNTgtZDQ2YTI0NDctMmU5ZS00OWMyLTk3ZjQtMmQ0OGI5NDUxNDkwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIyVDE4NTIyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWMwZDY4MGJhOWU3M2Q3ZjUwNTAxYjZhYjdlZTI3NGFhMGZhOTc3ZmI5Y2Q0Njc4MzcwMGJjMmYzMjdlM2RhZjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.vQrfAS9DP6V2wp_NOht_nD1xdEWNdug7jupBOTqpoOg> > [image: Screenshot 2024-07-22 at 1 22 47 PM] > <https://private-user-images.githubusercontent.com/11671931/351075091-20e2a17d-0f29-4bfc-9915-081d923c9130.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE2NzQ2NDcsIm5iZiI6MTcyMTY3NDM0NywicGF0aCI6Ii8xMTY3MTkzMS8zNTEwNzUwOTEtMjBlMmExN2QtMGYyOS00YmZjLTk5MTUtMDgxZDkyM2M5MTMwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzIyVDE4NTIyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQxNDA3NjhlMWNmODdiNmI1ZGM1YTRmNTI1MTc4YWIzYjVlNTNkOWM5OWExZWMyODc3YTc4NzA3MDJlNjEyZDcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Rjmzp26Tu5rPReCAacP7d0euX5NeRxszBJ5TnEoLSyw> > > The first thing that stands out to my are the # of GC collections. My > submission seems to be running GC much more frequently - 100,000 > incremental sweeps, 32,000 background sweeps, and around 1000 stop the > world GCs. The heap only grows to 180MB at its peak, and typically stays > around 72MB. In the otherSubmission, incremental GC only runs 6 times, and > stop the work is ~30 times. Their heap grows much larger (up to 3.5GB), and > GC happens less frequently. > > Also, looking at the flame graphs, it looks like there is much more time > being spent on managing the stack systemstack, runtime.morestack, and > runtime.mcall. I think this may be due to more scheduler overhead due to > the frequent GCs (waiting for routines to get to a stopping point, running > GC, then re-scheduling threads to start again). > > I believe the time discrepancy is mostly due to GC overhead and go > scheduler overhead. Although, I'm not sure why. > > Could anyone help a newbie figure out where I'm going wrong? Also could > share any tools or tricks that could help me figure this out? > I have not looked at your code. But if the GC is running a lot, the first step is to look at where you are allocating memory, using Go's memory profiler. It may help to read https://go.dev/blog/pprof. See if you can adjust your code to allocate less memory. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWFM4x5F_8TSxr76%3DERfWdUtWQxEbSTptSXs_S7Jrd5MA%40mail.gmail.com.
