Thanks Matt P, I hadn’t gotten a chance to try reverting that patch. I agree reverting it and running SE mode or using FS mode is the simplest solution in the meantime.
In terms of the deadlock: I think it’s just that many ticks because the threshold for deadlocks is very long/big. I wasn’t reading too much into that – and anyways since the develop branch seems to not have that failure, I don’t think debugging it is the top priority? We’ll definitely need to dig further into the CPUID stuff as you mentioned. Thanks, Matt S. From: Poremba, Matthew via gem5-users <gem5-users@gem5.org> Sent: Monday, November 7, 2022 5:01 PM To: The gem5 Users mailing list <gem5-users@gem5.org> Cc: 1575883782 <1575883...@qq.com>; Poremba, Matthew <matthew.pore...@amd.com> Subject: [gem5-users] Re: 回复：Re: 回复：Re: Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) [AMD Official Use Only - General] Hi, The rocclr, panic, and unimplemented instructions errors/warnings seem to be caused by this patch: https://gem5-review.googlesource.com/c/public/gem5/+/64831. It is likely the ROCm stack is taking a different code path with the different processor vendor string and doing things that gem5 doesn’t do properly yet. I don’t know what exactly the problem is but pagerank runs with this change reverted. It (probably) won’t be easy to track down the exact problem without more directed tests for the new processor features. Your options are (1) to revert this patch or do the equivalent of setting the string back in your python config but I think that means you cannot run this benchmark on Ubuntu 22.04. Alternately, (2) this application is known to work in the GPU fullsystem environment on the develop branch with the default parameters (4 CUs as well). You could try that if you don’t require SE mode. I am testing 16 CUs in fullsystem and it is slowly making progress, so that appears to be working too. Regarding the coalescer timeouts, I don’t have much advice. Based on the tick number being in the trillions and deadlock being triggered with a timeout in the hundreds of millions, it looks like the simulation is *very slowly* making progress until you eventually get unlucky with a deadlock. One thing to check when you are increasing the CU count is to ensure the rest of the system is balanced along with it, such as increasing the number of memory channels accordingly to get the bandwidth needed to avoid deadlock. -MattP From: 1575883782 via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Sent: Sunday, November 6, 2022 4:57 PM To: The gem5 Users mailing list <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Cc: 1575883782 <1575883...@qq.com<mailto:1575883...@qq.com>> Subject: [gem5-users] 回复：Re: 回复：Re: Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. Hi, Matt I didn't change Makefile for PageRank. Actually, I use the same PageRank obj in different gem5 (It was compiled only once). Looking forward to your good news. And please let me know if there's anything I can do. Thanks. ------------------ 原始邮件 ------------------ 发件人: "The gem5 Users mailing list" <gem5-users@gem5.org<mailto:gem5-users@gem5.org>>; 发送时间: 2022年11月7日(星期一) 上午6:16 收件人: "The gem5 Users mailing list"<gem5-users@gem5.org<mailto:gem5-users@gem5.org>>; 抄送: "1575883782"<1575883...@qq.com<mailto:1575883...@qq.com>>;"Matt Sinclair"<sincl...@cs.wisc.edu<mailto:sincl...@cs.wisc.edu>>; 主题: [gem5-users] Re: 回复：Re: Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) Thanks, this is helpful. Regarding the trace: if this is the failure on develop, then I don’t think you need to get a trace, as the failure is different here. But yes, ProtocolTrace would be the flag to use for this. Regarding PageRank, I am running just the PageRank SPMV variant from the weekly tests in isolation, to validate if that is working. If that works and what you ran doesn’t, then perhaps there is something wrong with the docker config – TBD though. In terms of the error, I don’t think it’s a problem with the compilation. The error comes from: https://github.com/ROCm-Developer-Tools/HIP/blob/rocm-4.0.x/rocclr/hip_global.cpp#L69<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FROCm-Developer-Tools%2FHIP%2Fblob%2Frocm-4.0.x%2Frocclr%2Fhip_global.cpp%23L69&data=05%7C01%7Cmatthew.poremba%40amd.com%7Cad001e803cd5430635ef08dac05b68a8%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638033796107693251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Fnw0lTQjG65%2B8g%2FHrGYwNtp6k0OUZ%2B5K5rQRssA0x8M%3D&reserved=0>, which is happening because the kernel is not being found by HIP. I don’t know exactly why this is happening yet, but unless you changed the Makefile for PageRank I don’t see why it would be the HIP version failing. More once I can dig further into this. Matt From: 1575883782 via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Sent: Sunday, November 6, 2022 11:37 AM To: The gem5 Users mailing list <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Cc: 1575883782 <1575883...@qq.com<mailto:1575883...@qq.com>> Subject: [gem5-users] 回复：Re: Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) Hi, Matt I tried to run pagerank in the develop branch (5d0a7b6a6cca0dc20e8b8c366db2ccc150c7480a, Thu Nov 3 16:42:53 2022). But I met a new error (details are below). The error message: ``` /HIP/rocclr/hip_global.cpp:69: guarantee(false && "Cannot find Symbol") build/GCN3_X86/sim/faults.cc:60: panic: panic condition !FullSystem occurred: fault (General-Protection) detected @ PC (0x7ffff6afa941=>0x7ffff6afa942).(0=>1) Memory Usage: 19719528 KBytes Program aborted at tick 1904281529500 ``` It seems the hip version is not correct. I wonder if this problem is because my docker image version is old. (I used gcn-gpu v22-0). The good news is that pagerank runs more instructions and prints more output (although it did not run successfully to the end). I am not sure whether it's random. But for now, I think it's good news. Finally, I'm relatively new to gem5 debugging. Could you give some tips about debugging the trace? For example, the debug flag (should I use the --debug-flags=ProtocolTrace or another accurate flag about GPU?). Thanks. ------------------ 原始邮件 ------------------ 发件人: "The gem5 Users mailing list" <gem5-users@gem5.org<mailto:gem5-users@gem5.org>>; 发送时间: 2022年11月6日(星期天) 下午2:15 收件人: "The gem5 Users mailing list"<gem5-users@gem5.org<mailto:gem5-users@gem5.org>>; 抄送: "1575883782"<1575883...@qq.com<mailto:1575883...@qq.com>>;"Matt Sinclair"<sincl...@cs.wisc.edu<mailto:sincl...@cs.wisc.edu>>; 主题: [gem5-users] Re: Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) Can you please try the develop branch as well? While this is good to know it doesn’t pass on stable, if develop solves already then that is good to know. Matt Sent from my iPhone On Nov 5, 2022, at 10:51 PM, 1575883782 via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> wrote: Thanks. I will try to use `--reg-alloc-policy=dynamic`(I didn't specify a specific policy, I just used the default policy). And I will further read the trace. Then, I am using the stable branch. The commit is: ``` commit 39f85b7a3be1ee0ff6e375c9791dd62d23eb8a3e (HEAD -> stable, tag: v22.0.0.1, origin/stable, origin/master, origin/HEAD) Author: Bobby R. Bruce <bbr...@ucdavis.edu<mailto:bbr...@ucdavis.edu>> Date: Sat Jun 18 04:59:02 2022 -0700 misc: Update version info to v22.0.0.1 ``` ------------------ Original ------------------ From: "The gem5 Users mailing list" <gem5-users@gem5.org<mailto:gem5-users@gem5.org>>; Date: Sun, Nov 6, 2022 02:55 AM To: "The gem5 Users mailing list"<gem5-users@gem5.org<mailto:gem5-users@gem5.org>>; Cc: "1575883782"<1575883...@qq.com<mailto:1575883...@qq.com>>;"Matt Sinclair"<sincl...@cs.wisc.edu<mailto:sincl...@cs.wisc.edu>>; Subject: [gem5-users] Re: Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) Hi, Ultimately this message is telling you there is a deadlock in the cache coherence protocol when running PageRank with the specifications you did. To fix it, you would need to get a trace (https://www.gem5.org/documentation/learning_gem5/part3/MSIdebugging/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gem5.org%2Fdocumentation%2Flearning_gem5%2Fpart3%2FMSIdebugging%2F&data=05%7C01%7Cmatthew.poremba%40amd.com%7Cad001e803cd5430635ef08dac05b68a8%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638033796107693251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ROE5gZ27OSiJ5tBTcXyibhbNqhUZX6wEVENq9k5Pfp8%3D&reserved=0>) and look through to see what the problem is. If you do this and find a fix, we definitely welcome any patches you may find to help with this! Having said that, I’ve been trying to replicate your problem. However, the input size you are running means that gem5 will be running for a while, so it will take a while before I can say something more definitive. We do test PageRank as part of the weekly tests, but not specifically for 16 CUs. What branch (stable vs. develop) are you using? Also, I recommend using --reg-alloc-policy=dynamic, as this is a more realistic register allocation policy than the simple one (which I can’t tell if you are using or not). In the meantime, if you can answer the above questions, that may help us debug. Thanks, Matt From: 1575883782 via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Sent: Saturday, November 5, 2022 3:58 AM To: gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Cc: 1575883782 <1575883...@qq.com<mailto:1575883...@qq.com>> Subject: [gem5-users] Gem5 GCN3 (GPUCoalescer detected deadlock when running pagerank.) Hi, I was trying to run PageRank benchmark with its GCN3 GPU model. I succeed running PageRank with 4 CUs, but when I run it with 16CUs, I met some problems. The key error message is "build/GCN3_X86/mem/ruby/system/GPUCoalescer.cc:292: warn: GPUCoalescer 10 Possible deadlock detected!" Was I missing something? I don't know how to solve it. Someone could help me? 4CUs command line (default CU number is 4) ``` command line: build/GCN3_X86/gem5.opt -n 3 --mem-size=8GB --benchmark-root=/home/ubuntu/lmy/gem5-gcn3/gem5-resources/src/gpu/pannotia -c pagerank/bin/pagerank_spmv '--options=/home/ubuntu/lmy/gem5-gcn3/gem5-resources/src/gpu/pannotia/pagerank/coAuthorsDBLP.graph 1' ``` 16CUs command line ``` command line: build/GCN3_X86/gem5.opt configs/example/apu_se.py -n 3 --num-compute-units 16 --mem-size=8GB --benchmark-root=/home/ubuntu/lmy/gem5-gcn3/gem5-resources/src/gpu/pannotia -c pagerank/bin/pagerank_spmv '--options=/home/ubuntu/lmy/gem5-resources/src/gpu/pannotia/pagerank/coAuthorsDBLP.graph 1' ``` gem5 version ``` gem5 version 22.0.0.1 gem5 compiled Jun 29 2022 10:34:02 gem5 started Nov 3 2022 14:32:39 gem5 executing on 1bcbbec61aaf, pid 1287240 ``` Error message: ``` build/GCN3_X86/mem/ruby/system/GPUCoalescer.cc:292: warn: GPUCoalescer 10 Possible deadlock detected! Printing out 763 outstanding requests in the coalesced table Addr: [0x3b8b1c0, line 0x3b8b1c0] Instruction sequence number: 16871 Type: LD Number of associated packets: 2 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b300, line 0x3b8b300] Instruction sequence number: 16871 Type: LD Number of associated packets: 3 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b380, line 0x3b8b380] Instruction sequence number: 16871 Type: LD Number of associated packets: 1 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b3c0, line 0x3b8b3c0] Instruction sequence number: 16871 Type: LD Number of associated packets: 3 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b440, line 0x3b8b440] Instruction sequence number: 16871 Type: LD Number of associated packets: 1 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b480, line 0x3b8b480] Instruction sequence number: 16871 Type: LD Number of associated packets: 2 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b4c0, line 0x3b8b4c0] Instruction sequence number: 16871 Type: LD Number of associated packets: 1 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b540, line 0x3b8b540] Instruction sequence number: 16871 Type: LD Number of associated packets: 1 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b5c0, line 0x3b8b5c0] Instruction sequence number: 16871 Type: LD Number of associated packets: 2 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b680, line 0x3b8b680] Instruction sequence number: 16871 Type: LD Number of associated packets: 1 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b740, line 0x3b8b740] Instruction sequence number: 16871 Type: LD Number of associated packets: 3 Issue time: 1732620214000 Difference from current tick: 280298000 Addr: [0x3b8b7c0, line 0x3b8b7c0] ................................... Difference from current tick: 17915000 Addr: [0x4c60b40, line 0x4c60b40] Instruction sequence number: 16552 Type: LD Number of associated packets: 1 Issue time: 1732882652000 Difference from current tick: 17860000Listing pending packets from 0 instructions build/GCN3_X86/mem/ruby/system/GPUCoalescer.cc:294: panic: Aborting due to deadlock! Memory Usage: 19939216 KBytes Program aborted at tick 1732900512000 --- BEGIN LIBC BACKTRACE --- /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x4fb330)[0x55f2ea122330] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x5297ee)[0x55f2ea1507ee] /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0)[0x7fe799cb63c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fe798e5e03b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fe798e3d859] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x512b15)[0x55f2ea139b15] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0xffa194)[0x55f2eac21194] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x515ed2)[0x55f2ea13ced2] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x553944)[0x55f2ea17a944] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x55469e)[0x55f2ea17b69e] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x1c5b422)[0x55f2eb882422] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x4a3e27)[0x55f2ea0cae27] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x2a8738)[0x7fe799f6f738] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x8dd8)[0x7fe799d44f48] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe799e91e3b] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x94)[0x7fe799f6f114] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7fe799d3bd6d] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x7d86)[0x7fe799d43ef6] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8fb)[0x7fe799e91e3b] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyEval_EvalCodeEx+0x42)[0x7fe799e921c2] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyEval_EvalCode+0x1f)[0x7fe799e925af] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x1cfbf1)[0x7fe799e96bf1] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x25f537)[0x7fe799f26537] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x74d6d)[0x7fe799d3bd6d] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x12fd)[0x7fe799d3d46d] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(+0x8006b)[0x7fe799d4706b] /lib/x86_64-linux-gnu/libpython3.8.so.1.0(PyVectorcall_Call+0x60)[0x7fe799f6f830] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x52b704)[0x55f2ea152704] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x423666)[0x55f2ea04a666] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fe798e3f0b3] /home/ubuntu/lmy/gem5-gcn3/gem5/build/GCN3_X86/gem5.opt(+0x492f0e)[0x55f2ea0b9f0e] --- END LIBC BACKTRACE --- ``` _______________________________________________ gem5-users mailing list -- gem5-users@gem5.org<mailto:gem5-users@gem5.org> To unsubscribe send an email to gem5-users-le...@gem5.org<mailto:gem5-users-le...@gem5.org>
_______________________________________________ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org
