OK, I should have read the email thread a little more carefully before I dove into the trace... I saw Ali's comments about the prefetcher after I sent that last email.
Anyway, here's a theory: it looks like the L2 prefetcher is generating a bunch of prefetches that are hitting in the L1 *data* cache. The L2's MSHRs are filling up with entries tracking these outstanding prefetches, so it can't handle new requests from either L1 cache. An individual cache prioritizes responses over requests, so normally this would allow the MSHRs to clear before new requests get sent. However, what's happening here is that whenever the L1-L2 bus is free, the bus sends the retry to the *instruction* cache which only has requests to send (which then get nacked). Meanwhile the L2 cache needs responses from the *data* cache, but the retry signal never gets sent there (I think because the bus maintains a FIFO list of who gets the next retry). To some extent this is a shortcoming of our flow control mechanism; the 'retry' is a blunt instrument that can't specify that we really only want to allow responses or any other specific subset. It's just on or off. This is actually the same basic situation for which people invented virtual channels; we want to unblock the "response" VC on the bus without unblocking the "request" VC. Except that we don't have those concepts ;-). One way to work around this (perhaps) would be to have a rotating priority on the bus instead of a FIFO, i.e., if you're at the head of the retry list and you get a chance to send your request but it fails, you go to the end of the retry list instead of going back on the head. At least I'm assuming that the problem is that the dcache is also on the retry list but is stuck behind the icache, not that the dcache is not requesting the bus at all. (It should be requesting the bus so it can send its responses to the prefetch hits.) Steve On Fri, Jun 22, 2012 at 9:10 AM, Steve Reinhardt <ste...@gmail.com> wrote: > I looked at the trace a bit, and the thing that's most puzzling to me is > why the L2 is sending the icache a retry (saying that it can take a request > now) but then denying the icache's request when it is sent. That's what I > believe is happening over and over again in this slice of the trace: > > 112977432749000: system.cpu.icache-mem_side_port-MemSidePort: Queue > system.cpu.icache-mem_side_port-MemSidePort received retry > 112977432749000: system.cpu.icache-mem_side_port-MemSidePort: Deferred > packet is not ready (time: 9223372036854775807) > 112977432749000: system.tol2bus: recvTimingReq: src system.tol2bus-p0 > ReadReq 0x4f000 > 112977432749000: system.switch_cpus_1.iew.lsq: received pkt for > addr:0x4f000 ReadReq > 112977432749000: system.l2-cpu_side_port: Scheduling a retry while blocked > 112977432749000: system.tol2bus: recvTimingReq: src system.tol2bus-p0 > ReadReq 0x4f000 RETRY > 112977432749000: system.tol2bus: The bus is now occupied from tick > 112977432749000 to 112977432750000 > 112977432749000: system.cpu.icache-mem_side_port-MemSidePort: now waiting > on a retry > > So to be a little more more specific, the icache's call to > masterPort.sendTimingReq() in MemSidePacketQueue::sendDeferredPacket() is > returning false, which (1) indicates that the L2 is rejecting the request, > when (2) the whole reason we're in sendDeferredPacket() is because the L2 > sent us a retry indicating that it was ready for another request. > > Note that (1) and (2) are my assumptions based on how I believe the code > should be working. If there is some corner case where something other than > the L2 is causing sendTimingReq() to fail, or something other than the L2 > is generating the retry, then that could be the cause of the problem. Of > course, these L2/icache interactions all happen indirectly via the bus, but > the bus itself shouldn't be initiating the retry or rejecting the request. > > I believe the iew.lsq part is just where we send the request to the lsq > for snooping; I don't think that's an issue. > > Steve > > On Thu, Jun 21, 2012 at 9:02 PM, Ali Saidi <sa...@umich.edu> wrote: > >> ** >> >> Not yet unfortunately. If you turn off the prefetcher, does it still >> happen? Can you use gdb to step through the entire call chain and see how >> the cache blocks itself? >> >> >> >> Thanks, >> >> Ali >> >> >> >> >> >> On 20.06.2012 10:31, Nathanaël Prémillieu wrote: >> >> Any new insights on what happens? >> >> Le 12/06/2012 16:20, Nathanaël Prémillieu a écrit : >> >> Ok, my bad. The new trace: >> http://www.irisa.fr/alf/downloads/premillieu/gem5/445.gobmk.score2.trace_All-Event.deferredtime.gzThe >> code look like that now: // if we have a response packet waiting we have to >> start with that if (deferredPacketReady()) { // use the normal approach >> from the timing port trySendTiming(); DPRINTF(Cache, "Got deferred packet >> ready\n"); } else { DPRINTF(Cache, "Deferred packet is not ready (time: >> %d)\n", deferredPacketReadyTime()); // check for request packets (requests >> & writebacks) PacketPtr pkt = cache.getTimingPacket(); if (pkt == NULL) { >> // can happen if e.g. we attempt a writeback and fail, but // before the >> retry, the writeback is eliminated because // we snoop another cache's >> ReadEx. waitingOnRetry = false; DPRINTF(Cache, "Deferred packet is >> null\n"); Nathanaël Le 12/06/2012 14:22, Andreas Hansson a écrit : >> >> Thanks. Actually I was thinking of the other branch, i.e. have the >> ReadyTime printed in the else part to find out if something is waiting but >> not ready. Andreas -----Original Message----- From: >> gem5-users-boun...@gem5.org [mailto:gem5-users-boun...@gem5.org] On >> Behalf Of Nathanaël Prémillieu Sent: 12 June 2012 12:50 To: >> gem5-users@gem5.org Subject: Re: [gem5-users] Bpred: RAS maybe not >> getting correctly squashed (in some case) A new trace with the >> deferredPacketReadyTime() information: >> http://www.irisa.fr/alf/downloads/premillieu/gem5/445.gobmk.score2.trace_All-Event.withtime.gzThe >> DPRINTF is now DPRINTF(Cache, "Got deferred packet ready (time: %d)\n", >> deferredPacketReadyTime()); Nathanaël Le 12/06/2012 10:49, Andreas Hansson >> a écrit : >> >> It seems there is no response waiting (or at least not ready). Is there >> any chance you could also add a deferredPacketReadyTime() to the DPRINTF >> just to find out if there is actually something waiting (and just not >> ready)? Andreas -----Original Message----- From: >> gem5-users-boun...@gem5.org [mailto:gem5-users-boun...@gem5.org] On >> Behalf Of Nathanaël Prémillieu Sent: 11 June 2012 21:59 To: >> gem5-users@gem5.org Subject: Re: [gem5-users] Bpred: RAS maybe not >> getting correctly squashed (in some case) Here is a new trace with more >> output before and after: >> http://www.irisa.fr/alf/downloads/premillieu/gem5/445.gobmk.score2.trace_All-Event.gzThe >> DPRINT statement is DPRINTF(Cache, "Got deferred packet ready\n"); >> Nathanaël Le 11/06/2012 18:33, Nathanaël Prémillieu a écrit : >> >> I use the arm_detailed cpu model with no modification to the cpu >> configuration. Here is the option I give to the se.py script: >> --cpu-type=arm_detailed --caches --l2cache >> --checkpoint-dir=/temp_dd/alf_2/npremill/spec2006/checkpoints/base/445.gobmk/cpt.gobmk.ref.score2 >> --checkpoint-restore=200875000000 --at-instruction -m 1000000 -W 25000000 >> --standard-switch -c >> /local/npremill/test_gen3.445.gobmk.score2.14.2009/gobmk_base.armv7_nothumb-gcc >> --input=score2.tst -o "--quiet --mode gtp" --output=score2.out >> --errout=score2.err The only modification I have made to the se.py script >> is to have a physical memory of 2048MB instead of 512MB. I have put the >> DPRINT statement in the code. I will send the trace output once the run is >> done. Le 11/06/2012 18:06, Ali Saidi a écrit : >> >> Well, the good news is that it should'n be too hard to debug as it's just >> looping and not making any progress. The bad news is I can only make some >> guesses from the trace. It appears as though the L2 cache is blocked (out >> of MSHRs). At least some of these responses are coming from the L1 cache >> (responses to prefetcher requests that hit in the L1 icache). However, for >> reasons I'm not sure about the L1 seems to be sending a demand request and >> not the responses. That isn't supposed to happen. How many MSHRs does your >> L2 have? Can you put a DPRINT in the cache and see if you get into the >> deferredPacketReady if statement? if (deferredPacketReady()) { // use the >> normal approach from the timing port trySendTiming(); } else { Thanks, Ali >> On 11.06.2012 11:06, Nathanaël Prémillieu wrote: >> >> The trace continues with the same lines again and again. I have waited >> for the simulator to finish for more than one day but it is in this >> infinite loop retrying to do the access. Nathanaël >> >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users-- >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users-- >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ gem5-users mailing list >> gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ >> gem5-users mailing >> listgem5-users@gem5.orghttp://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> >> >> >> _______________________________________________ >> gem5-users mailing list >> gem5-users@gem5.org >> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> > >
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
