Hi guys, I've been able to reproduce Ivan's issue in the latest gem5 (rev. 10235). It seems this may be the same bug as a report that I filed about a year ago <http://flyspray.gem5.org/index.php?do=details&task_id=7>. Previously, I had overlooked that Ivan's tests were using the classic memory system (and frankly, I had forgotten I had submitted that bug report). I'll second Emilio and recommend using Ruby for now.
For anyone's future reference: Contrary to Emilio's statement, I'm not clear that simulations necessarily make progress after terminal output interrupts are lost with the classic memory model. It is possible that unimplemented x86 atomics in the classic memory hierarchy could be the problem, and if so, many other problems besides hung terminal output could arise. Joel On Fri, Jun 6, 2014 at 10:49 AM, Castillo Villar, Emilio < emilio.casti...@unican.es> wrote: > Hello, > > I have seen similar issues when running X86 timing and detailed cpus with > the Classic memory system. Mostly due to X86 atomic memory accesses not > being implemented. The stdout freezes but instructions are still being > committed. > > If you want to run with timing or detailed cpus in X86 & FS & multi-core I > am afraid you will need to use Ruby. > > Emilio > ------------------------------ > *De:* gem5-users [gem5-users-boun...@gem5.org] en nombre de Ivan Stalev > via gem5-users [gem5-users@gem5.org] > *Enviado:* viernes, 06 de junio de 2014 1:14 > *Para:* Joel Hestness > *CC:* gem5 users mailing list > *Asunto:* Re: [gem5-users] System Hangs > > Hi Joel, > > Thanks for getting back to me. > > I ran it again with the ProtocolTrace flag and the only output there > is: 0: rtc: Real-time clock set to Sun Jan 1 00:00:00 2012 > > With the Exec flag, I do see spinlock output on and off in the beginning > during regular execution, so that is normal as you said. But once the > "problem" occurs shortly after, the Exec flag output is just continuous > spinlock forever as I posted previously. > > The exact gem5 command lines I use are posted in my previous post. The > kernel and disk image are the simply the default ones from the GEM5 > downloads page: http://www.m5sim.org/dist/current/x86/x86-system.tar.bz2 > > I have attached a zip file containing the following files: > > BOOT-config.ini - The config.ini from the first run, i.e. booting in > atomic mode, creating a checkpoint, and exiting. > BOOT-system.pc.com_1.terminal - The terminal output from the first run > CPT-config.ini - The config.ini when restoring from the checkpoint in > detailed mode > CPT-system.pc.com_1.terminal - The system output after restoring from the > checkpoint > run.c - The dummy program started by the run script > run.rcS - The run script > flag-exec-partial.out - The output from the Exec flag, right before the > "problem" occurs, The infinite spinlock starts at tick 5268700121500 > > Again, this problem occurs even without checkpointing. I have also tried > a few different kernels and disk images. I did the same test with both > alpha and arm64 and it works, so it appears to just be an issue with x86. > > Thank you, > > Ivan > > > > On Tue, Jun 3, 2014 at 7:53 PM, Joel Hestness <jthestn...@gmail.com> > wrote: > >> Hi Ivan, >> Sorry for the delay on this. >> >> I haven't had an opportunity to try to reproduce your problem, though >> the traces you've supplied here can help a bit. Specifically, the stalled >> LocalApics (plural, because 2 CPU cores) is fishy, because we'd expect >> periodic interrupts to continue. However, the last interrupt on CPU 1 >> appears to get cleared, which looks fine. The CPU spin lock is normal for >> threads that don't have any work to complete, but it's confusing why they >> wouldn't be doing something. >> >> The next thing to dig into would be to figure out what the CPUs were >> doing last before they entered the spin loop. For this we may need to >> trace a bit earlier in time using the Exec flags, and since it is likely >> that messages/responses may be getting lost in the memory hierarchy or >> devices, we'll need to use the ProtocolTrace flag to see what is being >> communicated. You could try playing around with these as a start. >> >> I may also have time to try to reproduce this over the next week, so >> I'm hoping you could give me some more information: can you send me your >> gem5 command line, config.ini, and system.pc.com_1.terminal output from >> your simulation, and details on the kernel and disk image that you're >> trying to use? >> >> >> Thanks! >> Joel >> >> >> >> >> On Sat, May 24, 2014 at 7:27 PM, Ivan Stalev <ids...@psu.edu> wrote: >> >>> Hi, >>> >>> Has anyone been able to reproduce this issue? >>> >>> Thanks, >>> >>> Ivan >>> >>> >>> On Sat, May 17, 2014 at 1:50 AM, Ivan Stalev <ids...@psu.edu> wrote: >>> >>>> Hi Joel, >>>> >>>> I am using revision 10124. I removed all of my own modifications just >>>> to be safe. >>>> >>>> Running with gem5.opt and restoring from a boot-up checkpoint >>>> with--debug-flag=Exec, it appears that the CPU is stuck in some sort of >>>> infinite loop, executing this continuously: >>>> >>>> 5268959012000: system.switch_cpus0 T0 : @_spin_lock_irqsave+18.0 : >>>> CMP_M_I : limm t2d, 0 : IntAlu : D=0x0000000000000000 >>>> 5268959012000: system.switch_cpus0 T0 : @_spin_lock_irqsave+18.1 : >>>> CMP_M_I : ld t1d, DS:[rdi] : MemRead : D=0x00000000fffffffe >>>> A=0xffffffff80822400 >>>> 5268959012000: system.switch_cpus0 T0 : @_spin_lock_irqsave+18.2 : >>>> CMP_M_I : sub t0d, t1d, t2d : IntAlu : D=0x0000000000000000 >>>> 5268959012000: system.switch_cpus0 T0 : @_spin_lock_irqsave+21.0 : >>>> JLE_I : rdip t1, %ctrl153, : IntAlu : D=0xffffffff80596897 >>>> 5268959012000: system.switch_cpus0 T0 : @_spin_lock_irqsave+21.1 : >>>> JLE_I : limm t2, 0xfffffffffffffff9 : IntAlu : D=0xfffffffffffffff9 >>>> 5268959012000: system.switch_cpus0 T0 : @_spin_lock_irqsave+21.2 : >>>> JLE_I : wrip , t1, t2 : IntAlu : >>>> 5268959012500: system.switch_cpus0 T0 : @_spin_lock_irqsave+16 : >>>> NOP : IntAlu : >>>> 5268959012500: system.switch_cpus0 T0 : @_spin_lock_irqsave+18.0 : >>>> CMP_M_I : limm t2d, 0 : IntAlu : D=0x0000000000000000 >>>> 5268959012500: system.switch_cpus0 T0 : @_spin_lock_irqsave+18.1 : >>>> CMP_M_I : ld t1d, DS:[rdi] : MemRead : D=0x00000000fffffffe >>>> A=0xffffffff80822400 >>>> 5268959012500: system.switch_cpus0 T0 : @_spin_lock_irqsave+18.2 : >>>> CMP_M_I : sub t0d, t1d, t2d : IntAlu : D=0x0000000000000000 >>>> 5268959012500: system.switch_cpus0 T0 : @_spin_lock_irqsave+21.0 : >>>> JLE_I : rdip t1, %ctrl153, : IntAlu : D=0xffffffff80596897 >>>> 5268959012500: system.switch_cpus0 T0 : @_spin_lock_irqsave+21.1 : >>>> JLE_I : limm t2, 0xfffffffffffffff9 : IntAlu : D=0xfffffffffffffff9 >>>> 5268959012000: system.switch_cpus1 T0 : @_spin_lock_irqsave+21.2 : >>>> JLE_I : wrip , t1, t2 : IntAlu : >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+16 : >>>> NOP : IntAlu : >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+18.0 : >>>> CMP_M_I : limm t2d, 0 : IntAlu : D=0x0000000000000000 >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+18.1 : >>>> CMP_M_I : ld t1d, DS:[rdi] : MemRead : D=0x00000000fffffffe >>>> A=0xffffffff80822400 >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+18.2 : >>>> CMP_M_I : sub t0d, t1d, t2d : IntAlu : D=0x0000000000000000 >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+21.0 : >>>> JLE_I : rdip t1, %ctrl153, : IntAlu : D=0xffffffff80596897 >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+21.1 : >>>> JLE_I : limm t2, 0xfffffffffffffff9 : IntAlu : D=0xfffffffffffffff9 >>>> 5268959012500: system.switch_cpus1 T0 : @_spin_lock_irqsave+21.2 : >>>> JLE_I : wrip , t1, t2 : IntAlu : >>>> 5268959013000: system.switch_cpus1 T0 : @_spin_lock_irqsave+16 : >>>> NOP : IntAlu : >>>> >>>> ....and so on repetitively without stopping. >>>> >>>> Using --debug-flag=LocalApic, the output does indeed stop shortly >>>> after restoring from the checkpoint. The last output is: >>>> .. >>>> 5269570990500: system.cpu1.interrupts: Reported pending regular >>>> interrupt. >>>> 5269570990500: system.cpu1.interrupts: Reported pending regular >>>> interrupt. >>>> 5269570990500: system.cpu1.interrupts: Generated regular interrupt >>>> fault object. >>>> 5269570990500: system.cpu1.interrupts: Reported pending regular >>>> interrupt. >>>> 5269570990500: system.cpu1.interrupts: Interrupt 239 sent to core. >>>> 5269571169000: system.cpu1.interrupts: Writing Local APIC register 5 at >>>> offset 0xb0 as 0. >>>> >>>> ...and no more output from this point on. >>>> >>>> I appreciate your help tremendously. >>>> >>>> Ivan >>>> >>>> >>>> >>>> On Fri, May 16, 2014 at 11:18 AM, Joel Hestness <jthestn...@gmail.com> >>>> wrote: >>>> >>>>> Hi Ivan, >>>>> I believe that the email thread you previously referenced was >>>>> related to a bug that we identified and fixed with changeset 9624 >>>>> <http://permalink.gmane.org/gmane.comp.emulators.m5.devel/19326>. >>>>> That bug was causing interrupts to be dropped in x86 when running with >>>>> the >>>>> O3 CPU. Are you using a version of gem5 after that changeset? If not, >>>>> I'd >>>>> recommend updating to a more recent version and trying to replicate this >>>>> issue again. >>>>> >>>>> If you are using a more recent version of gem5, first, please let >>>>> us know which changeset and whether you've made any changes. Then, I'd >>>>> recommend compiling gem5.opt and using the DPRINTF tracing functionality >>>>> to >>>>> see if you can zero in on what is happening. Specifically, first try >>>>> passing the flag --debug-flag=Exec to look at what the CPU cores are >>>>> executing (you may also want to pass --trace-start=<<tick>> with a >>>>> simulator tick time close to when the hang happens). This trace will >>>>> include Linux kernel symbols for at least some of the lines if executing >>>>> in >>>>> the kernel (e.g. handling an interrupt). If you've compiled your >>>>> benchmark >>>>> without debugging symbols, it may just show the memory addresses of >>>>> instructions being executed within the application. I will guess that >>>>> you'll see kernel symbols for at least some of the executed instructions >>>>> for interrupts. >>>>> >>>>> If it appears that the CPUs are continuing to execute, try running >>>>> with --debug-flag=LocalApic. This will print the interrupts that each >>>>> core >>>>> is receiving, and if it stops printing at any point, it means something >>>>> has >>>>> gone wrong and we'd have to do some deeper digging. >>>>> >>>>> Keep us posted on what you find, >>>>> Joel >>>>> >>>>> >>>>> >>>>> On Thu, May 15, 2014 at 11:16 PM, Ivan Stalev <ids...@psu.edu> wrote: >>>>> >>>>>> Hi Joel, >>>>>> >>>>>> I have tried several different kernels and disk images, including >>>>>> the default ones provided on the GEM5 website in the x86-system.tar.bz2 >>>>>> download. I run with these commands: >>>>>> >>>>>> build/X86/gem5.fast -d m5out/test_run configs/example/fs.py >>>>>> --kernel=/home/mdl/ids103/full_system_images/binaries/x86_64-vmlinux-2.6.22.9.smp >>>>>> -n 2 --mem-size=4GB --cpu-type=atomic --cpu-clock=2GHz >>>>>> --script=rcs_scripts/run.rcS --max-checkpoints=1 >>>>>> >>>>>> My run.rcS script simply contains: >>>>>> >>>>>> #!/bin/sh >>>>>> /sbin/m5 resetstats >>>>>> /sbin/m5 checkpoint >>>>>> echo 'booted' >>>>>> /extras/run >>>>>> /sbin/m5 exit >>>>>> >>>>>> where "/extras/run" is simply a C program with an infinite loop >>>>>> that prints a counter. >>>>>> >>>>>> I then restore: >>>>>> >>>>>> build/X86/gem5.fast -d m5out/test_run configs/example/fs.py >>>>>> --kernel=/home/mdl/ids103/full_system_images/binaries/x86_64-vmlinux-2.6.22.9.smp >>>>>> -r 1 -n 2 --mem-size=4GB --cpu-type=detailed --cpu-clock=2GHz --caches >>>>>> --l2cache --num-l2caches=1 --l1d_size=32kB --l1i_size=32kB --l1d_assoc=4 >>>>>> --l1i_assoc=4 --l2_size=4MB --l2_assoc=8 --cacheline_size=64 >>>>>> >>>>>> I specified the disk image file in Benchmarks.py. Restoring from >>>>>> the same checkpoint and running in atomic mode works fine. I also tried >>>>>> booting the system in detailed and letting it run for a while, but once >>>>>> it >>>>>> boots, there is no more output. So it seems that checkpointing is not the >>>>>> issue. The "run" program is just a dummy, and the same issue also >>>>>> persists >>>>>> when running SPEC benchmarks or any other program. >>>>>> >>>>>> My dummy program is simply: >>>>>> >>>>>> int count=0; >>>>>> printf("**************************** HEYY \n"); >>>>>> while(1) >>>>>> printf("\n %d \n", count++); >>>>>> >>>>>> Letting it run for a while, the only output is exactly this: >>>>>> >>>>>> booted >>>>>> ******* >>>>>> >>>>>> It doesn't even finish printing the first printf. >>>>>> >>>>>> Another thing to add: In another scenario, I modified the kernel to >>>>>> call an m5 pseudo instruction on every context switch, and then GEM5 >>>>>> prints >>>>>> that a context switch occurred. Once again, in atomic mode this worked as >>>>>> expected. However, in detailed, even the GEM5 (printf inside GEM5 itself) >>>>>> output stopped along with the system output in the terminal. >>>>>> >>>>>> Thank you for your help. >>>>>> >>>>>> Ivan >>>>>> >>>>>> >>>>>> On Thu, May 15, 2014 at 10:51 PM, Joel Hestness <jthestn...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Hi Ivan, >>>>>>> Can you please give more detail on what you're running? >>>>>>> Specifically, can you give your command line, and which kernel, disk >>>>>>> image >>>>>>> you're using? Are you using checkpointing? >>>>>>> >>>>>>> Joel >>>>>>> >>>>>>> >>>>>>> On Mon, May 12, 2014 at 10:52 AM, Ivan Stalev via gem5-users < >>>>>>> gem5-users@gem5.org> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I am running X86 in full system mode. When running just 1 CPU, >>>>>>>> both atomic and detailed mode work fine. However, with more than 1 CPU, >>>>>>>> atomic works fine, but in detailed mode the system appears to hang >>>>>>>> shortly >>>>>>>> after boot-up. GEM5 doesn't crash, but the system stops having any >>>>>>>> output. >>>>>>>> Looking at the stats, it appears that instructions are still being >>>>>>>> committed, but the actual applications/benchmarks are not making >>>>>>>> progress. >>>>>>>> The issue persists with the latest version of GEM5. I also tried two >>>>>>>> different kernel versions and several different disk images. >>>>>>>> >>>>>>>> I might be experiencing what seems to be the same issue that was >>>>>>>> found about a year ago but appears to not have been fixed: >>>>>>>> https://www.mail-archive.com/gem5-dev@gem5.org/msg08839.html >>>>>>>> >>>>>>>> Can anyone reproduce this or know of a solution? >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Ivan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> gem5-users mailing list >>>>>>>> gem5-users@gem5.org >>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joel Hestness >>>>>>> PhD Student, Computer Architecture >>>>>>> Dept. of Computer Science, University of Wisconsin - Madison >>>>>>> http://pages.cs.wisc.edu/~hestness/ >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Joel Hestness >>>>> PhD Student, Computer Architecture >>>>> Dept. of Computer Science, University of Wisconsin - Madison >>>>> http://pages.cs.wisc.edu/~hestness/ >>>>> >>>> >>>> >>> >> >> >> -- >> Joel Hestness >> PhD Student, Computer Architecture >> Dept. of Computer Science, University of Wisconsin - Madison >> http://pages.cs.wisc.edu/~hestness/ >> > > -- Joel Hestness PhD Student, Computer Architecture Dept. of Computer Science, University of Wisconsin - Madison http://pages.cs.wisc.edu/~hestness/
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
