Hi David, It sounds like there are plenty things going on here, and that you already got quite far in your debugging.
The bridge has no clue of any 32/64 bit BAR, it cares only about the physical 64-bit address ranges, and if you add a range that is unique (not in the normal memory range for example) it should work fine. The bus will tell you if there are multiple destinations for the same address. Could you perhaps post these few lines that you've added? If I remember correctly, you also have to ensure that the pciconfig is the default slave and that the address mapping is ignored for that range. You can also have a look at dev/arm/Realview.py for more examples of how this is done. I don't dare say what's the right way with the X86 interrupts, but hopefully the hints above will sort out the address-related issues and someone else can help with the latter problem. Andreas On 12/04/2013 21:40, "David Miller" <david.mil...@cl.cam.ac.uk> wrote: >Hi all, > >I'm trying to add a NIC to an x86 system, but apparently I'm missing a >step - probably to do with the memory range list of the membus-iobus >bridge - can anybody straighten me out? > >* So far, I've made a copy of configs/examples/fs.py, and added to it a >copy of the routines makeX86System() and makeLinuxX86System() from >configs/common/FSConfig.py so I can customise makeX86System(). > >(In addition to adding a NIC, I've also fiddled with the default kernel >images and disc configuration, including deleting the swap image and >adding a disc for /home with the COW layer disabled, even if, for now, >/home is mounted read-only.) > >After the disc-related code, I've added: > > self.eth0 = IGbE_igb( pci_bus=0, pci_dev=2, pci_func=0 ) > self.eth0.pio = self.iobus.master > self.eth0.config = self.iobus.master > self.eth0.dma = self.iobus.slave > >which is all that the other architectures appear to be doing. > >* Additionally, I've compiled a custom 2.6.28.4 kernel (based on the >configs available from m5sim.org) that includes the igb and e1000e >drivers, and built a custom ubuntu OS image using debootstrap. > >* What's going wrong: > >As shown in kernel boot log 1 below, the address range allocated to the >NIC's BAR0 (x10000000) is for some reason inaccessible, and also the >interrupt configuration isn't happy. > >The machine boots to a shell okay. The device appears in lspci, and the >relevant line of /proc/iomem is: > >> 10000000-1001ffff : 0000:00:02.0 > >there is no entry in /proc/interrupts, but that's probably not >surprising given the foregoing. > >So, the kernel is at least aware that there is a PCI device there. >During boot, as shown in the kernel boot log, attempts to read that BAR >go pear-shaped. After the machine has booted, an attempt to read that >address manually produces the following message in the gem5 trace log: > >> 1449801470671000: system.membus: Unable to find destination for addr >>0x10000000, will use default port >> warning: caught [10000000:10000003] BadAddressError > >the second line being a message I hacked into src/cpu/simple/atomic.cc >in lieu of the assert which otherwise causes such an access to kill the >simulation. > >At this point, my best guess is that the reason is that the assignment >of x86_sys.bridge.ranges in FSConfig.connectX86ClassicSystem doesn't >appear to allow for 32-bit BARs, and particularly not the physical >address 0x10000000. As shown in the trace log below, the iobus knows >about that range, but grep says membus does not. > >Short of manually adding that address range into bridge.ranges, what is >the right way to map IO space into the mem space? Is there some way of >forwarding the address range change through the bridge? Obviously, a >device with a 32-bit BAR can't take a 64-bit address, so I can't map it >into the IO space beginning at 800000000000000. > >How best to resolve this? > >By the way, is the kernel panic (boot log 2 below) important? I've seen >others mention it, but no resolution of it. > > >* I'm confused about how to connect the interrupt line appropriately. > >There seem to be two mechanisms for connecting interrupt lines: > * X86IntLine: eg as used in SouthBridge to attach the legacy devices to >the IOAPIC. I'm unclear what this struct does precisely, or why the >legacy devices can't be directly attached to the IOAPIC. > As far as I can tell, the list of struct X86IntLine instances, >system.pc.south_bridge.int_lines, is somehow used to make the >appropriate connections during elaboration, but I don't understand how >this mechanism works (or even where to start). > > * X86IntelMPIOIntAssignment: creates entries in the intel MP table so >the kernel can discover what interrupt sources there are and where >they're attached though, again, I can't see where the connections >between the bus/device and the IOAPIC are actually made. > >Using the IDE controller as a reference, I see it instantiated in >SouthBridge.py, and there is nothing about the IDE controller in Pc.py >(where SouthBridge is instantiated). The IDE controller is not >mentioned explicitly in makeX86system() either, but there is an >instantiation of X86IntelMPIOIntAssignment called pci_dev4_inta there, >which happens to be the device ID of the IDE controller. > >This X86IntelMPIOIntAssignment declaration really confuses me because >the interrupt number in this declaration (destination = IOAPIC pin 16) >doesn't appear to match the parameters reported by the kernel in >/proc/interrupts or as set in SouthBridge.py, both of which say that the >primary IDE controller is on int 14. > >Similarly, source_bus_id = 1 (earlier associated with the PCI bus ‹ so >far so good), but I don't see how this bus ID is associated with the >iobus instance (other than in the MP table entry, but I don't see how >that's connected with system.iobus either), nor do I see where PCI bus 0 >device ID 4 is associated in the declaration, other than by >source_bus_irq, but if that, then what's the << 2 for? (can't be >allowance for function number, that has 3 bits.) > > >I haven't long been using gem5 and, as I'm sure it was with the rest of >you, it's a pretty daunting amount of code to absorb all at once, >especially given the last time I did anything with even a moderately >complex C++ code base was something like 15 years ago in my >undergraduate degree. > >I've spent most of the intervening time messing with FPGAs and PCI >endpoints. My interests with gem5 are to model interconnect performance >and, if I can produce code of adequate quality, I hope to be able to >contribute those improved models back to the gem5 community. > >I'm grateful for any direct assistance you can provide, even >(especially) general advice wrt how to go about adapting the bus and >bridge models for these ends. > > >=== kernel boot log 1 === > >> Intel(R) Gigabit Ethernet Network Driver - version 1.2.45-k2 >> Copyright (c) 2008 Intel Corporation. >> igb 0000:00:02.0: enabling device (0000 -> 0002) >> igb 0000:00:02.0: can't find IRQ for PCI INT A; probably buggy MP table >> ioremap: invalid physical address 10000000 >> ------------[ cut here ]------------ >> WARNING: at arch/x86/mm/ioremap.c:212 __ioremap_caller+0x33a/0x380() >> Modules linked in: >> Pid: 1, comm: swapper Tainted: G W 2.6.28.4 #2 >> Call Trace: >> [<ffffffff8022ee3a>] warn_on_slowpath+0x5a/0x90 >> [<ffffffff80228bb5>] place_entity.clone.77+0xb5/0xd0 >> [<ffffffff805eac93>] igb_probe+0x251/0xb4c >> [<ffffffff805fb113>] printk+0x40/0x45 >> [<ffffffff8022198a>] __ioremap_caller+0x33a/0x380 >> [<ffffffff8055ca60>] ether_setup+0x0/0x80 >> [<ffffffff8054b6de>] alloc_netdev_mq+0x11e/0x160 >> [<ffffffff805eac93>] igb_probe+0x251/0xb4c >> [<ffffffff802ced3d>] sysfs_addrm_finish+0x1d/0x220 >> [<ffffffff802cef9d>] sysfs_find_dirent+0x2d/0x40 >> [<ffffffff802cf175>] __sysfs_add_one+0x75/0xb0 >> [<ffffffff80391898>] pci_device_probe+0x78/0xa0 >> [<ffffffff803c7322>] driver_sysfs_add+0x62/0xa0 >> [<ffffffff803c7549>] driver_probe_device+0x99/0x1b0 >> [<ffffffff803c7703>] __driver_attach+0xa3/0xb0 >> [<ffffffff803c7660>] __driver_attach+0x0/0xb0 >> [<ffffffff803c66c6>] bus_for_each_dev+0x56/0x80 >> [<ffffffff803c6f0a>] bus_add_driver+0xaa/0x250 >> [<ffffffff808222d2>] igb_init_module+0x0/0x58 >> [<ffffffff803c78d1>] driver_register+0x61/0x130 >> [<ffffffff808222d2>] igb_init_module+0x0/0x58 >> [<ffffffff80391b25>] __pci_register_driver+0x55/0x90 >> [<ffffffff8020917a>] do_one_initcall+0x4a/0x1c0 >> [<ffffffff8037d038>] ida_get_new_above+0x158/0x1c0 >> [<ffffffff802c7cc2>] proc_register+0x42/0x1c0 >> [<ffffffff802c82a3>] create_proc_entry+0x53/0xa0 >> [<ffffffff80256d49>] register_irq_proc+0x89/0xa0 >> [<ffffffff802c0000>] load_elf_binary+0x16d0/0x1950 >> [<ffffffff80806b21>] kernel_init+0x70/0xbf >> [<ffffffff8020bf79>] child_rip+0xa/0x11 >> [<ffffffff80806ab1>] kernel_init+0x0/0xbf >> [<ffffffff8020bf6f>] child_rip+0x0/0x11 >> ---[ end trace 4eaa2a86a8e2da22 ]--- >> igb: probe of 0000:00:02.0 failed with error -5 > >=== kernel boot log 2 === > >> WARNING: at arch/x86/kernel/cpu/mtrr/main.c:1604 >>mtrr_trim_uncached_memory+0x2f9/0x2fb() >> WARNING: strange, CPU MTRRs all blank? >> Modules linked in: >> Pid: 0, comm: swapper Not tainted 2.6.28.4 #2 >> Call Trace: >> [<ffffffff8022eda8>] warn_slowpath+0xb8/0xf0 >> [<ffffffff8021d464>] early_serial_write+0x24/0x50 >> [<ffffffff8021d464>] early_serial_write+0x24/0x50 >> [<ffffffff8022f0d9>] __call_console_drivers+0x49/0x60 >> [<ffffffff80218cf0>] post_set+0x20/0x40 >> [<ffffffff8080fe63>] mtrr_trim_uncached_memory+0x2f9/0x2fb >> [<ffffffff805fb113>] printk+0x40/0x45 >> [<ffffffff8080c8e1>] setup_arch+0x37a/0x5bb >> [<ffffffff80806837>] start_kernel+0x5e/0x2d8 >> [<ffffffff8080632f>] x86_64_start_kernel+0x0/0xe4 >> ---[ end trace 4eaa2a86a8e2da22 ]--- > >=== excerpts from gem5 trace output === > >> command line: build/X86/gem5.opt >>--debug-flags=PCIDEV,PciConfigAll,Bridge,BaseBus,BusAddrRanges,PCEvent,Pa >>cketQueue -d rt/fs-djm configs/fs-djm.py >> >> 0: system.iobus: Received range change from slave port >>system.eth0-pciconf >> 0: system.iobus: Adding range [0xc000000000001000 : >>0xc0000000000010ff] for id 19 >> 0: system.iobus: Received range change from slave port >>system.eth0.pio >> 0: system.iobus: Got address ranges from all slaves >> 0: system.iobus: Aggregating bus ranges >> 0: system.iobus: -- Adding default [0xc000000000000000 : >>0xc000000000ffffff] >> 0: system.iobus: -- Adding range [0 : 0x7ffffff] >> 0: system.iobus: -- Adding range [0xfec00000 : 0xfec00003] >> 0: system.iobus: -- Adding range [0xfec00010 : 0xfec00013] >> >> 12690582500: system.eth0: readConfig: dev 0x2 func 0 reg 0xa 2 bytes: >>data = 0x200 >> 12690609000: system.eth0: readConfig: dev 0x2 func 0 reg 0 2 bytes: >>data = 0x8086 >> 12690633000: system.eth0: readConfig: dev 0x2 func 0 reg 0x2 2 bytes: >>data = 0x10c9 >> (snip more PCI config cycles) >> >> 301374837000: system.iobus: Received range change from slave port >>system.eth0.pio >> 301374837000: system.iobus: Adding range [0x10000000 : 0x1001ffff] for >>id 18 >> 301374837000: system.iobus: Aggregating bus ranges >> 301374837000: system.iobus: -- Adding default [0xc000000000000000 : >>0xc000000000ffffff] >> 301374837000: system.iobus: -- Adding range [0 : 0x7ffffff] >> 301374837000: system.iobus: -- Adding range [0x10000000 : 0x1001ffff] >> 301374837000: system.iobus: -- Adding range [0xfec00000 : 0xfec00003] >> 301374837000: system.iobus: -- Adding range [0xfec00010 : 0xfec00013] >> 301374837000: system.iobus: -- Adding range [0x8000000000000000 : >>0x800000000000000f] >> 301374837000: system.iobus: -- Adding range [0x8000000000000020 : >>0x8000000000000021] >> >> 301374837000: system.eth0: writeConfig: dev 0x2 func 0 reg 0x10 4 >>bytes: data = 0x10000000 >> 301374928500: system.eth0: readConfig: dev 0x2 func 0 reg 0x10 4 >>bytes: data = 0x10000000 > >> 1449801470671000: system.membus: Unable to find destination for addr >>0x10000000, will use default port >> warning: caught [10000000:10000003] BadAddressError >_______________________________________________ >gem5-users mailing list >gem5-users@gem5.org >http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users > -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
