Re: [openib-general] Re: mthca and LinuxBIOS

2005-08-07 Thread Ivan Kokshaysky
On Fri, Aug 05, 2005 at 09:33:54PM -0700, Grant Grundler wrote: > > ISTR making comments before about the offending patch on linux-pci mailing > > list. Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ? > > I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i]. > not

Re: [openib-general] Re: mthca and LinuxBIOS

2005-08-06 Thread yhlu
In LinuxBIOS internal structure for resource, We have index member in resource. So the resource will be count from 0, 7 or etc, but index member will point to real BAR position. I would like to see Kernel has simmliar definintion. in LinuxBIOS typedef uint64_t resource_t; struct resource {

Re: [openib-general] Re: mthca and LinuxBIOS

2005-08-05 Thread Grant Grundler
On Fri, Aug 05, 2005 at 04:03:00PM -0700, Greg KH wrote: ... > > yesterday, someone add pci_restore_bars, that will call > > pci_update_resource, and it will overwirte upper 32 bit of BAR2 and > > BAR4 of IB card. > > Hm, perhaps that change should not do this? > > Dominik, care to weigh in h

Re: [openib-general] Re: mthca and LinuxBIOS

2005-08-05 Thread Grant Grundler
On Fri, Aug 05, 2005 at 04:59:37PM -0700, Grant Grundler wrote: > ISTR making comments before about the offending patch on linux-pci mailing > list. Is this the same patch that assumes pci_dev->resource[i] == BAR[i] ? I meant the patch assume 1:1 for pci_dev->resource[i] and BAR[i]. not that the

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
I remember last year when I used IBGOLD 0.5 with PCI-X IB card, it seems that it could support 64 bit pref mem. I will try IBGOLD 1.7 . YH On 8/5/05, Roland Dreier <[EMAIL PROTECTED]> wrote: >yhlu> Roland, what is the -16 mean? > >yhlu> is it /* Attempt to modify a QP/EE which is no

Re: mthca and LinuxBIOS

2005-08-05 Thread Roland Dreier
yhlu> Roland, what is the -16 mean? yhlu> is it /* Attempt to modify a QP/EE which is not in the yhlu> presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10, No, -16 is just -EBUSY. You could put a printk in event_timeout() in mthca_cmd.c to make sure, but I'm pretty sure that's wh

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
Roland, what is the -16 mean? is it /* Attempt to modify a QP/EE which is not in the presumed state: */ MTHCA_CMD_STAT_BAD_QPEE_STATE = 0x10, YH On 8/5/05, yhlu <[EMAIL PROTECTED]> wrote: > You are right. CONG_SPECIAL_QP > > ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June

Re: [openib-general] Re: mthca and LinuxBIOS

2005-08-05 Thread Grant Grundler
On Fri, Aug 05, 2005 at 04:06:06PM -0700, Linus Torvalds wrote: > > > On Fri, 5 Aug 2005, Greg KH wrote: > > On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > > > But what's the real problem we are trying to fix here? > > We're screwing up the top 32 bits of the BAR when you r

Re: mthca and LinuxBIOS

2005-08-05 Thread Linus Torvalds
On Fri, 5 Aug 2005, Greg KH wrote: > On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > But what's the real problem we are trying to fix here? We're screwing up the top 32 bits of the BAR when you resume it. Look at the patch, you'll see the fix (the other part of the patch loo

Re: mthca and LinuxBIOS

2005-08-05 Thread Greg KH
On Fri, Aug 05, 2005 at 03:25:02PM -0700, yhlu wrote: > In LinuxBIOS, We can allocate 64 bit region ( 0xfc000) to the > mellanox Infiniband card. Some range above 4G. So the mmio below 4G > is some smaller only 128M, Otherwise need 512M. If 4 IB cards are > used, the mmio will be 2G. For n

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
In LinuxBIOS, We can allocate 64 bit region ( 0xfc000) to the mellanox Infiniband card. Some range above 4G. So the mmio below 4G is some smaller only 128M, Otherwise need 512M. If 4 IB cards are used, the mmio will be 2G. For new opteron E stepping, We could use hareware memhole support.

Re: mthca and LinuxBIOS

2005-08-05 Thread Greg KH
On Fri, Aug 05, 2005 at 01:38:37PM -0700, Linus Torvalds wrote: > > Hmm.. This looks half-way sane, but too ugly for words. > > I'd much rather see that when we detect a 64-bit resource, we always mark > the next resource as being reserved some way, and then we just make > pci_update_resource()

Re: mthca and LinuxBIOS

2005-08-05 Thread Linus Torvalds
Hmm.. This looks half-way sane, but too ugly for words. I'd much rather see that when we detect a 64-bit resource, we always mark the next resource as being reserved some way, and then we just make pci_update_resource() ignore such reserved resources. The if((resno & 1)==1) {

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
please check the patch for fix overwrite upper 32bit YH --- drivers/pci/setup-res.c.orig2005-08-05 10:08:45.0 -0700 +++ drivers/pci/setup-res.c 2005-08-05 13:25:06.0 -0700 @@ -33,6 +33,18 @@ u32 new, check, mask; int reg; +if (resno < 6) { +

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
in drivers/pci/setup-res.c: pci_update_resource() why new = 0; /* currently everyone zeros the high address */ if ((new & (PCI_BASE_ADDRESS_SPACE|PCI_BASE_ADDRESS_MEM_TYPE_MASK)) == (PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64)) {

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
pci_restore_bars cause that. it didn't restore that according to if resource is 64 bit or not. So it overwirte upper 32 bit with 0. YH file:1b34fc56067ed8ae0ba9b32f46679e13068bb86c -> file:65ea7d25f6911d7396e19afbf4bb2738906376f7 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -222,6 +222,37 @

Re: mthca and LinuxBIOS

2005-08-05 Thread Roland Dreier
> ib_mthca :04:00.0: FW version 000400060002, max commands 64 This is FW 4.6.2 -- 4.7.0 has been released, so it might be worth trying that. > ib_mthca :04:00.0: NOP command IRQ test passed > ib_mthca :04:00.0: mthca_init_qp_table: mthca_CONF_SPECIAL_QP failed for 0/1024

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
before I do the cg-update this morning, it didn't mask out the upper 8 bit. YH On 8/5/05, Roland Dreier <[EMAIL PROTECTED]> wrote: > yhlu> ps. some kernel pci code patch broke sth yesterday night. > yhlu> it mask out bit [32-39] > > Is it possible that all your problems are coming from

Re: mthca and LinuxBIOS

2005-08-05 Thread Roland Dreier
yhlu> ps. some kernel pci code patch broke sth yesterday night. yhlu> it mask out bit [32-39] Is it possible that all your problems are coming from the PCI setup code incorrectly assigning BARs? - R. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body o

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
ps. some kernel pci code patch broke sth yesterday night. it mask out bit [32-39] ib_mthca :04:00.0: profile[ 0]--10/20 @ 0xe000 (size 0x 400) ib_mthca :04:00.0: profile[ 1]-- 0/16 @ 0xe400 (size 0x 100) ib_mthca :04:00.0: profile[ 2]-- 7/18 @ 0xe

Re: mthca and LinuxBIOS

2005-08-05 Thread yhlu
You are right. CONG_SPECIAL_QP ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (:04:00.0) ib_mthca :04:00.0: FW version 000400060002, max commands 64 ib_mthca :04:00.0: FW s

Re: mthca and LinuxBIOS

2005-08-04 Thread Roland Dreier
Hmm, that output all looks fine. Can you run with the patch below to see exactly where the QP table initialization fails? (I haven't actually compiled this patch so you may have to fix a typo or two) I'm guessing that the CONF_SPECIAL_QP command is failing, but let's make sure. Thanks, Roland

Re: mthca and LinuxBIOS

2005-08-04 Thread yhlu
ib_mthca: Mellanox InfiniBand HCA driver v0.06 (June 23, 2005) ib_mthca: Initializing Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (:04:00.0) ib_mthca :04:00.0: FW version 000400060002, max commands 64 ib_mthca :04:00.0: FW size 6143 KB (start fcefa0, e

Re: mthca and LinuxBIOS

2005-08-04 Thread Roland Dreier
yhlu> i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any yhlu> more debug info, is that depend other setting? It shouldn't depend on anything. mthca_dbg() gets turned into dev_dbg(), which just does dev_printk(KERN_DEBUG,...). Perhaps you have to change your console level to see

Re: mthca and LinuxBIOS

2005-08-04 Thread yhlu
i enable CCONFIG_INFINIBAND_MTHCA_DEBUG=y I didn't get any more debug info, is that depend other setting? YH On 8/4/05, yhlu <[EMAIL PROTECTED]> wrote: > The mellanox can use prefmem64, but the BIOS could only allocate the > some range below 4G, So 32 bit OS still can use the IB cards. > but for

Re: mthca and LinuxBIOS (was: [PATCH 1/2] [IB/cm]: Correct CM port redirect reject codes)

2005-08-04 Thread yhlu
YES. I will send you the output message later about "CONFIG_INFINIBAND_MTHCA_DEBUG=y" YH On 8/3/05, Roland Dreier <[EMAIL PROTECTED]> wrote: > yhlu> In LinuxBIOS, If I enable the prefmem64 to use real 64 > yhlu> range. the IB driver in Kernel can not be loaded. > > What does it mean to

Re: mthca and LinuxBIOS

2005-08-04 Thread yhlu
The mellanox can use prefmem64, but the BIOS could only allocate the some range below 4G, So 32 bit OS still can use the IB cards. but for 64bit OS, We could allocate range above 4G (0xfc), So the mmio below 4G can be smaller. ( for example from 512M to 128M, the user can get back some RAM

Re: mthca and LinuxBIOS

2005-08-04 Thread Roland Dreier
> "yhlu" == yhlu <[EMAIL PROTECTED]> writes: yhlu> YES. I will send you the output message later about yhlu> "CONFIG_INFINIBAND_MTHCA_DEBUG=y" Thanks. In the meantime, can you explain what it means to "enable the prefmem64 to use real 64 range"? What is the difference between this