RE: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-13 Thread Joachim Fenkes
"Caitlin Bestler" <[EMAIL PROTECTED]> wrote on 13.12.2007 22:08:34: > To clarify, an FMR Work Request is simply posted to the SendQ like > any other Work Request (of course the QP has to be privileged, or > it will complete in error). An SQ Post should never block. This would require hardware su

RE: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-13 Thread Caitlin Bestler
gt; Subject: Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize > HCA-related hCalls on POWER5 > > [EMAIL PROTECTED] wrote on 13.12.2007 20:22:49: > > > On Dec 13, 2007 12:30 AM, Or Gerlitz <[EMAIL PROTECTED]> wrote: > > > The current implementation of the open i

Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-13 Thread Joachim Fenkes
[EMAIL PROTECTED] wrote on 13.12.2007 20:22:49: > On Dec 13, 2007 12:30 AM, Or Gerlitz <[EMAIL PROTECTED]> wrote: > > The current implementation of the open iscsi initiator makes sure to > > issue commands in thread (sleepable) context, see iscsi_xmitworker and > > references to it in drivers/scsi

Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-13 Thread Caitlin Bestler
On Dec 13, 2007 12:30 AM, Or Gerlitz <[EMAIL PROTECTED]> wrote: > Roland Dreier wrote: > > I think the right fix for iSER would be to make iSER work even for > > devices that don't support FMRs. For example cxgb3 doesn't implement > > FMRs so if anyone ever updates iSER to work on iWARP and not ju

Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-13 Thread Or Gerlitz
Roland Dreier wrote: > I think the right fix for iSER would be to make iSER work even for > devices that don't support FMRs. For example cxgb3 doesn't implement > FMRs so if anyone ever updates iSER to work on iWARP and not just IB, > then this is something that has to be tackled anyway. Then ehc

Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-12 Thread Roland Dreier
> What is the fix you suggest, to add a device query that tells you for > which verbs the documentation does not apply? or enhance the code of the > map_phys_fmr verb within the ehca driver to return error if called > from non-sleepable context? I think the right fix for iSER would be to

Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-12 Thread Or Gerlitz
Joachim Fenkes wrote: > Roland Dreier <[EMAIL PROTECTED]> wrote on 10.12.2007 22:47:37: >> It's an optional device feature, so this should be OK >> (although the iSER driver currently seems to depend on a device >> supporting FMRs, which is probably going to be a problem with iWARP >> support in t

Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-12 Thread Christoph Raisch
Or Gerlitz <[EMAIL PROTECTED]> wrote on 12.12.2007 13:14:25: > Joachim Fenkes wrote: > > Roland Dreier <[EMAIL PROTECTED]> wrote on 10.12.2007 22:47:37: > > >> It's an optional device feature, so this should be OK > >> (although the iSER driver currently seems to depend on a device > >> supporting

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-11 Thread Joachim Fenkes
Roland Dreier <[EMAIL PROTECTED]> wrote on 10.12.2007 22:47:37: > It's a big problem. If you cannot implement FMRs in such a way that > you can handling having map_phys_fmr being called in a context that > can't sleep, then I think the only option is to remove your FMR > support. That's kind of

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-10 Thread Roland Dreier
> > map_phys_fmr > > In fact, we do use hCalls there. Our hardware doesn't actually support FMRs, > so we translate a "map FMR" into a "reallocate PMR", which doesn't work > without hCalls. What's more, the hCalls involved (e.g. H_FREE_RESOURCE) > might well return H_LONG_BUSY, so the wh

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-10 Thread Joachim Fenkes
Hi, guys, > We're taking this to the firmware architects at the moment, but they're not > very fond of the idea of reporting the absence of bugs through capability > flags, as this could quickly lead to the exhaustion of flag bits. We'll let > the discussion stew for a bit, but if we don't g

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-10 Thread Joachim Fenkes
On Monday 10 December 2007 00:22, Roland Dreier wrote: > Fair enough... according to Documentation/infiniband/core_locking.txt, > the only driver methods that cannot sleep are: > > [...] > map_phys_fmr In fact, we do use hCalls there. Our hardware doesn't actually support FMRs, so we tran

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-09 Thread Roland Dreier
> I think it needs some more inspection. The msleep in there is only called > for hcalls that return H_IS_LONG_BUSY(). In theory, you can call > ehca_plpar_hcall_norets() from inside an interrupt handler if the > hcall in question never returns long busy. Fair enough... according to Documentat

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-07 Thread Joachim Fenkes
Roland Dreier <[EMAIL PROTECTED]> wrote on 06.12.2007 19:27:09: > > > + ehca_lock_hcalls = !(cur_cpu_spec->cpu_user_features > > > +& PPC_FEATURE_ARCH_2_05); > > > We already talked about this yesterday, but I still feel that checking the >

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-07 Thread Arnd Bergmann
On Thursday 06 December 2007, Roland Dreier wrote: >  > Regarding the performance problem, have you checked whether converting all >  > your spin_lock_irqsave to spin_lock/spin_lock_irq improves your performance >  > on the older machines? Maybe it's already fast enough that way. > > It does seem

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-06 Thread Roland Dreier
> > +   ehca_lock_hcalls = !(cur_cpu_spec->cpu_user_features > > +        & PPC_FEATURE_ARCH_2_05); > We already talked about this yesterday, but I still feel that checking the > instruction set of the CPU should not be used to determine whether a > spe

Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-06 Thread Arnd Bergmann
On Thursday 06 December 2007, Joachim Fenkes wrote: > printk(KERN_INFO "eHCA Infiniband Device Driver " >       "(Version " HCAD_VERSION ")\n"); >   > +   /* Autodetect hCall locking -- we can't read the firmware version > +    * directly, but we know that starting with POW

[PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

2007-12-06 Thread Joachim Fenkes
All firmware versions on POWER5 systems have a locking issue in the HCA-related hCalls that can cause loss of Infiniband connectivity if allocate and free calls happen in parallel. This may for example be caused if two processes are using OpenMPI in parallel. Circumvent this by serializing all HCA-