On Fri, May 10, 2019 at 01:12:49PM -0700, Santosh Shilimkar wrote: > On 5/10/2019 12:47 PM, Jason Gunthorpe wrote: > > On Fri, May 10, 2019 at 12:38:31PM -0700, Santosh Shilimkar wrote: > > > On 5/10/2019 12:20 PM, Jason Gunthorpe wrote: > > > > On Fri, May 10, 2019 at 11:58:42AM -0700, santosh.shilim...@oracle.com > > > > wrote: > > > > > On 5/10/19 11:07 AM, Jason Gunthorpe wrote: > > > > > > On Fri, May 10, 2019 at 11:02:35AM -0700, > > > > > > santosh.shilim...@oracle.com wrote: > > > > > > [...] > > > > > > > > > Why would you need to detect FS DAX memory? GUP users are not > > > > > > supposed > > > > > > to care. > > > > > > > > > > > > GUP is supposed to work just 'fine' - other than the usual bugs we > > > > > > have with GUP and any FS backed memory. > > > > > > > > > > > Am not saying there is any issue with GUP. Let me try to explain the > > > > > issue first. You are aware of various discussions about doing DMA > > > > > or RDMA on FS DAX memory. e.g [1] [2] [3] > > > > > > > > > > One of the proposal to do safely RDMA on FS DAX memory is/was ODP > > > > > > > > It is not about safety. ODP is required in all places that would have > > > > used gup_longterm because ODP avoids th gup_longterm entirely. > > > > > > > > > Currently RDS doesn't have support for ODP MR registration > > > > > and hence we don't want user application to do RDMA using > > > > > fastreg/fmr on FS DAX memory which isn't safe. > > > > > > > > No, it is safe. > > > > > > > > The only issue is you need to determine if this use of GUP is longterm > > > > or short term. Longterm means userspace is in control of how long the > > > > GUP lasts, short term means the kernel is in control. > > > > > > > > ie posting a fastreg, sending the data, then un-GUP'ing on completion > > > > is a short term GUP and it is fine on any type of memory. > > > > > > > > So if it is a long term pin then it needs to be corrected and the only > > > > thing the comment needs to explain is that it is a long term pin. > > > > > > > Thanks for clarification. At least the distinction is clear to me now. Yes > > > the key can be valid for long term till the remote RDMA IO is issued and > > > finished. After that user can issue an invalidate/free key or > > > upfront specify a flag to free/invalidate the key on remote IO > > > completion. > > > > Again, the test is if *userspace* controls this. So if userspace is > > the thing that does the invalidate/free then it is long term. Sounds > > like if it specifies the free/invalidate flag then it short term. > > > > At this point you'd probably be better to keep both options. > > > Thats possible using the provided flag state but I am still not sure > whether its guaranteed to be safe when marked as short term even with > flag which tells kernel to invalidate/free the MR on remote IO > completion. Till the remote server finishes the IO,
This is fine. > there is still a window where userspace on local server can modify > the file mappings. Registered file handle say was ftuncated to zero > by another process and the backing memory was allocated by some > other process as part of fallocate. The FS is supposed to maintain sane semantics across GUP - fallocate should block until GUP is done. This is normal. > How do we avoid such an issue without GUP_longterm ? You don't, there is no problem using GUP for short term - ie a time frame entirely under control of the kernel. Jason
