To elaborate a bit on Patrick's answer, there is no mechanism to do this on the
*client*, because the performance difference between client RAM and server
storage is still fairly significant, especially if the application is doing
sub-page read or write operations.
However, on the *server* the OSS and MDS will *not* put flash storage into the
page cache, because using the kernel page cache has a measurable overhead, and
(at least in our testing) the performance of NVMe IOPS is actually better
*without* the page cache because more CPU is available to handle RPCs. This is
controlled on the server with
osd-ldiskfs.*.{read_cache_enable,writethrough_cache_enable}, default to 0 if
the block device is non-rotational, default to 1 if block device is rotational.
Separately, there is a tunable for avoiding the page cache for large read/write
RPCs, osd-ldiskfs.*.readcache_max_io_mb=8 by default, so RPCs >= 8MB go
directly to the disk, to avoid blowing out the page cache on the server.
Cheers, Andreas
> On May 19, 2022, at 12:21, Patrick Farrell via lustre-discuss
> <[email protected]> wrote:
>
> Well, you could use two file descriptors, one for O_DIRECT one otherwise. 🙂
>
> SSD is a fast medium but my instinct is the desirability of having data in
> RAM is much more about I/O pattern and hard to optimize for in advance - Do
> you read the data you wrote? (Or read data repeatedly?)
>
> In any case, there's no mechanism today. It's also relatively marginal if
> we're just doing buffered I/O then forcing the data out - it will reduce
> memory usage but it won't improve performance.
>
> -Patrick
>
> From: John Bauer <[email protected]>
> Sent: Thursday, May 19, 2022 1:16 PM
> To: Patrick Farrell <[email protected]>; [email protected]
> <[email protected]>
> Subject: Re: [lustre-discuss] Avoiding system cache when using ssd pfl extent
>
> Pat,
> No, not in general. It just seems that if one is storing data on an SSD it
> should be optional to have it not stored in memory ( why store in 2 fast
> mediums ).
> O_DIRECT is not of value as that would apply to all extents, whether on SSD
> on HDD. O_DIRECT on Lustre has been problematic for me in the past,
> performance wise.
> John
> On 5/19/22 13:05, Patrick Farrell wrote:
>> No, and I'm not sure I agree with you at first glance.
>>
>> Is this just generally an idea that data stored on SSD should not be in RAM?
>> If so, there's no mechanism for that other than using direct I/O.
>>
>> -Patrick
>> From: lustre-discuss <[email protected]> on behalf of
>> John Bauer <[email protected]>
>> Sent: Thursday, May 19, 2022 12:48 PM
>> To: [email protected] <[email protected]>
>> Subject: [lustre-discuss] Avoiding system cache when using ssd pfl extent
>>
>> When using PFL, and using an SSD as the first extent, it seems it would
>> be advantageous to not have that extent's file data consume memory in
>> the client's system buffers. It would be similar to using O_DIRECT, but
>> on a per-extent basis. Is there a mechanism for that already?
>>
>> Thanks,
>>
>> John
>>
>> _______________________________________________
>> lustre-discuss mailing list
>> [email protected]
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
> _______________________________________________
> lustre-discuss mailing list
> [email protected]
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
