Re: Keep a file in memory an any cost

2011-05-11 Thread Alain Spineux
On Wed, May 11, 2011 at 10:57 PM, JB wrote: > > I do not know if you are aware of: > $ man mmap > ... >       MAP_LOCKED (since Linux 2.5.37) >              Lock the pages of the mapped region into memory in the manner of >              mlock(2).  This flag is ignored in older kernels. > ... > > J

Re: Keep a file in memory an any cost

2011-05-11 Thread Joe Zeff
On 05/11/2011 01:54 PM, Tim wrote: > Back when I worked in a library, one of our fears was that some > miscreant would tip out the card file index, and we'd have to spend an > unimaginable amount of time putting the cards back in order. Yes, I know. Back in the mid '80s I did a little work for a

Re: Keep a file in memory an any cost

2011-05-11 Thread JB
Alain Spineux gmail.com> writes: > > Joe: The index is modified very often and need to be flushed at > regular interval, I don't want to handle read and write. > JB: I already use mmap. > JD: I already tried posix_fadvise with POSIX_FADV_WILLNEED, but this > is not rock solid, if not used for a

Re: Keep a file in memory an any cost

2011-05-11 Thread Tim
Tim: >> Tim, off to bubble-sort the junk piling up on my desk... Joe Zeff: > You do know, don't you, that the most succinct way to write a > bubble-sort is to use recursion? I think I still have the source code > sitting around from the time I did it in C. If you want, let me know > off-list

Re: Keep a file in memory an any cost

2011-05-11 Thread Alain Spineux
Joe: The index is modified very often and need to be flushed at regular interval, I don't want to handle read and write. JB: I already use mmap. JD: I already tried posix_fadvise with POSIX_FADV_WILLNEED, but this is not rock solid, if not used for a long time, data are not in the cache anymore. Br

Re: Keep a file in memory an any cost

2011-05-11 Thread Bryn M. Reeves
On 05/11/2011 08:33 PM, Joe Zeff wrote: > On 05/11/2011 12:19 PM, Bryn M. Reeves wrote: >> And have it all chucked onto the swap device when we're under memory pressure >> and reclaiming user pages that haven't been touched in a while? How does >> that help? > > It's much less likely, especially

Re: Keep a file in memory an any cost

2011-05-11 Thread Joe Zeff
On 05/11/2011 12:19 PM, Tim wrote: > Tim, off to bubble-sort the junk piling up on my desk... You do know, don't you, that the most succinct way to write a bubble-sort is to use recursion? I think I still have the source code sitting around from the time I did it in C. If you want, let me know

Re: Keep a file in memory an any cost

2011-05-11 Thread Joe Zeff
On 05/11/2011 12:19 PM, Bryn M. Reeves wrote: > And have it all chucked onto the swap device when we're under memory pressure > and reclaiming user pages that haven't been touched in a while? How does that > help? It's much less likely, especially if you're using that file often enough to worry

Re: Keep a file in memory an any cost

2011-05-11 Thread Tim
On Wed, 2011-05-11 at 20:19 +0200, Alain Spineux wrote: > How can I keep it in memory all the time ? "At any cost..." Is lots more RAM out of the question? Alternatively, I'd expect a programmer to ask whether there's a better way of indexing. My real programming days are long ago, though. Not

Re: Keep a file in memory an any cost

2011-05-11 Thread Bryn M. Reeves
On 05/11/2011 07:32 PM, Joe Zeff wrote: > On 05/11/2011 11:19 AM, Alain Spineux wrote: >> How can I keep it in memory all the time ? >> I can read it at regular interval, but this is a little over kill ? > > Is there a reason that you don't do the obvious: read it into an array, > use it and then

Re: Keep a file in memory an any cost

2011-05-11 Thread Bryn M. Reeves
On 05/11/2011 07:58 PM, JD wrote: > mlock(2): > int mlock(const void *addr, size_t len); > can lock user virtual pages to memory (if user has the privileges). > But when it comes to a file, you have no way of knowing the VA of the > pages in > which a file is stored, so you will not be able to cal

Re: Keep a file in memory an any cost

2011-05-11 Thread JD
On 05/11/11 11:19, Alain Spineux wrote: > Hello I have a file, that is an index, it is open all the time by the > application, > but sometime the access are slow because the file was moved out of the cache. > > How can I keep it in memory all the time ? > I can read it at regular interval, but this

Re: Keep a file in memory an any cost

2011-05-11 Thread Bryn M. Reeves
On 05/11/2011 07:47 PM, JD wrote: > On 05/11/11 11:19, Alain Spineux wrote: >> Hello I have a file, that is an index, it is open all the time by the >> application, >> but sometime the access are slow because the file was moved out of the cache. >> >> How can I keep it in memory all the time ? >> I

Re: Keep a file in memory an any cost

2011-05-11 Thread Bryn M. Reeves
On 05/11/2011 07:19 PM, Alain Spineux wrote: > Hello I have a file, that is an index, it is open all the time by the > application, > but sometime the access are slow because the file was moved out of the cache. > > How can I keep it in memory all the time ? > I can read it at regular interval, bu

Re: Keep a file in memory an any cost

2011-05-11 Thread JD
On 05/11/11 11:19, Alain Spineux wrote: > Hello I have a file, that is an index, it is open all the time by the > application, > but sometime the access are slow because the file was moved out of the cache. > > How can I keep it in memory all the time ? > I can read it at regular interval, but this

Re: Keep a file in memory an any cost

2011-05-11 Thread JB
Alain Spineux gmail.com> writes: > ... $ man shm_overview $ man mmap etc JB -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guideli

Re: Keep a file in memory an any cost

2011-05-11 Thread Joe Zeff
On 05/11/2011 11:19 AM, Alain Spineux wrote: > How can I keep it in memory all the time ? > I can read it at regular interval, but this is a little over kill ? Is there a reason that you don't do the obvious: read it into an array, use it and then if needed write it out again when the program exi