What application language are you using, and does it provide a method to load a file into memory. In perl for example, a file can be loaded into memory like:
#-------------------------- #!/usr/bin/perl -w #Experimenting with files print "Enter filename:"; chomp($fname = <STDIN>); ## Open a file. If unsuccessful, print an error message and quit. open (FPTR,$fname) || die "Can't Open File: $fname\n"; # First Technique: "Slurping" # This approach reads the ENTIRE file into memory # Caution... This is not a good method for BIG files!!! -- added by LFJ: unless you have enough memory. #------------------------------ Also, The linux file system will cache recently or frequently used files in memory regardless, and the effect is similar in performance to anapplication level cache. A separate application level cache would not necessarily be desirable since the file system cache would largely have the same items in it anyway; storing them again in an application level memory cache would simply leave less total memory available to file caching or other uses. http://www.ruby-forum.com/topic/190207 #----------------------------------- Other speedy methods to work with large files include: a) Have the information in a database file with proper indexes. b) Use a RAM disk as you mentioned. c) Use an array to hold the file across several smaller but very capable hard drives, like is provide in a RAID 0 configuration. A RAID 0 distributes data across several disks in a way that gives improved speed at any given instant. If one disk fails, however, all of the data on the array will be lost, as there is neither parity nor mirroring. d) Buy a solid state hard drive which is equivalent to using a RAM hard drive, or use solid state hard drives in a RAID 0 configuration. http://video.google.com/videosearch?q=solid+state+hard+drive&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&um=1&ie=UTF-8&ei=9WBWSoDQDobiMJCvyZ0I&sa=X&oi=video_result_group&ct=title&resnum=4# or http://tinyurl.com/mjp55l Regards, LelandJ On 07/09/2009 03:49 PM, Malcolm Greene wrote: > This is a Windows 2008 and Linux Ubuntu 8.10 related > question. The machine in question is an 8 core 64-bit > server with 48gb RAM. I have a large file (~16gb) that I > would like to load into cache so that scripts that I have > running on multiple cores can all efficiently read from > this file without drive contention. The file is a fixed > record width ASCII text file and each script is reading > from a specific physical section of this file. How can I > efficiently pre-load a large file into cache or is my > best bet to use the modern day equivalent of a RAM disk? > Any suggestions? Thanks, Malcolm > > > --- StripMime Report -- processed MIME parts --- > multipart/alternative text/plain (text body -- kept) > text/html --- > > _______________________________________________ Post > Messages to: [email protected] Subscription Maintenance: > http://leafe.com/mailman/listinfo/profox OT-free version > of this list: > http://leafe.com/mailman/listinfo/profoxtech Searchable > Archive: http://leafe.com/archives/search/profox This > message: > http://leafe.com/archives/byMID/profox/[email protected] > > ** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious. _______________________________________________ Post Messages to: [email protected] Subscription Maintenance: http://leafe.com/mailman/listinfo/profox OT-free version of this list: http://leafe.com/mailman/listinfo/profoxtech Searchable Archive: http://leafe.com/archives/search/profox This message: http://leafe.com/archives/byMID/profox/[email protected] ** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
