On 2012-10-28 19:21, Oscar Benjamin wrote:
On 28 October 2012 14:20, Virgil Stokes wrote:
On 28-Oct-2012 12:18, Dave Angel wrote:
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes writes:
Yes, I do wish to inverse the order, but the "forwar
On 28 October 2012 14:20, Virgil Stokes wrote:
> On 28-Oct-2012 12:18, Dave Angel wrote:
>>
>> On 10/24/2012 03:14 AM, Virgil Stokes wrote:
>>>
>>> On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes writes:
>
> Yes, I do wish to inverse the order, but the "forward in time" f
On 28-Oct-2012 12:18, Dave Angel wrote:
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes writes:
Yes, I do wish to inverse the order, but the "forward in time" file
will be in binary.
I really think it will be simplest to just write the file
On 10/24/2012 03:14 AM, Virgil Stokes wrote:
> On 24-Oct-2012 01:46, Paul Rubin wrote:
>> Virgil Stokes writes:
>>> Yes, I do wish to inverse the order, but the "forward in time" file
>>> will be in binary.
>> I really think it will be simplest to just write the file in forward
>> order, then use
> Data files have some sort of parsing, unless it's one huge dict, or
> list, so there has to be an average size to the parse.
>
Not meaning the dict, or list isn't parsed, but that the file should
be have parsable areas.
--
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
--
http
On 24-Oct-2012 01:46, Paul Rubin wrote:
Virgil Stokes writes:
Yes, I do wish to inverse the order, but the "forward in time" file
will be in binary.
I really think it will be simplest to just write the file in forward
order, then use mmap to read it one record at a time. It might be
possible
On 24-Oct-2012 17:11, rusi wrote:
On Oct 23, 7:52 pm, Virgil Stokes wrote:
I am working with some rather large data files (>100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data (e.g. moving med
On Oct 23, 7:52 pm, Virgil Stokes wrote:
> I am working with some rather large data files (>100GB) that contain time
> series
> data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I
> perform
> various types of processing on these data (e.g. moving median, moving average,
> an
Emile van Sebille writes:
>> probably somewhere close to 400-500Gb in memory
> I went looking for a machine capable of this and got about halfway
> there with http://www.tech-news.com/publib/pl2818.html which allows up
> to 248Gb memory -- near as I can tell the price for the maxed out
> syste
On 2012-10-23, Steven D'Aprano wrote:
> I would be very surprised if the poster will be able to fit 100
> gigabytes of data into even a single list comprehension, let alone
> two.
>
> This is a classic example of why the old external processing
> algorithms of the 1960s and 70s will never be obso
On 10/23/2012 4:35 PM, emile wrote:
So, let's see, at that point in time (building backward) you've got
probably somewhere close to 400-500Gb in memory.
My guess -- probably not so fast. Thrashing is sure to be a factor on
all but machines I'll never have a chance to work on.
I went looking
On 24/10/2012 08:07, Virgil Stokes wrote:
> On 23-Oct-2012 22:03, Cousin Stanley wrote:
>> Virgil Stokes wrote:
>>
>>> Not sure about "tac" --- could you provide more details on this
>>> and/or a simple example of how it could be used for fast reversed
>>> "reading" of a data file ?
>>tac is av
On Wed, 24 Oct 2012 01:23:58 -0400, Dennis Lee Bieber wrote:
> On Tue, 23 Oct 2012 16:35:40 -0700, emile declaimed the
> following in gmane.comp.python.general:
>
>> On 10/23/2012 04:19 PM, David Hutto wrote:
>> > forward = [line.rstrip('\n') for line in f.readlines()]
>>
>> f.readlines() will
On 23-Oct-2012 22:03, Cousin Stanley wrote:
Virgil Stokes wrote:
Not sure about "tac" --- could you provide more details on this
and/or a simple example of how it could be used for fast reversed
"reading" of a data file ?
tac is available as a command under linux
$ whatis tac
ta
On Wed, Oct 24, 2012 at 3:17 AM, Virgil Stokes wrote:
> On 24-Oct-2012 00:57, Demian Brecht wrote:
>>>
>>> This is a classic example of why the old external processing algorithms
>>> of the 1960s and 70s will never be obsolete. No matter how much memory
>>> you have, there will always be times whe
On Wed, Oct 24, 2012 at 3:05 AM, Virgil Stokes wrote:
> On 24-Oct-2012 00:36, David Hutto wrote:
>>>
>>> Don't forget to use timeit for an average OS utilization.
>>>
>>> I'd suggest two list comprehensions for now, until I've reviewed it some
>>> more:
>>>
>>> forward = ["%i = %s" % (i,chr(i)) f
On 24-Oct-2012 00:53, Steven D'Aprano wrote:
On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes wrote:
I am working with some rather large data files (>100GB)
[...]
Finally, to my question --- What is a fast way to write these variables
to
On 24-Oct-2012 00:57, Demian Brecht wrote:
This is a classic example of why the old external processing algorithms
of the 1960s and 70s will never be obsolete. No matter how much memory
you have, there will always be times when you want to process more data
than you can fit into memory.
But sur
On 24-Oct-2012 02:06, Oscar Benjamin wrote:
On 23 October 2012 15:31, Virgil Stokes wrote:
I am working with some rather large data files (>100GB) that contain time
series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
format. I perform various types of processing on these data
On 24-Oct-2012 00:36, David Hutto wrote:
Don't forget to use timeit for an average OS utilization.
I'd suggest two list comprehensions for now, until I've reviewed it some more:
forward = ["%i = %s" % (i,chr(i)) for i in range(33,126)]
backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-
On Tue, Oct 23, 2012 at 8:06 PM, Oscar Benjamin
wrote:
> On 23 October 2012 15:31, Virgil Stokes wrote:
>> I am working with some rather large data files (>100GB) that contain time
>> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
>> format. I perform various types of proce
On 10/23/12 13:37, Virgil Stokes wrote:
> Yes, I do wish to inverse the order, but the "forward in time"
> file will be in binary.
Your original post said:
> The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format
making it hard to know what sort of data is in this file.
So I guess it
On 23 October 2012 15:31, Virgil Stokes wrote:
> I am working with some rather large data files (>100GB) that contain time
> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
> format. I perform various types of processing on these data (e.g. moving
> median, moving average, an
On Tue, Oct 23, 2012 at 7:35 PM, emile wrote:
> On 10/23/2012 04:19 PM, David Hutto wrote:
>>
>> Whether this is fast enough, or not, I don't know:
>
>
> well, the OP's original post started with
> "I am working with some rather large data files (>100GB)..."
Well, is this a dedicated system, an
Virgil Stokes writes:
> Yes, I do wish to inverse the order, but the "forward in time" file
> will be in binary.
I really think it will be simplest to just write the file in forward
order, then use mmap to read it one record at a time. It might be
possible to squeeze out a little more performan
On 10/23/2012 04:19 PM, David Hutto wrote:
Whether this is fast enough, or not, I don't know:
well, the OP's original post started with
"I am working with some rather large data files (>100GB)..."
filename = "data_file.txt"
f = open(filename, 'r')
forward = [line.rstrip('\n') for line in f
On Tue, Oct 23, 2012 at 6:53 PM, Steven D'Aprano
wrote:
> On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
>
>> On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes wrote:
>>> I am working with some rather large data files (>100GB)
> [...]
>>> Finally, to my question --- What is a fast way to wr
Whether this is fast enough, or not, I don't know:
filename = "data_file.txt"
f = open(filename, 'r')
forward = [line.rstrip('\n') for line in f.readlines()]
backward = [line.rstrip('\n') for line in reversed(forward)]
f.close()
print forward, "\n\n", "\n\n", backward, "\n"
> This is a classic example of why the old external processing algorithms
> of the 1960s and 70s will never be obsolete. No matter how much memory
> you have, there will always be times when you want to process more data
> than you can fit into memory.
But surely nobody will *ever* need more t
On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
> On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes wrote:
>> I am working with some rather large data files (>100GB)
[...]
>> Finally, to my question --- What is a fast way to write these variables
>> to an external file and then read them in
> Missed the part about it being a file. Use:
>
> forward = ["%i = %s" % (i,chr(i)) for i in range(33,126)]
> backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]
>
> print forward,backward
>
This was a dud, let me rework it real quick, I deleted what i had, and
accidentally wrote the wro
> Don't forget to use timeit for an average OS utilization.
>
> I'd suggest two list comprehensions for now, until I've reviewed it some more:
>
> forward = ["%i = %s" % (i,chr(i)) for i in range(33,126)]
> backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]
>
> for var in forward:
>
On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes wrote:
> I am working with some rather large data files (>100GB) that contain time
> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
> format. I perform various types of processing on these data (e.g. moving
> median, moving ave
Virgil Stokes wrote:
> Not sure about "tac" --- could you provide more details on this
> and/or a simple example of how it could be used for fast reversed
> "reading" of a data file ?
tac is available as a command under linux
$ whatis tac
tac (1) - concatenate and print files in re
On 23-Oct-2012 19:56, Tim Chase wrote:
On 10/23/12 12:17, Virgil Stokes wrote:
On 23-Oct-2012 18:09, Tim Chase wrote:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in
backwards?
Am I missing something, or would the fairly-standar
On 10/23/12 12:17, Virgil Stokes wrote:
> On 23-Oct-2012 18:09, Tim Chase wrote:
>>> Finally, to my question --- What is a fast way to write these
>>> variables to an external file and then read them in
>>> backwards?
>> Am I missing something, or would the fairly-standard "tac"
>> utility do the r
On 23-Oct-2012 18:35, Dennis Lee Bieber wrote:
On Tue, 23 Oct 2012 16:31:17 +0200, Virgil Stokes
declaimed the following in gmane.comp.python.general:
Finally, to my question --- What is a fast way to write these variables to an
external file and then read them in backwards?
Stuff th
On 23-Oct-2012 18:17, Paul Rubin wrote:
Virgil Stokes writes:
Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in backwards?
Seeking backwards in files works, but the performance hit is
significant. There is also a performance hit
On 23-Oct-2012 18:09, Tim Chase wrote:
On 10/23/12 09:31, Virgil Stokes wrote:
I am working with some rather large data files (>100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data (e.g. moving
Tim Chase writes:
> Again, the conversion to/from decimal hasn't been a great cost in my
> experience, as it's overwhelmed by the I/O cost of shoveling the
> data to/from disk.
I've found that cpu costs both for processing and conversion are
significant. Also, using a binary format makes the fil
On 10/23/12 11:17, Paul Rubin wrote:
> Virgil Stokes writes:
>> Finally, to my question --- What is a fast way to write these
>> variables to an external file and then read them in backwards?
>
> Seeking backwards in files works, but the performance hit is
> significant. There is also a performa
Paul Rubin writes:
> Seeking backwards in files works, but the performance hit is
> significant. There is also a performance hit to scanning pointers
> backwards in memory, due to cache misprediction. If it's something
> you're just running a few times, seeking backwards the simplest
> approach.
Virgil Stokes writes:
> Finally, to my question --- What is a fast way to write these
> variables to an external file and then read them in backwards?
Seeking backwards in files works, but the performance hit is
significant. There is also a performance hit to scanning pointers
backwards in memor
On 10/23/12 09:31, Virgil Stokes wrote:
> I am working with some rather large data files (>100GB) that contain time
> series
> data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I
> perform
> various types of processing on these data (e.g. moving median, moving
> average,
I am working with some rather large data files (>100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data (e.g. moving median, moving average,
and Kalman-filter, Kalman-smoother) in a sequential man
45 matches
Mail list logo
