Since you are dealing with that much information perhaps pytables would
be useful to you.http://pytables.sourceforge.net
[EMAIL PROTECTED] wrote:
Hi
I need to process a really huge text file (4GB) and this is what i
need to do. It takes for ever to complete this. I read some where that
"l
[EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
...
> > > files (you see "huge" is really relative ;-)) on 2-4GB RAM boxes and
> > > setting a big buffer (1GB or more) reduces the wall time by 30 to 50%
> > > compared to the default value. BerkeleyDB should have a buffering
> > Out of curiosity, wh
On 3/18/07, Alex Martelli <[EMAIL PROTECTED]> wrote:
> George Sakkis <[EMAIL PROTECTED]> wrote:
> > On Mar 18, 12:11 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> > > I need to process a really huge text file (4GB) and this is what i
> > > need to do. It takes for ever to complete this. I re
George Sakkis <[EMAIL PROTECTED]> wrote:
> On Mar 18, 12:11 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
>
> > Hi
> > I need to process a really huge text file (4GB) and this is what i
> > need to do. It takes for ever to complete this. I read some where that
> > "list comprehension" can fa
On 18 Mar 2007 19:01:27 -0700, George Sakkis <[EMAIL PROTECTED]> wrote:
> On Mar 18, 12:11 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> > I need to process a really huge text file (4GB) and this is what i
> > need to do. It takes for ever to complete this. I read some where that
> > "list c
On Mar 18, 12:11 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> Hi
> I need to process a really huge text file (4GB) and this is what i
> need to do. It takes for ever to complete this. I read some where that
> "list comprehension" can fast up things. Can you point out how to do
> it in this
f<[EMAIL PROTECTED]> wrote:
> I wonder whether the following be more efficient if DB was a
> dictionnary:
>
> Splits = (line.split(' ') for line in open('file.text', 'r'))
> DB = dict([(S[0], S[-1]) for S in Splits])
You'd still be doing much more splitting work (and behind-the-scene
allocation
George Sakkis <[EMAIL PROTECTED]> wrote:
...
> > Unless each line is huge, how exactly you split it to get the first and
> > last blank-separated word is not going to matter much.
> >
> > Still, you should at least avoid repeating the splitting twice, that's
> > pretty obviously sheer waste: so,
On Mar 18, 1:40 pm, [EMAIL PROTECTED] (Alex Martelli) wrote:
> [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> > On 3/18/07, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> > > In <[EMAIL PROTECTED]>, Daniel Nogradi
> > > wrote:
>
> > > >> f = open('file.txt','r')
> > > >> for line in f:
> > >
I wonder whether the following be more efficient if DB was a
dictionnary:
Splits = (line.split(' ') for line in open('file.text', 'r'))
DB = dict([(S[0], S[-1]) for S in Splits])
--
http://mail.python.org/mailman/listinfo/python-list
[EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> On 3/18/07, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> > In <[EMAIL PROTECTED]>, Daniel Nogradi
> > wrote:
> >
> > >> f = open('file.txt','r')
> > >> for line in f:
> > >> db[line.split(' ')[0]] = line.split(' ')[-1]
> > >>
> > > I need to process a really huge text file (4GB) and this is what i
> > > "list comprehension" can fast up things. Can you point out how to do
> > > f = open('file.txt','r')
> > > for line in f:
> > > db[line.split(' ')[0]] = line.split(' ')[-1]
> > > db.sync()
> >
> > What is
On 3/18/07, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:
> In <[EMAIL PROTECTED]>, Daniel Nogradi
> wrote:
>
> >> f = open('file.txt','r')
> >> for line in f:
> >> db[line.split(' ')[0]] = line.split(' ')[-1]
> >> db.sync()
> >
> > What is db here? Looks like a dictionary but
In <[EMAIL PROTECTED]>, Daniel Nogradi
wrote:
>> f = open('file.txt','r')
>> for line in f:
>> db[line.split(' ')[0]] = line.split(' ')[-1]
>> db.sync()
>
> What is db here? Looks like a dictionary but that doesn't have a sync method.
Shelves (`shelve` module) have this API. And
On 3/18/07, Daniel Nogradi <[EMAIL PROTECTED]> wrote:
> > I need to process a really huge text file (4GB) and this is what i
> > "list comprehension" can fast up things. Can you point out how to do
> > f = open('file.txt','r')
> > for line in f:
> > db[line.split(' ')[0]] = line.split(' ')[
> I need to process a really huge text file (4GB) and this is what i
> need to do. It takes for ever to complete this. I read some where that
> "list comprehension" can fast up things. Can you point out how to do
> it in this case?
> thanks a lot!
>
>
> f = open('file.txt','r')
> for line in f:
>
In <[EMAIL PROTECTED]>,
[EMAIL PROTECTED] wrote:
> I need to process a really huge text file (4GB) and this is what i
> need to do. It takes for ever to complete this. I read some where that
> "list comprehension" can fast up things. Can you point out how to do
> it in this case?
No way I can see
17 matches
Mail list logo
