Bruno Desthuilliers wrote:
Simon Strobl a écrit :
(snip)
> I would prefer to be able to use the same type of
scripts with data of all sizes, though.
Since computers have a limited RAM, this is to remain a wish. You
can't obviously expect to deal with terabytes of data like you do with
a 1kb
Simon Strobl a écrit :
(snip)
> I would prefer to be able to use the same type of
scripts with data of all sizes, though.
Since computers have a limited RAM, this is to remain a wish. You can't
obviously expect to deal with terabytes of data like you do with a 1kb
text file.
--
http://mail.p
Simon Strobl wrote:
Well, as I was using Python, I did not expect to have to care about
the language's internal affairs that much. I thought I could simply do
always the same no matter how large my files get. In other words, I
thought Python was really scalable.
Python the language is indefi
Simon Strobl <[EMAIL PROTECTED]> wrote:
> Well, as I was using Python, I did not expect to have to care about
> the language's internal affairs that much. I thought I could simply do
> always the same no matter how large my files get. In other words, I
> thought Python was really scalable.
It's n
On Tue, 05 Aug 2008 01:20:08 -0700, Simon Strobl wrote:
>> > I thought it would be practical not to create the dictionary from a
>> > text file each time I needed it. I.e. I thought loading the .pyc-file
>> > should be faster. Yet, Python failed to create a .pyc-file
>>
>> Probably a good example
En Mon, 04 Aug 2008 11:02:16 -0300, Simon Strobl <[EMAIL PROTECTED]>
escribió:
I created a python file that contained the dictionary. The size of
this file was 6.8GB. I thought it would be practical not to create the
dictionary from a text file each time I needed it. I.e. I thought
loading the
> Have you considered that the operating system imposes per-process limits
> on memory usage? You say that your server has 128 GB of memory, but that
> doesn't mean the OS will make anything like that available.
According to our system administrator, I can use all of the 128G.
> > I thought it wo
On Mon, 04 Aug 2008 07:02:16 -0700, Simon Strobl wrote:
> I created a python file that contained the dictionary. The size of this
> file was 6.8GB.
Ah, that's what I thought you had done. That's not a dictionary. That's a
text file containing the Python code to create a dictionary.
My guess is
On 4 Aug., 00:51, Avinash Vora <[EMAIL PROTECTED]> wrote:
> On Aug 4, 2008, at 4:12 AM, Jörgen Grahn wrote:
>
> > (You might want to post this to comp.lang.python rather than to me --
> > I am just another c.l.p reader. If you already have done to, please
> > disregard this.)
>
> Yeah, I hit "repl
On Aug 4, 2008, at 4:12 AM, Jörgen Grahn wrote:
(You might want to post this to comp.lang.python rather than to me --
I am just another c.l.p reader. If you already have done to, please
disregard this.)
Yeah, I hit "reply" by mistake and didn't realize it. My bad.
(I assume here that Berk
On 3 Aug 2008 20:40:02 GMT, Jorgen Grahn <[EMAIL PROTECTED]> wrote:
> On 3 Aug 2008 20:36:33 GMT, Jorgen Grahn <[EMAIL PROTECTED]> wrote:
> > On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl <[EMAIL PROTECTED]>
> wrote:
>
> ...
>
> >> If there is no other way to do it, I will have to learn h
On 3 Aug 2008 20:36:33 GMT, Jorgen Grahn <[EMAIL PROTECTED]> wrote:
> On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl <[EMAIL PROTECTED]>
> wrote:
...
>> If there is no other way to do it, I will have to learn how to use
>> databases in Python.
>
> If you use Berkeley DB ("import bsddb"), yo
On Fri, 1 Aug 2008 01:05:07 -0700 (PDT), Simon Strobl <[EMAIL PROTECTED]> wrote:
>> What does "load a dictionary" mean?
>
> I had a file bigrams.py with a content like below:
>
> bigrams = {
> ", djy" : 75 ,
> ", djz" : 57 ,
> ", djzoom" : 165 ,
> ", dk" : 28893 ,
> ", dk.au" : 854 ,
> ", dk.b." :
On Fri, 01 Aug 2008 00:46:09 -0700, Simon Strobl wrote:
> Hello,
>
> I tried to load a 6.8G large dictionary on a server that has 128G of
> memory. I got a memory error. I used Python 2.5.2. How can I load my
> data?
How do you know the dictionary takes 6.8G?
I'm going to guess an answer to my
Simon Strobl wrote:
Hello,
I tried to load a 6.8G large dictionary on a server that has 128G of
memory. I got a memory error. I used Python 2.5.2. How can I load my
data?
SImon
Take a look at the python bsddb module. Uing btree tables is fast, and
it has the benefit that once the table is o
On Fri, 01 Aug 2008 14:47:17 +0100, Sion Arrowsmith wrote:
> Simon Strobl <[EMAIL PROTECTED]> wrote:
>>I tried to load a 6.8G large dictionary on a server that has 128G of
>>memory. I got a memory error. I used Python 2.5.2. How can I load my
>>data?
> Let's just eliminate one thing here: this s
On Fri, 01 Aug 2008 14:47:17 +0100, Sion Arrowsmith wrote:
> Simon Strobl <[EMAIL PROTECTED]> wrote:
>>I tried to load a 6.8G large dictionary on a server that has 128G of
>>memory. I got a memory error. I used Python 2.5.2. How can I load my
>>data?
>
> Let's just eliminate one thing here: this
Simon Strobl <[EMAIL PROTECTED]> wrote:
>I tried to load a 6.8G large dictionary on a server that has 128G of
>memory. I got a memory error. I used Python 2.5.2. How can I load my
>data?
Let's just eliminate one thing here: this server is running a
64-bit OS, isn't it? Because if it's a 32-bit OS
Simon Strobl:
> I had a file bigrams.py with a content like below:
> bigrams = {
> ", djy" : 75 ,
> ", djz" : 57 ,
> ", djzoom" : 165 ,
> ", dk" : 28893 ,
> ", dk.au" : 854 ,
> ", dk.b." : 3668 ,
> ...
> }
> In another file I said:
> from bigrams import bigrams
Probably there's a limit in the modu
> What does "load a dictionary" mean?
I had a file bigrams.py with a content like below:
bigrams = {
", djy" : 75 ,
", djz" : 57 ,
", djzoom" : 165 ,
", dk" : 28893 ,
", dk.au" : 854 ,
", dk.b." : 3668 ,
...
}
In another file I said:
from bigrams import bigrams
> How about using a database in
On Fri, 01 Aug 2008 00:46:09 -0700, Simon Strobl wrote:
> I tried to load a 6.8G large dictionary on a server that has 128G of
> memory. I got a memory error. I used Python 2.5.2. How can I load my
> data?
What does "load a dictionary" mean? Was it saved with the `pickle`
module?
How about usi
21 matches
Mail list logo
