> I had the (mis)pleasure of dealing with a multi-terabyte postgresql
> instance many years ago and figuring out why random scripts were eating
> up system memory became quite common.
>
> All of our "ETL" scripts were either written in Perl, Java, or Python
> but the results were always the sa
On 3/29/21 5:12 AM, Alexey wrote:
Hello everyone!
I'm experiencing problems with memory consumption.
I have a class which is doing ETL job. What`s happening inside:
- fetching existing objects from DB via SQLAchemy
- iterate over raw data
- create new/update existing objects
- commit cha
четверг, 1 апреля 2021 г. в 15:56:23 UTC+3, Marco Ippolito:
> > > Are you running with systemd?
> >
> > I really don't know.
> An example of how to check:
>
> ```
> $ readlink /sbin/init
> /lib/systemd/systemd
> ```
>
> You want to check which program runs as PID 1.
Thank you Marco
--
h
четверг, 1 апреля 2021 г. в 15:46:21 UTC+3, Marco Ippolito:
> I suspect the high watermark of `` needs to be reachable still and,
> secondly, that a forceful constraint whilst running would crash the
> container?
Exactly.
--
https://mail.python.org/mailman/listinfo/python-list
четверг, 1 апреля 2021 г. в 17:21:59 UTC+3, Mats Wichmann:
> On 4/1/21 5:50 AM, Alexey wrote:
> > Found it. As I said before the problem was lurking in the cache.
> > Few days ago I read about circular references and things like that and
> > I thought to myself that it might be the case. To buil
четверг, 1 апреля 2021 г. в 16:02:15 UTC+3, Barry:
> > On 1 Apr 2021, at 13:46, Marco Ippolito wrote:
> >
> >
> >>
> What if you increase the machine's (operating system's) swap space? Does
> that take care of the problem in practice?
> >>>
> >>> I can`t do that because it will aff
четверг, 1 апреля 2021 г. в 15:27:01 UTC+3, Chris Angelico:
> On Thu, Apr 1, 2021 at 10:56 PM Alexey wrote:
> >
> > Found it. As I said before the problem was lurking in the cache.
> > Few days ago I read about circular references and things like that and
> > I thought to myself that it might
On 4/1/21 5:50 AM, Alexey wrote:
Found it. As I said before the problem was lurking in the cache.
Few days ago I read about circular references and things like that and
I thought to myself that it might be the case. To build the cache I was
using lots of 'setdefault' methods chained together
s
> On 1 Apr 2021, at 13:46, Marco Ippolito wrote:
>
>
>>
What if you increase the machine's (operating system's) swap space? Does
that take care of the problem in practice?
>>>
>>> I can`t do that because it will affect other containers running on this
>>> host.
>>> In my opinion i
> > Are you running with systemd?
>
> I really don't know.
An example of how to check:
```
$ readlink /sbin/init
/lib/systemd/systemd
```
You want to check which program runs as PID 1.
```
ps 1
```
--
https://mail.python.org/mailman/listinfo/python-list
четверг, 1 апреля 2021 г. в 14:57:29 UTC+3, Barry:
> > On 31 Mar 2021, at 09:42, Alexey wrote:
> >
> > среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
> >>> On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote:
> >>>
> >>>
> >>> I'm sorry. I didn't understand your question right. If I have
> >> What if you increase the machine's (operating system's) swap space? Does
> >> that take care of the problem in practice?
> >
> > I can`t do that because it will affect other containers running on this
> > host.
> > In my opinion it may significantly reduce their performance.
>
> Assuming thi
On Thu, Apr 1, 2021 at 10:56 PM Alexey wrote:
>
> Found it. As I said before the problem was lurking in the cache.
> Few days ago I read about circular references and things like that and
> I thought to myself that it might be the case. To build the cache I was
> using lots of 'setdefault' methods
> On 31 Mar 2021, at 09:42, Alexey wrote:
>
> среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
>>> On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote:
>>>
>>>
>>> I'm sorry. I didn't understand your question right. If I have 4 workers,
>>> they require 4Gb
>>> in idle state and some ex
Found it. As I said before the problem was lurking in the cache.
Few days ago I read about circular references and things like that and
I thought to myself that it might be the case. To build the cache I was
using lots of 'setdefault' methods chained together
self.__cache.setdefault(cluster_name,
On 31/03/2021 09:35, Alexey wrote:
среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
What if you increase the machine's (operating system's) swap space? Does
that take care of the problem in practice?
I can`t do that because it will affect other containers running on this host.
In my
среда, 31 марта 2021 г. в 18:17:46 UTC+3, Dieter Maurer:
> Alexey wrote at 2021-3-31 02:43 -0700:
> >среда, 31 марта 2021 г. в 06:54:52 UTC+3, Inada Naoki:
> > ...
> >> You can get some hints from sys._debugmallocstats(). It prints
> >> obmalloc (allocator for small objects) stats to stderr.
> >
Alexey wrote at 2021-3-31 02:43 -0700:
>среда, 31 марта 2021 г. в 06:54:52 UTC+3, Inada Naoki:
> ...
>> You can get some hints from sys._debugmallocstats(). It prints
>> obmalloc (allocator for small objects) stats to stderr.
>> Try printing stats before and after 1st run, and after 2nd run. And
>>
среда, 31 марта 2021 г. в 14:16:30 UTC+3, Inada Naoki:
> > ** Before first run:
> > # arenas allocated total = 776
> > # arenas reclaimed = 542
> > # arenas highwater mark = 234
> > # arenas allocated current = 234
> > 234 arenas * 262144 bytes/arena = 61,341,696
> > ** After fi
> ** Before first run:
> # arenas allocated total = 776
> # arenas reclaimed = 542
> # arenas highwater mark= 234
> # arenas allocated current = 234
> 234 arenas * 262144 bytes/
среда, 31 марта 2021 г. в 11:52:43 UTC+3, Marco Ippolito:
> > > At which point does the problem start manifesting itself?
> > The problem spot is my cache(dict). I simplified my code to just load
> > all the objects to this dict and then clear it.
> What's the memory utilisation just _before_ per
среда, 31 марта 2021 г. в 06:54:52 UTC+3, Inada Naoki:
> First of all, I recommend upgrading your Python. Python 3.6 is a bit old.
I was thinking about that.
> As you saying, Python can not return the memory to OS until the whole
> arena become unused.
> If your task releases all objects alloc
среда, 31 марта 2021 г. в 05:45:27 UTC+3, cameron...@gmail.com:
> Since everyone is talking about vague OS memory use and not at all about
> working set size of Python objects, let me ...
> On 29Mar2021 03:12, Alexey wrote:
> >I'm experiencing problems with memory consumption.
> >
> >I have a
> > At which point does the problem start manifesting itself?
> The problem spot is my cache(dict). I simplified my code to just load
> all the objects to this dict and then clear it.
What's the memory utilisation just _before_ performing this load? I am assuming
it's much less than this 1 GB you
среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
> On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote:
>
> >
> > I'm sorry. I didn't understand your question right. If I have 4 workers,
> > they require 4Gb
> > in idle state and some extra memory when they execute other tasks. If I
> > inc
вторник, 30 марта 2021 г. в 18:43:54 UTC+3, Alan Gauld:
> On 29/03/2021 11:12, Alexey wrote:
> The first thing you really need to tell us is which
> OS you are using? Memory management varies wildly
> depending on OS. Even different flavours of *nix
> do it differently.
I'm using Ubuntu(5.8.
вторник, 30 марта 2021 г. в 18:43:51 UTC+3, Marco Ippolito:
> Have you tried to identify where in your code the surprising memory
> allocations
> are made?
Yes.
> You could "bisect search" by adding breakpoints:
>
> https://docs.python.org/3/library/functions.html#breakpoint
>
> At which po
On Mon, Mar 29, 2021 at 7:16 PM Alexey wrote:
>
> Problem. Before executing, my interpreter process weighs ~100Mb, after first
> run memory increases up to 500Mb
> and after second run it weighs 1Gb. If I will continue to run this class,
> memory wont increase, so I think
> it's not a memory lea
Since everyone is talking about vague OS memory use and not at all about
working set size of Python objects, let me ...
On 29Mar2021 03:12, Alexey wrote:
>I'm experiencing problems with memory consumption.
>
>I have a class which is doing ETL job. What`s happening inside:
> - fetching existing o
On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote:
>
> I'm sorry. I didn't understand your question right. If I have 4 workers,
> they require 4Gb
> in idle state and some extra memory when they execute other tasks. If I
> increase workers
> count up to 16, they`ll eat all the memory I have (16GB) on
On 30/03/2021 16:50, Chris Angelico wrote:
>> A 1GB process on modern computers is hardly a big problem?
>> Most machines have 4G and many have 16G or even 32G
>> nowadays.
>>
>
> Desktop systems maybe, but if you rent yourself a worker box, it might
> not have anything like that much. Especially
On Wed, Mar 31, 2021 at 2:44 AM Alan Gauld via Python-list
wrote:
>
> On 29/03/2021 11:12, Alexey wrote:
> > Hello everyone!
> > I'm experiencing problems with memory consumption.
> >
>
> The first thing you really need to tell us is which
> OS you are using? Memory management varies wildly
> depe
Have you tried to identify where in your code the surprising memory allocations
are made?
You could "bisect search" by adding breakpoints:
https://docs.python.org/3/library/functions.html#breakpoint
At which point does the problem start manifesting itself?
--
https://mail.python.org/mailman/lis
On 29/03/2021 11:12, Alexey wrote:
> Hello everyone!
> I'm experiencing problems with memory consumption.
>
The first thing you really need to tell us is which
OS you are using? Memory management varies wildly
depending on OS. Even different flavours of *nix
do it differently.
However, most do i
> I'm sorry. I didn't understand your question right. If I have 4 workers,
> they require 4Gb
> in idle state and some extra memory when they execute other tasks. If I
> increase workers
> count up to 16, they`ll eat all the memory I have (16GB) on my machine and
> will crash as soon
> as system ge
понедельник, 29 марта 2021 г. в 19:56:52 UTC+3, Stestagg:
> > > 2. Can you try a test with 16 or 32 active workers (i.e. number of
> > > workers=2x available memory in GB), do they all still end up with 1gb
> > > usage? or do you get any other memory-related issues running this?
> > Yes. They wi
понедельник, 29 марта 2021 г. в 19:37:03 UTC+3, Dieter Maurer:
> Alexey wrote at 2021-3-29 06:26 -0700:
> >понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
> >> It looks like the problem is on celery.
> >> The mentioned issue is still open, so not sure if it was corrected.
> >>
> >>
> > 2. Can you try a test with 16 or 32 active workers (i.e. number of
> > workers=2x available memory in GB), do they all still end up with 1gb
> > usage? or do you get any other memory-related issues running this?
> Yes. They will consume 1Gb each. It doesn't matter how many workers I
> have,
> t
Alexey wrote at 2021-3-29 06:26 -0700:
>понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
>> It looks like the problem is on celery.
>> The mentioned issue is still open, so not sure if it was corrected.
>>
>> https://manhtai.github.io/posts/memory-leak-in-celery/
>
>As I mentioned in my f
понедельник, 29 марта 2021 г. в 17:19:02 UTC+3, Stestagg:
> On Mon, Mar 29, 2021 at 2:32 PM Alexey wrote:
> Some questions here to help understand more:
>
> 1. Do you have any actual problems caused by running 8 celery workers
> (beyond high memory reports)? What are they?
No. Everything work
On Mon, Mar 29, 2021 at 2:32 PM Alexey wrote:
> понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
> > It looks like the problem is on celery.
> > The mentioned issue is still open, so not sure if it was corrected.
> >
> > https://manhtai.github.io/posts/memory-leak-in-celery/
>
> As I me
понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
> It looks like the problem is on celery.
> The mentioned issue is still open, so not sure if it was corrected.
>
> https://manhtai.github.io/posts/memory-leak-in-celery/
As I mentioned in my first message, I tried to run
this task(cla
It looks like the problem is on celery.
The mentioned issue is still open, so not sure if it was corrected.
https://manhtai.github.io/posts/memory-leak-in-celery/
Julio
El lun, 29 de mar. de 2021 a la(s) 08:31, Alexey (zen.supag...@gmail.com)
escribió:
> Hello Lars!
> Thanks for your interest.
Hello Lars!
Thanks for your interest.
The problem appears when all celery workers
require 1Gb of RAM each in idle state. They
hold this memory constantly and when they do
something useful, they grab more memory. I
think 8Gb+ in idle state is quite a lot for my
app.
> Did it crash your system or p
Hello Alexej,
May I stupidly ask, why you care about that in general? Please don't get
me wrong I don't want to criticize you, this is rather meant to be a
(thought) provoking question.
Normally your OS-Kernel and the Python-Interpreter get along pretty well
and whenthere is free memory to be had,
In article ,
Wolodja Wentland wrote:
>
>I have a problem with the memory consumption of multiprocessing.Pool()'s
>worker processes. I have a parent process that has to handle big data
>structures and would like to use a pool of processes for computations.
>
>The problem is, that all worker proces
46 matches
Mail list logo
