Hello everyone!
I'm experiencing problems with memory consumption.
I have a class which is doing ETL job. What`s happening inside:
- fetching existing objects from DB via SQLAchemy
- iterate over raw data
- create new/update existing objects
- commit changes
Before processing data I create in
Hello Lars!
Thanks for your interest.
The problem appears when all celery workers
require 1Gb of RAM each in idle state. They
hold this memory constantly and when they do
something useful, they grab more memory. I
think 8Gb+ in idle state is quite a lot for my
app.
> Did it crash your system or p
понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
> It looks like the problem is on celery.
> The mentioned issue is still open, so not sure if it was corrected.
>
> https://manhtai.github.io/posts/memory-leak-in-celery/
As I mentioned in my first message, I tried to run
this task(cla
понедельник, 29 марта 2021 г. в 17:19:02 UTC+3, Stestagg:
> On Mon, Mar 29, 2021 at 2:32 PM Alexey wrote:
> Some questions here to help understand more:
>
> 1. Do you have any actual problems caused by running 8 celery workers
> (beyond high memory reports)? What are they?
понедельник, 29 марта 2021 г. в 19:37:03 UTC+3, Dieter Maurer:
> Alexey wrote at 2021-3-29 06:26 -0700:
> >понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
> >> It looks like the problem is on celery.
> >> The mentioned issue is still open, so n
понедельник, 29 марта 2021 г. в 19:56:52 UTC+3, Stestagg:
> > > 2. Can you try a test with 16 or 32 active workers (i.e. number of
> > > workers=2x available memory in GB), do they all still end up with 1gb
> > > usage? or do you get any other memory-related issues running this?
> > Yes. They wi
вторник, 30 марта 2021 г. в 18:43:51 UTC+3, Marco Ippolito:
> Have you tried to identify where in your code the surprising memory
> allocations
> are made?
Yes.
> You could "bisect search" by adding breakpoints:
>
> https://docs.python.org/3/library/functions.html#breakpoint
>
> At which po
вторник, 30 марта 2021 г. в 18:43:54 UTC+3, Alan Gauld:
> On 29/03/2021 11:12, Alexey wrote:
> The first thing you really need to tell us is which
> OS you are using? Memory management varies wildly
> depending on OS. Even different flavours of *nix
> do it differently.
I
среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
> On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote:
>
> >
> > I'm sorry. I didn't understand your question right. If I have 4 workers,
> > they require 4Gb
> > in idle state and some extra me
среда, 31 марта 2021 г. в 05:45:27 UTC+3, cameron...@gmail.com:
> Since everyone is talking about vague OS memory use and not at all about
> working set size of Python objects, let me ...
> On 29Mar2021 03:12, Alexey wrote:
> >I'm experiencing problems with memory consumptio
среда, 31 марта 2021 г. в 06:54:52 UTC+3, Inada Naoki:
> First of all, I recommend upgrading your Python. Python 3.6 is a bit old.
I was thinking about that.
> As you saying, Python can not return the memory to OS until the whole
> arena become unused.
> If your task releases all objects alloc
среда, 31 марта 2021 г. в 11:52:43 UTC+3, Marco Ippolito:
> > > At which point does the problem start manifesting itself?
> > The problem spot is my cache(dict). I simplified my code to just load
> > all the objects to this dict and then clear it.
> What's the memory utilisation just _before_ per
среда, 31 марта 2021 г. в 14:16:30 UTC+3, Inada Naoki:
> > ** Before first run:
> > # arenas allocated total = 776
> > # arenas reclaimed = 542
> > # arenas highwater mark = 234
> > # arenas allocated current = 234
> > 234 arenas * 262144 bytes/arena = 61,341,696
> > ** After fi
среда, 31 марта 2021 г. в 18:17:46 UTC+3, Dieter Maurer:
> Alexey wrote at 2021-3-31 02:43 -0700:
> >среда, 31 марта 2021 г. в 06:54:52 UTC+3, Inada Naoki:
> > ...
> >> You can get some hints from sys._debugmallocstats(). It prints
> >> obmalloc (allocator f
Found it. As I said before the problem was lurking in the cache.
Few days ago I read about circular references and things like that and
I thought to myself that it might be the case. To build the cache I was
using lots of 'setdefault' methods chained together
self.__cache.setdefault(cluster_name,
четверг, 1 апреля 2021 г. в 14:57:29 UTC+3, Barry:
> > On 31 Mar 2021, at 09:42, Alexey wrote:
> >
> > среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
> >>> On Tue, Mar 30, 2021 at 1:25 AM Alexey wrote:
> >>>
> >>>
> >>
четверг, 1 апреля 2021 г. в 15:27:01 UTC+3, Chris Angelico:
> On Thu, Apr 1, 2021 at 10:56 PM Alexey wrote:
> >
> > Found it. As I said before the problem was lurking in the cache.
> > Few days ago I read about circular references and things like that and
> > I thou
четверг, 1 апреля 2021 г. в 16:02:15 UTC+3, Barry:
> > On 1 Apr 2021, at 13:46, Marco Ippolito wrote:
> >
> >
> >>
> What if you increase the machine's (operating system's) swap space? Does
> that take care of the problem in practice?
> >>>
> >>> I can`t do that because it will aff
четверг, 1 апреля 2021 г. в 17:21:59 UTC+3, Mats Wichmann:
> On 4/1/21 5:50 AM, Alexey wrote:
> > Found it. As I said before the problem was lurking in the cache.
> > Few days ago I read about circular references and things like that and
> > I thought to myself that it
четверг, 1 апреля 2021 г. в 15:46:21 UTC+3, Marco Ippolito:
> I suspect the high watermark of `` needs to be reachable still and,
> secondly, that a forceful constraint whilst running would crash the
> container?
Exactly.
--
https://mail.python.org/mailman/listinfo/python-list
четверг, 1 апреля 2021 г. в 15:56:23 UTC+3, Marco Ippolito:
> > > Are you running with systemd?
> >
> > I really don't know.
> An example of how to check:
>
> ```
> $ readlink /sbin/init
> /lib/systemd/systemd
> ```
>
> You want to check which program runs as PID 1.
Thank you Marco
--
h
> I had the (mis)pleasure of dealing with a multi-terabyte postgresql
> instance many years ago and figuring out why random scripts were eating
> up system memory became quite common.
>
> All of our "ETL" scripts were either written in Perl, Java, or Python
> but the results were always the sa
Hi!
I've tried to build Python 2.7.3rc2 on cygwin and got the following errors:
$ CFLAGS=-I/usr/include/ncursesw/ CPPFLAGS=-I/usr/include/ncursesw/ ./configure
$ make
...
gcc -shared -Wl,--enable-auto-image-base
build/temp.cygwin-1.7.11-i686-2.7/Python-2.7.3rc2/Modules/_io/bufferedio.o
build/t
On 28.03.2012 14:50, Alexey Luchko wrote:
Hi!
I've tried to build Python 2.7.3rc2 on cygwin and got the following errors:
$ CFLAGS=-I/usr/include/ncursesw/ CPPFLAGS=-I/usr/include/ncursesw/
./configure
$ make
...
gcc -shared -Wl,--enable-auto-image-base
build/temp.cygwin-1.7.11-i686-2.7/P
JFI
Reported as
http://bugs.python.org/issue14437
http://bugs.python.org/issue14438
--
Regars,
Alex
--
http://mail.python.org/mailman/listinfo/python-list
On 28.03.2012 18:42, David Robinow wrote:
> On Wed, Mar 28, 2012 at 7:50 AM, Alexey Luchko wrote:
>> I've tried to build Python 2.7.3rc2 on cygwin and got the following errors:
>>
>> $ CFLAGS=-I/usr/include/ncursesw/ CPPFLAGS=-I/usr/include/ncursesw/
>> ./configu
On 29.03.2012 21:29, David Robinow wrote:
Have you included the patch to Include/py_curses.h ?
If you don't know what that is, download the cygwin src package for
Python-2.6 and look at the patches. Not all of them are still
Thanks for the hint. With cygwin's 2.6.5-ncurses-abi6.patch it wor
Greets!
Since i'm new to Python, i've decided to create a handy plugin for
Elipse SDK which is my primary dev environment.
Practically the plugin is a simple html archive from python
documentation website running
inside Eclipse so you can call it using Eclipse help system.
As for now it is pr
Greets!
Since i'm new to Python, i've decided to create a handy plugin for
Elipse SDK which is my primary dev environment.
Practically the plugin is a simple html archive from python
documentation website running
inside Eclipse so you can call it using Eclipse help system.
As for now it is pr
On Sun, 10 Jun 2012 05:02:35 -0500, Andrew Berg wrote:
> On 6/10/2012 4:22 AM, Alexey Gaidamaka wrote:
>> Practically the plugin is a simple html archive from python
>> documentation website running
>> inside Eclipse so you can call it using Eclipse help system. As for now
On Sun, 10 Jun 2012 12:14:15 +0300, Alexey Gaidamaka wrote:
> Greets!
>
> Since i'm new to Python, i've decided to create a handy plugin for
> Elipse SDK which is my primary dev environment. Practically the plugin
> is a simple html archive from python documentation
On Sun, 10 Jun 2012 15:37:50 +, Alexey Gaidamaka wrote:
> On Sun, 10 Jun 2012 05:02:35 -0500, Andrew Berg wrote:
>
>> On 6/10/2012 4:22 AM, Alexey Gaidamaka wrote:
>>> Practically the plugin is a simple html archive from python
>>> documentation website running
se ... finally".
Currently, with "else", it is almost impossible to guess the meaning
without looking into the documentation.
Off course, it should not be changed in Python 3, maybe in Python 4 or
5, but in Python 3 `then` could be an alias of `else` in these contexts.
Alexey.
--
https://mail.python.org/mailman/listinfo/python-list
tries to look up the *name*
> end. But it will run that line of code - if you quote it, it will
> work.
You see how people are confused over "for ... else".
Alexey.
--
https://mail.python.org/mailman/listinfo/python-list
e that is to be executed when the
main condition of `if` is false.
So, in
try:
do_something
except:
catch_exception
else:
continue_doing_something
when no exception occurs in `do_something`, is `do_something` more
true, or more false?
Alexey.
--
https://mail.pyt
e better than
"try/except/then/finally", but "try/except/else/finally" does not make
a perfect sense IMHO.
Alexey.
--
https://mail.python.org/mailman/listinfo/python-list
ss some extraordinary
situation prevents it from happening, for example:
try:
go_to_the_bakery()
then:
buy_croissants(2)
except BakeryClosed:
go_to_the_grociery()
buy_baguette(1)
finally:
come_back()
I know this is a poor program example (why not to use a boo
On Fri, 2017-11-03 at 22:03 +1100, Chris Angelico wrote:
> On Fri, Nov 3, 2017 at 8:48 PM, Alexey Muranov com> wrote:
> > 'Then' describes what happens next indeed, unless some
> > extraordinary
> > situation prevents it from happening, for example:
> >
&
I have discovered the following bug or problem: it looks like i am
forced to choose different names for class attributes and function
arguments, and i see no workaround. Am i missing some special syntax
feature ?
Alexey.
---
x = 42
class C1:
y = x # Works
class C2:
x = x # Works
To be more exact, i do see a few workarounds, for example:
def f4(a):
b = a
class D:
a = b # Works
return D
But this is not what i was hoping for.
Alexey.
On Tue, 8 May, 2018 at 12:02 AM, Alexey Muranov
wrote:
I have discovered the following bug or
e, probably the solution with
def f(a):
_a = a
class D:
a = _a
return D
is good enough, if Python does not allow to refer "simultaneously" to
variables from different scopes if they have the same name.
Alexey.
On Tue, 8 May, 2018 at 12:21 AM, Alexe
write instead just
f(x) = x*x
(like in Haskell and such).
Have this idea been discussed before?
I do not see any conflicts with the existing syntax. The following
would also work:
incrementer(m)(n) = n + m
instead of
incrementer = lambda m: lambda n: n + m
Alexey.
--
https
On mer., mars 27, 2019 at 10:10 AM, Paul Moore
wrote:
On Wed, 27 Mar 2019 at 08:25, Alexey Muranov
wrote:
Whey you need a simple function in Python, there is a choice
between a
normal function declaration and an assignment of a anonymous
function
(defined by a lambda-expression) to
On mer., Mar 27, 2019 at 5:00 PM, python-list-requ...@python.org wrote:
On 27/03/19 09:21, Alexey Muranov wrote:
Whey you need a simple function in Python, there is a choice
between a
normal function declaration and an assignment of a anonymous
function
(defined by a lambda-expression) to
syntax i can think of are currently
illegal, so i suppose there is no conflicts. (I would appreciate a
counterexample, if any.)
Thanks for the reference to PEP 8, this is indeed an argument against.
Alexey.
--
https://mail.python.org/mailman/listinfo/python-list
On jeu., mars 28, 2019 at 5:00 PM, python-list-requ...@python.org wrote:
On 2019-03-27 10:42 a.m., Paul Moore wrote:
On Wed, 27 Mar 2019 at 12:27, Alexey Muranov
wrote:
On mer., mars 27, 2019 at 10:10 AM, Paul Moore
wrote:
On Wed, 27 Mar 2019 at 08:25, Alexey Muranov
wrote:
Whey
tanding.
I believe there is no semantic conflict either, or could you be more
specific?
Alexey.
--
https://mail.python.org/mailman/listinfo/python-list
On jeu., mars 28, 2019 at 8:57 PM, Terry Reedy wrote:
On 3/28/2019 12:29 PM, Alexey Muranov wrote:
On jeu., Mar 28, 2019 at 5:00 PM, python-list-requ...@python.org
wrote:
So my opinion is that lambda expressions should only be used within
larger expressions and never directly bound.
It
On jeu., mars 28, 2019 at 5:00 PM, python-list-requ...@python.org wrote:
So documentation of that syntax would 100% be required
Regarding documentation, i believe there would be 3 line to add:
() =
is a syntactic sugar for
= lambda :
Alexey.
--
https://mail.python.org
On ven., Mar 29, 2019 at 4:51 PM, python-list-requ...@python.org wrote:
On Thu, Mar 28, 2019 at 2:30 PM Alexey Muranov
wrote:
On jeu., mars 28, 2019 at 8:57 PM, Terry Reedy
wrote:
> Throwing the name away is foolish. Testing functions is another
> situation in which function nam
On ven., Mar 29, 2019 at 4:51 PM, python-list-requ...@python.org wrote:
On Thu, Mar 28, 2019 at 2:30 PM Alexey Muranov
wrote:
On jeu., mars 28, 2019 at 8:57 PM, Terry Reedy
wrote:
> Throwing the name away is foolish. Testing functions is another
> situation in which fu
On dim., Mar 31, 2019 at 6:00 PM, python-list-requ...@python.org wrote:
On Sat, Mar 30, 2019, 5:32 AM Alexey Muranov
wrote:
On ven., Mar 29, 2019 at 4:51 PM, python-list-requ...@python.org
wrote:
>
> There could perhaps be a special case for lambda expressions such
> that
On lun., avril 1, 2019 at 6:00 PM, python-list-requ...@python.org wrote:
On Sun, Mar 31, 2019 at 1:09 PM Alexey Muranov
wrote:
On dim., Mar 31, 2019 at 6:00 PM, python-list-requ...@python.org
wrote:
> On Sat, Mar 30, 2019, 5:32 AM Alexey Muranov
>
> wrote:
>
>>
&
if Python did not have docstrings, the programmer could still
use comments to tell a fellow programmer what kind of code the fellow
programmer is looking at. Even languages like Brainfuck have comments
:).
Alexey.
--
https://mail.python.org/mailman/listinfo/python-list
On Mon, Apr 1, 2019 at 3:52 PM Alexey Muranov gmail.com>
wrote:
>
> I only see a superficial analogy with `super()`, but perhaps it is
> because you did not give much details of you suggestion.
No, it's because the analogy was not meant to be anything more than
superficial. Bo
On mar., Apr 2, 2019 at 6:00 PM, python-list-requ...@python.org wrote:
On Tue, Apr 2, 2019 at 1:43 AM Alexey Muranov
wrote:
> On Mon, Apr 1, 2019 at 3:52 PM Alexey Muranov gmail.com>
> wrote:
> >
> > I only see a superficial analogy with `super()`, but perhaps it
On mer., Apr 3, 2019 at 6:00 PM, python-list-requ...@python.org wrote:
On Wed, Apr 3, 2019 at 3:55 AM Alexey Muranov
wrote:
I clarified what i meant by an assignment, and i believe it to be a
usual meaning.
1. `def` is not an assignment, there is no left-hand side or
right-hand side. I
]> writes:
>
> > http://www-900.ibm.com/developerworks/cn/linux/sdk/python/charm-28/index_eng.shtml
>
> I can't reach it. Is there an alternative URL?
>
> Tschö,
> Torsten.
>
> --
> Torsten Bronger, aquisgrana, europa vetus
> --
> http://mail.pyt
oughts.
The test case is attached, the main file is test.bat. Python is
expected to be in PATH. Stderr of readers is redirected to *.log. You
may need to run several times to hit the issue.
Alexey Izbyshev,
research assistant,
ISP RAS
--
https://mail.python.org/mailman/listinfo/python-list
During developing web-projects, manager always need to test new features.
Typically, this is done using test servers. Often, manager cannot run a test
server himself to see new features and has to ask the developers for the help,
distracting them from their work. Besides purchased test equipment
Example script.py: """
def f(arg):
return g(arg)
def g(arg):
return arg
"""
Reading the Lib/runpy.py I've found, that the temporary module created
inside the run_path() calls, is destroyed right after the script.py code
executed in the resulting namespace.
I've got an idea. It would
Hi!
I've just had fun with the runpy module in Python 2.7. I'm writing to
share it :)
What I've tried is to "load" a python script using runpy.run_path(), take a
function from the resulting namespace and call it with arbitrary arguments.
All the functions in the namespace seem to be ok. r
After reading PEP-0328 I wanted to give relative imports a try:
# somepkg/__init__.py
# somepkg/test1.py
from __future__ import absolute_import
from . import test2
if __name__ == "__main__":
print "Test"
# somepkg/test2.py
But it complaints:
C:\1\somepkg>test1.py
Traceback (most recent c
[EMAIL PROTECTED] wrote:
> Compared to the Python I know and love, Ruby isn't quite the same.
> However, it has at least one terrific feature: "blocks".
Well, I particularly like how Boo (http://boo.codehaus.org) has done
it:
func(a, b, c) def(p1, p2, p3):
stmts
I was so attached to these
[EMAIL PROTECTED] wrote:
> but maybe it reduces code readabilty a bit for people
> that have just started to program:
>
> mul2 = def(a, b):
> return a * b
>
> Instead of:
>
> def mul2(a, b):
> return a * b
For such simple cases, yes. What about:
button.click += def(obj):
# do stuff
On Oct 25, 10:00 pm, "David S." <[EMAIL PROTECTED]> wrote:
> Does something like operator.getattr exist to perform a chained attr
> lookup?
Do you mean something like
class cattrgetter:
def __init__(self, name):
self.names = name.split('.')
def __call__(self, obj):
for nam
о увиденного показалось komodo. Да и то, подглюкивает
периодически, подвисает, из vcs знает только css да svn.
SourceOffSite'а не умеет :( Человеку, привыкшему к нормальному ide от
вижуал студии тяжко :)
--
С уважением,
Alexey mailto:[EMAIL PROTECTED]
--
http://mail.python.org/mailman/listinfo/python-list
On May 21, 8:12 am, "Silver Rock" <[EMAIL PROTECTED]> wrote:
> yes, that is the way I a solving the problem. using lists. so it seems
> that there is no way around it then..
There's at least one way to do it that I can think of straight away:
selfmodule = __import__(__name__, None, None, (None,))
;, line 9, in
[Sat Nov 29 19:41:15 2008] [error] import flup
[Sat Nov 29 19:41:15 2008] [error] ImportError: No module named flup
If you start it with console, you get the same, but there appears also
another path:
/home/username/python/lib64/python2.5/site-packages/flup-1.0.1-py2.5.egg
As I u
name/python/lib/python2.5/site-packages")
Thanks for help.
--
BRGDS. Alexey Vlasov.
--
http://mail.python.org/mailman/listinfo/python-list
Martin, thanks for fast reply, now anything is ok!
On Oct 6, 1:30 am, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> > I have a set of strings (all letters are capitalized) at utf-8,
>
> That's the problem. If these are really utf-8 encoded byte strings,
> then .lower likely won't work. It uses the
Hi!
I have a set of strings (all letters are capitalized) at utf-8,
russian language. I need to lower it, but
my_string.lower(). Doesn't work.
See sample script:
# -*- coding: utf-8 -*-
[skip]
s1 = self.title
s2 = self.title.lower()
print s1 == s2
returns true.
I have no problems with lower() for
72 matches
Mail list logo
