Re: memory consumption

[Alexey]

Found it. As I said before the problem was lurking in the cache.
Few days ago I read about circular references and things like that and
I thought to myself that it might be the case. To build the cache I was
using lots of 'setdefault' methods chained together

self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})...

and instead of wring a long lines I decided to divide it to increase 
readability

cluster = self.__cache.setdefault(cluster_name, {})
database = database.setdefault(database_name, {})
...
and I guess that was the problem.

First thing I did was to rewrite this back to single line. And it helped.
In the morning I tried different approach and decided to clear cache
with different way. So instead of doing self.__cache.clear(), 
self.__cache = None or even 'del self.__cache' I did:

for item in list(self.__cache.keys()):
del self.__cache[item]

and againg effect was positive. As a result I decided to rewrite all the
methods to build,update and get from cache without 'setdefault' and 
to use "for loop" instead of dict.clear().

I don't understand underlying processes and why I had this issues.
In my opinion if you're leaving the function scope you don't have to 
worry about what variables you're leaving there. There are other guys at
 StackOverflow who has similar problems with big dictionaries and memory 
 but they're using different approaches.

Thanks you everyone for your time and advices! I think the problem 
is solved and I hope some one will find this helpful
-- 
https://mail.python.org/mailman/listinfo/python-list

XanaNews Statistic for comp.lang.python. 4/1/2021 5:52:47 AM

[The Doctor via Python-list]

XanaNews Statistic for comp.lang.python.  4/1/2021 5:52:47 AM

>From article 587730 (3/1/2021 4:09:02 AM) to article 588588 (3/31/2021
11:40:40 PM)

Number of threads  ... 385
Number of articles  .. 895
Average articles per thread  . 2.32
Number of unanswered posts  .. 326
Number of posts from XanaNews users .. 0


Top Threads

Ranking  Articles  Subject
---    --
  185  .title() - annoying mistake
  240  Why assert is not a function?
  333  python documentation
  433  neonumeric - C++ arbitrary precision arithmetic
library
  531  memory consumption
  623  convert script awk in python
  717  Ann: New Python curses book
  817  Re: weirdness with list()
  916  Canonical conversion of dict of dicts to list of
dicts
 1014  How to loop over a text file (to remove tags and
normalize) using Python
 1111  port to PDOS (especially mainframe)
 1211  Choosable dependency
 13 9  Re: editor recommendations?
 14 9  Re: XML RPC changes between 3.7 and 3.9 yield 401
http error
 15 9  Looking for people interested in a Python register
virtual machine project
 16 9  Application problems
 17 9  A 35mm film camera represented in Python object
 18 9  Code Formatter Questions
 19 8  How to create both a c extension and a pure python
package
 20 8  How to implement logging for an imported module?
 21 8  How do I read .csv files
 22 7  Button placement
 23 7  program python
 24 7  Pips for python2 and python3
 25 6  Re: É DA ARRESTARE SUBITO L'AVVOCATO PEDOFILO ED
ASSASSINO DANIELE MINOTTI DI GENOVA, RAPALLO E CRIMINALE STUDIO LEGALE
LISI (FOTO
https://ecampus.aicel.org/wp-content/uploads/Daniele-Minotti-300x300.jpg
)!
 26 6  Problem with printing statement when condition is
false
 27 6  Horrible abuse of __init_subclass__, or elegant hack?
 28 6  Question about generators
 29 5  Best practices regarding PYTHONPATH
 30 5  Apriori Algorithm
 31 5  python curses constant names for mouse wheel
movements?
 32 5  file location/directory
 33 5  Retrieving non-/etc/passwd users with Python 3?
 34 5  "unexpected argument"
 35 4  a + not b
 36 4  REPL peculiarity
 37 4  Uninstall error
 38 4  SSL certificate issue
 39 4  Regarding Python installation issue
 40 4  firewall in python
 41 4  trouble using pygame
 42 4  Found a problem
 43 4  Re: try to install Python3.9.2 / setup failed
 44 4  Compare word to word files
 45 4  Help Please
 46 4  pygame font issue
 47 3  Unable to find 'edit with ide' option in the Context
menu
 48 3  i am getting the following error code while trying
to reinstall pip
 49 3  Apriori Algorithm
 50 3  Access to python37_d.lib
 51 3  thought
 52 3  Getting Modify Setup message
 53 3  Re: is not recognized as an internal or external
command, operable program or batch file.
 54 3  How to set up a 'listening' Unix domain socket
 55 3  pygame "font not intialized"
 56 3  pygame errors
 57 3  IDLE python
 58 2  Re: solution manual for Horngren’s Cost Accounting:
A Managerial Emphasis 16th Edition
 59 2  PYTHONBREAKPOINT=remote_pdb.set_trace


Top Posters

Ranking  Articles  NameMost Used Newsreader
---    --  
  1   167  TestBank store  G2   
  261  Vagrant G2   
  354  Chris Angelico   
  451  students help   G2   
  519  Robert Latest   slrn 
  619  Terry Reedy Mozilla  
  718  dn  Mozilla  
  817  Grant Edwards   slrn 
  916  Cameron Simpson Mutt 
 1016  Avi Gross   Microsoft Outlook
 1115  Alan Gauld  Mozilla  
 1214  Alexey  G2   
 1313  MRABMozilla  
 1412  Mr Flibble  Mozilla  
 1512  Peter Otten

Re: memory consumption

[Barry]

> On 31 Mar 2021, at 09:42, Alexey  wrote:
> 
> среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
>>> On Tue, Mar 30, 2021 at 1:25 AM Alexey  wrote: 
>>> 
>>> 
>>> I'm sorry. I didn't understand your question right. If I have 4 workers, 
>>> they require 4Gb 
>>> in idle state and some extra memory when they execute other tasks. If I 
>>> increase workers 
>>> count up to 16, they`ll eat all the memory I have (16GB) on my machine and 
>>> will crash as soon 
>>> as system get swapped. 
>>> 
>> What if you increase the machine's (operating system's) swap space? Does 
>> that take care of the problem in practice?
> 
> I can`t do that because it will affect other containers running on this host.
> In my opinion it may significantly reduce their performance.

Assuming this a modern linux then you should have control groups that allow you 
to set limits on memory and swap for each container.

Are you running with systemd?

Barry

> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: XanaNews Statistic for comp.lang.python. 4/1/2021 5:52:47 AM

[Chris Angelico]

On Thu, Apr 1, 2021 at 10:56 PM The Doctor via Python-list
 wrote:
> Top Posters
>
> Ranking  Articles  NameMost Used Newsreader
> ---    --  
>   1   167  TestBank store  G2
>   261  Vagrant G2
>   354  Chris Angelico
>   451  students help   G2

Ouch. I'm up there among the spambots.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Chris Angelico]

On Thu, Apr 1, 2021 at 10:56 PM Alexey  wrote:
>
> Found it. As I said before the problem was lurking in the cache.
> Few days ago I read about circular references and things like that and
> I thought to myself that it might be the case. To build the cache I was
> using lots of 'setdefault' methods chained together
>
> self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})...
>
> and instead of wring a long lines I decided to divide it to increase
> readability
>
> cluster = self.__cache.setdefault(cluster_name, {})
> database = database.setdefault(database_name, {})
> ...
> and I guess that was the problem.
>
> First thing I did was to rewrite this back to single line.

If the cache is always and only used in this way, it might be cleaner
to use a defaultdict(dict) instead of the setdefault calls. Or, since
this appears to be a two-level cache:

self.__cache = defaultdict(lambda: defaultdict(dict))

and then you can simply reference
self.__cache[cluster_name][database_name] to read or update the cache.

> And it helped.
> In the morning I tried different approach and decided to clear cache
> with different way. So instead of doing self.__cache.clear(),
> self.__cache = None or even 'del self.__cache' I did:
>
> for item in list(self.__cache.keys()):
> del self.__cache[item]
>
> and againg effect was positive. As a result I decided to rewrite all the
> methods to build,update and get from cache without 'setdefault' and
> to use "for loop" instead of dict.clear().

That seems very strange. Why should this be more effective than
self.__cache.clear()? I don't get it.

Having that be more efficient than either self.__cache=None or del
self.__cache (which will be equivalent), I can understand. But better
than clearing the dict? Seems very odd.

Ideally, though, you'd want to NOT have those reference loops. I
presume the database objects need to have a reference to whatever
'self' is, but perhaps the cache can be done externally to the object,
which would make all the references one-way instead of circular. But
that's something only you can investigate.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Marco Ippolito]

> >> What if you increase the machine's (operating system's) swap space? Does
> >> that take care of the problem in practice?
> > 
> > I can`t do that because it will affect other containers running on this
> > host.
> > In my opinion it may significantly reduce their performance.
> 
> Assuming this a modern linux then you should have control groups that allow
> you to set limits on memory and swap for each container.
> 
> Are you running with systemd?

If their problem is that their memory goes from `` to `` and then
back down to `` rather than ``, how could `cgroups` have helped in
that case?

I suspect the high watermark of `` needs to be reachable still and,
secondly, that a forceful constraint whilst running would crash the container?

How else could one approach it?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 14:57:29 UTC+3, Barry:
> > On 31 Mar 2021, at 09:42, Alexey  wrote: 
> > 
> > среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg:
> >>> On Tue, Mar 30, 2021 at 1:25 AM Alexey  wrote: 
> >>> 
> >>> 
> >>> I'm sorry. I didn't understand your question right. If I have 4 workers, 
> >>> they require 4Gb 
> >>> in idle state and some extra memory when they execute other tasks. If I 
> >>> increase workers 
> >>> count up to 16, they`ll eat all the memory I have (16GB) on my machine 
> >>> and 
> >>> will crash as soon 
> >>> as system get swapped. 
> >>> 
> >> What if you increase the machine's (operating system's) swap space? Does 
> >> that take care of the problem in practice? 
> > 
> > I can`t do that because it will affect other containers running on this 
> > host. 
> > In my opinion it may significantly reduce their performance.
> Assuming this a modern linux then you should have control groups that allow 
> you to set limits on memory and swap for each container. 
> 
> Are you running with systemd? 

I really don't know.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Marco Ippolito]

> > Are you running with systemd? 
> 
> I really don't know.

An example of how to check:

```
$ readlink /sbin/init
/lib/systemd/systemd
```

You want to check which program runs as PID 1.

```
ps 1
```
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Barry]

> On 1 Apr 2021, at 13:46, Marco Ippolito  wrote:
> 
> 
>> 
 What if you increase the machine's (operating system's) swap space? Does
 that take care of the problem in practice?
>>> 
>>> I can`t do that because it will affect other containers running on this
>>> host.
>>> In my opinion it may significantly reduce their performance.
>> 
>> Assuming this a modern linux then you should have control groups that allow
>> you to set limits on memory and swap for each container.
>> 
>> Are you running with systemd?
> 
> If their problem is that their memory goes from `` to `` and then
> back down to `` rather than ``, how could `cgroups` have helped in
> that case?
> 
> I suspect the high watermark of `` needs to be reachable still and,
> secondly, that a forceful constraint whilst running would crash the container?
> 
> How else could one approach it?
> 

I was responding to the assertion that adding swap to the system would impact 
other containers.
The solution I have used is to set service/container resource limits to ensure 
they work as expected.

I was not suggestion this a fix for the memory leak.

Barry
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Mats Wichmann]

On 4/1/21 5:50 AM, Alexey wrote:

Found it. As I said before the problem was lurking in the cache.
Few days ago I read about circular references and things like that and
I thought to myself that it might be the case. To build the cache I was
using lots of 'setdefault' methods chained together

self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})...

and instead of wring a long lines I decided to divide it to increase
readability

cluster = self.__cache.setdefault(cluster_name, {})
database = database.setdefault(database_name, {})
...
and I guess that was the problem.


I guess it is worth mentioning here that there are people who feel you 
shouldn't use setdefault() this way. setdefault is primarily a "getter", 
which has the side effect of filling in dict entry if one did not exist 
before returning it, and there some folks feel that using it primarily 
as a "setter" is abusing the interface...


Do with that what you will :)
--
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 15:27:01 UTC+3, Chris Angelico:
> On Thu, Apr 1, 2021 at 10:56 PM Alexey  wrote: 
> > 
> > Found it. As I said before the problem was lurking in the cache. 
> > Few days ago I read about circular references and things like that and 
> > I thought to myself that it might be the case. To build the cache I was 
> > using lots of 'setdefault' methods chained together 
> > 
> > self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})... 
> > 
> > and instead of wring a long lines I decided to divide it to increase 
> > readability 
> > 
> > cluster = self.__cache.setdefault(cluster_name, {}) 
> > database = database.setdefault(database_name, {}) 
> > ... 
> > and I guess that was the problem. 
> > 
> > First thing I did was to rewrite this back to single line.
> If the cache is always and only used in this way, it might be cleaner 
> to use a defaultdict(dict) instead of the setdefault calls. Or, since 
> this appears to be a two-level cache: 
> 
> self.__cache = defaultdict(lambda: defaultdict(dict)) 
> 
> and then you can simply reference 
> self.__cache[cluster_name][database_name] to read or update the cache.

I agree

> Having that be more efficient than either self.__cache=None or del 
> self.__cache (which will be equivalent), I can understand. But better 
> than clearing the dict? Seems very odd. 

In this particular case 'cache.clear()' just don't work (in context of 
releasing memory). If someone can tell me why, I'll be very thankful 

> Ideally, though, you'd want to NOT have those reference loops. I 
> presume the database objects need to have a reference to whatever 
> 'self' is, but perhaps the cache can be done externally to the object, 
> which would make all the references one-way instead of circular. But 
> that's something only you can investigate. 

I did some refactoring and changed my code already.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 16:02:15 UTC+3, Barry:
> > On 1 Apr 2021, at 13:46, Marco Ippolito  wrote: 
> > 
> >
> >> 
>  What if you increase the machine's (operating system's) swap space? Does 
>  that take care of the problem in practice? 
> >>> 
> >>> I can`t do that because it will affect other containers running on this 
> >>> host. 
> >>> In my opinion it may significantly reduce their performance. 
> >> 
> >> Assuming this a modern linux then you should have control groups that 
> >> allow 
> >> you to set limits on memory and swap for each container. 
> >>
> >> Are you running with systemd? 
> >
> > If their problem is that their memory goes from `` to `` and 
> > then 
> > back down to `` rather than ``, how could `cgroups` have helped 
> > in 
> > that case? 
> > 
> > I suspect the high watermark of `` needs to be reachable still and, 
> > secondly, that a forceful constraint whilst running would crash the 
> > container? 
> > 
> > How else could one approach it? 
> >
> I was responding to the assertion that adding swap to the system would impact 
> other containers. 
> The solution I have used is to set service/container resource limits to 
> ensure they work as expected. 
> 
> I was not suggestion this a fix for the memory leak. 

I think it's good advice anyway. Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 17:21:59 UTC+3, Mats Wichmann:
> On 4/1/21 5:50 AM, Alexey wrote: 
> > Found it. As I said before the problem was lurking in the cache. 
> > Few days ago I read about circular references and things like that and 
> > I thought to myself that it might be the case. To build the cache I was 
> > using lots of 'setdefault' methods chained together 
> > 
> > self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})... 
> > 
> > and instead of wring a long lines I decided to divide it to increase 
> > readability 
> > 
> > cluster = self.__cache.setdefault(cluster_name, {}) 
> > database = database.setdefault(database_name, {}) 
> > ... 
> > and I guess that was the problem.
> I guess it is worth mentioning here that there are people who feel you 
> shouldn't use setdefault() this way. setdefault is primarily a "getter", 
> which has the side effect of filling in dict entry if one did not exist 
> before returning it, and there some folks feel that using it primarily 
> as a "setter" is abusing the interface... 
> 
> Do with that what you will :)

Ok )
-- 
https://mail.python.org/mailman/listinfo/python-list

[ANN] Free Python tutorial with exercises, cscx.org

[Rudy Matela]

Hello python-list members,

I would like to announce the following educational project:

Computer Science by Example https://cscx.org/ is a collection of short
programming exercises.  The site can automatically grade students'
solutions and it accepts submissions in Python.

The exercises start simple, then increase in difficulty and complexity
gradually.  Here are some examples:

* Print "Hello, World!": https://cscx.org/hello
* Add two numbers: https://cscx.org/add1
* Compute the factorial of a number: https://cscx.org/factorial
* Compute the GCD of two numbers: https://cscx.org/gcd
* Solve the change-making problem: https://cscx.org/cash

The website has a tutorial section covering Python's basics.

I tried to make the content easy to use by instructors/lecturers, feel free
to use this with your students.  The backend of the website is open source
and you can find it on
https://github.com/rudymatela/udge

-- Rudy
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Horrible abuse of __init_subclass__, or elegant hack?

[Alan Gauld via Python-list]

On 01/04/2021 00:14, Chris Angelico wrote:

> On a scale of 1 to "submit this to The Daily WTF immediately", how bad
> is this code? :)

The only worthwhile test of code quality is whether a new member
of the team, competent in the language but not an expert can
understand the code in two readings or less. If they can it's
good enough. Maintenance is always the highest cost in any
significant project so good code must be maintainable.

In this case I'd venture they'd encounter init_sublass() for the
first time, do some research and on the second reading make sense
of it. So, it's good to go.

Whether it's an optimal design or not is a different matter.
There's always more than one way to do it.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Horrible abuse of __init_subclass__, or elegant hack?

[Chris Angelico]

On Thu, Apr 1, 2021 at 3:36 PM dn via Python-list
 wrote:
>
> On 01/04/2021 13.54, Chris Angelico wrote:
> > Real and imaginary are the same thing, just rotated a quarter turn
>
> In which dimension(s)?

Cartesian.

> >> Without looking into the details/context: surely there's a more
> >> straightforward approach?
> >
> > Perhaps, but there are potentially a LOT of recipes, and I needed to
> > be able to cleanly edit those, even if the code at the top was a mess.
> > (The goal here is to map out production patterns in the game
> > "Satisfactory", for the curious. It's easy to add other things, like
> > computer manufacturing or bauxite processing, simply by adding more
> > recipes.)
> >
> > My original plan was basically pairwise tuple summing (deriving a set
> > of "oil in, water in, rubber out, fuel out" for each set of recipes,
> > where some might be zero), but it turned out that that wasn't flexible
> > enough, and it really needed more options than that.
>
> Which was where my mind was going*. Why not a dict of inputs, processes,
> and outputs**? Each dict having variable length, from None, and
> key:values assigned at declaration/init. In the case of process, the
> contained objects could be Python-functions. With "compact
> representation" (3.6+) the functions could also be relied upon to
> represent a 'production line' or pipeline of functions.

Perhaps, but the key here is the input method. It wouldn't look nearly as clean.

> > I already have certificates from Rutledge's Asylum and MaayaInsane's
> > (unnamed) asylum, so that seems pretty likely.
>
> Noted you on the list of lauded alumni at the latter.

Hmm, where do you see that list? I'm curious.

> When you left the former, did they allow you to keep the t-shirt, or did
> you have to buy your own memorabilia?
> (https://mysterious.americanmcgee.com/products/rutledge-asylum-mug)

I beg your pardon? What is this "left"? I'm still there! Actually I
pay my membership on a monthly basis, and in return, my walls are
lavishly decorated in Alice art.

> The latter's treatment list sounds remarkably like .mil training. I know
> of plenty with that t-shirt - but can't think of a one sporting a mug...
> Should you have one, kindly bring it (with appropriate contents) come
> ANZAC Day at the end of this month...

https://streamlabs.com/maayainsane/merch/1053635
This is what I'd bring. They're the standard mugs that I offer to
guests. Well, I would if ever I had guests, but hermitism is a
thing...

> Magic, you ask? Well, maybe more "sinister". We did manage to find a
> loose floor-board, but a sad life-lesson was learned, when certain ones
> (un-named*) took it upon themselves to eat all of the contraband
> secreted there.

Uh oh. How old was the contraband?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 15:46:21 UTC+3, Marco Ippolito:

> I suspect the high watermark of `` needs to be reachable still and, 
> secondly, that a forceful constraint whilst running would crash the 
> container? 

Exactly.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Alexey]

четверг, 1 апреля 2021 г. в 15:56:23 UTC+3, Marco Ippolito:
> > > Are you running with systemd? 
> > 
> > I really don't know.
> An example of how to check: 
> 
> ``` 
> $ readlink /sbin/init 
> /lib/systemd/systemd 
> ``` 
> 
> You want to check which program runs as PID 1. 

Thank you Marco
-- 
https://mail.python.org/mailman/listinfo/python-list

Issues in starting Python application

[Mahira Pamnani]

Sir
I have been trying hard to install Python on my PC since a long time. The
application gets installed but, for reasons unknown, it doesn't start.
It keeps asking to repair, modify or uninstall the application.
I have tried doing that too, but it still doesn't give any results.
Please guide me on how to solve this issue.

Thank you
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Issues in starting Python application

[Marco Ippolito]

On 2021-04-01, Mahira Pamnani wrote:
> Sir
> I have been trying hard to install Python on my PC since a long time. The
> application gets installed but, for reasons unknown, it doesn't start.
> It keeps asking to repair, modify or uninstall the application.
> I have tried doing that too, but it still doesn't give any results.
> Please guide me on how to solve this issue.
> Thank you

What do you do that takes you to this offer to repair?

For example: I click on the Start menu and try to select... .

Maybe a screenshot of the moment in which the problem manifests itself?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: memory consumption

[Sam]

On 3/29/21 5:12 AM, Alexey wrote:

Hello everyone!
I'm experiencing problems with memory consumption.

I have a class which is doing ETL job. What`s happening inside:
  - fetching existing objects from DB via SQLAchemy
  - iterate over raw data
  - create new/update existing objects
  - commit changes

Before processing data I create internal cache(dictionary) and store all 
existing objects in it.
Every 1 items I do bulk insert and flush. At the end I run commit command.

Problem. Before executing, my interpreter process weighs ~100Mb, after first 
run memory increases up to 500Mb
and after second run it weighs 1Gb. If I will continue to run this class, 
memory wont increase, so I think
it's not a memory leak, but rather Python wont release allocated memory back to 
OS. Maybe I'm wrong.

What I tried after executing:
  - gc.collect()
  - created snapshots with tracemalloc and searched for some garbage, diff =
smapshot_before_run - smapshot_after_run
  - searched for links with "objgraph" library to internal cache(dictionary
containing elements from DB)
  - cleared the cache(dictionary)
  - db.session.expire_all()

This class is a periodic celery task. So when each worker executes this class 
at least two times,
all celery workers need 1Gb of RAM. Before celery there was a cron script and 
this class was executed via API call
and the problem was the same. So no matter how I run, interpreter consumes 1Gb 
of RAM after two runs.

I see few solutions to this problem
1. Execute this class in separate process. But I had few errors when the same 
SQLAlchemy connection being shared
between different processes.
2. Restart celery worker after executing this task by throwing exception.
3. Use separate queue for such tasks, but then worker will stay idle most of 
the time.
All this is looks like a crutch. Do I have any other options ?

I'm using:
Python - 3.6.13
Celery - 4.1.0
Flask-RESTful - 0.3.6
Flask-SQLAlchemy - 2.3.2

Thanks in advance!



I had the (mis)pleasure of dealing with a multi-terabyte postgresql 
instance many years ago and figuring out why random scripts were eating 
up system memory became quite common.


All of our "ETL" scripts were either written in Perl, Java, or Python 
but the results were always the same, if a process grew to using 1gb of 
memory (as your case), then it never "released" it back to the OS. What 
this basically means is that your script at one time did in fact 
use/need 1GB of memory. That becomes the "high watermark" and in most 
cases usage will stay at that level. And if you think about it, it makes 
sense. Your python program went through the trouble of requesting memory 
space from the OS, it makes no sense for it to give it back to the OS as 
if it needed 1GB in the past, it will probably need 1GB in the future so 
you will just waste time with syscalls. Even the glibc docs state 
calling free() does not necessarily mean that the OS will allocate the 
"freed" memory back to the global memory space.


There are basically two things you can try. First, try working in 
smaller batch sizes. 10,000 is a lot, try 100. Second, as you hinted, 
try moving the work to a separate process. The simple way to do this 
would be to move away from modules that use threads and instead use 
something that creates child processes with fork().


Regards,



--
https://mail.python.org/mailman/listinfo/python-list

Re: Issues in starting Python application

[Igor Korot]

Hi,

On Thu, Apr 1, 2021 at 10:41 AM Marco Ippolito  wrote:
>
> On 2021-04-01, Mahira Pamnani wrote:
> > Sir
> > I have been trying hard to install Python on my PC since a long time. The
> > application gets installed but, for reasons unknown, it doesn't start.
> > It keeps asking to repair, modify or uninstall the application.
> > I have tried doing that too, but it still doesn't give any results.
> > Please guide me on how to solve this issue.
> > Thank you
>
> What do you do that takes you to this offer to repair?
>
> For example: I click on the Start menu and try to select... .
>
> Maybe a screenshot of the moment in which the problem manifests itself?

Don't send a screenshot to the list - it will get rejected.
After you install python do this:

Open Windows Command Prompt.
Type "python" (without quotes) and press Enter.

Will you get a python prompt (liike >>>)?

Thank you.

> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: NOT ABLE TO DOWNLOAD speech_recognition ON MY COMPUTER

[Marco Ippolito]

On 2021-04-01, ᗷᑌᑎᑎY wrote:
> When I enter the command  pip install speech_recognition  it say’s we cannot
> find a compatible version.

Try:

```
pip install SpeechRecognition
```

What's the output?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Issues in starting Python application

[Mats Wichmann]

On 4/1/21 9:08 AM, Mahira Pamnani wrote:

Sir
I have been trying hard to install Python on my PC since a long time. The
application gets installed but, for reasons unknown, it doesn't start.
It keeps asking to repair, modify or uninstall the application.
I have tried doing that too, but it still doesn't give any results.
Please guide me on how to solve this issue.


You're ending up rerunning the installer - this seems to have been 
happening to quite a few Windows users recently.  The program you're 
launching is doing what it's intended to do - manage the Python 
_installation_, but it's not Python itself.


Try launching Python from the start menu - navigate to "Python 3.9" and 
start either "Python 3.9 (64-bit)" or "IDLE (Python 3.9 64-bit)" 
depending on whether you want the bare interpreter or the IDLE 
development environment,


or

From a command shell type "py"  to activate the Python Launcher.


Also see here:

https://docs.python.org/3/using/windows.html


--
https://mail.python.org/mailman/listinfo/python-list

Re: Issues in starting Python application

[Grant Edwards]

On 2021-04-01, Mahira Pamnani  wrote:

> I have been trying hard to install Python on my PC since a long time. The
> application gets installed but, for reasons unknown, it doesn't start.
> It keeps asking to repair, modify or uninstall the application.

That's because you're re-running the installer, not the installed
Python application.

--
Grant




-- 
https://mail.python.org/mailman/listinfo/python-list

NOT ABLE TO DOWNLOAD speech_recognition ON MY COMPUTER

[ᗷᑌᑎᑎY]

   Hello everyone. I am not able to download  speech_recognition . I am not
   professional just a beggnier but I decided and started developing a voice
   commanding software and I need to download speech_recognition. When I
   enter the command  pip install speech_recognition  it say's we cannot find
   a compatible version. My windows version is
   windows [Version 10.0.19042.844] . is there any way  I can  download it. I
   will be waiting for your advice.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: NOT ABLE TO DOWNLOAD speech_recognition ON MY COMPUTER

[Igor Korot]

Hi,

On Thu, Apr 1, 2021 at 12:23 PM ᗷᑌᑎᑎY  wrote:
>
>Hello everyone. I am not able to download  speech_recognition . I am not
>professional just a beggnier but I decided and started developing a voice
>commanding software and I need to download speech_recognition. When I
>enter the command  pip install speech_recognition  it say's we cannot find
>a compatible version. My windows version is
>windows [Version 10.0.19042.844] . is there any way  I can  download it. I
>will be waiting for your advice.

Someone just answered on another thread. ;-)

Try to install SpeechRecognition

Thank you.

>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [ANN] Free Python tutorial with exercises, cscx.org

[Chris Angelico]

On Fri, Apr 2, 2021 at 2:14 AM Rudy Matela  wrote:
>
> Hello python-list members,
>
> I would like to announce the following educational project:
>
> Computer Science by Example https://cscx.org/ is a collection of short
> programming exercises.  The site can automatically grade students'
> solutions and it accepts submissions in Python.
>

What versions of Python does it support? The setup page is not clear.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Unable to use pip to install packages

[Md Sohail Ansari via Python-list]

Hi Team,
Thanks for your support. I am having trouble installing any packages using pip 
in Python 3.x (earlier had Python 3.8.3, so changed to 3.8.8) in Windows 10.
Please help me with below issues:
---C:\Users\mohsohai>pip install 
pytestTraceback (most recent call last):  File 
"C:\Users\mohsohai\AppData\Local\Programs\Python\Python38\Scripts\pip-script.py",
 line 11, in     load_entry_point('pip==21.0.1', 'console_scripts', 
'pip')()  File 
"c:\users\mohsohai\appdata\local\programs\python\python38\lib\site-packages\pip\_internal\cli\main.py",
 line 71, in main    command = create_command(cmd_name, isolated=("--isolated" 
in cmd_args))  File 
"c:\users\mohsohai\appdata\local\programs\python\python38\lib\site-packages\pip\_internal\commands\__init__.py",
 line 96, in create_command    module = importlib.import_module(module_path)  
File 
"c:\users\mohsohai\appdata\local\programs\python\python38\lib\importlib\__init__.py",
 line 127, in import_module    return _bootstrap._gcd_import(name[level:], 
package, level)  File "", line 1014, in 
_gcd_import  File "", line 991, in _find_and_load  
File "", line 975, in _find_and_load_unlocked  
File "", line 671, in _load_unlocked  File 
"", line 783, in exec_module  File 
"", line 219, in _call_with_frames_removed  File 
"c:\users\mohsohai\appdata\local\programs\python\python38\lib\site-packages\pip\_internal\commands\install.py",
 line 12, in     from pip._internal.cache import 
WheelCacheModuleNotFoundError: No module named 
'pip._internal.cache'-
Note: Have included pip in Enviromnet Variables
Thank you,Sohail.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unable to use pip to install packages

[MRAB]

On 2021-04-01 19:42, Md Sohail Ansari via Python-list wrote:

Hi Team,
Thanks for your support. I am having trouble installing any packages using pip 
in Python 3.x (earlier had Python 3.8.3, so changed to 3.8.8) in Windows 10.
Please help me with below issues:
---C:\Users\mohsohai>pip install pytestTraceback (most recent call last):  File "C:\Users\mohsohai\AppData\Local\Programs\Python\Python38\Scripts\pip-script.py", line 11, in     
load_entry_point('pip==21.0.1', 'console_scripts', 'pip')()  File "c:\users\mohsohai\appdata\local\programs\python\python38\lib\site-packages\pip\_internal\cli\main.py", line 71, in main    command = create_command(cmd_name, 
isolated=("--isolated" in cmd_args))  File "c:\users\mohsohai\appdata\local\programs\python\python38\lib\site-packages\pip\_internal\commands\__init__.py", line 96, in create_command    module = 
importlib.import_module(module_path)  File "c:\users\mohsohai\appdata\local\programs\python\python38\lib\importlib\__init__.py", line 127, in import_module    return _bootstrap._gcd_import(name[level:], package, level)  File 
"", line 1014, in _gcd_import  File "", line 991, in _find_and_load  File "", line 975, in _find_and_load_unlocked  
File "", line 671, in _load_unlocked  File "", line 783, in exec_module  File "", line 219, in 
_call_with_frames_removed  File "c:\users\mohsohai\appdata\local\programs\python\python38\lib\site-packages\pip\_internal\commands\install.py", line 12, in     from pip._internal.cache import WheelCacheModuleNotFoundError: No 
module named 'pip._internal.cache'-
Note: Have included pip in Enviromnet Variables
Thank you,Sohail.


I recommend you use the pip module with Python launcher:

py -m pip install pytest

It just makes life easier.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Horrible abuse of __init_subclass__, or elegant hack?

[David L Neil via Python-list]

Officially April-Fools Day is over (here), but...


On 01/04/2021 19.25, Chris Angelico wrote:
> On Thu, Apr 1, 2021 at 3:36 PM dn via Python-list
>  wrote:
>>
>> On 01/04/2021 13.54, Chris Angelico wrote:
>>> Real and imaginary are the same thing, just rotated a quarter turn
>>
>> In which dimension(s)?
> 
> Cartesian.

Isn't this water you get out of the ground and then deliver using an
horse-pulled vehicle?


 Without looking into the details/context: surely there's a more
 straightforward approach?
>>>
>>> Perhaps, but there are potentially a LOT of recipes, and I needed to
>>> be able to cleanly edit those, even if the code at the top was a mess.
>>> (The goal here is to map out production patterns in the game
>>> "Satisfactory", for the curious. It's easy to add other things, like
>>> computer manufacturing or bauxite processing, simply by adding more
>>> recipes.)
>>>
>>> My original plan was basically pairwise tuple summing (deriving a set
>>> of "oil in, water in, rubber out, fuel out" for each set of recipes,
>>> where some might be zero), but it turned out that that wasn't flexible
>>> enough, and it really needed more options than that.
>>
>> Which was where my mind was going*. Why not a dict of inputs, processes,
>> and outputs**? Each dict having variable length, from None, and
>> key:values assigned at declaration/init. In the case of process, the
>> contained objects could be Python-functions. With "compact
>> representation" (3.6+) the functions could also be relied upon to
>> represent a 'production line' or pipeline of functions.
> 
> Perhaps, but the key here is the input method. It wouldn't look nearly as 
> clean.

Not sure about that. For example, is it immediately-obvious which of
"Crude", "Residue", and "Plastic" are inputs; and which outputs?


>>> I already have certificates from Rutledge's Asylum and MaayaInsane's
>>> (unnamed) asylum, so that seems pretty likely.
>>
>> Noted you on the list of lauded alumni at the latter.
> 
> Hmm, where do you see that list? I'm curious.

Appeared to be some sort of 'leader board' for the game. Recognised your
handle there. Think it was https://www.twitch.tv/maayainsane. Perhaps I
was looking at the same time as you logged-in? Daren't disappear down
that rabbit-hole again...


>> When you left the former, did they allow you to keep the t-shirt, or did
>> you have to buy your own memorabilia?
>> (https://mysterious.americanmcgee.com/products/rutledge-asylum-mug)
> 
> I beg your pardon? What is this "left"? I'm still there! Actually I
> pay my membership on a monthly basis, and in return, my walls are
> lavishly decorated in Alice art.

Whereas us lateral-thinkers don't like the feeling of being fenced-in.


>> The latter's treatment list sounds remarkably like .mil training. I know
>> of plenty with that t-shirt - but can't think of a one sporting a mug...
>> Should you have one, kindly bring it (with appropriate contents) come
>> ANZAC Day at the end of this month...
> 
> https://streamlabs.com/maayainsane/merch/1053635
> This is what I'd bring. They're the standard mugs that I offer to
> guests. Well, I would if ever I had guests, but hermitism is a
> thing...

Is that the same as (anti-) club-ism
(cue Groucho:
https://www.goodreads.com/quotes/6517787-i-wouldn-t-want-to-belong-to-a-club-that-would)

After suffering the RSA's ?quaint tradition of rum-in-milk (that BEFORE
Dawn Parade, and thereafter marching on a less-than flat surface) I have
learned to guard against the pre-dawn cold by heading for the Hot
Chocolate counter - can't say the mugs look like that though.


>> Magic, you ask? Well, maybe more "sinister". We did manage to find a
>> loose floor-board, but a sad life-lesson was learned, when certain ones
>> (un-named*) took it upon themselves to eat all of the contraband
>> secreted there.
> 
> Uh oh. How old was the contraband?

Was between 'tuck days' (when we were allowed to acquire such goodies -
once?twice per week). The theory being that we would all contribute, in
anticipation of some later "midnight feast") - such gyre and gimble-ing
being more Jabberwock than Alice!
-- 
Regards =dn
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: XanaNews Statistic for comp.lang.python. 4/1/2021 5:52:47 AM

[dn via Python-list]

On 02/04/2021 01.19, Chris Angelico wrote:
> On Thu, Apr 1, 2021 at 10:56 PM The Doctor via Python-list
>  wrote:
>> Top Posters
>>
>> Ranking  Articles  NameMost Used Newsreader
>> ---    --  
>>   1   167  TestBank store  G2
>>   261  Vagrant G2
>>   354  Chris Angelico
>>   451  students help   G2
> 
> Ouch. I'm up there among the spambots.


No, not true!
We all know you are ham-ming it up*...

However, after some reflection, is the bot part not true?


* slang term: https://idioms.thefreedictionary.com/hamming+it+up
-- 
Regards,
=dn
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: XanaNews Statistic for comp.lang.python. 4/1/2021 5:52:47 AM

[Chris Angelico]

On Fri, Apr 2, 2021 at 7:55 AM dn via Python-list
 wrote:
>
> On 02/04/2021 01.19, Chris Angelico wrote:
> > On Thu, Apr 1, 2021 at 10:56 PM The Doctor via Python-list
> >  wrote:
> >> Top Posters
> >>
> >> Ranking  Articles  NameMost Used Newsreader
> >> ---    --  
> >>   1   167  TestBank store  G2
> >>   261  Vagrant G2
> >>   354  Chris Angelico
> >>   451  students help   G2
> >
> > Ouch. I'm up there among the spambots.
>
>
> No, not true!
> We all know you are ham-ming it up*...
>
> However, after some reflection, is the bot part not true?
>

Oh yes, that's a known fact. Particularly on Twitch.tv, where I'm
known for developing myself on livestream.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Horrible abuse of __init_subclass__, or elegant hack?

[Chris Angelico]

On Fri, Apr 2, 2021 at 7:52 AM David L Neil via Python-list
 wrote:
>
> Officially April-Fools Day is over (here), but...

This wasn't a prank post, although it was intended to give amusement
rather than real education or debate or anything. So there's nothing
wrong with keeping it going. :)

> On 01/04/2021 19.25, Chris Angelico wrote:
> > On Thu, Apr 1, 2021 at 3:36 PM dn via Python-list
> >  wrote:
> >>
> >> On 01/04/2021 13.54, Chris Angelico wrote:
> >>> Real and imaginary are the same thing, just rotated a quarter turn
> >>
> >> In which dimension(s)?
> >
> > Cartesian.
>
> Isn't this water you get out of the ground and then deliver using an
> horse-pulled vehicle?

Yes. Also, this is the guy who said "I think not" and promptly
vanished from the world.

(For those not familiar, the Cartesian coordinate system or the
Cartesian plane is what you'd normally use for graphing a mathematical
function. You can label the axes as "x" and "y" to graph something
like "y = sin(x)", or you can label them as "real" and "imaginary" and
use it to depict complex numbers. Though David's response has the
elegance of true *art*, if you examine the words carefully.)

> > Perhaps, but the key here is the input method. It wouldn't look nearly as 
> > clean.
>
> Not sure about that. For example, is it immediately-obvious which of
> "Crude", "Residue", and "Plastic" are inputs; and which outputs?

Well, it's a simple matter of chronology. First you have crude oil,
then time passes, and then you have plastic and residue. It makes
sense ONLY if you think of it with a specific ordering, which implies
Python 3.7 or later.

> >>> I already have certificates from Rutledge's Asylum and MaayaInsane's
> >>> (unnamed) asylum, so that seems pretty likely.
> >>
> >> Noted you on the list of lauded alumni at the latter.
> >
> > Hmm, where do you see that list? I'm curious.
>
> Appeared to be some sort of 'leader board' for the game. Recognised your
> handle there. Think it was https://www.twitch.tv/maayainsane. Perhaps I
> was looking at the same time as you logged-in? Daren't disappear down
> that rabbit-hole again...

Makes sense. I'm very active there, you'll see me chatting a good bit
during her streams. I'm not sure what leaderboard you were seeing, but
it's probably something involving recent financial support (she's a
highly competent digital artist and absolutely deserves the support).

> >> The latter's treatment list sounds remarkably like .mil training. I know
> >> of plenty with that t-shirt - but can't think of a one sporting a mug...
> >> Should you have one, kindly bring it (with appropriate contents) come
> >> ANZAC Day at the end of this month...
> >
> > https://streamlabs.com/maayainsane/merch/1053635
> > This is what I'd bring. They're the standard mugs that I offer to
> > guests. Well, I would if ever I had guests, but hermitism is a
> > thing...
>
> Is that the same as (anti-) club-ism
> (cue Groucho:
> https://www.goodreads.com/quotes/6517787-i-wouldn-t-want-to-belong-to-a-club-that-would)
>
> After suffering the RSA's ?quaint tradition of rum-in-milk (that BEFORE
> Dawn Parade, and thereafter marching on a less-than flat surface) I have
> learned to guard against the pre-dawn cold by heading for the Hot
> Chocolate counter - can't say the mugs look like that though.

Hot chocolate? Ahh, now, that's the other thing I drink a lot of. I
even have a dedicated mug with a Cadbury logo on it. Sadly, not easily
purchasable, but this one was a gift from a family member who knows
full well that I have my priorities straight.

> >> Magic, you ask? Well, maybe more "sinister". We did manage to find a
> >> loose floor-board, but a sad life-lesson was learned, when certain ones
> >> (un-named*) took it upon themselves to eat all of the contraband
> >> secreted there.
> >
> > Uh oh. How old was the contraband?
>
> Was between 'tuck days' (when we were allowed to acquire such goodies -
> once?twice per week). The theory being that we would all contribute, in
> anticipation of some later "midnight feast") - such gyre and gimble-ing
> being more Jabberwock than Alice!

Ah, I see. So you'd get some tucker, and tuck it away for later, but
the whole scheme came... unstuck.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[D.M. Procida]

D.M. Procida  wrote:

> Hi everyone, I've created  -
> a representation of a Canonet G-III QL17 in Python.
> 
> There's also documentation: .
> 
> It's a pure Python model of the physical sub-systems of a camera and
> their interactions. So far it's at a fairly high level - I haven't yet
> got down to the level of individual springs and levers yet.

I'm now working on the exposure system, which in the published version
so far just models inputs and outputs of the system as a whole. I want
to start to model its functionality by modelling the interactions of its
components - the levers, cams and so on.

It seems to me I have fundamentally two approaches that I could take:

The first is to model the movement of each lever as a series of discrete
steps (hundreds of tiny steps, to maintain accuracy). Pro: it models
movement through physical positions; con: it's ugly that it breaks
continuous movement into arbitrary steps.

The second is not to move each lever in such detail, but to consider its
interactions: given the state of each of the other parts of the system,
what *would* be the outcome if something were to try to move the lever
from such-and-such a position to another? Pro: it feels more elegant
than the brute-force stepping of the alternative; con: it also feels
like a bit of a cheat.

But neither approach really captures for me the interaction of moving
analog components, which in the case of this camera also have some
circular logic to them - the movement of A determines that of B which
determines that of C which determines that of D which finally also
affects the movement of A.

Any thoughts or wise ideas? 

Daniele
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Richard Damon]

On 4/1/21 5:47 PM, D.M. Procida wrote:
> D.M. Procida  wrote:
>
>> Hi everyone, I've created  -
>> a representation of a Canonet G-III QL17 in Python.
>>
>> There's also documentation: .
>>
>> It's a pure Python model of the physical sub-systems of a camera and
>> their interactions. So far it's at a fairly high level - I haven't yet
>> got down to the level of individual springs and levers yet.
> I'm now working on the exposure system, which in the published version
> so far just models inputs and outputs of the system as a whole. I want
> to start to model its functionality by modelling the interactions of its
> components - the levers, cams and so on.
>
> It seems to me I have fundamentally two approaches that I could take:
>
> The first is to model the movement of each lever as a series of discrete
> steps (hundreds of tiny steps, to maintain accuracy). Pro: it models
> movement through physical positions; con: it's ugly that it breaks
> continuous movement into arbitrary steps.
>
> The second is not to move each lever in such detail, but to consider its
> interactions: given the state of each of the other parts of the system,
> what *would* be the outcome if something were to try to move the lever
> from such-and-such a position to another? Pro: it feels more elegant
> than the brute-force stepping of the alternative; con: it also feels
> like a bit of a cheat.
>
> But neither approach really captures for me the interaction of moving
> analog components, which in the case of this camera also have some
> circular logic to them - the movement of A determines that of B which
> determines that of C which determines that of D which finally also
> affects the movement of A.
>
> Any thoughts or wise ideas? 
>
> Daniele

If you keep track of the positions as a floating point number, the
precision will be more than you could actually measure it.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Mike Dewhirst]

On 2/04/2021 8:47 am, D.M. Procida wrote:

D.M. Procida  wrote:

Hi everyone, I've created  

-

a representation of a Canonet G-III QL17 in Python.

There's also documentation: .

It's a pure Python model of the physical sub-systems of a camera and
their interactions. So far it's at a fairly high level - I haven't yet
got down to the level of individual springs and levers yet.

I'm now working on the exposure system, which in the published version
so far just models inputs and outputs of the system as a whole. I want
to start to model its functionality by modelling the interactions of its
components - the levers, cams and so on.

It seems to me I have fundamentally two approaches that I could take:

The first is to model the movement of each lever as a series of discrete
steps (hundreds of tiny steps, to maintain accuracy). Pro: it models
movement through physical positions; con: it's ugly that it breaks
continuous movement into arbitrary steps.


If in doubt model the real world.



The second is not to move each lever in such detail, but to consider its
interactions: given the state of each of the other parts of the system,
what *would* be the outcome if something were to try to move the lever
from such-and-such a position to another? Pro: it feels more elegant
than the brute-force stepping of the alternative; con: it also feels
like a bit of a cheat.


Usually taking shortcuts means losing information. You can only drop 
intermediate data which doesn't trigger other events.


Go with your instinct



But neither approach really captures for me the interaction of moving
analog components, which in the case of this camera also have some
circular logic to them - the movement of A determines that of B which
determines that of C which determines that of D which finally also
affects the movement of A.


Connect things which are connected and recalculate A in passive mode.



Any thoughts or wise ideas?


May the force be with you



Daniele



--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.



OpenPGP_signature
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[2QdxY4RzWzUUiLuE]

On 2021-04-01 at 18:10:46 -0400,
Richard Damon  wrote:

> On 4/1/21 5:47 PM, D.M. Procida wrote:
> > D.M. Procida  wrote:
> >
> >> Hi everyone, I've created  -
> >> a representation of a Canonet G-III QL17 in Python.
> >>
> >> There's also documentation: .
> >>
> >> It's a pure Python model of the physical sub-systems of a camera and
> >> their interactions. So far it's at a fairly high level - I haven't yet
> >> got down to the level of individual springs and levers yet.
> > I'm now working on the exposure system, which in the published version
> > so far just models inputs and outputs of the system as a whole. I want
> > to start to model its functionality by modelling the interactions of its
> > components - the levers, cams and so on.
> >
> > It seems to me I have fundamentally two approaches that I could take:
> >
> > The first is to model the movement of each lever as a series of discrete
> > steps (hundreds of tiny steps, to maintain accuracy). Pro: it models
> > movement through physical positions; con: it's ugly that it breaks
> > continuous movement into arbitrary steps.
> >
> > The second is not to move each lever in such detail, but to consider its
> > interactions: given the state of each of the other parts of the system,
> > what *would* be the outcome if something were to try to move the lever
> > from such-and-such a position to another? Pro: it feels more elegant
> > than the brute-force stepping of the alternative; con: it also feels
> > like a bit of a cheat.
> >
> > But neither approach really captures for me the interaction of moving
> > analog components, which in the case of this camera also have some
> > circular logic to them - the movement of A determines that of B which
> > determines that of C which determines that of D which finally also
> > affects the movement of A.
> >
> > Any thoughts or wise ideas? 
> >
> > Daniele
> 
> If you keep track of the positions as a floating point number, the
> precision will be more than you could actually measure it.

I won't disagree with Richard, although I will give you a general
warning about floating point rounding issues:  if you do, in fact, end
up with your first solution and lots and lots (millions?  billions?
more?) of discrete calculations, be aware that what looks like a lot of
precision in the beginning may not be all that precise (or accurate) in
the end.

Also, doesn't the overall motion of the camera as a whole also depend on
external factors, such as whether/how it's mounted or handheld, the
nature of the "ground" (e.g., soft wet sand vs. hard concrete
vs. someone standing on a boat in the water), an experienced
photographer "squeezing" the shutter release vs. a newbie "pressing the
button"?  I can think of plenty of variables; I guess it depends on what
you're trying to model and how accurate you intend to be (or not to be).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Chris Angelico]

On Fri, Apr 2, 2021 at 9:43 AM <2qdxy4rzwzuui...@potatochowder.com> wrote:
>
> On 2021-04-01 at 18:10:46 -0400,
> Richard Damon  wrote:
>
> > On 4/1/21 5:47 PM, D.M. Procida wrote:
> > > D.M. Procida  wrote:
> > >
> > >> Hi everyone, I've created  -
> > >> a representation of a Canonet G-III QL17 in Python.
> > >>
> > >> There's also documentation: .
> > >>
> > >> It's a pure Python model of the physical sub-systems of a camera and
> > >> their interactions. So far it's at a fairly high level - I haven't yet
> > >> got down to the level of individual springs and levers yet.
> > > I'm now working on the exposure system, which in the published version
> > > so far just models inputs and outputs of the system as a whole. I want
> > > to start to model its functionality by modelling the interactions of its
> > > components - the levers, cams and so on.
> > >
> > > It seems to me I have fundamentally two approaches that I could take:
> > >
> > > The first is to model the movement of each lever as a series of discrete
> > > steps (hundreds of tiny steps, to maintain accuracy). Pro: it models
> > > movement through physical positions; con: it's ugly that it breaks
> > > continuous movement into arbitrary steps.
> > >
> > > The second is not to move each lever in such detail, but to consider its
> > > interactions: given the state of each of the other parts of the system,
> > > what *would* be the outcome if something were to try to move the lever
> > > from such-and-such a position to another? Pro: it feels more elegant
> > > than the brute-force stepping of the alternative; con: it also feels
> > > like a bit of a cheat.
> > >
> > > But neither approach really captures for me the interaction of moving
> > > analog components, which in the case of this camera also have some
> > > circular logic to them - the movement of A determines that of B which
> > > determines that of C which determines that of D which finally also
> > > affects the movement of A.
> > >
> > > Any thoughts or wise ideas?
> > >
> > > Daniele
> >
> > If you keep track of the positions as a floating point number, the
> > precision will be more than you could actually measure it.
>
> I won't disagree with Richard, although I will give you a general
> warning about floating point rounding issues:  if you do, in fact, end
> up with your first solution and lots and lots (millions?  billions?
> more?) of discrete calculations, be aware that what looks like a lot of
> precision in the beginning may not be all that precise (or accurate) in
> the end.

Inaccuracy introduced by FP rounding is likely to be dwarfed by
inaccuracy of measurement. 53-bit precision is pretty accurate
compared to most real-world measurement.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: XanaNews Statistic for comp.lang.python. 4/1/2021 5:52:47 AM

[Abdur-Rahmaan Janhangeer]

So, is that mail spam or ...?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Richard Damon]

On 4/1/21 6:41 PM, 2qdxy4rzwzuui...@potatochowder.com wrote:
> On 2021-04-01 at 18:10:46 -0400,
> Richard Damon  wrote:
>
>> On 4/1/21 5:47 PM, D.M. Procida wrote:
>>> D.M. Procida  wrote:
>>>
 Hi everyone, I've created  -
 a representation of a Canonet G-III QL17 in Python.

 There's also documentation: .

 It's a pure Python model of the physical sub-systems of a camera and
 their interactions. So far it's at a fairly high level - I haven't yet
 got down to the level of individual springs and levers yet.
>>> I'm now working on the exposure system, which in the published version
>>> so far just models inputs and outputs of the system as a whole. I want
>>> to start to model its functionality by modelling the interactions of its
>>> components - the levers, cams and so on.
>>>
>>> It seems to me I have fundamentally two approaches that I could take:
>>>
>>> The first is to model the movement of each lever as a series of discrete
>>> steps (hundreds of tiny steps, to maintain accuracy). Pro: it models
>>> movement through physical positions; con: it's ugly that it breaks
>>> continuous movement into arbitrary steps.
>>>
>>> The second is not to move each lever in such detail, but to consider its
>>> interactions: given the state of each of the other parts of the system,
>>> what *would* be the outcome if something were to try to move the lever
>>> from such-and-such a position to another? Pro: it feels more elegant
>>> than the brute-force stepping of the alternative; con: it also feels
>>> like a bit of a cheat.
>>>
>>> But neither approach really captures for me the interaction of moving
>>> analog components, which in the case of this camera also have some
>>> circular logic to them - the movement of A determines that of B which
>>> determines that of C which determines that of D which finally also
>>> affects the movement of A.
>>>
>>> Any thoughts or wise ideas? 
>>>
>>> Daniele
>> If you keep track of the positions as a floating point number, the
>> precision will be more than you could actually measure it.
> I won't disagree with Richard, although I will give you a general
> warning about floating point rounding issues:  if you do, in fact, end
> up with your first solution and lots and lots (millions?  billions?
> more?) of discrete calculations, be aware that what looks like a lot of
> precision in the beginning may not be all that precise (or accurate) in
> the end.
>
> Also, doesn't the overall motion of the camera as a whole also depend on
> external factors, such as whether/how it's mounted or handheld, the
> nature of the "ground" (e.g., soft wet sand vs. hard concrete
> vs. someone standing on a boat in the water), an experienced
> photographer "squeezing" the shutter release vs. a newbie "pressing the
> button"?  I can think of plenty of variables; I guess it depends on what
> you're trying to model and how accurate you intend to be (or not to be).

Actually, I would contend that due to all the factors that you can't
take into account accurately makes the use of floating point more
applicable. Yes, you need to realize that just because the value has
many digits, you KNOW that is only an approximation, and you process
accordingly, knowing you just has an approximation.

The real question comes, what is the purpose of the simulation? You can
NEVER simulate everything, and some parts of 'simulating' requires
actual hardware to interact with. Sometimes the real thing is the best
simulation available.

-- 
Richard Damon

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A 35mm film camera represented in Python object

[Eli the Bearded]

In comp.lang.python, Richard Damon   wrote:
> On 4/1/21 6:41 PM, 2qdxy4rzwzuui...@potatochowder.com wrote:
>> Richard Damon  wrote:
>>> If you keep track of the positions as a floating point number, the
>>> precision will be more than you could actually measure it.
>> I won't disagree with Richard, although I will give you a general
>> warning about floating point rounding issues:  if you do, in fact, end
>> up with your first solution and lots and lots (millions?  billions?
>> more?) of discrete calculations, be aware that what looks like a lot of
>> precision in the beginning may not be all that precise (or accurate) in
>> the end.

I'm having a hard time figuring where the floating point rounding issues
enter in. My reading is that instead of N discrete steps (level 12 is 
1% moved, lever 12 is 2% moved, lever 12 is 3% moved and makes contact
to cam 3, lever 12 is 4% moved and cam 3 is 5% moved; or what not) using
floating points lever 12 could move 0.0 to > 1, and cam 3 start moving
at lever 12 => 0.04.

>> Also, doesn't the overall motion of the camera as a whole also depend on
>> external factors, such as whether/how it's mounted or handheld, the
>> nature of the "ground" (e.g., soft wet sand vs. hard concrete
>> vs. someone standing on a boat in the water), an experienced
>> photographer "squeezing" the shutter release vs. a newbie "pressing the
>> button"?  I can think of plenty of variables; I guess it depends on what
>> you're trying to model and how accurate you intend to be (or not to be).

I suspect very little of the motion of parts *inside* the camera see
meaningful changes from that. The motion of the camera relative to the
scene is meaningful for how motion blurred a shot will be. But the
springs and levers that move as the shutter release button is pushed
will be basically only moving relative to the camera, and not much
changed by tripod or handheld.

> Actually, I would contend that due to all the factors that you can't
> take into account accurately makes the use of floating point more
> applicable. Yes, you need to realize that just because the value has
> many digits, you KNOW that is only an approximation, and you process
> accordingly, knowing you just has an approximation.

All of the parts of the camera were built with some engineering
tolerance. Using a precision in code that exceeds that tolerance fails
to accurately model the camera.

> The real question comes, what is the purpose of the simulation? You can
> NEVER simulate everything, and some parts of 'simulating' requires
> actual hardware to interact with. Sometimes the real thing is the best
> simulation available.

The purpose was stated in the original post: model a particular camera in
software from the point of view of someone who has done repair work. If
lever 12 is bent, and only touches cam 3 after 15% (or => 0.15) motion
that changes the way the camera works. I believe those sorts of things
are meant to be visible in this model.

Elijah
--
would use floating point, or fixed point as if floating
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Horrible abuse of __init_subclass__, or elegant hack?

[dn via Python-list]

On 02/04/2021 10.13, Chris Angelico wrote:
> On Fri, Apr 2, 2021 at 7:52 AM David L Neil via Python-list
>  wrote:
>> Officially April-Fools Day is over (here), but...
> This wasn't a prank post, although it was intended to give amusement
> rather than real education or debate or anything. So there's nothing
> wrong with keeping it going. :)

The Python 'education' is offered at a lower density than the
explanations of slang in variants of ?the? English-language...


>>> Cartesian.
>>
>> Isn't this water you get out of the ground and then deliver using an
>> horse-pulled vehicle?
> 
> Yes. Also, this is the guy who said "I think not" and promptly
> vanished from the world.
> 
> (For those not familiar, the Cartesian coordinate system or the
> Cartesian plane is what you'd normally use for graphing a mathematical
> function. You can label the axes as "x" and "y" to graph something
> like "y = sin(x)", or you can label them as "real" and "imaginary" and
> use it to depict complex numbers. Though David's response has the
> elegance of true *art*, if you examine the words carefully.)

The Cartesian coordinate system, as used in geometry, mapping, etc, was
named after the French philosopher René Descartes.

Possibly Descartes' most famous quotation is "je pense, donc je suis",
which some reason (best known to others) we were taught in Latin, as
"cogito, ergo sum". Most will be familiar with the English translation,
"I think, therefore I am".

A[n] horse-drawn (pulled) vehicle might be a cART, and cARTESIAN water
comes from an underground supply or aquifer.


Whilst we are 'messing around' with [human] languages, my French
vocabulary/translation capability has been extended by discovering that
"artesian" is "artois". Artois was a kingdom/principality that is part
of modern France, 'up' by the Belgian border. A man with French
heritage, but purported to be Dutch, built a brewery in Belgium. Hence
today one of that country's most famous exports is "Stella Artois" beer
("stella" meaning "star"!).

(but this isn't teaching me much Python - or OpenCV: this weekend's
objective...)


>>> Perhaps, but the key here is the input method. It wouldn't look nearly as 
>>> clean.
>>
>> Not sure about that. For example, is it immediately-obvious which of
>> "Crude", "Residue", and "Plastic" are inputs; and which outputs?
> 
> Well, it's a simple matter of chronology. First you have crude oil,
> then time passes, and then you have plastic and residue. It makes
> sense ONLY if you think of it with a specific ordering, which implies
> Python 3.7 or later.

My anxiety over 'ordering' comes to the fore: if there are multiple
inputs and/or multiple outputs, how can one tell where the former series
ends and the latter begins?

"Explicit" cf "implicit"?


> Hot chocolate? Ahh, now, that's the other thing I drink a lot of. I
> even have a dedicated mug with a Cadbury logo on it. Sadly, not easily
> purchasable, but this one was a gift from a family member who knows
> full well that I have my priorities straight.

Cadbury do not make chocolate!
Neither do Hershey, for that matter!
There, now that I've upset most of the English-speaking world...

PS they use the milk to 'water down' the chocolate! It's much like
buying Easter Eggs: the density of chocolate is much lower for the price.

View
https://www.whittakers.co.nz/en_NZ/products/72-dark-ghana/block-250g
before questioning my curmugeonly credentials!


 Magic, you ask? Well, maybe more "sinister". We did manage to find a
 loose floor-board, but a sad life-lesson was learned, when certain ones
 (un-named*) took it upon themselves to eat all of the contraband
 secreted there.
>>>
>>> Uh oh. How old was the contraband?
>>
>> Was between 'tuck days' (when we were allowed to acquire such goodies -
>> once?twice per week). The theory being that we would all contribute, in
>> anticipation of some later "midnight feast") - such gyre and gimble-ing
>> being more Jabberwock than Alice!
> 
> Ah, I see. So you'd get some tucker, and tuck it away for later, but
> the whole scheme came... unstuck.

Time for another language lesson?

"tucker" is Australian slang for food. Hence "tucker-bag", the
'waltzer's' equivalent of a lunch-box or tiffin tin.

Back when Australia was re-considering their choice for National Anthem,
I was in full expectation (and great hope) that they would select
"Waltzing Matilda". Sadly, they did not (instead: "Advance Australia
Fair").

However, within one poem/song it is possible to learn a considerable
number of (100-year old) Australian slang terms...

Whilst visiting Freemantle (Western Australia), as a guest of Royal
Australian Navy Vets, we were marched-away from the Remembrance Day
Parade to the tune of Waltzing Matilda - something of an highlight (for
me at least)!


Contrarily "tuck" in (old) English slang represented "sweets" (or
"lollies", "candy", etc, in other forms of English). Some say it had
wider application, but I don't recall it ever being