Parallel(?) programming with python
tI would like to write a program, that reads from the network a fixed amount of bytes and appends them to a list. This should happen once a second. Another part of the program should take the list, as it has been filled so far, every 6 hours or so, and do some computations on the data (a FFT). Every so often (say once a week) the list should be saved to a file, shorthened in the front by so many items, and filled further with the data coming fom the network. After the first saving of the whole list, only the new part (the data that have come since the last saving) should be appended to the file. A timestamp is in the data, so it's easy to say what is new and what was already there. I'm not sure how to do this properly: can I write a part of a program that keeps doing its job (appending data to the list once every second) while another part computes something on the data of the same list, ignoring the new data being written? Basically the question boils down to wether it is possible to have parts of a program (could be functions) that keep doing their job while other parts do something else on the same data, and what is the best way to do this. -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallel(?) programming with python
Thanks for your reply. On 08.08.22 13:20, Stefan Ram wrote: Yes, but this is difficult. If you ask this question here, you might not be ready for this. Indeed. I haven't learned it yet myself, but nevertheless tried to write a small example program quickly, which might still contain errors because of my lack of education. import threading import time def write_to_list( list, lock, event ): for i in range( 10 ): lock.acquire() try: list.append( i ) finally: lock.release() event.set() time.sleep( 3 ) def read_from_list( list, lock, event ): while True: event.wait() print( "Waking up." ) event.clear() if len( list ): print( "List contains " + str( list[ 0 ]) + "." ) lock.acquire() try: del list[ 0 ] finally: lock.release() else: print( "List is empty." ) list = [] lock = threading.Lock() event = threading.Event() threading.Thread( target=write_to_list, args=[ list, lock, event ]).start() threading.Thread( target=read_from_list, args=[ list, lock, event ]).start() If I understand some things correctly, a "lock" would be something that, as the name says, locks, meaning prevents parts of the program from executing on the locked resource until ohter parts have finished doing their things and have released the lock. If this is correct, it's not exactly what I wanted, because this way "parts of the program" would not "keep doing their things, while other parts do other things on the same data". I'm in principle ok with locks, if it must be. What I fear is that the lock could last long and prevent the function that writes into the list from doing so every second. With an FFT on a list that contains a few bytes taken every second over one week time (604.800 samples), I believe it's very likely that the FFT function takes longer than a second to return. Then I would have to import all the data I have missed since the lock was aquired, which is doable, but I would like to avoid it if possible. In basketball, first you must learn to dribble and pass, before you can begin to shoot. Sure. With certain reservations, texts that can be considered to learn Python are: "Object-Oriented Programming in Python Documentation" - a PDF file, Introduction to Programming Using Python - Y Daniel Liang (2013), How to Think Like a Computer Scientist - Peter Wentworth (2012-08-12), The Coder's Apprentice - Pieter Spronck (2016-09-21), and Python Programming - John Zelle (2009). Thank you for the list. I an currently taking a Udemy course and at the same time reading the tutorials on python.org. I hope I will some day come to any of the books you suggest (I'm doing this only in my spare time and it will take forever). -- https://mail.python.org/mailman/listinfo/python-list
Re: Parallel(?) programming with python
Thank you for your reply.
On 08.08.22 14:55, Julio Di Egidio wrote:
Concurrent programming is quite difficult, plus you better think
in terms of queues than shared data...
Do you mean queues in the sense of deque (the data structure)? I ask
because I can see the advantage there when I try to pop data from the
front of it, but I don't see the sense of the following statement ("than
shared data"). I mean, I called my structure a list, but it may well be
a queue instead. That wouldn't prevent it from being shared in the idea
I described: one function would still append data to it while the other
is reading what is there up to a certain point and calculate the FFT of it.
But, an easier and often
better option for concurrent data access is use a (relational)
database, then the appropriate transaction isolation levels
when reading and/or writing.
That would obviusly save some coding (but would introduce the need to
code the interaction with the database), but I'm not sure it would speed
up the thing. Would the RDBMS allow to read a table while something else
is writing to it? I doubt it and I'm not sure it doesn't flush the cache
before letting you read, which would include a normally slow disk access.
Andreas
Julio
--
https://mail.python.org/mailman/listinfo/python-list
Re: Parallel(?) programming with python
I would like to thank everybody who answered my question. The insight was very informative. This seems to be one of the few newsgroups still alive and kicking, with a lot of knowledgeable people taking the time to help others. I like how quick and easy it is to post questions and receive answers here as compared to web-based forums (although there are some disadvantages too). I'm implementing some of the ideas received here and I will surely have other questions as I go. But the project will take a long time because I'm doing this as a hobby during my vacation, that are unfortunately about to end. Thanks again, Community. On 08.08.22 12:47, Andreas Croci wrote: tI would like to write a program, that reads from the network a fixed amount of bytes and appends them to a list. This should happen once a second. Another part of the program should take the list, as it has been filled so far, every 6 hours or so, and do some computations on the data (a FFT). Every so often (say once a week) the list should be saved to a file, shorthened in the front by so many items, and filled further with the data coming fom the network. After the first saving of the whole list, only the new part (the data that have come since the last saving) should be appended to the file. A timestamp is in the data, so it's easy to say what is new and what was already there. I'm not sure how to do this properly: can I write a part of a program that keeps doing its job (appending data to the list once every second) while another part computes something on the data of the same list, ignoring the new data being written? Basically the question boils down to wether it is possible to have parts of a program (could be functions) that keep doing their job while other parts do something else on the same data, and what is the best way to do this. -- https://mail.python.org/mailman/listinfo/python-list
