I am working on moving some of the computation-intensive processes to a cron-launched scheme. I have some that run every minute, every 2 minutes, every 5 minutes and every 15 minutes.
In production, it's possible that the volume of incoming data would result in the task not finishing before its next cron slot fires again. In development , however, I can flood the system quite easily by importing data in larger chunks. At the moment, my laptop (fairly decent machine) is essentially unusable because there are so many python processes consuming RAM. Here's a slightly cleaned-up sample of ps: 6716 ttys000 0:01.70 ../Python web2py.py -a 1234 -i 127.0.0.1 -p 8000 6722 ttys000 0:36.46 ../Python ../web2py.py -J -M -S Debug/DAQ/setup_DAQ -a <recycle> 6724 ttys000 0:00.00 (Python) 7239 ttys000 0:00.00 (Python) 7543 ttys000 0:00.00 (Python) 7641 ttys000 0:00.00 (Python) 7720 ttys000 0:00.00 (Python) 7819 ttys000 0:00.00 (Python) 8781 ttys000 0:00.00 (Python) 8783 ttys000 0:00.00 (Python) 8784 ttys000 0:00.00 (Python) 8787 ttys000 0:00.00 (Python) 8789 ttys000 0:00.00 (Python) 8790 ttys000 0:00.00 (Python) 8791 ttys000 0:00.00 (Python) 8792 ttys000 0:00.00 (Python) 8867 ttys000 0:00.00 (Python) 8868 ttys000 0:00.00 (Python) 8872 ttys000 0:00.00 (Python) 8946 ttys000 0:00.00 (Python) 8947 ttys000 0:00.00 (Python) 8949 ttys000 0:00.00 (Python) 8950 ttys000 0:00.00 (Python) 8951 ttys000 0:00.00 (Python) 8948 ttys000 0:00.46 ../Python ../web2py.py -J -M -S Debug/rounding/array_1minsRollup -a <recycle> 7320 ttys000 1:04.75 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 7424 ttys000 0:56.54 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 7544 ttys000 0:50.19 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 7643 ttys000 0:42.32 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 7722 ttys000 0:36.26 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 7820 ttys000 0:30.36 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 7899 ttys000 0:25.12 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8013 ttys000 0:20.99 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8090 ttys000 0:17.28 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8189 ttys000 0:14.05 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8271 ttys000 0:11.55 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8364 ttys000 0:07.50 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8455 ttys000 0:07.51 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8557 ttys000 0:05.08 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8635 ttys000 0:00.73 ../Python ../web2py.py -J -M -S Debug/rounding/data_tableRollup -a <recycle> 8458 ttys000 0:18.24 ../Python ../web2py.py -J -M -S Debug/rounding/panel_1minsRollup -a <recycle> 8638 ttys000 0:16.68 ../Python ../web2py.py -J -M -S Debug/rounding/panel_1minsRollup -a <recycle> 8786 ttys000 0:00.57 ../Python ../web2py.py -J -M -S Debug/rounding/panel_1minsRollup -a <recycle> 7061 ttys000 0:43.78 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7140 ttys000 0:12.82 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7240 ttys000 0:12.35 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7319 ttys000 0:12.37 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7423 ttys000 0:12.20 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7541 ttys000 0:12.10 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7640 ttys000 0:12.26 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7721 ttys000 0:12.31 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7818 ttys000 0:12.59 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 7898 ttys000 0:12.53 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8012 ttys000 0:12.46 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8092 ttys000 0:12.46 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8188 ttys000 0:12.26 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8270 ttys000 0:11.96 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8363 ttys000 0:12.05 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8454 ttys000 0:12.12 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8556 ttys000 0:12.02 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8632 ttys000 0:11.49 ../Python ../web2py.py -J -M -S Debug/rounding/stringdataRollup -a <recycle> 8785 ttys000 0:00.55 ../Python ../web2py.py -J -M -S Debug/rounding/string_1minsRollup -a <recycle> 8788 ttys000 0:00.57 ../Python ../web2py.py -J -M -S Debug/rounding/string_5minsRollup -a <recycle>
