Hi, I'm in need of a system for logging the step-wise results and diagnostic metadata about a python function implementation of an algorithm that I'm developing. The specific algorithm is not of great consequence except that it's for scientific computing and may produce large (e.g., '00s or maybe '000s, but not "big data" scale) amounts of intermediate numerical data that can be complex to understand when debugging its progress.
In fact, I'm trying to build a general purpose tool for exploring the inner workings of numerical algorithms for teaching and learning purposes, e.g. for graduate student training or for figuring out parameter choices in difficult applications. I want to be able to insert commands inside of the loops that log certain variable states, completed stages of the algorithm (assume it's hierarchical), text warnings, and possibly even 'pointers' to graphical output objects of some of the intermediate data (e.g. matplotlib object handles for lines, points). Then I can trace the work done afterwards or step through the process in an IDE debugger and make interactive calls to access recent steps, plot certain relationships in the current state, etc. The basic logger's "levels" of output don't really apply here. I at least want categories, if not hierarchical sub-categories. I don't think the built-in logger is sophisticated enough for this, being a flat record of freeform text AFAIU, but the API looks appealing. I'm considering an in-memory sqlite DB to store structured records at any logged step, and an accompanying dictionary to store references to any python object metadata, keyed by a unique ID in the DB log. Then I'd write an API for it that resembles the logger's. It's not super hard for me to write my own thing here, but I'm wondering if anyone has come across any existing solutions in this vein, or has any advice before I go further in designing a solution? I can't really believe that no-one has attempted this before, but it's been really hard to find any existing work through online search. Thanks, Rob -- https://mail.python.org/mailman/listinfo/python-list
