Hi all, I have encountered a major problem with version 3.1 just released. This problem occurs in the revamped optimizers. One of the OptimizationData implementations we can pass to multivariate vector optimizers is the weight of the various observations.
In all cases I can think of, this weight is a vector, but it is in fact stored as a matrix in which only the diagonal is set. It's main constructor uses a simple double array (which is used in all test cases and is also used under the hood by curve fitters), and some optimizers also assume only the diagonal is set (for example the Gauss-Newton optimizer extract only the diagonal elements and ignores the other elements. However, it is also possible to set a non-diagonal weight matrix, and one class (AbstractLeastSquaresOptimizer) performs an eigen dcomposition on the matrix to extract its square root. I don't know any use case for this, but it has been set up this way, so I guess someone has a use for non-diagonal weights. However, merging all cases (vector and matrix) into one (despite the fact Gauss-Newton ignores non-diagonal elements) leads to huge problems. I discovered this when attempting a simple polynomial curve fitting on a large number of points (41201 points). In this case, a full weight matrix with about 1.7 billions entries was created, and copied... I ended up with a memory error for something that should have been completely trivial (fitting a polynomial, when in fact all weights were equal to 1.0)! So for now, the new curve fitter is really impossible to use with a large number of points. So I would like to be able to cope with vector-only weight easily. As I don't understand if matrix weight are useful or not, I have simply added a new WeightVector class implementing OptimizationData and added it to the various parseOptimizationData, while preserving the former matrix weights class. This does work well and changing the optimizers to directly use a vector was really straightforward as in fact the weights are not really used as a matrix (except in the strange eigen decomposition and associated Jacobian multiplication of AbstractLeastSquaresOptimizer). I wonder if I should simply add this as is or if we should rather remove the non-diagonal weights feature and support only vector weights. I also wonder if we should release a bug fix version early. best regards, Luc --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
