On Tue, Feb 25, 2014 at 8:39 AM, luc <l...@spaceroots.org> wrote: > Also, I think testing should be done on an actual large problem where >>> scaling issuing would show up. The 1000x2 jaccobian would results in a >>> 2x2 >>> normal equation. Surely this is not a good test case. >>> >>> Konstantin >>> >>> >> As you point out, the test case in question shows how copying dominates >> computation for massively over-determined systems. >> > > You are right. Massively over-determined systems is also an important > class of problems, > so it needs to be addressed. I am aware there are many other important > classes of > problems, though, so there are probably no silver bullets and what is > important here > is not important elsewhere. There are also cases for which forming the > normal equations > is avoided (mainly for the sake of numerical robustness if I remember > correctly), so > once again no silver bullets.
Underdetermined systems, for instance, have pretty much the opposite problem in that the normal equations are very large. These systems are often solved using least squares with an L_2 regularizer.
