Not having source is a mere inconvenience; you can always decompile the program, read the assembly, translate it back into C, etc. Not being able to distribute the program is only an inconvenience; you can always rewrite it from scratch.
Those words are simply an indirect way of declining to recognize the difference between loss of freedom and practical inconvenience. Where we draw the line, when judging licenses as free or not, is whether you can practically speaking make the code or the manual do the substantive job you want. If license restrictions make it impossible to make the technical changes you want, then the license is non-free. If they make it possible, but with conditions you might not necessarily like, it is free but with a practical inconvenience. Invariant text that isn't part of the technical job that the program or the manual does is just a practical inconvenience. That's why I decided that the unchangable text required by Reiser's license is just a practical inconvenience (though it is getting close to the limit). A number of free software licenses also themselves contain text that is more or less equivalent to an invariant nontechnical section in the source code and executable versions of the program. So what divides "egregious practical inconveniences" and "non-free"? The practical inconveniences of the GFDL are similar to those Debian accepts from other licenses that we agree are free. They are not egregious.