If you provide the source code with the modified program, but the receiver loses it, he may ask for it again.
Under the GPL, if you distribute the source with the binaries, nobody can insist on getting anything from you subsequently. If you distribute just binaries, you must provide a written offer for source code. However, that written offer only needs to be valid for three years. So you only have to save the source code for three years.