I am thrilled to report that with 2.4.20-ben10 rsynced this evening I can quit X and get the console back, or switch between X and virtual consoles, thanks apparently to the "major rivafb updates from Paul Richards & others".
I'll try to find out on my own why this is, but now the console has problems: it boots up into what looks like 800x600 or possibly even 640x480 (80x24 characters) centered on the screen, and it can't really handle text properly -- certainly not ncurses applications, but even "reset" and "clear" and ^L don't clean it up, so it's unusable. But I'm happy to be able to quit X, now. Thanks, O.
