I see nothing in the Gerontocracy rule that prevents me from successfully forming a party or otherwise carrying on ordinary Agoran gameplay, although I see how a group of five Elders could choose to step in and make it not happen.
Right.
Care to explain what I'm overlooking?
You have to give them 4 days notice (from the time intent was announced) so that those hypothetical 5 Elders have a chance to object. See R1728 a).