On Fri, Sep 4, 2015 at 6:52 PM, Ozgun Erdogan <oz...@citusdata.com> wrote: > Distributed shuffles (Map/Reduce) are hard. When we looked at using FDWs for > pg_shard, we thought that Map/Reduce would require a comprehensive revamp of > the APIs.
Well, so you've said. But what kind of API do you want to see? Taking control at some very high-level hook like ExecutorRun() is not really a maintainable solution - it's fine if you've only got one guy doing it, perhaps, but if you have several FDWs talking to different kinds of remote systems, they can't all seize overall control. >> For Citus, a second part of the question is as FDW writers. We implemented > cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't benefit from > even the simple join pushdown that doesn't require Map/Reduce. > > The PostgreSQL wiki lists 85 foreign data wrappers, and only 18 of these > have support for joins: > https://wiki.postgresql.org/wiki/Foreign_data_wrappers What do you mean by "support for joins"? Do you mean that only 18 of the remote data sources can do joins? If so, why does that matter? I'd be quite happy if a join pushdown or "distributed shuffle" API had as many as 18 users - I'd be quite happy if it had one (postgres_fdw). The fact that not all FDWs can support every operation because of limitations on the remote side isn't a reason not to support those operations when the remote side is capable. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
