On Wed, Jan 16, 2013 at 8:19 PM, Robert Haas wrote:
> On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane wrote:
>> I find the argument that this supports compression-over-the-wire to be
>> quite weak, because COPY is only one form of bulk data transfer, and
>> one that a lot of applications don't ever use
On Wed, Jan 16, 2013 at 06:19:09PM -0500, Robert Haas wrote:
> On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane wrote:
> > I find the argument that this supports compression-over-the-wire to be
> > quite weak, because COPY is only one form of bulk data transfer, and
> > one that a lot of applications don
* Bruce Momjian (br...@momjian.us) wrote:
> On Wed, Jan 16, 2013 at 06:19:09PM -0500, Robert Haas wrote:
> > But presumably this would transparently compress at one end and
> > decompress at the other end, which is again a somewhat different use
> > case. To get compressed output on the client sid
On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane wrote:
> I find the argument that this supports compression-over-the-wire to be
> quite weak, because COPY is only one form of bulk data transfer, and
> one that a lot of applications don't ever use. If we think we need to
> support transmission compressi
On Wed, Jan 16, 2013 at 5:15 PM, Robert Haas wrote:
> On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne
> wrote:
>> That points towards a fix that involves having a set of non-arbitrary
>> commands
>> that we allow plain users to use.
>>
>> Hmm. There's an interesting thought...
>>
>> How ab
On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne wrote:
> That points towards a fix that involves having a set of non-arbitrary commands
> that we allow plain users to use.
>
> Hmm. There's an interesting thought...
>
> How about having a "pg_filters" table in pg_catalog which allows capturing
On Mon, Jan 14, 2013 at 11:28 AM, Stephen Frost wrote:
> * Tom Lane (t...@sss.pgh.pa.us) wrote:
>> Stephen Frost writes:
>> > I do like the idea of a generalized answer which just runs a
>> > user-provided command on the server but that's always going to require
>> > superuser privileges.
>>
>> T
* Peter Eisentraut (pete...@gmx.net) wrote:
> On 1/15/13 2:53 PM, Stephen Frost wrote:
> > You're right, to clarify, for *file_fdw*, which is a backend-only
> > operation, the popen patch is great (thought I made that clear before).
>
> I would think that if we get writable FDWs, you would want fi
On 1/15/13 2:53 PM, Stephen Frost wrote:
> You're right, to clarify, for *file_fdw*, which is a backend-only
> operation, the popen patch is great (thought I made that clear before).
I would think that if we get writable FDWs, you would want file_fdw to
go through zlib so that it can write directl
On Tue, Jan 15, 2013 at 7:46 PM, Tom Lane wrote:
>> Compressing every small packet seems like it'd be overkill and might
>> surprise people by actually reducing performance in the case of lots of
>> small requests.
>
> Yeah, proper selection and integration of a compression method would be
> criti
On 01/15/2013 06:22 PM, Tom Lane wrote:
Stephen Frost writes:
Perhaps I'm taking a bit too narrow view of the world, but my thinking
is OLTP won't want things compressed, as it increases latency of
requests, while OLAP users are operating with enough data that they'll
go through the effort to
Stephen Frost writes:
> Perhaps I'm taking a bit too narrow view of the world, but my thinking
> is OLTP won't want things compressed, as it increases latency of
> requests, while OLAP users are operating with enough data that they'll
> go through the effort to use COPY.
Also, if there are so man
Stephen Frost writes:
> * Tom Lane (t...@sss.pgh.pa.us) wrote:
>> Really? Given that libpq provides no useful support for doing anything
>> with COPY data, much less higher-level packages such as Perl DBI, I'd
>> venture that the real-world ratio is more like 90/10. If not 99/1.
> Perhaps I'm t
* Tom Lane (t...@sss.pgh.pa.us) wrote:
> Really? Given that libpq provides no useful support for doing anything
> with COPY data, much less higher-level packages such as Perl DBI, I'd
> venture that the real-world ratio is more like 90/10. If not 99/1.
Perhaps I'm taking a bit too narrow view of
Stephen Frost writes:
> * Tom Lane (t...@sss.pgh.pa.us) wrote:
>> I find the argument that this supports compression-over-the-wire to be
>> quite weak, because COPY is only one form of bulk data transfer, and
>> one that a lot of applications don't ever use. If we think we need to
>> support tran
* Bruce Momjian (br...@momjian.us) wrote:
> COPY with a file name is super-user-only. I am unclear how you would
> use STDIN/STDOUT in any meaningful way with binary data produced by
> compression. I guess you could with libpq.
The patch that I posted provided this:
psql -h myhost -c "COPY myta
On Tue, Jan 15, 2013 at 04:22:48PM -0500, Stephen Frost wrote:
> * Bruce Momjian (br...@momjian.us) wrote:
> > Well, COPY is super-user only, so it seems only useful for FDW, no? We
> > already have lots of user-configuration FDW commands, so I can see
> > adding this one too.
>
> COPY is most ce
* Tom Lane (t...@sss.pgh.pa.us) wrote:
> I find the argument that this supports compression-over-the-wire to be
> quite weak, because COPY is only one form of bulk data transfer, and
> one that a lot of applications don't ever use. If we think we need to
> support transmission compression for ours
Stephen Frost writes:
> * Bruce Momjian (br...@momjian.us) wrote:
>> And this leads to support-my-compression-binary-of-the-day mess. Why
>> not just allow them to do '|compression-binary'?
> The popen patch doesn't support the '|compression-binary' option through
> the FE protocol. Even if it
* Bruce Momjian (br...@momjian.us) wrote:
> Well, COPY is super-user only, so it seems only useful for FDW, no? We
> already have lots of user-configuration FDW commands, so I can see
> adding this one too.
COPY is most certainly not superuser-only.. COPY w/ popen, if that
popen can call anythin
* Christopher Browne (cbbro...@gmail.com) wrote:
> How about having a "pg_filters" table in pg_catalog which allows capturing
> labels and names of known-to-be-safe binary filters:
I was considering that (though I was thinking they'd be
"transformations" rather than filters; filter implies that yo
On Tue, Jan 15, 2013 at 03:37:07PM -0500, Christopher Browne wrote:
> On Tue, Jan 15, 2013 at 2:53 PM, Stephen Frost wrote:
> > The popen patch doesn't support the '|compression-binary' option through
> > the FE protocol. Even if it did, it would only be available for
> > superusers as we can't a
On Tue, Jan 15, 2013 at 2:53 PM, Stephen Frost wrote:
> The popen patch doesn't support the '|compression-binary' option through
> the FE protocol. Even if it did, it would only be available for
> superusers as we can't allow regular users to run arbitrary commands on
> the server-side.
That poi
* Bruce Momjian (br...@momjian.us) wrote:
> On Tue, Jan 15, 2013 at 10:55:04AM -0500, Stephen Frost wrote:
> > * Peter Eisentraut (pete...@gmx.net) wrote:
> > > Operating on compressed files transparently in file_fdw is obviously
> > > useful, but why only gzip?
> >
> > This isn't really an argume
On Tue, Jan 15, 2013 at 10:55:04AM -0500, Stephen Frost wrote:
> * Peter Eisentraut (pete...@gmx.net) wrote:
> > Operating on compressed files transparently in file_fdw is obviously
> > useful, but why only gzip?
>
> This isn't really an argument, imv. It's only gzip *right this moment*
> because
* Claudio Freire (klaussfre...@gmail.com) wrote:
> On Tue, Jan 15, 2013 at 1:08 PM, Stephen Frost wrote:
> > Where it does work well is when you move into a bulk-data mode (ala
> > COPY) and can compress relatively large amounts of data into a smaller
> > number of full-size packets to be sent.
>
On Tue, Jan 15, 2013 at 01:35:57PM +, Greg Stark wrote:
> On Tue, Jan 15, 2013 at 2:33 AM, Stephen Frost wrote:
> > * Peter Eisentraut (pete...@gmx.net) wrote:
> >> On 1/14/13 11:28 AM, Stephen Frost wrote:
> >> > While there is no option currently for having the server do the
> >> > compressi
On Tue, Jan 15, 2013 at 1:08 PM, Stephen Frost wrote:
> Where it does work well is when you move into a bulk-data mode (ala
> COPY) and can compress relatively large amounts of data into a smaller
> number of full-size packets to be sent.
Well... exactly. COPY is one case, big result sets is anot
* Claudio Freire (klaussfre...@gmail.com) wrote:
> Postgres' packetized protocol lends itself quite well for this kind of
> thing. It could even be done on a packet-by-packet basis. The only
> drawback I see, is that it pretty much rules out piping through
> arbitrary commands (a protocol needs to
* Peter Eisentraut (pete...@gmx.net) wrote:
> Operating on compressed files transparently in file_fdw is obviously
> useful, but why only gzip?
This isn't really an argument, imv. It's only gzip *right this moment*
because that's all that I implemented. I've already offered to add
bzip2 or whate
On 1/13/13 9:16 PM, Stephen Frost wrote:
> On top of this I plan to submit a trivial patch to add support for
> this to file_fdw, allowing creation of FDW tables which operate
> directly on compressed files (including CSVs, which is what I need
> this patch for).
>
> I've also begun workin
On Mon, Jan 14, 2013 at 11:33 PM, Stephen Frost wrote:
> Now, protocol-level on-the-wire compression
> is another option, but there's quite a few drawbacks to that and quite a
> bit of work involved. Having support for COPY-based compression could
> be an answer for many cases where on-the-wire c
On Tue, Jan 15, 2013 at 2:33 AM, Stephen Frost wrote:
> * Peter Eisentraut (pete...@gmx.net) wrote:
>> On 1/14/13 11:28 AM, Stephen Frost wrote:
>> > While there is no option currently for having the server do the
>> > compression before sending the data over the wire.
>>
>> OpenSSL?
>
> To be hon
* Peter Eisentraut (pete...@gmx.net) wrote:
> On 1/14/13 11:28 AM, Stephen Frost wrote:
> > While there is no option currently for having the server do the
> > compression before sending the data over the wire.
>
> OpenSSL?
To be honest, I expected that to come up earlier in this discussion.
It'd
On 1/14/13 11:28 AM, Stephen Frost wrote:
> While there is no option currently for having the server do the
> compression before sending the data over the wire.
OpenSSL?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgr
* Tom Lane (t...@sss.pgh.pa.us) wrote:
> Stephen Frost writes:
> > I do like the idea of a generalized answer which just runs a
> > user-provided command on the server but that's always going to require
> > superuser privileges.
>
> The design that was being kicked around allowed pipes to be used
* Claudio Freire (klaussfre...@gmail.com) wrote:
> On Mon, Jan 14, 2013 at 1:01 PM, Stephen Frost wrote:
> > I do like the idea of a generalized answer which just runs a
> > user-provided command on the server but that's always going to require
> > superuser privileges.
>
> Unless it's one of a s
Stephen Frost writes:
> I do like the idea of a generalized answer which just runs a
> user-provided command on the server but that's always going to require
> superuser privileges.
The design that was being kicked around allowed pipes to be used on the
client side too, ie \copy foo to '| gzip ..
On Mon, Jan 14, 2013 at 1:01 PM, Stephen Frost wrote:
> I do like the idea of a generalized answer which just runs a
> user-provided command on the server but that's always going to require
> superuser privileges.
Unless it's one of a set of superuser-authorized compression tools.
--
Sent via
* Simon Riggs (si...@2ndquadrant.com) wrote:
> There is a new option being added to pre/post process data, so it
> seems like the best way to add new features - in general.
That structure appears to have no option for passing compressed data to
or from a client connection. Instead, it actually ov
On 14 January 2013 13:43, Stephen Frost wrote:
> Tom,
>
> * Tom Lane (t...@sss.pgh.pa.us) wrote:
>> Stephen Frost writes:
>> > Attached is a patch to add a 'COMPRESSED' option to COPY which will
>> > cause COPY to expect a gzip'd file on input and which will output a
>> > gzip'd file on out
Tom,
* Tom Lane (t...@sss.pgh.pa.us) wrote:
> Stephen Frost writes:
> > Attached is a patch to add a 'COMPRESSED' option to COPY which will
> > cause COPY to expect a gzip'd file on input and which will output a
> > gzip'd file on output. Included is support for backend COPY, psql's
> >
Stephen Frost writes:
> Attached is a patch to add a 'COMPRESSED' option to COPY which will
> cause COPY to expect a gzip'd file on input and which will output a
> gzip'd file on output. Included is support for backend COPY, psql's
> \copy, regression tests for both, and documentation.
I
43 matches
Mail list logo
