Re: custom background worker task for \copy?

2023-06-03 Thread Joe Carlson
Thanks. Yes, an option. Was considering that as well. Joe > On Jun 2, 2023, at 9:41 PM, Laurenz Albe wrote: > > On Fri, 2023-06-02 at 11:36 -0700, Joe Carlson wrote: >> I have SQL that does it with \copy (select * from where…) … And that >> is fine. >> But it

custom background worker task for \copy?

2023-06-02 Thread Joe Carlson
Hi Folks, I’m writing a little utility for dumping parts of tables into files which will be later slurped into another primary instance. The primary source is our referential data (big, and subject to heavy loads when adding data), the second is a smaller version used in driving our web app. (s

Re: TEXT column > 1Gb

2023-04-12 Thread Joe Carlson
> On Apr 12, 2023, at 12:21 PM, Rob Sargent wrote: > > On 4/12/23 13:02, Ron wrote: >> Must the genome all be in one big file, or can you store them one line per >> table row? The assumption in the schema I’m using is 1 chromosome per record. Chromosomes are typically strings of continuous s

Re: TEXT column > 1Gb

2023-04-12 Thread Joe Carlson
is still important and we need to keep it. > On Apr 12, 2023, at 10:04 AM, Mark Dilger > wrote: > > > >> On Apr 12, 2023, at 7:59 AM, Joe Carlson wrote: >> >> The use case is genomics. Extracting substrings is common. So going to >> chunked st

Re: TEXT column > 1Gb

2023-04-12 Thread Joe Carlson
storage makes sense. And we have a menagerie of code to deal with. Legacy Perl for loading. Clients in Perl, Java, node, python. A whole zoo. Thanks > On Apr 11, 2023, at 10:51 AM, Rob Sargent wrote: > >  >> On 4/11/23 11:41, Joe Carlson wrote: >> Hello, >> >&

TEXT column > 1Gb

2023-04-11 Thread Joe Carlson
Hello, I’ve recently encountered the issue of trying to insert more than 1 Gb into a TEXT column. While the docs say TEXT is unlimited length, I had been unaware of the 1Gb buffer size limitations. We can debate whether or not saving something this big in a single column is a good idea (spoile