Re: pg12 - migrate tables to partitions structure

2019-09-18 Thread Michael Lewis
Is this being done because it can be, or is it solving a real-life pain point? Just wondering what the perspective is here. Much of partitioning strategy seems to me to revolve around how the system is used, and not just the schema and what is possible. For instance, you can mimic primary and fore

Re: Question regarding fast-hashing in PGSQL

2019-09-18 Thread Tom Lane
Stephen Conley writes: > My idea was to hash the string to a bigint, because the likelihood of all 3 > columns colliding is almost 0, and if a duplicate does crop up, it isn't > the end of the world. > However, Postgresql doesn't seem to have any 'native' hashing calls that > result in a bigint.

Re: pg12 - migrate tables to partitions structure

2019-09-18 Thread Mariel Cherkassky
Well, if u have 10M rows, and all your queries use the same column in the query and the data can split pretty even between the partitions, any specific reason not to use is ? An index will help u reach a complexity of (logn) while partition + index can be in complexity of (logm) when m = rows in pa

Re: pg12 - migrate tables to partitions structure

2019-09-18 Thread Michael Lewis
> > All the queries uses the vendor product and thats why this column is a > perfect fit as a partition column. > My main table is big (10M+) (Product), but other tables can also be > big(1M+).. > I assume you have query performance problems and are hoping partitioning will help? Are you read heav

Re: pg12 - migrate tables to partitions structure

2019-09-18 Thread Mariel Cherkassky
Hey Michael, first of all thanks for the quick response. Right now the production env is on a different version(10). I'm doing all my tests on a test environment. I'm familiar with the hash partitions but my queries doesnt involve the product.id therefore iti isnt relevant. All the queries uses the

Re: pg12 - migrate tables to partitions structure

2019-09-18 Thread Michael Lewis
How many rows are you dealing with currently? What are your queries like? Have you looked at doing a hash partition on product.id? Is this on a test system or destined for a production environment in the near future? I ask because PG12 is still in beta.

pg12 - migrate tables to partitions structure

2019-09-18 Thread Mariel Cherkassky
Hey, Thanks to the new partitions features in pg12 (referencing partition table is possible) I was trying to migrate some of my tables into a partitions structure. Lets assume I have the following non partitions structure : Product(id int PK,vendor int references Vendor(id),price int) ProductPic(

Re: Question regarding fast-hashing in PGSQL

2019-09-18 Thread Stephen Conley
This should work perfectly for me. Thank you so much! On Wed, Sep 18, 2019 at 12:50 PM Adam Brusselback wrote: > I've had a similar issue in the past. > > I used the md5 hash function and stored it in a UUID column for my > comparisons. Bigger than a bigint, but still much faster than string >

Re: Question regarding fast-hashing in PGSQL

2019-09-18 Thread Adam Brusselback
I've had a similar issue in the past. I used the md5 hash function and stored it in a UUID column for my comparisons. Bigger than a bigint, but still much faster than string comparisons directly for my use case. UUID works fine for storing md5 hashes and gives you the ability to piggyback on all t

Question regarding fast-hashing in PGSQL

2019-09-18 Thread Stephen Conley
Hey there; I have a weird use case where I am basically taking data from many different sources and merging it into a single table, while trying to avoid duplicates as much as possible. None of them share any kind of primary key, but I have determined 3 columns that, together, will almost always