psql \copy

[Steve Clark]

Hello,

I am using psql to copy data extracted from an InfluxDB in csv format into 
postgresql.
I have a key field on the time field which I have defined as a bigint since the 
time I get
from InfluxDB is an epoch time.

My question is does psql abort the copy if it hits a duplicate key, or does it 
keep processing?


Thanks,
--
Stephen Clark
NetWolves Managed Services, LLC.
Sr. Applications Architect
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Email Confidentiality Notice: The information contained in this transmission 
may contain privileged and confidential and/or protected health information 
(PHI) and may be subject to protection under the law, including the Health 
Insurance Portability and Accountability Act of 1996, as amended (HIPAA). This 
transmission is intended for the sole use of the individual or entity to whom 
it is addressed. If you are not the intended recipient, you are notified that 
any use, dissemination, distribution, printing or copying of this transmission 
is strictly prohibited and may subject you to criminal or civil penalties. If 
you have received this transmission in error, please contact the sender 
immediately and delete this email and any attachments from any computer. Vaso 
Corporation and its subsidiary companies are not responsible for data leaks 
that result from email messages received that contain privileged and 
confidential and/or protected health information (PHI).

Re: psql \copy

[Steve Clark]

On 04/24/2020 11:59 AM, Steve Crawford wrote:
On Fri, Apr 24, 2020 at 8:55 AM Steve Clark 
mailto:steve.cl...@netwolves.com>> wrote:
Hello,

I am using psql to copy data extracted from an InfluxDB in csv format into 
postgresql.
I have a key field on the time field which I have defined as a bigint since the 
time I get
from InfluxDB is an epoch time.

My question is does psql abort the copy if it hits a duplicate key, or does it 
keep processing?


The copy will fail. You could import into a temporary table and preprocess then 
copy to your permanent table or use an ETL solution to remove unwanted data 
before importing. I don't know the nature of your data or project but perhaps 
that column isn't suitable for a key.

Cheers,
Steve
I am attempting to periodically pull time series data from an InfluxDB.
The column at issue is the timestamp. I have a script that pulls the last 15 
minutes of data from the InfluxDB
as csv data and pipe it into a psql -c "\copy" command. I was looking for 
the simplest way to do this.

--
Stephen Clark
NetWolves Managed Services, LLC.
Sr. Applications Architect
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com<mailto:steve.cl...@netwolves.com>
http://www.netwolves.com

Email Confidentiality Notice: The information contained in this transmission 
may contain privileged and confidential and/or protected health information 
(PHI) and may be subject to protection under the law, including the Health 
Insurance Portability and Accountability Act of 1996, as amended (HIPAA). This 
transmission is intended for the sole use of the individual or entity to whom 
it is addressed. If you are not the intended recipient, you are notified that 
any use, dissemination, distribution, printing or copying of this transmission 
is strictly prohibited and may subject you to criminal or civil penalties. If 
you have received this transmission in error, please contact the sender 
immediately and delete this email and any attachments from any computer. Vaso 
Corporation and its subsidiary companies are not responsible for data leaks 
that result from email messages received that contain privileged and 
confidential and/or protected health information (PHI).

Re: psql \copy

[Steve Clark]

On 04/24/2020 12:15 PM, Adrian Klaver wrote:

On 4/24/20 9:12 AM, Steve Clark wrote:


On 04/24/2020 11:59 AM, Steve Crawford wrote:


On Fri, Apr 24, 2020 at 8:55 AM Steve Clark 
mailto:steve.cl...@netwolves.com>
<mailto:steve.cl...@netwolves.com><mailto:steve.cl...@netwolves.com>> wrote:

   Hello,

   I am using psql to copy data extracted from an InfluxDB in csv
   format into postgresql.
   I have a key field on the time field which I have defined as a
   bigint since the time I get
   from InfluxDB is an epoch time.

   My question is does psql abort the copy if it hits a duplicate
   key, or does it keep processing?


The copy will fail. You could import into a temporary table and
preprocess then copy to your permanent table or use an ETL solution to
remove unwanted data before importing. I don't know the nature of your
data or project but perhaps that column isn't suitable for a key.

Cheers,
Steve


I am attempting to periodically pull time series data from an InfluxDB.
The column at issue is the timestamp. I have a script that pulls the
last 15 minutes of data from the InfluxDB
as csv data and pipe it into a psql -c "\copy" command. I was
looking for the simplest way to do this.



Then as suggested above pull into staging table that has no constraints
e.g. PK. Verify data and then push into permanent table.


Thanks for the tip. I'll head down that road. Stay safe everyone.






--
Stephen Clark
*NetWolves Managed Services, LLC.*
Sr. Applications Architect
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com<mailto:steve.cl...@netwolves.com>
http://www.netwolves.com







--
Stephen Clark
NetWolves Managed Services, LLC.
Sr. Applications Architect
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com<mailto:steve.cl...@netwolves.com>
http://www.netwolves.com

Email Confidentiality Notice: The information contained in this transmission 
may contain privileged and confidential and/or protected health information 
(PHI) and may be subject to protection under the law, including the Health 
Insurance Portability and Accountability Act of 1996, as amended (HIPAA). This 
transmission is intended for the sole use of the individual or entity to whom 
it is addressed. If you are not the intended recipient, you are notified that 
any use, dissemination, distribution, printing or copying of this transmission 
is strictly prohibited and may subject you to criminal or civil penalties. If 
you have received this transmission in error, please contact the sender 
immediately and delete this email and any attachments from any computer. Vaso 
Corporation and its subsidiary companies are not responsible for data leaks 
that result from email messages received that contain privileged and 
confidential and/or protected health information (PHI).

no libpq.pc for Centos 6 or Centos 7

[Steve Clark]

Hello List,

I can't seem to find  a pkgconfig file for postgresql on centos 6 or centos 7.

Centos packages installed
$ rpm -qa|grep postgresql
postgresql-server-9.2.24-1.el7_5.x86_64
postgresql-9.2.24-1.el7_5.x86_64
postgresql-devel-9.2.24-1.el7_5.x86_64
postgresql-libs-9.2.24-1.el7_5.x86_64

Centos 6 packages installed
postgresql-server-8.4.20-8.el6_9.x86_64
postgresql-devel-8.4.20-8.el6_9.x86_64
postgresql-8.4.20-8.el6_9.x86_64
postgresql-libs-8.4.20-8.el6_9.x86_64
postgresql-contrib-8.4.20-8.el6_9.x86_64


Any ideas?

Thanks,
Steve

Re: no libpq.pc for Centos 6 or Centos 7

[Steve Clark]

On 10/24/2018 09:38 AM, Adrian Klaver wrote:
> On 10/24/18 6:28 AM, Steve Clark wrote:
>> Hello List,
>>
>> I can't seem to find  a pkgconfig file for postgresql on centos 6 or centos 
>> 7.
>>
>> Centos packages installed
>> $ rpm -qa|grep postgresql
>> postgresql-server-9.2.24-1.el7_5.x86_64
>> postgresql-9.2.24-1.el7_5.x86_64
>> postgresql-devel-9.2.24-1.el7_5.x86_64
>> postgresql-libs-9.2.24-1.el7_5.x86_64
>>
>> Centos 6 packages installed
>> postgresql-server-8.4.20-8.el6_9.x86_64
>> postgresql-devel-8.4.20-8.el6_9.x86_64
>> postgresql-8.4.20-8.el6_9.x86_64
>> postgresql-libs-8.4.20-8.el6_9.x86_64
>> postgresql-contrib-8.4.20-8.el6_9.x86_64
>>
>>
>> Any ideas?
>  From this:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=977115
>
> it appears that it did not show up until 9.3.
>
> FYI 8.4 and 9.2 are EOL.
>
Yes, but those are the versions that ship with CentOS.
>> Thanks,
>> Steve
>>
>>
>>
>


-- 
Stephen Clark
*NetWolves Managed Services, LLC.*
Sr. Applications Architect 
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

postgres operational

[Steve Clark]

Hi List,

Is there a sure fire way to tell if postgres server is up an operational. I was 
testing to see if the
socket at /tmp/.s.PGSQL.5432 existed - but I ran into a recent problem on 
CentOS 7.5, postgresql 9.2.24, where the
socket was there but my script couldn't read from my database yet.

Thanks,
Steve
--