|
This is the first time I post to the list. I’ve done a
brief search and didn’t find my issue treated already, so here it goes. Apologies
if this has been reported before. I have a pretty big file, around 2 million rows, in
tab-separated format, with 4 columns, that I read into a table in Postgres
using the copy command. I’ve started to notice missing info sometimes. I’ll
truncate the table, read from the file, and notice that sometimes there are
less rows in the table than in the file. This is not well reproducible. If I truncate again, and
reread, I may get all the lines, or I may get a different amount of missing
lines. I concluded that there was a bug in the copy command, and
wrote a replacement in Ruby, using the pure-ruby Postgres-pr library. I run into the same issue. Some lines seem to be dropped,
but no exceptions nor SQL errors are reported by the program. In order to improve throughput, in my ruby program I connect
to the server just once, and send the INSERT statements to the server in batches
of 2000. I’ve checked that the file doesn’t contain any
SQL escape sequences or anything else that would invalidate an INSERT. The version running in the server is 8.1.3 on Linux 2.6.5 on
an Intel platform. The imports are being run from windows machines in the same
network. Has somebody seen this before? Thanks Jaime |
*********************************************************************** Bear Stearns is not responsible for any recommendation, solicitation, offer or agreement or any information about any transaction, customer account or account activity contained in this communication.
Bear Stearns does not provide tax, legal or accounting advice. You should consult your own tax, legal and accounting advisors before engaging in any transaction. In order for Bear Stearns to comply with Internal Revenue Service Circular 230 (if applicable), you are notified that any discussion of U.S. federal tax issues contained or referred to herein is not intended or written to be used, and cannot be used, for the purpose of: (A) avoiding penalties that may be imposed under the Internal Revenue Code; nor (B) promoting, marketing or recommending to another party any transaction or matter addressed herein. ***********************************************************************
---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
