Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-09 Thread richard coleman
Nanina, If you can do it through the psql prompt, then the problem is most likely pgAdmin4, otherwise you might want to get in touch with the postgreSQL devs/support lists. For example, you can run the COPY command both with and without the HEADERS option and see if that works. I hope that helps

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-09 Thread Nanina Tron
Hi, thanks for trying. Yes the backup included the invalid data, but we've just found the problem. The 'special' characters in the data are handled just fine, but not if they are present in the column names. So does pgAdmin handle both data differently or is it Postgres? I thought if the db

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-09 Thread Dave Page
Hi On Wed, Jan 9, 2019 at 3:29 PM Nanina Tron wrote: > > Hi, > thanks for trying. Yes the backup included the invalid data, but we've just > found the problem. > The 'special' characters in the data are handled just fine, but not if they > are present in the column names. > So does pgAdmin hand

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Linus Hicks
I'm not a PostgreSQL expert, but especially since python plays a role here, the system language must be taken into account. If you are wanting no conversion to be done on your "ASCII" data, then you will probably have to go through special hoops. On Tue, Jan 8, 2019, 5:19 PM Doug Easterbrook Had

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Doug Easterbrook
Had a rethink of this. We had sql ascii databases with non ascii data in it. When converting that data to utf8 and sticking into Postgres, all that windows extended characters could not be used unless converted to utf8 before hand. It may be that your case depends 9n the character coding

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread richard coleman
Doug, Hi. Is this a typo? We’ve Postgres reject putting such stuff into the database using anything . PostgreSQL allow *any text* into a column if the database has been set to use SQL_ASCII as the encoding. This was the default encoding on Windows machines, a long long time ago. All that set

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Doug Easterbrook
https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html Smart quotes are not ascii. They are Unicode, strictly speaking . We’ve Postgres reject putting such stuff into the database using anything . Sent from my iPad > On Jan 8, 2019, at 9:23 AM, richard coleman > wrote: > > Dave, > > Thanks f

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Dave Page
On Tue, Jan 8, 2019 at 8:55 PM Nanina Tron wrote: > > > Hi, > > yes thank you I will scan the table so at least I will see the 'bad' > characters. We will see how many there are. > Sorry, i forgot to append the data. Thanks. Did that include the "bad" data? I can load it just fine into a SQL_ASC

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Dave Page
Hi On Tue, Jan 8, 2019 at 7:32 PM richard coleman wrote: > > Dave, > > I would imagine Nanina would be in a better position to provide you with > problematic import/export data in the short term. I don't tend to > import/export that often these days, preferring to use SQL statements for > mos

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Nanina Tron
Hi, thanks for the multiple responses and interesting discussion I am struggeling to keep up with all the new information. I double checked, all settings I could find and the .csv files I want to import. Everything is manually, or at creation of the db set to UTF8. I tried to copy one of th

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread richard coleman
Nanina, I am glad to hear that you are finding the discussion illuminating. You might want to send a copy of your backup file (assuming it isn't sensitive data) to Dave so that he can test pgAdmin4 against it. You might also want to provide your OS and postgreSQL information as well. I've notice

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread richard coleman
Dave, I would imagine Nanina would be in a better position to provide you with problematic import/export data in the short term. I don't tend to import/export that often these days, preferring to use SQL statements for most things short of a full backup/restore (in my case I've found it to be muc

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread Dave Page
Hi Rik On Tue, Jan 8, 2019 at 6:53 PM richard coleman wrote: > > Dave, > > Thanks for continuing this discussion, but I think you misunderstand the > situation. I am storing valid non-UTF8 data in a SQL_ASCII encoded > postgreSQL database (please re-read what I had previously written). This is

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-08 Thread richard coleman
Dave, Thanks for continuing this discussion, but I think you misunderstand the situation. I am storing *valid* *non-UTF8* data in a SQL_ASCII encoded postgreSQL database (please re-read what I had previously written). This is why psql has *NO* problem dealing with it. This is also why Windows OD

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread Dave Page
Hi On Tue, Jan 8, 2019 at 12:47 AM richard coleman wrote: > > Dave, > > Thanks for taking the time to respond, but I don't see anywhere that > SQL_ASCII is recommended against doing. Here's the documentation listing the > supported encoding schemas: > https://www.postgresql.org/docs/current/mu

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread richard coleman
Dave, Thanks for taking the time to respond, but I don't see anywhere that SQL_ASCII is recommended against doing. Here's the documentation listing the supported encoding schemas: https://www.postgresql.org/docs/current/multibyte.html . The *only* caveats listed for SQL_ASCII are: > In most case

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread Dave Page
Hi On Mon, Jan 7, 2019 at 11:30 PM richard coleman wrote: > > Dave, > > I can't speak to Nania's specific issue, but I believe it's a pgAdmin4 > specific problem, at least in so far as SQL_ASCII is concerned. I say this > because I can usually work with the data just fine from the psql prompt,

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread richard coleman
Dave, I can't speak to Nania's specific issue, but I believe it's a *pgAdmin4* specific problem, at least in so far as SQL_ASCII is concerned. I say this because I can usually work with the data just fine from the psql prompt, but *not* through pgAdmin4 (or other postgreSQL GUI's like dBeaver tha

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread Dave Page
On Mon, Jan 7, 2019 at 9:11 PM richard coleman wrote: > > Nania, > > Welcome to the wonderful world of pgAdmin4. I have been bitten often by this > particular shortcoming in pgAdmin4. :( My issue seems to stems from the fact > that I use ASCII tables as a back end for a .Net windows applicatio

Re: Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread richard coleman
Nania, Welcome to the wonderful world of pgAdmin4. I have been bitten often by this particular shortcoming in pgAdmin4. :( My issue seems to stems from the fact that I use ASCII tables as a back end for a .Net windows application and perfectly valid windows (Word/Excel) characters cause pgAdmin4

Import/Export failure due to UTF-8 error in pgAdmin4 but not in pgAdmin3

2019-01-07 Thread Nanina Tron
Hi, I am pretty new to PostgreSQL so I might just miss something basic here. My problem is that, I cannot import or export some of the tables in my db with pgAdmin4, as it raises the “ERROR: unvalid byte-sequenz for coding »UTF8«: 0xdf 0x67“”. The table was originally created with Excel and im