Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Yeah I figured that its python as per the message. I didn't know I had to add it to every source that has the chars but thanks for the tip. I was trying it out to see if the issue was related to the database encoding or not, a UTF-8 conversion on the data completely fixed my issue. This is the

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Michael Radziej
Hi, this has nothing to do with Django at all, it's a python thing. The encoding declaration needs to go on top of EVERY source code file if you the source code contains non-ASCII encoding. This has NOTHING to do with Django encoding or the encoding your database uses! Now, slowly: When you excha

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
# Django settings for mandala project. # -*- coding: utf-8 -*- But django says its not declared. Where should I put this? -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send a

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Michael I already have this on top of my settings.py file all along, but djanog does not pick it up.. #django project # encoding line is 2nd in settings.py -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Michael Radziej
Hi Radomir, > So when I changed the encoding in the sql file from LATIN1 to UTF-8 I get > import errors: > > (1 row) > > ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x67 0x75 > CONTEXT: COPY locations_location, line 31 > setval Well, this is not a UTF8 sequence, but probably iso-8

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Thanks a bunch. Yeah when I did a full convert to UTF-8 I no longer have the issue. I still have the case sensitive login issue which came with the PostgreSQL upgrade. Do you have the same issue? Is this normal? Logging in as Rad no longer works, I have to use exact case as it is in the databa

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Xavier Ordoquy
I'm using full utf8. Whenever I need to import data, I make sure they are utf8 encoded. an extract from a database dump: SET statement_timeout = 0; SET client_encoding = 'UTF8'; SET standard_conforming_strings = off; SET check_function_bodies = false; SET client_min_messages = warning; SET escape

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
This fixed it iconv -f ISO_8859-1 -t UTF-8 mandala.postgres.sql > mandala.postgres.utf8.sql Now it takes the accents in as utf-8. I guess it didnt' like the LATIN1 encoding mixed with UTF-8 -- You received this message because you are subscribed to the Google Groups "Django users" group. To

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Xavier , which encoding do you use for french characters with django? UTF-8 does not like *é*, But it says it exists in utf-8:U+00E9éc3 a9LATIN SMALL LETTER E WITH ACUTE http://www.utf8-chartable.de -- You received this message because you are subscribed to the Google Groups "Django users" gr

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Xavier, what do you mean file is latin1 encoded? You lost me. By the way, one of the Non UTF-8 data that it complains about is the accent over the e in Neguac: * Néguac* New Brunswick See my error above given by psql db < file.sql -- You received this message because you are subscribed to t

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
So when I changed the encoding in the sql file from LATIN1 to UTF-8 I get import errors: (1 row) ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x67 0x75 CONTEXT: COPY locations_location, line 31 setval So what encoding can I use that is safe with french accents but at the same t

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Xavier Ordoquy
Hi Le 22 août 2013 à 09:52, Radomir Wojcik a écrit : > Are french characters UTF-8 Compliant? As I said, yes. Been using Django + Postgres with french texts and faced no issue so far. You also have a decent Django french user base which would have noticed that sort of issues. Back on our topi

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
So is it normal that without adding the u' to a string that contains french accents will give the error: ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) ? Are french characters UTF-8 Compliant? Would this insertion process not fail if the data contained within t

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
MySQL would also somehow make the built in django contrib views login case insensitive, and now its case sensitive. Since it takes the request as a parameter and request.POST is immutable I will have to write my own login without using the batteries built in solution since I cannot do a .lower()

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
This must be my problem all along: *By the way, if you're using python 2, you shouldn't be using 'string' notation for character strings and should be using the u'string' one. 'string' is a binary string while u'string' is a text string. This is misleading with python 2 as there's implicit conv

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
> > Russ like I said before, I used COPY to get the data back in and it was > accepted by postgres with the schema it crated using django (See step 5 > below) 1) Export each table individually from MySQL into csv format. 2) Have Django re-create the models schema from django on its own. 3) E

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
> > From what I read, SQLite is production grade and its the most used > database in the world.. besides the point. So other than Postgres not picking up my encoding (UTF-8 or Latin-1) , both did not work. How can I get the login to be case insensitive again? This will really be a bummer if I

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Russell Keith-Magee
On Thu, Aug 22, 2013 at 2:06 PM, Radomir Wojcik wrote: > Its not a problem with the conversion process. >> > Well, with all due respect -- yes, it is. Allow me to assure you that PostgreSQL handles UTF-8 and unicode just fine. If you're getting errors, it's either a problem with your original sour

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-22 Thread Radomir Wojcik
Thanks Xavier, I had this at the top of my settings.py file, tried changing it to latin-1 and it didn't make a difference. # -*- coding: utf-8 -*- So right now I am using the UTF-8 database as shown above: mandala | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | The data contains speci

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Russell Keith-Magee
On Thu, Aug 22, 2013 at 2:11 PM, Radomir Wojcik wrote: > Also why did from django.contrib.auth.views.login become case sensitive? I > have users that login with User and user, in PostgreSQL this is treated as > 2 different things. Using MySQL it always saw it as one.. all very > frustrating. > Dj

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
Also getting a bunch of these too: Non-ASCII character '\xc3' in file /var/web/mandala_env/mandala/alarms/views.py on line 329, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details (views.py, line 329) Is this normal? PostgreSQL is more strict with character types

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Xavier Ordoquy
Hi, Le 22 août 2013 à 08:25, Radomir Wojcik a écrit : > Adding the unicode u to the 'string' as such u'string' fixes the issue but > now I have to add this u to every string that will potentially have a > character with a french accent in it? How come MySQL didn't need this? There > must be

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
Adding the unicode u to the 'string' as such u'string' fixes the issue but now I have to add this u to every string that will potentially have a character with a french accent in it? How come MySQL didn't need this? There must be a straight forward answer to this, something that will make Post

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
Also why did from django.contrib.auth.views.login become case sensitive? I have users that login with User and user, in PostgreSQL this is treated as 2 different things. Using MySQL it always saw it as one.. all very frustrating. I will try SQLite tomorrow maybe instead, been up 2 days straigh

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
> > Its not a problem with the conversion process. I re-created the schema > using django syncdb and then I went table by table putting the data back in > using csv files I exported from MySQL. I took the long way to get this all > in and PostgreSQL had no problems taking my csv data as input.

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Russell Keith-Magee
On Thu, Aug 22, 2013 at 1:37 PM, Radomir Wojcik wrote: > > And I tried with both UTF8 and LATIN1 database encodings, both produce the > same error: > > List of databases >Name| Owner | Encoding | Collate |Ctype| Access > privileg > es > >

Re: Can't decode french characters after switching to PostgrSQL from MySQL

2013-08-21 Thread Radomir Wojcik
And I tried with both UTF8 and LATIN1 database encodings, both produce the same error: List of databases Name| Owner | Encoding | Collate |Ctype| Access privileg es ---+--+--+-+-+---