Re: [web2py] Problem with Does web2py modify python strings?

[Vinicius Assef]

Anthony, I'm using Ubuntu server 12.04 inside a Vagrant machine with
provided by Virtualbox.

Michele, my $LANG is exactly en_US.UTF-8, like yours.

Would it be a problem with my OS? As I explained in first message, it
works with standard Python shell, without web2py environment loaded.

The problem occurs only when I use web2py from shell.



On Fri, Sep 27, 2013 at 4:08 PM, Michele Comitini
<michele.comit...@gmail.com> wrote:
> On your terminal you have to look at locales setup. Issue the following:
>
> $ locales
>
> You should have utf-8 based locale. For instance:
>
> LANG=en_US.UTF-8
>
> On the browser look on the prefereces, see if you have locale/language
> settings supporting UTF-8.
> If you have UTF-8 both on terminal and browser, then there is a problem...
>
>
> 2013/9/27 Vinicius Assef <vinicius...@gmail.com>
>>
>> I was making a maintenance in table data through shell.
>>
>> Despite that, I'm still curious to discover why this behaviour is
>> different from natural Python behaviour.
>>
>>
>> On Fri, Sep 27, 2013 at 3:53 PM, Richard Vézina
>> <ml.richard.vez...@gmail.com> wrote:
>> > Ok, but why do you need to do that in web2py? I remember have read
>> > something
>> > about encoding in the book that was saying don't need to concern about
>> > that
>> > in web2py.
>> >
>> > Richard
>> >
>> >
>> > On Fri, Sep 27, 2013 at 2:46 PM, Vinicius Assef <vinicius...@gmail.com>
>> > wrote:
>> >>
>> >> On Fri, Sep 27, 2013 at 3:37 PM, Richard Vézina
>> >> <ml.richard.vez...@gmail.com> wrote:
>> >> > Ha ok, I was miss leading by the linux shell... Could it be that you
>> >> > need
>> >> > u('éà') ??
>> >>
>> >> No. the "áéíóú".decode("utf-8") should do that.
>> >> And really does, in standard Python shell, outside web2py.
>> >>
>> >>
>> >> >
>> >> >
>> >> > On Fri, Sep 27, 2013 at 1:47 PM, Vinicius Assef
>> >> > <vinicius...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> It wasn't a bash script, Richard.
>> >> >>
>> >> >> I made exactly as I described: straight from terminal.
>> >> >>
>> >> >> On Fri, Sep 27, 2013 at 2:23 PM, Richard Vézina
>> >> >> <ml.richard.vez...@gmail.com> wrote:
>> >> >> > Could it be the encoding of the file that content your bash
>> >> >> > script?
>> >> >> >
>> >> >> > Richard
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Sep 27, 2013 at 12:23 PM, Vinicius Assef
>> >> >> > <vinicius...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hey all.
>> >> >> >>
>> >> >> >> I'm facing a strange problem concerning special characters.
>> >> >> >>
>> >> >> >> Using web2py shell, through command line (linux shell), I do
>> >> >> >> this:
>> >> >> >>
>> >> >> >> # ----------------  start
>> >> >> >> $ python web2py.py -S welcome
>> >> >> >> web2py Web Framework
>> >> >> >> Created by Massimo Di Pierro, Copyright 2007-2013
>> >> >> >> Version 2.6.4-stable+timestamp.2013.09.22.01.43.37
>> >> >> >> Database drivers available: SQLite(sqlite3), MySQL(pymysql),
>> >> >> >> PostgreSQL(pg8000), IMAP(imaplib)
>> >> >> >> WARNING:web2py:import IPython error; use default python shell
>> >> >> >> Python 2.7.3 (default, Apr 10 2013, 06:20:15)
>> >> >> >> [GCC 4.6.3] on linux2
>> >> >> >> Type "help", "copyright", "credits" or "license" for more
>> >> >> >> information.
>> >> >> >> (InteractiveConsole)
>> >> >> >> >>> print "áéíóú"   # note the strange chars!
>> >> >> >> áéíóú
>> >> >> >> >>> from unicodedata import normalize
>> >> >> >> >>> normalize("NFKD", "áéuíóú".decode("utf-8")).encode("ASCII",
>> >> >> >> >>> "ignore")
>> >> >> >> >>> # again!
>> >> >> >> 'AAuAA3Ao'
>> >> >> >> # ----------------  end
>> >> >> >>
>> >> >> >> Note the noisy characters on the last line.
>> >> >> >>
>> >> >> >>
>> >> >> >> Then, at the same machine, I enter into the naked Python shell
>> >> >> >> and:
>> >> >> >> # ----------------  start
>> >> >> >> $ python
>> >> >> >> Python 2.7.3 (default, Apr 10 2013, 06:20:15)
>> >> >> >> [GCC 4.6.3] on linux2
>> >> >> >> Type "help", "copyright", "credits" or "license" for more
>> >> >> >> information.
>> >> >> >> >>> print "áéíóú" # looks ok
>> >> >> >> áéíóú
>> >> >> >> >>> from unicodedata import normalize
>> >> >> >> >>> normalize("NFKD", "áéuíóú".decode("utf-8")).encode("ASCII",
>> >> >> >> >>> "ignore")
>> >> >> >> >>> # ok again.
>> >> >> >> 'aeiou'
>> >> >> >> # ----------------  end
>> >> >> >>
>> >> >> >> Everything works fine.
>> >> >> >>
>> >> >> >> The same code.
>> >> >> >>
>> >> >> >> I found this problem because I'm working to strip special
>> >> >> >> characters
>> >> >> >> off of names, with unicodedata.normalize().
>> >> >> >>
>> >> >> >> If it wan't strange enough, the unicodedata.normalize() works
>> >> >> >> fine
>> >> >> >> from web environment, throug app admin. There the normalization
>> >> >> >> works
>> >> >> >> as expected (the same way as the standard naked Python shell).
>> >> >> >>
>> >> >> >> What could be the root of this situation?
>> >> >> >>
>> >> >> >> --
>> >> >> >> Resources:
>> >> >> >> - http://web2py.com
>> >> >> >> - http://web2py.com/book (Documentation)
>> >> >> >> - http://github.com/web2py/web2py (Source code)
>> >> >> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> >> >> >> ---
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "web2py-users" group.
>> >> >> >> To unsubscribe from this group and stop receiving emails from it,
>> >> >> >> send
>> >> >> >> an
>> >> >> >> email to web2py+unsubscr...@googlegroups.com.
>> >> >> >> For more options, visit https://groups.google.com/groups/opt_out.
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Resources:
>> >> >> > - http://web2py.com
>> >> >> > - http://web2py.com/book (Documentation)
>> >> >> > - http://github.com/web2py/web2py (Source code)
>> >> >> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>> >> >> > ---
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "web2py-users" group.
>> >> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> >> > send
>> >> >> > an
>> >> >> > email to web2py+unsubscr...@googlegroups.com.
>> >> >> > For more options, visit https://groups.google.com/groups/opt_out.
>> >> >>
>> >> >> --
>> >> >> Resources:
>> >> >> - http://web2py.com
>> >> >> - http://web2py.com/book (Documentation)
>> >> >> - http://github.com/web2py/web2py (Source code)
>> >> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> >> >> ---
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "web2py-users" group.
>> >> >> To unsubscribe from this group and stop receiving emails from it,
>> >> >> send
>> >> >> an
>> >> >> email to web2py+unsubscr...@googlegroups.com.
>> >> >> For more options, visit https://groups.google.com/groups/opt_out.
>> >> >
>> >> >
>> >> > --
>> >> > Resources:
>> >> > - http://web2py.com
>> >> > - http://web2py.com/book (Documentation)
>> >> > - http://github.com/web2py/web2py (Source code)
>> >> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>> >> > ---
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "web2py-users" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to web2py+unsubscr...@googlegroups.com.
>> >> > For more options, visit https://groups.google.com/groups/opt_out.
>> >>
>> >> --
>> >> Resources:
>> >> - http://web2py.com
>> >> - http://web2py.com/book (Documentation)
>> >> - http://github.com/web2py/web2py (Source code)
>> >> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "web2py-users" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to web2py+unsubscr...@googlegroups.com.
>> >> For more options, visit https://groups.google.com/groups/opt_out.
>> >
>> >
>> > --
>> > Resources:
>> > - http://web2py.com
>> > - http://web2py.com/book (Documentation)
>> > - http://github.com/web2py/web2py (Source code)
>> > - https://code.google.com/p/web2py/issues/list (Report Issues)
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "web2py-users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to web2py+unsubscr...@googlegroups.com.
>> > For more options, visit https://groups.google.com/groups/opt_out.
>>
>> --
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to web2py+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.