Re: Putting Unicode characters in JSON

[Chris Angelico]

On Fri, Mar 23, 2018 at 4:35 PM, Steven D'Aprano
 wrote:
> On Fri, 23 Mar 2018 12:05:34 +1100, Chris Angelico wrote:
>
>> Latin-1 is not "arbitrary bytes". It is a very specific encoding that
>> cannot decode every possible byte value.
>
> Yes it can.
>
> py> blob = bytes(range(256))
> py> len(blob)
> 256
> py> blob[45:55]
> b'-./0123456'
> py> s = blob.decode('latin1')
> py> len(s)
> 256
> py> s[45:55]
> '-./0123456'
>

That doesn't seem to be a strictly-correct Latin-1 decoder, then.
There are a number of unassigned byte values in ISO-8859-1.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Python 3.6: How to expand f-string literals read from a file vs inline statement

[Malcolm Greene]

Looking for advice on how to expand f-string literal strings whose
values I'm reading from a configuration file vs hard coding into
my script as statements. I'm using f-strings as a very simple
template language.
I'm currently using the following technique to expand these f-strings.
Is there a better way?
Bonus if anyone has suggestions on how my expand() function can access
the locals() value of the caller so this parameter doesn't have to be
passed in explicitly.
*def *expand(expression, values):

   """Expand expression using passed in locals()"""

   triple_quote = *"'" ** 3
   expression = dedent(expression)

   *return *eval(*f'f{triple_quote}{expression}{triple_quote}'*,
   *None*, values)

product_name = 'Bike ABC'

product_sku = '123456'

product_price = 299.95

discount = 0.85



# read in product description template 

# product_description_template might look like: {product_sku} :
# {product_name}: ${product_price * discount}product_description_template = 
config('product_description_template')



# expand the {expressions} in product_description_template using my
# locals()product_description = expand(product_description_template, locals())


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.6: How to expand f-string literals read from a file vs inline statement

[Malcolm Greene]

My original post reformatted for text mode:

Looking for advice on how to expand f-string literal strings whose values I'm 
reading from a configuration file vs hard coding into 
my script as statements. I'm using f-strings as a very simple template language.

I'm currently using the following technique to expand these f-strings. Is there 
a better way?

Bonus if anyone has suggestions on how my expand() function can access the 
locals() value of the caller so this parameter doesn't have to be passed in 
explicitly.

def expand(expression, values):
   """Expand expression using passed in locals()"""
   triple_quote = "'" * 3
   expression = dedent(expression)
   return eval(f'f{triple_quote}{expression}{triple_quote}', None, values)

product_name = 'Bike ABC'
product_sku = '123456'
product_price = 299.95
discount = 0.85

# read in product description template 
# product_description_template might look like: {product_sku} : {product_name}: 
${product_price * discount}
product_description_template = config('product_description_template')

# expand the {expressions} in product_description_template using my locals()
product_description = expand(product_description_template, locals())
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.6: How to expand f-string literals read from a file vs inline statement

[Chris Angelico]

On Fri, Mar 23, 2018 at 7:37 PM, Malcolm Greene  wrote:
> My original post reformatted for text mode:
>
> Looking for advice on how to expand f-string literal strings whose values I'm 
> reading from a configuration file vs hard coding into
> my script as statements. I'm using f-strings as a very simple template 
> language.
>
> I'm currently using the following technique to expand these f-strings. Is 
> there a better way?

They're not a simple template language; they're a special form of
expression. If you want something you can read from a config file,
look instead at str.format().

# Just an ordinary string at this point, and can be read from a file or anything
product_description_template = "{product_sku} : {product_name}:
${discounted_price}"

# Need to pre-calculate any compound values
discounted_price = product_price * discount
print(product_description_template.format(**locals()))


You can't execute arbitrary code this way, but that's generally a
_good_ thing. If you really want a templating language that can
execute arbitrary code, the best way is to actually make your
templating language BE code; for instance, you can make your config
file be a Python script that creates a bunch of functions, or use
exec/eval directly (though I still wouldn't recommend the latter if
you call this in any sort of tight loop; eval and exec are not cheap,
so it's a lot more efficient to compile once).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Paul Moore]

On 23 March 2018 at 00:27, Thomas Jollans  wrote:
> On 22/03/18 20:46, Tobiah wrote:
>> I was reading though, that JSON files must be encoded with UTF-8.  So
>> should I be doing string.decode('latin-1').encode('utf-8')?  Or does
>> the json module do that for me when I give it a unicode object?
>
> Definitely not. In fact, that won't even work.
>
 import json
 s = 'déjà vu'.encode('latin1')
 s
> b'd\xe9j\xe0 vu'
 json.dumps(s.decode('latin1').encode('utf8'))
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
> return _default_encoder.encode(obj)
>   File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
> chunks = self.iterencode(o, _one_shot=True)
>   File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
> return _iterencode(o, 0)
>   File "/usr/lib/python3.6/json/encoder.py", line 180, in default
> o.__class__.__name__)
> TypeError: Object of type 'bytes' is not JSON serializable

>
> You should make sure that either the file you're writing to is opened as
> UTF-8 text, or the ensure_ascii parameter of dumps() or dump() is set to
> True (the default) – and then write the data in ASCII or any
> ASCII-compatible encoding (e.g. UTF-8).
>
> Basically, the default behaviour of the json module means you don't
> really have to worry about encodings at all once your original data is
> in unicode strings.

From my analysis of the OP's comments, I suspect he's using Python 2,
which muddles the distinction between bytes and (Unicode) text, and
that's why he is seeing such strange results.

Getting this right in Python 2 is going to involve having a clear
understanding of how text and bytes differ, and carefully tracking
which values are conceptually text and which are conceptually bytes.
In my view one of the easiest ways of doing this is to try writing the
code you want in Python 3, and watch how it breaks (as you've
demonstrated above, it will!) Then, if you need your code to work in
Python 2, apply the knowledge you've gained to the Python 2 codebase.
Unfortunately, that may not be practical (people can be locked on
Python 2 for all sorts of reasons). If that's the case, then I can't
offer much help to the OP beyond "learn how Unicode works" - which
isn't much help, as that's basically what he asked in the first
place...

Paul
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Steven D'Aprano]

On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote:

> That doesn't seem to be a strictly-correct Latin-1 decoder, then. There
> are a number of unassigned byte values in ISO-8859-1.

That's incorrect, but I don't blame you for getting it wrong. Who thought 
that it was a good idea to distinguish between "ISO 8859-1" and 
"ISO-8859-1" as two related but distinct encodings?

https://en.wikipedia.org/wiki/ISO/IEC_8859-1

The old ISO 8859-1 standard, the one with undefined values, is mostly of 
historical interest. For the last twenty years or so, anyone talking 
about either Latin-1 or ISO-8859-1 (with or without dashes) is almost 
meaning the 1992 IANA superset version which defines all 256 characters:

"In 1992, the IANA registered the character map ISO_8859-1:1987,
more commonly known by its preferred MIME name of ISO-8859-1
(note the extra hyphen over ISO 8859-1), a superset of ISO 
8859-1, for use on the Internet. This map assigns the C0 and C1
control characters to the unassigned code values thus provides
for 256 characters via every possible 8-bit value."


Either that, or they actually mean Windows-1252, but let's not go there.



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Chris Angelico]

On Fri, Mar 23, 2018 at 9:29 PM, Steven D'Aprano
 wrote:
> On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote:
>
>> That doesn't seem to be a strictly-correct Latin-1 decoder, then. There
>> are a number of unassigned byte values in ISO-8859-1.
>
> That's incorrect, but I don't blame you for getting it wrong. Who thought
> that it was a good idea to distinguish between "ISO 8859-1" and
> "ISO-8859-1" as two related but distinct encodings?
>
> https://en.wikipedia.org/wiki/ISO/IEC_8859-1
>
> The old ISO 8859-1 standard, the one with undefined values, is mostly of
> historical interest. For the last twenty years or so, anyone talking
> about either Latin-1 or ISO-8859-1 (with or without dashes) is almost
> meaning the 1992 IANA superset version which defines all 256 characters:
>
> "In 1992, the IANA registered the character map ISO_8859-1:1987,
> more commonly known by its preferred MIME name of ISO-8859-1
> (note the extra hyphen over ISO 8859-1), a superset of ISO
> 8859-1, for use on the Internet. This map assigns the C0 and C1
> control characters to the unassigned code values thus provides
> for 256 characters via every possible 8-bit value."
>
>
> Either that, or they actually mean Windows-1252, but let's not go there.
>

Wait, whaaa...

Though in my own defense, MySQL itself seems to have a bit of a
problem with encoding names. Its "utf8" is actually "UTF-8 with a
maximum of three bytes per character", in contrast to "utf8mb4" which
is, well, UTF-8.

In any case, abusing "Latin-1" to store binary data is still wrong.
That's what BLOB is for.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Entering a very large number

[ast]

Hi

I found this way to put a large number in
a variable.

C = int(
"28871482380507712126714295971303939919776094592797"
"22700926516024197432303799152733116328983144639225"
"94197780311092934965557841894944174093380561511397"
"4215424169339729054237110027510420801349667317"
"5515285922696291677532547505856101949404200039"
"90443211677661994962953925045269871932907037356403"
"22737012784538991261203092448414947289768854060249"
"76768122077071687938121709811322297802059565867")

It works but is it not optimal since there is a
string to int conversion.

I was not able to put an integer directly because
character '\' for line cut doesnt work inside an
integer

C = \
28871482380507712126714295971303939919776094592797\
22700926516024197432303799152733116328983144639225\
...
76768122077071687938121709811322297802059565867

doesn't work

Do you have a better idea ?

Thx


--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Wolfgang Maier]

On 03/23/2018 01:16 PM, ast wrote:

Hi

I found this way to put a large number in
a variable.

C = int(
"28871482380507712126714295971303939919776094592797"
"22700926516024197432303799152733116328983144639225"
"94197780311092934965557841894944174093380561511397"
"4215424169339729054237110027510420801349667317"
"5515285922696291677532547505856101949404200039"
"90443211677661994962953925045269871932907037356403"
"22737012784538991261203092448414947289768854060249"
"76768122077071687938121709811322297802059565867")



A very simple improvement would be to use a single
triple-quoted string. Assuming you are copy/pasting
the number from somewhere that will save a lot of your
time.


It works but is it not optimal since there is a
string to int conversion.

I was not able to put an integer directly because
character '\' for line cut doesnt work inside an
integer

C = \
28871482380507712126714295971303939919776094592797\
22700926516024197432303799152733116328983144639225\
...
76768122077071687938121709811322297802059565867

doesn't work

Do you have a better idea ?

Thx




--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Rustom Mody]

On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:
> Hi
> 
> I found this way to put a large number in
> a variable.

What stops you from entering the number on one single (v long) line?

In case there is a religious commitment to PEP 8 dicta, the recommended 
meditation is this line (also from PEP8):

"However, know when to be inconsistent -- sometimes style guide recommendations 
just aren't applicable"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Wolfgang Maier]

On 03/23/2018 01:30 PM, Wolfgang Maier wrote:

On 03/23/2018 01:16 PM, ast wrote:

Hi

I found this way to put a large number in
a variable.

C = int(
"28871482380507712126714295971303939919776094592797"
"22700926516024197432303799152733116328983144639225"
"94197780311092934965557841894944174093380561511397"
"4215424169339729054237110027510420801349667317"
"5515285922696291677532547505856101949404200039"
"90443211677661994962953925045269871932907037356403"
"22737012784538991261203092448414947289768854060249"
"76768122077071687938121709811322297802059565867")



A very simple improvement would be to use a single
triple-quoted string. Assuming you are copy/pasting
the number from somewhere that will save a lot of your
time.


Like this, for example:

n = int(
''.join("""
37107287533902102798797998220837590246510135740250
46376937677490009712648124896970078050417018260538
74324986199524741059474233309513058123726617309629
91942213363574161572522430563301811072406154908250
23067588207539346171171980310421047513778063246676
89261670696623633820136378418383684178734361726757
28112879812849979408065481931592621691275889832738
44274228917432520321923589422876796487670272189318
47451445736001306439091167216856844588711603153276
70386486105843025439939619828917593665686757934951
62176457141856560629502157223196586755079324193331
64906352462741904929101432445813822663347944758178
92575867718337217661963751590579239728245598838407
58203565325359399008402633568948830189458628227828
80181199384826282014278194139940567587151170094390
35398664372827112653829987240784473053190104293586
86515506006295864861532075273371959191420517255829
71693888707715466499115593487603532921714970056938
54370070576826684624621495650076471787294438377604
53282654108756828443191190634694037855217779295145
36123272525000296071075082563815656710885258350721
45876576172410976447339110607218265236877223636045
17423706905851860660448207621209813287860733969412
""".split())
)

--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[ast]

Le 23/03/2018 à 13:43, Rustom Mody a écrit :

On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:

Hi

I found this way to put a large number in
a variable.


What stops you from entering the number on one single (v long) line?



It is not beautiful and not very readable. It is better to
have a fixed number of digits per line (eg 50)

import this

Beautiful is better than ugly.
Readability counts.




In case there is a religious commitment to PEP 8 dicta, the recommended
meditation is this line (also from PEP8):

"However, know when to be inconsistent -- sometimes style guide recommendations just 
aren't applicable"



Yes I am using pylint which flags too long lines (80 characters)
--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[ast]

Le 23/03/2018 à 13:30, Wolfgang Maier a écrit :

On 03/23/2018 01:16 PM, ast wrote:




A very simple improvement would be to use a single
triple-quoted string. Assuming you are copy/pasting
the number from somewhere that will save a lot of your
time.


no, it seems that sone \n are inserted inside the number

>>> C = int("""
1234
5678""")

Traceback (most recent call last):
  File "", line 3, in 
5678""")
ValueError: invalid literal for int() with base 10: '\n1234\n5678'


--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Johannes Bauer]

On 23.03.2018 14:01, ast wrote:

> It is not beautiful and not very readable. It is better to
> have a fixed number of digits per line (eg 50)

Oh yes, because clearly a 400-digit number becomes VERY beautiful and
readable once you add line breaks to it.

Cheers,
Joe

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht öffentlich!
Ah, der neueste und bis heute genialste Streich unsere großen
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos über Rüdiger Thomas in dsa 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Antoon Pardon]

On 23-03-18 14:01, ast wrote:
> Le 23/03/2018 à 13:43, Rustom Mody a écrit :
>> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:
>>> Hi
>>>
>>> I found this way to put a large number in
>>> a variable.
>>
>> What stops you from entering the number on one single (v long) line?
>
>
> It is not beautiful and not very readable. It is better to
> have a fixed number of digits per line (eg 50)

Numbers that large are not readable, no matter how you put then in your
code. Such a blob of digits just doesn't carry much meaning to humans.

What meaningful information from number can you easily retrieve from
representing the number in some kind of table form that you can't from
just writing the number on one line?

-- 
Antoon.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[ast]

Le 23/03/2018 à 13:55, Wolfgang Maier a écrit :

On 03/23/2018 01:30 PM, Wolfgang Maier wrote:

On 03/23/2018 01:16 PM, ast wrote:




n = int(
     ''.join("""
37107287533902102798797998220837590246510135740250
46376937677490009712648124896970078050417018260538
74324986199524741059474233309513058123726617309629

...


45876576172410976447339110607218265236877223636045
17423706905851860660448207621209813287860733969412
""".split())
)



yes, good idea

--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[ast]

Le 23/03/2018 à 14:16, Antoon Pardon a écrit :

On 23-03-18 14:01, ast wrote:

Le 23/03/2018 à 13:43, Rustom Mody a écrit :

On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:




What meaningful information from number can you easily retrieve from
representing the number in some kind of table form that you can't from
just writing the number on one line?



digit n° 103 is 1
digit n° 150 is 7
...

--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Antoon Pardon]

On 23-03-18 14:35, ast wrote:
> Le 23/03/2018 à 14:16, Antoon Pardon a écrit :
>> On 23-03-18 14:01, ast wrote:
>>> Le 23/03/2018 à 13:43, Rustom Mody a écrit :
 On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:
>
>
>> What meaningful information from number can you easily retrieve from
>> representing the number in some kind of table form that you can't from
>> just writing the number on one line?
>>
>
> digit n° 103 is 1
> digit n° 150 is 7
> ...
>
In what way is that meaningful information? How will that information
help you in any way with writing your program?

-- 
Antoon.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Richard Damon]

On 3/23/18 9:35 AM, ast wrote:

Le 23/03/2018 à 14:16, Antoon Pardon a écrit :

On 23-03-18 14:01, ast wrote:

Le 23/03/2018 à 13:43, Rustom Mody a écrit :

On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:




What meaningful information from number can you easily retrieve from
representing the number in some kind of table form that you can't from
just writing the number on one line?



digit n° 103 is 1
digit n° 150 is 7
...

If the value of the specific digits is meaningful, they you likely don't 
really have a number, but a digital representation, for which a string 
is probably the better way to define it, and take the cost of the 
conversion as part of the cost to be pretty.


--
Richard Damon

--
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Richard Damon]

On 3/23/18 6:35 AM, Chris Angelico wrote:

On Fri, Mar 23, 2018 at 9:29 PM, Steven D'Aprano
 wrote:

On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote:


That doesn't seem to be a strictly-correct Latin-1 decoder, then. There
are a number of unassigned byte values in ISO-8859-1.

That's incorrect, but I don't blame you for getting it wrong. Who thought
that it was a good idea to distinguish between "ISO 8859-1" and
"ISO-8859-1" as two related but distinct encodings?

https://en.wikipedia.org/wiki/ISO/IEC_8859-1

The old ISO 8859-1 standard, the one with undefined values, is mostly of
historical interest. For the last twenty years or so, anyone talking
about either Latin-1 or ISO-8859-1 (with or without dashes) is almost
meaning the 1992 IANA superset version which defines all 256 characters:

 "In 1992, the IANA registered the character map ISO_8859-1:1987,
 more commonly known by its preferred MIME name of ISO-8859-1
 (note the extra hyphen over ISO 8859-1), a superset of ISO
 8859-1, for use on the Internet. This map assigns the C0 and C1
 control characters to the unassigned code values thus provides
 for 256 characters via every possible 8-bit value."


Either that, or they actually mean Windows-1252, but let's not go there.


Wait, whaaa...

Though in my own defense, MySQL itself seems to have a bit of a
problem with encoding names. Its "utf8" is actually "UTF-8 with a
maximum of three bytes per character", in contrast to "utf8mb4" which
is, well, UTF-8.

In any case, abusing "Latin-1" to store binary data is still wrong.
That's what BLOB is for.

ChrisA


One comment on this whole argument, the original poster asked how to get 
data from a database that WAS using Latin-1 encoding into JSON (which 
wants UTF-8 encoding) and was asking if something needed to be done 
beyond using .decode('Latin-1'), and in particular if they need to use a 
.encode('UTF-8'). The answer should be a simple Yes or No.


Instead, someone took the opportunity to advocate that a wholesale 
change to the database was the only reasonable course of action.


First comment, when someone is proposing a change, it is generally put 
on them the burden of proof that the change is warranted. This is 
especially true when they are telling someone else they need to make 
such a change.


Absolute statements are very hard to prove (but the difficulty of proof 
doesn't relieve the need to provide it), and in fact are fairly easy to 
disprove (one counter example disproves an absolute statement). Counter 
examples to the absolute statement have been provided.


When dealing with a code base, backwards compatibility IS important, and 
casually changing something that fundamental isn't the first thing that 
someone should be thinking about, We weren't given any details about the 
overall system this was part of, and they easily could be other code 
using the database that such a change would break. One easy Python 
example is to look back at the change from Python 2 to Python 3, how 
many years has that gone on, and how many more will people continue to 
deal with it? This was over a similar issue, that at least for today, 
Unicode is the best solution for storing arbitrary text, and forcing 
that change down to the fundamental level.


--
Richard Damon

--
https://mail.python.org/mailman/listinfo/python-list

Re: Entering a very large number

[Thomas Jollans]

On 2018-03-23 14:01, ast wrote:
> Le 23/03/2018 à 13:43, Rustom Mody a écrit :
>> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote:
>>> Hi
>>>
>>> I found this way to put a large number in
>>> a variable.
>>
>> What stops you from entering the number on one single (v long) line?
> 
> 
> It is not beautiful and not very readable. It is better to
> have a fixed number of digits per line (eg 50)
> 
> import this
> 
> Beautiful is better than ugly.
> Readability counts.

I would tend to read this as a reason not to have extremely large
numbers in your code in the first place if you can avoid it.

> 
> 
>>
>> In case there is a religious commitment to PEP 8 dicta, the recommended
>> meditation is this line (also from PEP8):
>>
>> "However, know when to be inconsistent -- sometimes style guide
>> recommendations just aren't applicable"
>>
> 
> Yes I am using pylint which flags too long lines (80 characters)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.6: How to expand f-string literals read from a file vs inline statement

[Ned Batchelder]

On 3/23/18 4:30 AM, Malcolm Greene wrote:

Looking for advice on how to expand f-string literal strings whose
values I'm reading from a configuration file vs hard coding into
my script as statements. I'm using f-strings as a very simple
template language.
I'm currently using the following technique to expand these f-strings.
Is there a better way?
Bonus if anyone has suggestions on how my expand() function can access
the locals() value of the caller so this parameter doesn't have to be
passed in explicitly.
*def *expand(expression, values):

"""Expand expression using passed in locals()"""

triple_quote = *"'" ** 3
expression = dedent(expression)

*return *eval(*f'f{triple_quote}{expression}{triple_quote}'*,
*None*, values)

product_name = 'Bike ABC'

product_sku = '123456'

product_price = 299.95

discount = 0.85



# read in product description template

# product_description_template might look like: {product_sku} :
# {product_name}: ${product_price * discount}product_description_template = 
config('product_description_template')



# expand the {expressions} in product_description_template using my
# locals()product_description = expand(product_description_template, locals())




Perhaps it doesn't need to be said, but just to be sure: don't use eval 
if you don't trust the people writing the configuration file. They can 
do nearly unlimited damage to your environment.  They are writing code 
that you are running.


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: PyQt4 QWebView cant load google maps markers

[Anssi Saari]

Xristos Xristoou  writes:

> I want to create a simple python app using pyqt,QWebView and google maps with 
> markers.
>
> The problem is that,the markers does not load inside the QWebView, as
> you can see they load just fine in the browser.

Well, since you got a javascript error, maybe Qt4 doesn't support the
version of javascript Google uses today? Qt4 was released over a decade
ago.

I tried with PyQt5 and the page loads and a marker shows up. It then
disappears as Google puts up a couple of popups. There's a complaint
about browser capabilities and also a privacy note from Google.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Tobiah]

On 03/22/2018 12:46 PM, Tobiah wrote:

I have some mailing information in a Mysql database that has
characters from various other countries.  The table says that
it's using latin-1 encoding.  I want to send this data out
as JSON.

So I'm just taking each datum and doing 'name'.decode('latin-1')
and adding the resulting Unicode value right into my JSON structure
before doing .dumps() on it.  This seems to work, and I can consume
the JSON with another program and when I print values, they look nice
with the special characters and all.

I was reading though, that JSON files must be encoded with UTF-8.  So
should I be doing string.decode('latin-1').encode('utf-8')?  Or does
the json module do that for me when I give it a unicode object?



Thanks for all the discussion.  A little more about our setup:
We have used a LAMP stack system for almost 20 years to deploy
hundreds of websites.  The database tables are latin-1 only because
at the time we didn't know how or care to change it.

More and more, 'special' characters caused a problem.  They would
not come out correctly in a .csv file or wouldn't print correctly.
Lately, I noticed that a JSON file I was sending out was delivering
unreadable characters.  That's when I started to look into Unicode
a bit more.  From the discussion, and my own guesses, it looks
as though all have to do is string.decode('latin-1'), and stuff
that Unicode object right into my structure that gets handed to
json.dumps().

If I changed my database tables to all be UTF-8 would this
work cleanly without any decoding?  Whatever people are doing
to get these characters in, whether it's foreign keyboards,
or fancy escape sequences in the web forms, would their intended
characters still go into the UTF-8 database as the proper characters?
Or now do I have to do a conversion on the way in to the database?

We also get import data that often comes in .xlsx format.  What
encoding do I get when I dump a .csv from that?  Do I have to
ask the sender?  I already know that they don't know.


Toby
--
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Grant Edwards]

On 2018-03-23, Chris Angelico  wrote:
> On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano
> wrote:
>> On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote:
>>
 I was reading though, that JSON files must be encoded with UTF-8.  So
 should I be doing string.decode('latin-1').encode('utf-8')?  Or does
 the json module do that for me when I give it a unicode object?
>>>
>>> Reconfigure your MySQL database to use UTF-8. There is no reason to use
>>> Latin-1 in the database.
>>
>> You don't know that. You don't know what technical, compatibility, policy
>> or historical constraints are on the database.
>
> Okay. Give me a good reason for the database itself to be locked to
> Latin-1.

Because the DB administrator wont change things without orders from
his boss, who won't order changes without because there's no budget
for that.

OK, perhaps it's not a _good_ reason by your metrics, but reasons like
that are what you find in the real world.

-- 
Grant Edwards   grant.b.edwardsYow! ... My pants just went
  at   on a wild rampage through a
  gmail.comLong Island Bowling Alley!!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Grant Edwards]

On 2018-03-23, Richard Damon  wrote:

> One comment on this whole argument, the original poster asked how to get 
> data from a database that WAS using Latin-1 encoding into JSON (which 
> wants UTF-8 encoding) and was asking if something needed to be done 
> beyond using .decode('Latin-1'), and in particular if they need to use a 
> .encode('UTF-8'). The answer should be a simple Yes or No.
>
> Instead, someone took the opportunity to advocate that a wholesale 
> change to the database was the only reasonable course of action.

Well, this is (somewhat) Usenet, where the most common answer to "How
do I do X?"  is usually "X is stupid. You don't want to do X. You
should do Y."  This is generally followed by somebody proposing Z
instead of Y and a long debate about the relative merits of Y and Z.

One learns to either justify X or just ingore the subthreads about Y
and Z.  Except sometimes the answer _is_ that you really don't want to
do X, and probably should do Y or Z.

-- 
Grant Edwards   grant.b.edwardsYow! Oh my GOD -- the
  at   SUN just fell into YANKEE
  gmail.comSTADIUM!!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Chris Angelico]

On Sat, Mar 24, 2018 at 1:46 AM, Tobiah  wrote:
> On 03/22/2018 12:46 PM, Tobiah wrote:
>>
>> I have some mailing information in a Mysql database that has
>> characters from various other countries.  The table says that
>> it's using latin-1 encoding.  I want to send this data out
>> as JSON.
>>
>> So I'm just taking each datum and doing 'name'.decode('latin-1')
>> and adding the resulting Unicode value right into my JSON structure
>> before doing .dumps() on it.  This seems to work, and I can consume
>> the JSON with another program and when I print values, they look nice
>> with the special characters and all.
>>
>> I was reading though, that JSON files must be encoded with UTF-8.  So
>> should I be doing string.decode('latin-1').encode('utf-8')?  Or does
>> the json module do that for me when I give it a unicode object?
>
>
>
> Thanks for all the discussion.  A little more about our setup:
> We have used a LAMP stack system for almost 20 years to deploy
> hundreds of websites.  The database tables are latin-1 only because
> at the time we didn't know how or care to change it.
>
> More and more, 'special' characters caused a problem.  They would
> not come out correctly in a .csv file or wouldn't print correctly.
> Lately, I noticed that a JSON file I was sending out was delivering
> unreadable characters.  That's when I started to look into Unicode
> a bit more.  From the discussion, and my own guesses, it looks
> as though all have to do is string.decode('latin-1'), and stuff
> that Unicode object right into my structure that gets handed to
> json.dumps().

Yep, this is sounding more and more like you need to go UTF-8 everywhere.

> If I changed my database tables to all be UTF-8 would this
> work cleanly without any decoding?  Whatever people are doing
> to get these characters in, whether it's foreign keyboards,
> or fancy escape sequences in the web forms, would their intended
> characters still go into the UTF-8 database as the proper characters?
> Or now do I have to do a conversion on the way in to the database?

The best way to do things is to let your Python-MySQL bridge do the
decoding for you; you'll simply store and get back Unicode strings.
That's how things happen by default in Python 3 (I believe; been a
while since I used MySQL, but it's like that with PostgreSQL). My
recommendation is to give it a try; most likely, things will just
work.

> We also get import data that often comes in .xlsx format.  What
> encoding do I get when I dump a .csv from that?  Do I have to
> ask the sender?  I already know that they don't know.

Ah, now, that's a potential problem. A CSV file can't tell you what
encoding it's in. Fortunately, UTF-8 is designed to be fairly
dependable: if you attempt to decode something as UTF-8 and it works,
you can be confident that it really is UTF-8. But ultimately, you have
to just ask the person who exports it: "please export it in UTF-8".

Generally, things should "just work" as long as you're consistent with
encodings, and the easiest way to be consistent is to use UTF-8
everywhere. It's a simple rule that everyone can follow. (Hopefully.
:) )

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons

[Malcolm Greene]

Looking for your suggestions on best practice techniques for managing
secrets used by Python daemon scripts. Use case is Windows scripts
running as NT Services, but interested in Linux options as well.
Here's what we're considering

1. Storing secrets in environment vars
2. Storing secrets in config file only in subfolder with access limited
   to daemon account only3. Using a 3rd party vault product

Thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.6: How to expand f-string literals read from a file vs inline statement

[Malcolm Greene]

> Perhaps it doesn't need to be said, but just to be sure: don't use eval  if 
> you don't trust the people writing the configuration file. They can do nearly 
> unlimited damage to your environment.  They are writing code that you are 
> running.

Of course! Script and config file are running in a private subnet and both are 
maintained by a single developer.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.6: How to expand f-string literals read from a file vs inline statement

[Ned Batchelder]

On 3/23/18 12:39 PM, Malcolm Greene wrote:

Perhaps it doesn't need to be said, but just to be sure: don't use eval  if you 
don't trust the people writing the configuration file. They can do nearly 
unlimited damage to your environment.  They are writing code that you are 
running.

Of course! Script and config file are running in a private subnet and both are 
maintained by a single developer.


Then why make your life difficult?  Put the "configuration" in a .py file.

--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: PyQt4 QWebView cant load google maps markers

[Xristos Xristoou]

tell some solution ?yes you are correct for some seconds show me the mark but 
only for some seconds.can i update my pyqt4 to show me the mark ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.6: How to expand f-string literals read from a file vs inline statement

[Steven D'Aprano]

On Fri, 23 Mar 2018 10:39:05 -0600, Malcolm Greene wrote:

>> Perhaps it doesn't need to be said, but just to be sure: don't use eval
>>  if you don't trust the people writing the configuration file. They can
>> do nearly unlimited damage to your environment.  They are writing code
>> that you are running.
> 
> Of course! Script and config file are running in a private subnet 


Okay. So only users who have access to the private subnet can inject code 
into your application. That covers a *lot* of ground:

"The private subnet is used by me and my wife, and we both have root on 
the system and trust each other implicitly."

"The private subnet is used by five thousand really smart and technically 
savvy but emotionally immature teens who are constantly trying to 
escalate privileges and take over the system."

I always find it amusing when techies imagine that hackers on the 
internet are the only security threat.

http://www.zdnet.com/article/the-top-five-internal-security-threats/

https://blog.trendmicro.com/most-data-security-threats-are-internal-
forrester-says/




> and both are maintained by a single developer.

And this is relevant to the security risk in what way?



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons

[Dan Stromberg]

I'd put them in a file with access to the daemon..

Putting credentials in an environment variable is insecure on Linux,
because ps auxwwe lists environment variables.

On Fri, Mar 23, 2018 at 9:37 AM, Malcolm Greene  wrote:
> Looking for your suggestions on best practice techniques for managing
> secrets used by Python daemon scripts. Use case is Windows scripts
> running as NT Services, but interested in Linux options as well.
> Here's what we're considering
>
> 1. Storing secrets in environment vars
> 2. Storing secrets in config file only in subfolder with access limited
>to daemon account only3. Using a 3rd party vault product
>
> Thanks
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

ANN: Wing Python IDEs version 6.0.11 released

[Wingware]

Hi,

We've just released Wing 6.0.11 , 
which implements auto-save and restore for remote files, adds a Russian 
translation of the UI (thanks to Alexandr Dragukin), improves remote 
development error reporting and recovery after network breaks, correctly 
terminates SSH tunnels when switching projects or quitting, fixes severe 
network slowdown seen on High Sierra, auto-reactivates expired annual 
licenses without restarting Wing, and makes about 20 other 
improvements.  For details, see 
https://wingware.com/pub/wingide/6.0.11/CHANGELOG.txt


Download Now 

About Wing

Wing is a family of cross-platform 
 Python IDEs with 
powerful integrated editing, debugging, unit testing, and project 
management features. Wing runs on Windows, Linux, and OS X, and can be 
used to develop any kind of Python code for web, desktop, embedded 
scripting, and other applications.


Wing 101  and Wing Personal 
 omit some features and 
are free to download and use without a license. Wing Pro 
 requires purchasing 
 or upgrading 
 a license, or obtaining a 30-day 
trial at startup.


Version 6 introduces many new features, including improved 
multi-selection, much easier remote development 
, debugging from the Python 
Shell, recursive debugging, PEP 484 and 526 type hinting, support for 
Python 3.6 and 3.7, Vagrant , 
Jupyter , and Django 
 1.10+, easier Raspberry Pi 
 development, optimized 
debugger, OS X full screen mode, One Dark color palette, Russian 
localization (thanks to Alexandr Dragukin), expanded free product line, 
and much more. For details, see What's New in Wing Version 6 
.


Wing 6 works with Python versions 2.5 through 2.7 and 3.2 through 3.7, 
including also Anaconda, ActivePython, EPD, Stackless, and others 
derived from the CPython implementation.


For more product information, please visit wingware.com 



Upgrading

You can try Wing 6 without removing older versions. Wing 6 will read and 
convert your old preferences, settings, and projects. Projects should be 
saved to a new name since previous versions of Wing cannot read Wing 6 
projects.


See also Migrating from Older Versions 
 and Upgrading 
.


Links

Release notice: https://wingware.com/news/2018-03-21
Downloads and Free Trial: https://wingware.com/downloads
Buy: https://wingware.com/store/purchase
Upgrade: https://wingware.com/store/upgrade

Questions?  Don't hesitate to email us at supp...@wingware.com.

Thanks,

--

Stephan Deibel
Wingware | Python IDE

The Intelligent Development Environment for Python Programmers

wingware.com

--
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Steven D'Aprano]

On Fri, 23 Mar 2018 07:46:16 -0700, Tobiah wrote:

> If I changed my database tables to all be UTF-8 would this work cleanly
> without any decoding?

Not reliably or safely. It will appear to work so long as you have only 
pure ASCII strings from the database, and then crash when you don't:

py> text_from_database = u"hello wörld".encode('latin1')
py> print text_from_database
hello w�rld
py> json.dumps(text_from_database)
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
  File "/usr/local/lib/python2.7/json/encoder.py", line 195, in encode
return encode_basestring_ascii(o)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 7: 
invalid start byte


> Whatever people are doing to get these characters
> in, whether it's foreign keyboards, or fancy escape sequences in the web
> forms, would their intended characters still go into the UTF-8 database
> as the proper characters? Or now do I have to do a conversion on the way
> in to the database?

There is no way to answer that, because it depends on how you are getting 
the characters, what you are doing to them, and how you put them in the 
database.

In the best possible scenario, your process is:

- user input comes in as UTF-8;
- you store it in the database;
- which converts it to Latin-1 (sometimes losing data: see below)

in which case, changing the database field to utf8mb4 (NOT plain utf8, 
thanks to a ludicrously idiotic design flaw in MySQL utf8 is not actually 
utf8) should work nicely.

I mentioned losing data: if your user enters, let's say the Greek letters 
'αβγ' (or emojis, or any of about a million other characters) then Latin1 
cannot represent them. Presumably your database is throwing them away:


py> s = 'αβγ'  # what the user wanted
py> db = s.encode('latin-1', errors='replace')  # what the database 
recorded
py> json.dumps(db.decode('latin-1'))  # what you end up with
'"???"'


Or, worse, you're getting moji-bake:

py> s = 'αβγ'  # what the user wanted
py> json.dumps(s.encode('utf-8').decode('latin-1'))
'"\\u00ce\\u00b1\\u00ce\\u00b2\\u00ce\\u00b3"'



> We also get import data that often comes in .xlsx format.  What encoding
> do I get when I dump a .csv from that?  Do I have to ask the sender?  I
> already know that they don't know.

They never do :-(

In Python 2, I believe the CSV module will assume ASCII-plus-random-crap, 
and it will work fine so long as it actually is ASCII. Otherwise you'll 
get random-crap: possibly an exception, possibly moji-bake.

The sad truth is that as soon as you leave the nice, clean world of pure 
Unicode, and start dealing with legacy encodings, everything turns to 
quicksand.

If you haven't already done so, you really should start by reading Joel 
Spolsky's introduction to Unicode:

http://global.joelonsoftware.com/English/Articles/Unicode.html

and Ned Batchelder's post on dealing with Unicode and Python:

https://nedbatchelder.com/text/unipain.html



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Chris Angelico]

On Sat, Mar 24, 2018 at 11:11 AM, Steven D'Aprano
 wrote:
> On Fri, 23 Mar 2018 07:46:16 -0700, Tobiah wrote:
>
>> If I changed my database tables to all be UTF-8 would this work cleanly
>> without any decoding?
>
> Not reliably or safely. It will appear to work so long as you have only
> pure ASCII strings from the database, and then crash when you don't:
>
> py> text_from_database = u"hello wörld".encode('latin1')
> py> print text_from_database
> hello w�rld
> py> json.dumps(text_from_database)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/local/lib/python2.7/json/__init__.py", line 231, in dumps
> return _default_encoder.encode(obj)
>   File "/usr/local/lib/python2.7/json/encoder.py", line 195, in encode
> return encode_basestring_ascii(o)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 7:
> invalid start byte
>

If the database has been configured to use UTF-8 (as mentioned, that's
"utf8mb4" in MySQL), you won't get that byte sequence back. You'll get
back valid UTF-8. At least, if ever you don't, that's a MySQL bug, and
not your fault. So yes, it WILL work cleanly. Reliably and safely.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Since when builtin dict preserve key order?

[Arkadiusz Bulski]

I already asked on PYPY and they confirmed that any version of pypy,
including 2.7, has dict preserving insertion order. I am familiar with
ordered **kw which was introduced in 3.6 but I also heard that builtin dict
preserves order since 3.5. Is that true?

-- 
~ Arkadiusz Bulski
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Since when builtin dict preserve key order?

[Chris Angelico]

On Sat, Mar 24, 2018 at 3:34 PM, Arkadiusz Bulski  wrote:
> I already asked on PYPY and they confirmed that any version of pypy,
> including 2.7, has dict preserving insertion order. I am familiar with
> ordered **kw which was introduced in 3.6 but I also heard that builtin dict
> preserves order since 3.5. Is that true?
>

I don't think 3.5 had it. According to the tracker, it landed in 3.6:

https://bugs.python.org/issue27350

But yes, current versions of CPython preserve insertion order.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Putting Unicode characters in JSON

[Steven D'Aprano]

On Sat, 24 Mar 2018 11:21:09 +1100, Chris Angelico wrote:

>>> If I changed my database tables to all be UTF-8 would this work
>>> cleanly without any decoding?
>>
>> Not reliably or safely. It will appear to work so long as you have only
>> pure ASCII strings from the database, and then crash when you don't:
[...]
> If the database has been configured to use UTF-8 (as mentioned, that's
> "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get
> back valid UTF-8.

Of course you will -- sorry I got confused.




-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list