EuroPython 2015: Educational Summit

[M.-A. Lemburg]

We are proud to announce an educational summit at EuroPython
2015. This is being organized together with the Raspberry Pi
Foundation (https://www.raspberrypi.org/).


*** EuroPython 2015 Educational Summit ***

https://ep2015.europython.eu/en/events/educational-summit/


We will have Educational Summit focused talks, trainings, birds of a
feather sessions to debate and also Educational Sprints for the
building of education focused projects during the weekend.

Reduced ticket prices for teachers and kids
---

To to facilitate attendance, teacher can attend the conference at the
reduced student rate. If you want to bring along kids, we also have
special tickets for them.

Spanish track
-

For those who don’t feel comfortable with English, we also provide
Spanish tracks to get you up to speed with Python.

Summary of the agenda
-

 * July 21 - Trainings in Spanish (including an introduction to Python)
 * July 22 - Talks in Spanish
 * July 23 - Educational Summit Talks (English, all day)
 + Trainings (English, afternoon)
 * July 24 - relax, discuss, attend other conference talks
 * July 25 - Educational Summit Sprints
 * July 26 - Educational Summit Sprints

More information is available on the Education Summit page. If you
have questions, please write to helpd...@europython.eu.

Enjoy,
--
EuroPython 2015 Team
http://ep2015.europython.eu/
http://www.europython-society.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parser needed.

[Ryan Stuart]

On Tue, Jun 9, 2015 at 12:30 PM, Skybuck Flying 
wrote:

> Anyway... I am trying a more robust parser... because my own parser right
> now didn't work out for new inputs.
>

You should take a look at lrparsing:
https://pypi.python.org/pypi/lrparsing/1.0.11

Cheers


>
> It almost worked except for first item... don't know what problem was
> maybe just this...
>
> But I'll try and do it the usually way.
>
> "Tokenize", "Parse" etc.
>
> It's significantly slower though.
>
> Maybe an idea for a new kind of parser could be:
>
> Find all recgonized token keywords in one go... and stuff all find indexes
> into a list... and then perhaps sort the list...
>
> and then use the sorted list... to return token found at that position...
>
> and then start processing like that...
>
> Benefit of this idea is less characters to compare... and could be done
> with just integer compare... or even lookup table processing.
>
> Added benefit is still in-order which is nice.
>
> For now I won't do it... cause I am not sure it would be an improvemt.
>
> Another idea which might be an improvement is.
>
> Parallel searching... ofcourse that might not be possible... though...
> multi core does it exist and threading too.
>
> But to mimic parallel searching and to prevent problems.
>
> A sliding window approach could be taken.
>
> And perhaps items found that way in a certain buffer or so and then still
> added to processing list or so...
>
> which kinda mimics parallel search... but just uses data cache nicely.
>
> Though it's somewhat of a complex method... I will avoid it for now.
>
> The split() routine is real nice... to get rid of fudd/white space... and
> just tokens which is nice.
>
> So for now I will use that as my tokenizer ;) =D
>
> and bracket level counting and sections and stuff like that yeah...
>
>
> Bye,
>  Skybuck.
> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Ryan Stuart, B.Eng
Software Engineer

ABN: 81-206-082-133
W: http://www.textisbeautiful.net/
M: +61-431-299-036
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cheat sheet for the new string formatting?

[Skip Montanaro]

Skip> Why don't floats support "{:.Ns}"? (I know I can use "{!s}".)

random832> Why would they? The old style didn't support %.Ns either.

Well, the old style does, though it appears the N is ignored:

>>> "%5s" % -0.00666762259822
'-0.00666762259822'

It doesn't raise an exception though.

(This is with Python 2.7.2. Haven't tried to see if that behavior
changed with 3.x.)

One thing which seems obvious now is that since format() delegates to
the individual types for formatting, much of the documentation of this
stuff must now be delegated to the individual types. However, I can't
find anything about the formatting syntax supported by
float.__format__.  Where might I find it documented that "{:.3}" is
synonymous with "%.3g"?

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parser needed.

[Skybuck Flying]

Euhm...

My parser is already done... since today

Loving it too

Wrote it myself... based on the c# code technique explained somewhere in 
this thread too


Bye,
 Skybuck.


--
https://mail.python.org/mailman/listinfo/python-list

Re: Cheat sheet for the new string formatting?

[random832]

On Tue, Jun 9, 2015, at 08:15, Skip Montanaro wrote:
> Skip> Why don't floats support "{:.Ns}"? (I know I can use "{!s}".)
> 
> random832> Why would they? The old style didn't support %.Ns either.
> 
> Well, the old style does, though it appears the N is ignored:
> 
> >>> "%5s" % -0.00666762259822
> '-0.00666762259822'

There's no dot there. I was mistaken, though, it does work.

>>> "%.5s" % -0.00666762259822
'-0.00'

The equivalent new-style format is {!s:.5} - the reason is that !s is
fundamentally different from other formats since it performs a type
conversion rather than delegating to the argument's __format__ method.
Whereas _all_ old-style format specifiers did type conversions [see
'%.23f' % Decimal('0.1') vs '{:.23f}'.format(Decimal('0.1'))]

> One thing which seems obvious now is that since format() delegates to
> the individual types for formatting, much of the documentation of this
> stuff must now be delegated to the individual types. However, I can't
> find anything about the formatting syntax supported by
> float.__format__.  Where might I find it documented that "{:.3}" is
> synonymous with "%.3g"?

Where have you looked? Have you read
https://docs.python.org/3/library/string.html#formatspec ?

NoneSimilar to 'g', except that fixed-point notation, when used, has
at least one digit past the decimal point. The default precision is as
high as needed to represent the particular value. The overall effect is
to match the output of str() as altered by the other format modifiers.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cheat sheet for the new string formatting?

[Skip Montanaro]

On Tue, Jun 9, 2015 at 7:29 AM,   wrote:
> Where have you looked? Have you read
> https://docs.python.org/3/library/string.html#formatspec ?

Yes, but I missed the None section. I looked closely at 'g', but
didn't see anything like "this is the default". I will admit I was a
bit frustrated to see things like the documentation for the format()
builtin indicating that it calls value.__format__, but then couldn't
find any documentation in the numeric types for float.__format__. From
the documentation of the format() builtin: "The interpretation of
format_spec will depend on the type of the value argument...", which
suggested to me that at least the builtin types would have some stub
documentation of their __format__ method, even if all it did was
redirect the user to the section in the string doc:
https://docs.python.org/3.5/library/string.html#format-string-syntax

I'll open a doc issue about that. Thanks for the help.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Cheat sheet for the new string formatting?

[Mark Lawrence]

On 09/06/2015 13:15, Skip Montanaro wrote:


One thing which seems obvious now is that since format() delegates to
the individual types for formatting, much of the documentation of this
stuff must now be delegated to the individual types. However, I can't
find anything about the formatting syntax supported by
float.__format__.  Where might I find it documented that "{:.3}" is
synonymous with "%.3g"?



You won't, at least not that I'm aware of.  I think part of the problem 
is that there was a lot of confusion as to whether or not the old style 
formatting was to be deprecated.  See http://bugs.python.org/issue14123 
which refers to http://www.gossamer-threads.com/lists/python/dev/969817


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

enhancement request: make py3 read/write py2 pickle format

[Neal Becker]

One of the most annoying problems with py2/3 interoperability is that the 
pickle formats are not compatible.  There must be many who, like myself, 
often use pickle format for data storage.

It certainly would be a big help if py3 could read/write py2 pickle format.  
You know, backward compatibility? 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Mark Lawrence]

On 09/06/2015 19:08, Neal Becker wrote:

One of the most annoying problems with py2/3 interoperability is that the
pickle formats are not compatible.  There must be many who, like myself,
often use pickle format for data storage.

It certainly would be a big help if py3 could read/write py2 pickle format.
You know, backward compatibility?



http://bugs.python.org/issue13566

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Laura Creighton]

In a message of Tue, 09 Jun 2015 14:08:25 -0400, Neal Becker writes:
>One of the most annoying problems with py2/3 interoperability is that the 
>pickle formats are not compatible.  There must be many who, like myself, 
>often use pickle format for data storage.
>
>It certainly would be a big help if py3 could read/write py2 pickle format.  
>You know, backward compatibility? 
>
>-- 
>https://mail.python.org/mailman/listinfo/python-list

We have an issue about that.
https://bugs.python.org/issue13566

Go there and say you want it too. :)

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Chris Warrick]

On Tue, Jun 9, 2015 at 8:08 PM, Neal Becker  wrote:
> One of the most annoying problems with py2/3 interoperability is that the
> pickle formats are not compatible.  There must be many who, like myself,
> often use pickle format for data storage.
>
> It certainly would be a big help if py3 could read/write py2 pickle format.
> You know, backward compatibility?

Don’t use pickle. It’s unsafe — it executes arbitrary code, which
means someone can give you a pickle file that will delete all your
files or eat your cat.

Instead, use a safe format that has no ability to execute code, like
JSON. It will also work with other programming languages and
environments if you ever need to talk to anyone else.

But, FYI: there is backwards compatibility if you ask for it, in the
form of protocol versions. That’s all you should know — again, don’t
use pickle.

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Zachary Ware]

On Tue, Jun 9, 2015 at 1:08 PM, Neal Becker  wrote:
> One of the most annoying problems with py2/3 interoperability is that the
> pickle formats are not compatible.  There must be many who, like myself,
> often use pickle format for data storage.
>
> It certainly would be a big help if py3 could read/write py2 pickle format.
> You know, backward compatibility?

Uhh...

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('test.pkl', 'wb') as f:
...  pickle.dump({'test': [1, 2, {3}]}, f)
...
>>> with open('test.pkl', 'rb') as f:
...  pickle.load(f)
...
{'test': [1, 2, set([3])]}
>>> ^D
$ python3
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('test.pkl', 'rb') as f:
...  pickle.load(f)
...
{'test': [1, 2, {3}]}
>>> with open('test2.pkl', 'wb') as f:
...  pickle.dump(['test', {2: {3.4}}], f, protocol=2)
...
>>> with open('test2.pkl', 'rb') as f:
...  pickle.load(f)
...
['test', {2: {3.4}}]
>>> ^D
✔ ~
13:35 $ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('test2.pkl', 'rb') as f:
...  pickle.load(f)
...
[u'test', {2: set([3.4])}]
>>>

-- 
Zach
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Serhiy Storchaka]

On 09.06.15 21:08, Neal Becker wrote:

One of the most annoying problems with py2/3 interoperability is that the
pickle formats are not compatible.  There must be many who, like myself,
often use pickle format for data storage.

It certainly would be a big help if py3 could read/write py2 pickle format.
You know, backward compatibility?


Pickle format is mostly compatible. What is your issue?


--
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Serhiy Storchaka]

On 09.06.15 21:31, Laura Creighton wrote:

We have an issue about that.
https://bugs.python.org/issue13566

Go there and say you want it too. :)


I afraid issue title is too general. :) The issue is only about one 
minor detail of py3 to py2 compatibility.


--
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Devin Jeanpierre]

There's a lot of subtle issues with pickle compatibility. e.g.
old-style vs new-style classes. It's kinda hard and it's better to
give up. I definitely agree it's better to use something else instead.
For example, we switched to using protocol buffers, which have much
better compatibility properties and are a bit more testable to boot
(since text format protobufs are always output in a canonical (sorted)
form.)

-- Devin

On Tue, Jun 9, 2015 at 11:35 AM, Chris Warrick  wrote:
> On Tue, Jun 9, 2015 at 8:08 PM, Neal Becker  wrote:
>> One of the most annoying problems with py2/3 interoperability is that the
>> pickle formats are not compatible.  There must be many who, like myself,
>> often use pickle format for data storage.
>>
>> It certainly would be a big help if py3 could read/write py2 pickle format.
>> You know, backward compatibility?
>
> Don’t use pickle. It’s unsafe — it executes arbitrary code, which
> means someone can give you a pickle file that will delete all your
> files or eat your cat.
>
> Instead, use a safe format that has no ability to execute code, like
> JSON. It will also work with other programming languages and
> environments if you ever need to talk to anyone else.
>
> But, FYI: there is backwards compatibility if you ask for it, in the
> form of protocol versions. That’s all you should know — again, don’t
> use pickle.
>
> --
> Chris Warrick 
> PGP: 5EAAEA16
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Want to readout the control file of aria2.

[Hongyi Zhao]

Hi all,

I want to read out the control file of aria2, which is a binary file 
including the data format depicted here:

http://aria2.sourceforge.net/manual/en/html/technical-notes.html

Could you please give me some snipped codes for this purpose?

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.





-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Joel Goldstick]

On Tue, Jun 9, 2015 at 6:21 PM, Hongyi Zhao  wrote:
> Hi all,
>
> I want to read out the control file of aria2, which is a binary file
> including the data format depicted here:
>
> http://aria2.sourceforge.net/manual/en/html/technical-notes.html
>
> Could you please give me some snipped codes for this purpose?
>
> Regards
> --
> .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
>
>
>
>
>
> --
> .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
> --
> https://mail.python.org/mailman/listinfo/python-list

I found this: http://aria.pasteur.fr/developers/aria-python-api

Does that help?


-- 
Joel Goldstick
http://joelgoldstick.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Laura Creighton]

No promises -- I never used this myself, but it looks like:

https://github.com/killuahzl/pyaria2

parses pyaria files for its own purposes, so you could steal code from there.

Laura

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Chris Angelico]

On Wed, Jun 10, 2015 at 6:07 AM, Devin Jeanpierre
 wrote:
> There's a lot of subtle issues with pickle compatibility. e.g.
> old-style vs new-style classes. It's kinda hard and it's better to
> give up. I definitely agree it's better to use something else instead.
> For example, we switched to using protocol buffers, which have much
> better compatibility properties and are a bit more testable to boot
> (since text format protobufs are always output in a canonical (sorted)
> form.)

Or use JSON, if your data fits within that structure. It's easy to
read and write, it's human-readable, and it's safe (no chance of
arbitrary code execution). Forcing yourself to use a format that can
basically be processed by ast.literal_eval() is a good discipline -
means you don't accidentally save/load too much.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Hongyi Zhao]

On Tue, 09 Jun 2015 18:52:00 -0400, Joel Goldstick wrote:

> I found this: http://aria.pasteur.fr/developers/aria-python-api
> 
> Does that help?

The aria mentioned by your above links is completely different thing WRT 
the one I posted here ;-(

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Joel Goldstick]

On Tue, Jun 9, 2015 at 7:19 PM, Hongyi Zhao  wrote:
> On Tue, 09 Jun 2015 18:52:00 -0400, Joel Goldstick wrote:
>
>> I found this: http://aria.pasteur.fr/developers/aria-python-api
>>
>> Does that help?
>
> The aria mentioned by your above links is completely different thing WRT
> the one I posted here ;-(
>
> Regards
> --
> .: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
> --
> https://mail.python.org/mailman/listinfo/python-list

Well, maybe Laura's link will be what you need?

-- 
Joel Goldstick
http://joelgoldstick.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Hongyi Zhao]

On Wed, 10 Jun 2015 00:58:41 +0200, Laura Creighton wrote:

> No promises -- I never used this myself, but it looks like:
> 
> https://github.com/killuahzl/pyaria2
> 
> parses pyaria files for its own purposes, so you could steal code from
> there.
> 
> Laura

Thanks, but it doesn't do the same thing that I want.

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parser needed.

[Michael Torrie]

On 06/09/2015 06:20 AM, Skybuck Flying wrote:
> Euhm...
> 
> My parser is already done... since today
> 
> Loving it too
> 
> Wrote it myself... based on the c# code technique explained somewhere in 
> this thread too

I'm glad you're having fun, and making good progress.  And it's good to
hear of success with Python.  However, I wouldn't really call this a
"thread;" more a "monologue."  I'm not sure list members really
appreciate the blow-by-blow description of a project, for the future.  I
know I find it rather noisy.  I don't think anyone has actually read any
of your messages (I'm pretty sure some have just blacklisted your
messages), except to determine you're trying to parse something rather
vague.  Might I suggest a blog is a more appropriate place to ruminate
over a project (and posting code and examples) and use the list for
questions and advice?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Hongyi Zhao]

On Tue, 09 Jun 2015 19:21:30 -0400, Joel Goldstick wrote:

>> https://mail.python.org/mailman/listinfo/python-list
> 
> Well, maybe Laura's link will be what you need?

Not still.  Laura's link is a python script which interactively using 
aria2 for downloading via rpc.  I want to readout the binary control file 
of aria2 ( *.aria2 ).

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Want to readout the control file of aria2.

[Hongyi Zhao]

On Tue, 09 Jun 2015 23:29:43 +, Hongyi Zhao wrote:

> On Wed, 10 Jun 2015 00:58:41 +0200, Laura Creighton wrote:
> 
>> No promises -- I never used this myself, but it looks like:
>> 
>> https://github.com/killuahzl/pyaria2
>> 
>> parses pyaria files for its own purposes, so you could steal code from
>> there.
>> 
>> Laura
> 
> Thanks, but it doesn't do the same thing that I want.
> 
> Regards

Thanks again, based on your hints, I do a further search on the github.  
And meet the following one:

https://github.com/atmouse-/aria2-trunc/blob/master/aria2_cutfile.py

It seems this one is just do the things that I want. I'll read it further 
for confirmation.

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Irmen de Jong]

On 10-6-2015 1:06, Chris Angelico wrote:
> On Wed, Jun 10, 2015 at 6:07 AM, Devin Jeanpierre
>  wrote:
>> There's a lot of subtle issues with pickle compatibility. e.g.
>> old-style vs new-style classes. It's kinda hard and it's better to
>> give up. I definitely agree it's better to use something else instead.
>> For example, we switched to using protocol buffers, which have much
>> better compatibility properties and are a bit more testable to boot
>> (since text format protobufs are always output in a canonical (sorted)
>> form.)
> 
> Or use JSON, if your data fits within that structure. It's easy to
> read and write, it's human-readable, and it's safe (no chance of
> arbitrary code execution). Forcing yourself to use a format that can
> basically be processed by ast.literal_eval() is a good discipline -
> means you don't accidentally save/load too much.
> 
> ChrisA
> 

I made a specialized serializer for this, which is more expressive than JSON. 
It outputs
python literal expressions that can be directly parsed by ast.literal_eval(). 
You can
find it on pypi (https://pypi.python.org/pypi/serpent).  It's the default 
serializer of
Pyro, and it includes a Java and .NET version as well as an added bonus.


Irmen


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Devin Jeanpierre]

Passing around data that can be put into ast.literal_eval is
synonymous with passing around data taht can be put into eval. It
sounds like a trap.

Other points against JSON / etc.: the lack of schema makes it easier
to stuff anything in there (not as easily as pickle, mind), and by
returning a plain dict, it becomes easier to require a field than to
allow a field to be missing, which is bad for robustness and bad for
data format migrations. (Protobuf (v3) has schemas and gives every
field a default value.)

For human readable serialized data, text format protocol buffers are
seriously underrated. (Relatedly: underdocumented, too.)

/me lifts head out of kool-aid and gasps for air

-- Devin

On Tue, Jun 9, 2015 at 5:17 PM, Irmen de Jong  wrote:
> On 10-6-2015 1:06, Chris Angelico wrote:
>> On Wed, Jun 10, 2015 at 6:07 AM, Devin Jeanpierre
>>  wrote:
>>> There's a lot of subtle issues with pickle compatibility. e.g.
>>> old-style vs new-style classes. It's kinda hard and it's better to
>>> give up. I definitely agree it's better to use something else instead.
>>> For example, we switched to using protocol buffers, which have much
>>> better compatibility properties and are a bit more testable to boot
>>> (since text format protobufs are always output in a canonical (sorted)
>>> form.)
>>
>> Or use JSON, if your data fits within that structure. It's easy to
>> read and write, it's human-readable, and it's safe (no chance of
>> arbitrary code execution). Forcing yourself to use a format that can
>> basically be processed by ast.literal_eval() is a good discipline -
>> means you don't accidentally save/load too much.
>>
>> ChrisA
>>
>
> I made a specialized serializer for this, which is more expressive than JSON. 
> It outputs
> python literal expressions that can be directly parsed by ast.literal_eval(). 
> You can
> find it on pypi (https://pypi.python.org/pypi/serpent).  It's the default 
> serializer of
> Pyro, and it includes a Java and .NET version as well as an added bonus.
>
>
> Irmen
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Chris Angelico]

On Wed, Jun 10, 2015 at 10:47 AM, Devin Jeanpierre
 wrote:
> Passing around data that can be put into ast.literal_eval is
> synonymous with passing around data taht can be put into eval. It
> sounds like a trap.

Except that it's hugely restricted. There are JSON parsing libraries
all over the place, and one of the points in favour of JSON is that it
can be dumped into JavaScript as-is and it evaluates correctly.
Doesn't mean you have to use eval() to parse it, and usually you
won't.

> Other points against JSON / etc.: the lack of schema makes it easier
> to stuff anything in there (not as easily as pickle, mind), and by
> returning a plain dict, it becomes easier to require a field than to
> allow a field to be missing, which is bad for robustness and bad for
> data format migrations. (Protobuf (v3) has schemas and gives every
> field a default value.)

This is true. But there are plenty of cases where you can manage on a
simple dictionary that maps keywords to values that are either
strings, numbers, or lists/dicts of same - and without any schema to
tell you what keywords are valid. It's simple, extensible, and scales
reasonably well to mid-sized use cases. Sure, the biggest cases (and
some of the smaller ones) benefit nicely from schema definitions, but
there's room to manage without.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parser needed.

[Rustom Mody]

On Wednesday, June 10, 2015 at 5:04:17 AM UTC+5:30, Michael Torrie wrote:
> On 06/09/2015 06:20 AM, Skybuck Flying wrote:
> > Euhm...
> > 
> > My parser is already done... since today
> > 
> > Loving it too
> > 
> > Wrote it myself... based on the c# code technique explained somewhere in 
> > this thread too
> 
> I'm glad you're having fun, and making good progress.  And it's good to
> hear of success with Python.  However, I wouldn't really call this a
> "thread;" more a "monologue."  I'm not sure list members really
> appreciate the blow-by-blow description of a project, for the future.  I
> know I find it rather noisy.  I don't think anyone has actually read any
> of your messages (I'm pretty sure some have just blacklisted your
> messages), except to determine you're trying to parse something rather
> vague.  Might I suggest a blog is a more appropriate place to ruminate
> over a project (and posting code and examples) and use the list for
> questions and advice?

+1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Steven D'Aprano]

On Wednesday 10 June 2015 10:47, Devin Jeanpierre wrote:

> Passing around data that can be put into ast.literal_eval is
> synonymous with passing around data taht can be put into eval. It
> sounds like a trap.

In what way?

literal_eval will cleanly and safely refuse to evaluate strings like:

"len(None)"
"100**100**100"
"__import__('os').system('rm this')"


and so on, which makes it significantly safer when given untrusted data. I 
suppose that one might be able to perform a DOS attack by passing it:

"1000 ... 0"

where the ... represents, say, a gigabyte of zeroes, but if an attacker has 
the ability to feed you gigabytes of data, they don't need literal_eval to 
DOS you.

If you can think of an actual attack against literal_eval, please tell us or 
report it, so it can be fixed.


> For human readable serialized data, text format protocol buffers are
> seriously underrated. (Relatedly: underdocumented, too.)

Ironically, literal_eval is designed to process text-format protocols using 
human-readable Python syntax for common data types like int, str, and dict.



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[random832]

On Tue, Jun 9, 2015, at 23:52, Steven D'Aprano wrote:
> > For human readable serialized data, text format protocol buffers are
> > seriously underrated. (Relatedly: underdocumented, too.)
> 
> Ironically, literal_eval is designed to process text-format protocols
> using 
> human-readable Python syntax for common data types like int, str, and
> dict.

"protocol buffers" is the name of a specific tool.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Chris Angelico]

On Wed, Jun 10, 2015 at 1:57 PM,   wrote:
> On Tue, Jun 9, 2015, at 23:52, Steven D'Aprano wrote:
>> > For human readable serialized data, text format protocol buffers are
>> > seriously underrated. (Relatedly: underdocumented, too.)
>>
>> Ironically, literal_eval is designed to process text-format protocols
>> using
>> human-readable Python syntax for common data types like int, str, and
>> dict.
>
> "protocol buffers" is the name of a specific tool.

Yes, it is. But the point is that literal_eval, JSON, and other such
tools are _also_ text-format protocols that serialize to/from human
readable data. I'm not sure what the advantage of protocol buffers is,
but it's not like "human readable" is such a rarity. (It is still a
strike against pickle.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Devin Jeanpierre]

On Tue, Jun 9, 2015 at 8:52 PM, Steven D'Aprano
 wrote:
> On Wednesday 10 June 2015 10:47, Devin Jeanpierre wrote:
>
>> Passing around data that can be put into ast.literal_eval is
>> synonymous with passing around data taht can be put into eval. It
>> sounds like a trap.
>
> In what way?

I misspoke, and instead of "synonymous", meant "also means".
(Implication, not equivalence.)

>> For human readable serialized data, text format protocol buffers are
>> seriously underrated. (Relatedly: underdocumented, too.)
>
> Ironically, literal_eval is designed to process text-format protocols using
> human-readable Python syntax for common data types like int, str, and dict.

"Protocol buffers" are a specific technology, not an abstract concept,
and literal_eval is not a great idea.

* the common serializer (repr) does not output a canonical form, and
  can serialize things in a way that they can't be deserialized
* there is no schema
* there is no well understood migration story for when the data you
  load and store changes
* it is not usable from other programming languages
* it encourages the use of eval when literal_eval becomes inconvenient
  or insufficient
* It is not particularly well specified or documented compared to the
  alternatives.
* The types you get back differ in python 2 vs 3

For most apps, the alternatives are better. Irmen's serpent library is
strictly better on every front, for example. (Except potentially
security, who knows.)

At least it's better than pickle, security wise. Reliability wise,
repr is a black hole, so no dice. :(

-- Devin
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: enhancement request: make py3 read/write py2 pickle format

[Steven D'Aprano]

On Wednesday 10 June 2015 13:57, random...@fastmail.us wrote:

> On Tue, Jun 9, 2015, at 23:52, Steven D'Aprano wrote:
>> > For human readable serialized data, text format protocol buffers are
>> > seriously underrated. (Relatedly: underdocumented, too.)
>> 
>> Ironically, literal_eval is designed to process text-format protocols
>> using
>> human-readable Python syntax for common data types like int, str, and
>> dict.
> 
> "protocol buffers" is the name of a specific tool.

It is? It sounds like a generic term for, you know, a buffer used by a 
protocol. I live and learn.

https://developers.google.com/protocol-buffers/docs/pythontutorial

You have to:

- write a data template, in a separate file; just don't call it a schema, 
because this isn't XML;

- don't forget the technically-optional-but-recommended (and required if you 
use other languages) "package" header, which is completely redundant in 
Python;

- run a separate compiler over that template, which will generate Python 
classes for you; just don't think that these classes are first class 
citizens that you can extend using inheritance, because they're not;

- import the generated module containing those classes;

- and now you have you're very own private pickle-like format, yay!


I'm sure that this has its uses for big, complex projects, but for 
lightweight needs, it seems over-engineered and unPythonic.

-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list