How to quickly search over a large number of files using python?

[dwivedi . devika]

Hi all,

I am a newbie to python.

I have about 500 search queries, and about 52000 files in which I have to find 
all matches for each of the 500 queries.

How should I approach this? Seems like the straightforward way to do it would 
be to loop through each of the files and go line by line comparing all the 
terms to the query, but this seems like it would take too long.

Can someone give me a suggestion as to how to minimize the search time?

Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Possibilities of Building "Stacked Neural Networks"

[Robert Kern]

On 2013-09-24 19:03, Michael Lamport Commons wrote:

Dear Members of this list-serve:

Would it be possible to build “stacked neural networks” like the one 
shown in the attached document?

You may have a few questions about the stacked neural network. For 
example, what is a stacked neural network? What is the difference between 
stacked neural networks and the existing neural network? A brief description is 
provided in the attached document.

Based on this brief description, I would like to know how would one go 
about building such stacked neural networks cheaply and easily?  Is there any 
software available that can do this?  How much would it cost?

Please feel free to contact me if you think that it would be possible 
or easier to apply stacked neural network into a more practical field? 
Suggestions are welcome as well.


The term of art for these kind of architectures is "deep learning" (and 
associated terms like "deep architecture", "deep networks", etc.). It's an 
active field of research that is showing promising preliminary results, and we 
are beginning to see its limits as well. Google and other big machine learning 
players are putting a lot of resources into building these systems.


  http://arxiv.org/pdf/1112.6209v3.pdf

A good resource would be the Deep Learning Tutorial which shows you how to build 
these systems using Theano, a Python package for computing with GPUs, one that 
is particularly well-suited to building deep neural networks.


  http://deeplearning.net/tutorial/

Unfortunately, there is nothing cheap or easy about deep networks. They are 
*very* computationally expensive. You will probably need a small cluster of GPUs 
to solve interesting problems, and training one will probably take a couple of 
days of computation (for the final run, *after* you have debugged your code and 
done the initial experiments to find all of the right hyperparameters for your 
problem).


Good luck!

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
https://mail.python.org/mailman/listinfo/python-list

Re: How to quickly search over a large number of files using python?

[Oscar Benjamin]

On 25 September 2013 09:41,   wrote:
> I am a newbie to python.
>
> I have about 500 search queries, and about 52000 files in which I have to 
> find all matches for each of the 500 queries.
>
> How should I approach this? Seems like the straightforward way to do it would 
> be to loop through each of the files and go line by line comparing all the 
> terms to the query, but this seems like it would take too long.

That would be the obvious way to do this.

> Can someone give me a suggestion as to how to minimize the search time?

What do you mean by a "query"? (Code indicating how a query would
match would be helpful here.)


Oscar
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to quickly search over a large number of files using python?

[Dave Angel]

On 25/9/2013 04:41, dwivedi.dev...@gmail.com wrote:

> Hi all,
>
> I am a newbie to python.
>
> I have about 500 search queries, and about 52000 files in which I have to 
> find all matches for each of the 500 queries.
>
> How should I approach this? Seems like the straightforward way to do it would 
> be to loop through each of the files and go line by line comparing all the 
> terms to the query, but this seems like it would take too long.
>
> Can someone give me a suggestion as to how to minimize the search time?
>

Are these files text or binary?  Are they an 8bit character set or
Unicode?

Without more information about what these "queries" are, it's not even
possible to say whether the above approach could work at all.

Please specify the nature of these queries, and whether all the queries
are of the same form.  For example, it may be that each of the queries
is a simple search string, not containing newline or wildcard.

Or it may be that the queries are arbitrary regular expressions, with
some of them potentially matching a multi-line block of text.

Have you implemented the brute-force approach you describe, and is it
indeed too slow?  By what factor?  Does it take 1000 times as long as
desired, or 5 times?  How about if you do one query for those 52000
files, is it still too slow?  And by what factor?

Assuming each of the queries is independent, and that none of them need
more than one line to process, it might be possible to combine some or
all of those queries into a siimpler filter or filters.  Then one could
speed up the process by applying the filter to each line, and only if
it triggers, to check the line with the individual queries.

You also don't indicate whether this is a one--time query, or whether
the same files might need later to be searched for a different set of
queries, or whether the queries might need to be applied to a
different set of files.  Or whether the same search may need to be
repeated on a very similar set of files, or ...

Even in the most ideally placed set of constraints, some queries may
produce filters that are either harder to apply than the original
queries, or filters that produce so many candidates that this process
takes longer than just applying the queries brute-force.

Many times, optimization efforts focus on the wrong problem, or ignore
the relative costs of programmer time and machine time.  Other times,
the problem being optimized is simply intractiable with current
technology.

-- 
DaveA


-- 
https://mail.python.org/mailman/listinfo/python-list

removing BOM prepended by codecs?

[J. Bagg]

So it is just a random sequence of "junk".

It will be a matter of finding the real start of the record (in this 
case a %) and throwing the "junk" away. I was misled by the note in the 
codecs class that BOMs were being prepended. Should have looked more 
carefully.


Mea culpa.

--
https://mail.python.org/mailman/listinfo/python-list

Re: removing BOM prepended by codecs?

[Dave Angel]

On 25/9/2013 06:38, J. Bagg wrote:

> So it is just a random sequence of "junk".
>
> It will be a matter of finding the real start of the record (in this 
> case a %) and throwing the "junk" away.

Please join the list.  Your present habit of starting a new thread for
each of your messages is getting old.


You still need to find the source of the "junk" if you want anything
approaching reliability.

The open() call you showed in one of the other four threads had a
"append" mode.  Is it possible that you're "creating" a file without
deleting pre-existing junk?


-- 
DaveA


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Possibilities of Building "Stacked Neural Networks"

[William Ray Wing]

On Sep 25, 2013, at 5:43 AM, Robert Kern  wrote:

> On 2013-09-24 19:03, Michael Lamport Commons wrote:
>> Dear Members of this list-serve:
>> 
>>Would it be possible to build “stacked neural networks” like the one 
>> shown in the attached document?
>> 
>>You may have a few questions about the stacked neural network. For 
>> example, what is a stacked neural network? What is the difference between 
>> stacked neural networks and the existing neural network? A brief description 
>> is provided in the attached document.
>> 
>>Based on this brief description, I would like to know how would one 
>> go about building such stacked neural networks cheaply and easily?  Is there 
>> any software available that can do this?  How much would it cost?
>> 
>>Please feel free to contact me if you think that it would be possible 
>> or easier to apply stacked neural network into a more practical field? 
>> Suggestions are welcome as well.
> 
> The term of art for these kind of architectures is "deep learning" (and 
> associated terms like "deep architecture", "deep networks", etc.). It's an 
> active field of research that is showing promising preliminary results, and 
> we are beginning to see its limits as well. Google and other big machine 
> learning players are putting a lot of resources into building these systems.
> 
>  http://arxiv.org/pdf/1112.6209v3.pdf
> 
> A good resource would be the Deep Learning Tutorial which shows you how to 
> build these systems using Theano, a Python package for computing with GPUs, 
> one that is particularly well-suited to building deep neural networks.
> 
>  http://deeplearning.net/tutorial/
> 
> Unfortunately, there is nothing cheap or easy about deep networks. They are 
> *very* computationally expensive. You will probably need a small cluster of 
> GPUs to solve interesting problems, and training one will probably take a 
> couple of days of computation (for the final run, *after* you have debugged 
> your code and done the initial experiments to find all of the right 
> hyperparameters for your problem).
> 
> Good luck!
> 
> -- 
> Robert Kern


The OP might also want to look at Nvidia's CUDO units (which package GPUs into 
massive parallel accelerators - currently well over 2500 GPUs in a single fat 
card) and PyCUDA which makes the CUDA software available to python.

-Bill
-- 
https://mail.python.org/mailman/listinfo/python-list

nexec on Android is ready for Python.

[Tomohiko Sumi]

I announce that nexec[1] is ready for Python.

nexec is a system to transfer system call requests. You can use 
applications in an nexec server as same as those in your local machine.


nexec is ready for Python now. Furthermore, nexec client for Android is 
available. These mean that you can use Python on Android via nexec.


The demo application[2] is good to start nexec on Android. The 
documentation includes the tutorial for Python[3].


[1] http://neko-daisuki.ddo.jp/~SumiTomohiko/nexec/index.html
[2] 
http://neko-daisuki.ddo.jp/~SumiTomohiko/android-nexec-client-demo/index.html
[3] 
http://neko-daisuki.ddo.jp/~SumiTomohiko/android-nexec-client-demo/tutorial/python/index.html

--
Tomohiko Sumi
sumitomoh...@neko-daisuki.ddo.jp
http://neko-daisuki.ddo.jp/~SumiTomohiko/index.html
--
https://mail.python.org/mailman/listinfo/python-list

Re: How to quickly search over a large number of files using python?

[Roy Smith]

In article <60f36178-b584-4fcb-8ad9-2dac6052e...@googlegroups.com>,
 dwivedi.dev...@gmail.com wrote:

> Hi all,
> 
> I am a newbie to python.
> 
> I have about 500 search queries, and about 52000 files in which I have to 
> find all matches for each of the 500 queries.

Before anybody can even begin to answer this question, we need to know 
what you mean by "search query".  Are you talking pattern matching, 
keyword matching, fuzzy hits OK, etc?  Give us a couple of examples of 
the kind of searches you'd like to execute.

Also, is this a one-off thing, or are you planning to do many searches 
over the same collection of files?  If so, you will want to do some sort 
of pre-processing or indexing to speed up the search execution.  It's 
extremely unlikely you want to reinvent the wheel here.  There are tons 
of search packages out there that do this sort of thing.  Just a few to 
check out include Apache Lucene, Apache Solr, and Xapian.
-- 
https://mail.python.org/mailman/listinfo/python-list

Referrer key missing form os.environ dictionary?

[Νίκος]

Hello, i decided am ong other os.environ variables to also grab the 
'HTTP_REFERER' fiel but when i try to run my script i was seeing a 
KeyError complaining that 'HTTP_REFERER' didnt exist.


So, to see what existed in the os.environ dictionary i issues a print( 
os.environ ) to see all available keys and their values:


environ({'SERVER_PORT': '80', 'REQUEST_URI': '/', 
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch', 'SERVER_ADDR': 
'84.200.17.58', 'DOCUMENT_ROOT': '/home/nikos/public_html', 
'HTTP_CONNECTION': 'keep-alive', 'SCRIPT_FILENAME': 
'/home/nikos/public_html/cgi-bin/metrites.py', 'SERVER_NAME': 
'superhost.gr', 'REMOTE_PORT': '58896', 'HTTP_ACCEPT': 
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.2; WOW64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.2 
Safari/537.36', 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8,el;q=0.6', 
'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_HOST': 'superhost.gr', 
'REDIRECT_URL': '/index.html', 'SERVER_PROTOCOL': 'HTTP/1.1', 
'SERVER_SIGNATURE': '
Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips 
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server 
at superhost.gr Port 80
\n', 'HTTP_DNT': '1', 'REQUEST_METHOD': 'GET', 'QUERY_STRING': 
'file=/home/nikos/public_html/index.html', 'PATH': 
'/sbin:/usr/sbin:/bin:/usr/bin', 'HTTP_COOKIE': 'cf_use_ob=0; 
__cfduid=da37079bb377f13e9c50224189ab46ac71379783691866; 
__utma=210786583.1207352568.1379497319.1380099225.1380108255.32; 
__utmc=210786583; 
__utmz=210786583.1379497319.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 
'REMOTE_ADDR': '176.92.73.41', 'REDIRECT_QUERY_STRING': 
'file=/home/nikos/public_html/index.html', 'SERVER_SOFTWARE': 
'Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips 
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635', 
'SERVER_ADMIN': 'webmas...@superhost.gr', 'SCRIPT_NAME': 
'/cgi-bin/metrites.py', 'REDIRECT_STATUS': '200'})



i dont see anywhere a refferer key so to catch it in a variable sting 
like this:


referrer = os.environ['HTTP_REFERER']

Do i miss something? its a suprise to me that the environ dictioanry has 
almost anythign but a referrer key.


I need your help please.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 3:45 μμ, ο/η Νίκος έγραψε:

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a
KeyError complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:

environ({'SERVER_PORT': '80', 'REQUEST_URI': '/',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch', 'SERVER_ADDR':
'84.200.17.58', 'DOCUMENT_ROOT': '/home/nikos/public_html',
'HTTP_CONNECTION': 'keep-alive', 'SCRIPT_FILENAME':
'/home/nikos/public_html/cgi-bin/metrites.py', 'SERVER_NAME':
'superhost.gr', 'REMOTE_PORT': '58896', 'HTTP_ACCEPT':
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.2
Safari/537.36', 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8,el;q=0.6',
'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_HOST': 'superhost.gr',
'REDIRECT_URL': '/index.html', 'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SIGNATURE': '
Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server
at superhost.gr Port 80
\n', 'HTTP_DNT': '1', 'REQUEST_METHOD': 'GET', 'QUERY_STRING':
'file=/home/nikos/public_html/index.html', 'PATH':
'/sbin:/usr/sbin:/bin:/usr/bin', 'HTTP_COOKIE': 'cf_use_ob=0;
__cfduid=da37079bb377f13e9c50224189ab46ac71379783691866;
__utma=210786583.1207352568.1379497319.1380099225.1380108255.32;
__utmc=210786583;
__utmz=210786583.1379497319.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
'REMOTE_ADDR': '176.92.73.41', 'REDIRECT_QUERY_STRING':
'file=/home/nikos/public_html/index.html', 'SERVER_SOFTWARE':
'Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635',
'SERVER_ADMIN': 'webmas...@superhost.gr', 'SCRIPT_NAME':
'/cgi-bin/metrites.py', 'REDIRECT_STATUS': '200'})


i dont see anywhere a refferer key so to catch it in a variable sting
like this:

referrer = os.environ['HTTP_REFERER']

Do i miss something? its a suprise to me that the environ dictioanry has
almost anythign but a referrer key.

I need your help please.


I also tried this:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anythign either.
--
https://mail.python.org/mailman/listinfo/python-list

Re: How to login to a website using Python 3.x?

[Chris “Kwpolska” Warrick]

On Tue, Sep 24, 2013 at 10:39 AM, Mark Lawrence  wrote:
> On 24/09/2013 09:09, Osumo Clement wrote:
>>
>>   Hi. I am new to Python. I am making a script where logging in to a
>> website is the first step.. I am using Python 3.3 All of the help I have
>> seen online uses urllib2 which in Python 3 aint there. I will greatly
>> appreciate any help
>>
>
> urllib2 has been renamed in Python 3 see
> http://www.python.org/dev/peps/pep-3108/#urllib-package

Note that, for sanity, you should use  (a
third-party package) instead.

-- 
Chris “Kwpolska” Warrick 
PGP: 5EAAEA16
stop html mail | always bottom-post | only UTF-8 makes sense
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Chris “Kwpolska” Warrick]

On Wed, Sep 25, 2013 at 2:45 PM, Νίκος  wrote:
> Hello, i decided am ong other os.environ variables to also grab the
> 'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
> complaining that 'HTTP_REFERER' didnt exist.
>
> So, to see what existed in the os.environ dictionary i issues a print(
> os.environ ) to see all available keys and their values:

The Referer header is not mandatory by any means.  Your client
probably does not send it.

-- 
Chris “Kwpolska” Warrick 
PGP: 5EAAEA16
stop html mail | always bottom-post | only UTF-8 makes sense
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:

On Wed, Sep 25, 2013 at 2:45 PM, Νίκος  wrote:

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:


The Referer header is not mandatory by any means.  Your client
probably does not send it.

I would like to check for its existence and retrieve it if possible, if 
its not there then default to the string "UnKnown Ref".


I try to do this with:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anything either.

Can you verify that its the correct way to grab the referral string?
--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:

On Wed, Sep 25, 2013 at 2:45 PM, Νίκος  wrote:

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:


The Referer header is not mandatory by any means.  Your client
probably does not send it.


But one other problem appeared too:

ni...@superhost.gr [~/www/cgi-bin]# python metrites.py
  File "metrites.py", line 27
host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 
'UnKnown Host'

   ^
SyntaxError: invalid syntax


i dont see anything wrong with that line, and the carret is actually 
pointing to the "host".

--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[John Gordon]

In  =?UTF-8?B?zp3Or866zr/Pgg==?= 
 writes:

> referrer = os.environ['HTTP_REFERER']

> Do i miss something? its a suprise to me that the environ dictioanry has 
> almost anythign but a referrer key.

HTTP_REFERER is used to indicate the URL containing the link that led
the user to the current URL.  So for example if the user is viewing
foo.html and then clicks a link on that page which leads to bar.html, the
HTTP_REFERER for that request would be foo.html.

However, if the user did not arrive from another page, then HTTP_REFERER
will be missing.  This happens when the user types the web address directly
into their browser, or clicks on a bookmark, or many other ways.

Also, obviously, it's up to the browser to truthfully report HTTP_REFERER;
the server itself has no idea what page you were on previously.  What
browser are you using?

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Robert Kern]

On 2013-09-25 15:07, Νίκος wrote:

Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:

On Wed, Sep 25, 2013 at 2:45 PM, Νίκος  wrote:

Hello, i decided am ong other os.environ variables to also grab the
'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
complaining that 'HTTP_REFERER' didnt exist.

So, to see what existed in the os.environ dictionary i issues a print(
os.environ ) to see all available keys and their values:


The Referer header is not mandatory by any means.  Your client
probably does not send it.


But one other problem appeared too:

ni...@superhost.gr [~/www/cgi-bin]# python metrites.py
   File "metrites.py", line 27
 host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnKnown 
Host'
^
SyntaxError: invalid syntax


i dont see anything wrong with that line, and the carret is actually pointing to
the "host".


As has been explained to you before, SyntaxErrors just point to the place where 
the parser saw something unexpected, not the exact point where you made the 
error. It is not omniscient. If you don't see the problem on the reported line, 
you have probably left off a closing bracket or something similar in a previous 
line. Look back a few lines.


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[John Gordon]

In  =?UTF-8?B?zp3Or866zr/Pgg==?= 
 writes:

> referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

> but that doesn't return anything either.

When you say it "doesn't return anything", what exactly do you mean?  Does
it return None?  Does it raise KeyError?  Something else?

In any case, I'm surprised that doesn't work.  That should be the correct
way to do it.

You could try this:

try:
referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
except KeyError:
referer = None

if not referer:
referer = 'UnknownRef'

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[John Gordon]

In  =?UTF-8?B?zp3Or866zr/Pgg==?= 
 writes:

> But one other problem appeared too:

> ni...@superhost.gr [~/www/cgi-bin]# python metrites.py
>File "metrites.py", line 27
>  host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 
> 'UnKnown Host'
> ^
> SyntaxError: invalid syntax

> i dont see anything wrong with that line, and the carret is actually 
> pointing to the "host".

Your post has the code broken up into two separate lines.  Is that how
the code actually appears?  If so, then that's the problem.  You can't
arbitrarily break a statement into separate lines.

If not, then you'll have to post more of the code; the problem isn't on
this line.

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Steven D'Aprano]

On Wed, 25 Sep 2013 17:07:38 +0300, Νίκος wrote:

> ni...@superhost.gr [~/www/cgi-bin]# python metrites.py
>File "metrites.py", line 27
>  host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
> 'UnKnown Host'
> ^
> SyntaxError: invalid syntax
> 
> 
> i dont see anything wrong with that line, and the carret is actually
> pointing to the "host".

If the caret is pointing to "host", then the syntax error isn't 
discovered until that point. That means the actual syntax error occurs 
before that point, probably on the previous line. 

How long have you been doing web development with Python? Six months? A 
year? You're not a beginner any more. If you can't solve syntax errors by 
yourself by now, it's probably time to give up and find a job more suited 
to your skills.



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Steven D'Aprano]

On Wed, 25 Sep 2013 17:04:55 +0300, Νίκος wrote:

> I would like to check for its existence and retrieve it if possible, if
> its not there then default to the string "UnKnown Ref".
> 
> I try to do this with:
> 
> referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
> 
> but that doesn't return anything either.
> 
> Can you verify that its the correct way to grab the referral string?


The Referer is not an environment variable. How would your shell know 
what URL you were just browsing?

Have you googled for HTTP Referer? Do you understand what it is?


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Tim Chase]

On 2013-09-25 14:18, John Gordon wrote:
> However, if the user did not arrive from another page, then
> HTTP_REFERER will be missing.  This happens when the user types the
> web address directly into their browser, or clicks on a bookmark,
> or many other ways.
> 
> Also, obviously, it's up to the browser to truthfully report
> HTTP_REFERER;

There are browser plugins that allow blocking or manually-overriding
the outbound refer[r]er header which help mitigate data leakage such
as search-engine query strings or work around website limitations.  So
server-side code should always assume that the HTTP_REFERER header can
be absent or easily be spoofed, treating it as a hint, not absolute
truth.

-tkc





-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 5:52 μμ, ο/η Steven D'Aprano έγραψε:

On Wed, 25 Sep 2013 17:04:55 +0300, Νίκος wrote:


I would like to check for its existence and retrieve it if possible, if
its not there then default to the string "UnKnown Ref".

I try to do this with:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anything either.

Can you verify that its the correct way to grab the referral string?



The Referer is not an environment variable. How would your shell know
what URL you were just browsing?

Have you googled for HTTP Referer? Do you understand what it is?



Yes Steven googleign for 2 hours now.

You were rigth about the carret it was a synatx error above which i 
missed and i was breakign my head to the wall to understand what was wot 
with the 'host' variable. why python doestn detect the exact synxtax 
error and ispoitnign to me to another line making me think the error is 
elsewhere?


This indeed works now:

ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')

but iam wondering why this doesnt work also:

ref = os.environ('HTTP_REFERER')

Shouldnt both work?
--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Steven D'Aprano]

On Wed, 25 Sep 2013 14:26:23 +, John Gordon wrote:

> You could try this:
> 
> try:
> referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
> except KeyError:
> referer = None
> if not referer:
> referer = 'UnknownRef'

The get method will not raise KeyError. If the key is missing, 
'UnknownRef' will be returned instead.

So when Nikos reports that os.environ.get('HTTP_REFERER', 'UnknownRef') 
"doesn't return anything", he is mistaken. It returns 'UnknownRef'.


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Ned Batchelder]

On 9/25/13 10:26 AM, John Gordon wrote:

You could try this:

 try:
 referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
 except KeyError:
 referer = None

 if not referer:
 referer = 'UnknownRef'
There's no need for the "except KeyError" clause.  dict.get never raises 
KeyError, this code will assign 'UnknownRef' in the case the environment 
variable is missing.


--Ned.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Xaxa Urtiz]

Le mercredi 25 septembre 2013 17:02:45 UTC+2, Ferrous Cranus a écrit :
> Στις 25/9/2013 5:52 μμ, ο/η Steven D'Aprano έγραψε:
> 
> > On Wed, 25 Sep 2013 17:04:55 +0300, Νίκος wrote:
> 
> >
> 
> >> I would like to check for its existence and retrieve it if possible, if
> 
> >> its not there then default to the string "UnKnown Ref".
> 
> >>
> 
> >> I try to do this with:
> 
> >>
> 
> >> referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
> 
> >>
> 
> >> but that doesn't return anything either.
> 
> >>
> 
> >> Can you verify that its the correct way to grab the referral string?
> 
> >
> 
> >
> 
> > The Referer is not an environment variable. How would your shell know
> 
> > what URL you were just browsing?
> 
> >
> 
> > Have you googled for HTTP Referer? Do you understand what it is?
> 
> >
> 
> >
> 
> Yes Steven googleign for 2 hours now.
> 
> 
> 
> You were rigth about the carret it was a synatx error above which i 
> 
> missed and i was breakign my head to the wall to understand what was wot 
> 
> with the 'host' variable. why python doestn detect the exact synxtax 
> 
> error and ispoitnign to me to another line making me think the error is 
> 
> elsewhere?
> 
> 
> 
> This indeed works now:
> 
> 
> 
> ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')
> 
> 
> 
> but iam wondering why this doesnt work also:
> 
> 
> 
> ref = os.environ('HTTP_REFERER')
> 
> 
> 
> Shouldnt both work?

http://www.tutorialspoint.com/python/dictionary_get.htm
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Tim Chase]

On 2013-09-25 18:02, Νίκος wrote:
> This indeed works now:
> 
> ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')
> 
> but iam wondering why this doesnt work also:
> 
> ref = os.environ('HTTP_REFERER')
> 
> Shouldnt both work?

No...that calls os.environ.  You likely *mean*

  ref = os.environ['HTTP_REFERER']

using square brackets.  However, as previously noted by multiple
respondents, this header is not guaranteed to be sent by the browser
for a variety of reasons, so it may not be in the environment
dictionary at all. Thus you are likely to get KeyError exceptions if
you don't use .get() unless you wrap it in a check:

  if "HTTP_REFERER" in os.environ:
ref = os.environ["HTTP_REFERER"]
  else:
deal_with_this_situation()

-tkc









-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 6:04 μμ, ο/η Steven D'Aprano έγραψε:

On Wed, 25 Sep 2013 14:26:23 +, John Gordon wrote:


You could try this:

 try:
 referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
 except KeyError:
 referer = None
 if not referer:
 referer = 'UnknownRef'


The get method will not raise KeyError. If the key is missing,
'UnknownRef' will be returned instead.

So when Nikos reports that os.environ.get('HTTP_REFERER', 'UnknownRef')
"doesn't return anything", he is mistaken. It returns 'UnknownRef'.



Not actually Steven,

the referer key works well but it was the simlutaneous "host" issues 
that was giving me false understanding. now it works ok?


So you say if the key is missign the get method wotn return an error but 
the default string instead. That is nice to hear.


But answer me plz this:

how caom the http_referer thing works ok now but when i just print all 
the key listing of .os.environ ket the http_referer key isnt inside?



--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Grant Edwards]

On 2013-09-25, Steven D'Aprano  wrote:
> On Wed, 25 Sep 2013 17:04:55 +0300, ?? wrote:
>
>> I would like to check for its existence and retrieve it if possible, if
>> its not there then default to the string "UnKnown Ref".
>> 
>> I try to do this with:
>> 
>> referer = os.environ.get('HTTP_REFERER', 'UnknownRef')
>> 
>> but that doesn't return anything either.

And of course that's bolloks:

   Python 2.7.5 (default, Aug 29 2013, 15:13:35) 
   [GCC 4.6.3] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import os
   >>> os.environ.get('HTTP_REFERER','UnkownRef')
   'UnkownRef'
   >>> 

   Python 3.2.5 (default, Aug 29 2013, 15:19:46) 
   [GCC 4.6.3] on linux2
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import os
   >>> os.environ.get('HTTP_REFERER','UnkownRef')
   'UnkownRef'
   >>> 

>> Can you verify that its the correct way to grab the referral string?

It is.

> The Referer is not an environment variable.

It is when you're writing a CGI app.

> How would your shell know what URL you were just browsing?

Because the HTTP server sets those environment variables before
invoking the CGI app.

-- 
Grant Edwards   grant.b.edwardsYow! Boy, am I glad it's
  at   only 1971...
  gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 6:14 μμ, ο/η Tim Chase έγραψε:

On 2013-09-25 18:02, Νίκος wrote:

This indeed works now:

ref = os.environ.get('HTTP_REFERER', 'Άγνωστο Ref')

but iam wondering why this doesnt work also:

ref = os.environ('HTTP_REFERER')

Shouldnt both work?


No...that calls os.environ.  You likely *mean*

   ref = os.environ['HTTP_REFERER']

using square brackets.  However, as previously noted by multiple
respondents, this header is not guaranteed to be sent by the browser
for a variety of reasons, so it may not be in the environment
dictionary at all. Thus you are likely to get KeyError exceptions if
you don't use .get() unless you wrap it in a check:

   if "HTTP_REFERER" in os.environ:
 ref = os.environ["HTTP_REFERER"]
   else:
 deal_with_this_situation()


Thank you very much for the clarification.
I just wanted to know the differecne between the 2 ways of using the 
HTTP_REFERER.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Tim Chase]

On 2013-09-25 18:16, Νίκος wrote:
> how caom the http_referer thing works ok now but when i just print
> all the key listing of .os.environ ket the http_referer key isnt
> inside?

Well, first off, it's entirely possible (based on reading that
paragraph) that you typed something wrong.

That said, it depends on whether you're looking in your local
environment, or the CGI environment provided by the server.  As a
simple example of what the server-side has/puts in your environment,
you can use this CGI script:


#!/usr/bin/env python
import cgitb; cgitb.enable()
import cgi
import os
import sys
print "Content-Type: text/html\n\n"
print """

 
  Test CGI
  dt{font-weight: bold;}
 
 
 """
for k,v in sorted(os.environ.items()):
print "%s" % cgi.escape(k)
print "%s" % cgi.escape(v)
print """
 
 
"""

Note that, if you go directly to the page, you shouldn't have a
refer[r]er header, while if you arrive at it from some previous page,
you might (modulo the aforementioned chicanery that plugins can
induce)

-tkc







-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Νίκος]

Στις 25/9/2013 6:18 μμ, ο/η Grant Edwards έγραψε:

On 2013-09-25, Steven D'Aprano  wrote:

On Wed, 25 Sep 2013 17:04:55 +0300, ?? wrote:


I would like to check for its existence and retrieve it if possible, if
its not there then default to the string "UnKnown Ref".

I try to do this with:

referer = os.environ.get('HTTP_REFERER', 'UnknownRef')

but that doesn't return anything either.


And of course that's bolloks:

Python 2.7.5 (default, Aug 29 2013, 15:13:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ.get('HTTP_REFERER','UnkownRef')
'UnkownRef'
>>>

Python 3.2.5 (default, Aug 29 2013, 15:19:46)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ.get('HTTP_REFERER','UnkownRef')
'UnkownRef'
>>>


Can you verify that its the correct way to grab the referral string?


It is.


The Referer is not an environment variable.


It is when you're writing a CGI app.


How would your shell know what URL you were just browsing?


Because the HTTP server sets those environment variables before
invoking the CGI app.

So you mean that even if i run it via shell this stement will  also work 
because it happens to be in the same enviroment with the HTTP server?


No need to run it via web browser and check the Apache's error log?
--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[John Gordon]

In  =?UTF-8?B?zp3Or866zr/Pgg==?= 
 writes:

> So you mean that even if i run it via shell this stement will  also work 
> because it happens to be in the same enviroment with the HTTP server?

No.  Each process has its own environment.  Your shell's environment
knows nothing about the web server's environment.

-- 
John Gordon   A is for Amy, who fell down the stairs
gor...@panix.com  B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

-- 
https://mail.python.org/mailman/listinfo/python-list

Newline interpretation issue with MIMEApplication with binary data, Python 3.3.2

[nilsbunger]

Hi, 

I'm having trouble encoding a MIME message with a binary file.  Newline 
characters are being interpreted even though the content is supposed to be 
binary. This is using Python 3.3.2

Small test case:

app = MIMEApplication(b'Q\x0dQ', _encoder=encode_noop)
b = io.BytesIO()
g = BytesGenerator(b)
g.flatten(app)
for i in b.getvalue()[-3:]:
print ("%02x " % i, end="")
print ()

This prints 51 0a 51,  meaning the 0x0d character got reinterpreted as a 
newline. 

I've tried setting an email policy of HTTP policy, but that goes even further, 
converting \r to \r\n

This is for HTTP transport, so binary encoding is normal.

Any thoughts how I can do this properly?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Grant Edwards]

On 2013-09-25, ??  wrote:
>  25/9/2013 6:18 , ??/?? Grant Edwards :

>>> The Referer is not an environment variable.
>>
>> It is when you're writing a CGI app.
>>
>>> How would your shell know what URL you were just browsing?
>>
>> Because the HTTP server sets those environment variables before
>> invoking the CGI app.
>
> So you mean that even if i run it via shell this stement will also
> work

If the shell was started by the HTTP server, yes.  If you logged in
normally, no.

> because it happens to be in the same enviroment with the HTTP server?

The shell will only have that environment if the shell was run by the
HTTP server.

> No need to run it via web browser and check the Apache's error log?

You can set the environemnt variables appropriately in the shell and
then invoke a CGI application directly for testing purposes.

-- 
Grant Edwards   grant.b.edwardsYow! I was born in a
  at   Hostess Cupcake factory
  gmail.combefore the sexual
   revolution!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Piet van Oostrum]

Νίκος  writes:

> Στις 25/9/2013 5:01 μμ, ο/η Chris “Kwpolska” Warrick έγραψε:
>> On Wed, Sep 25, 2013 at 2:45 PM, Νίκος  wrote:
>>> Hello, i decided am ong other os.environ variables to also grab the
>>> 'HTTP_REFERER' fiel but when i try to run my script i was seeing a KeyError
>>> complaining that 'HTTP_REFERER' didnt exist.
>>>
>>> So, to see what existed in the os.environ dictionary i issues a print(
>>> os.environ ) to see all available keys and their values:
>>
>> The Referer header is not mandatory by any means.  Your client
>> probably does not send it.
>>
> But one other problem appeared too:
>
> ni...@superhost.gr [~/www/cgi-bin]# python metrites.py
>   File "metrites.py", line 27
> host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or
> UnKnown Host'
>^
> SyntaxError: invalid syntax
>
>
> i dont see anything wrong with that line, and the carret is actually
> pointing to the "host".

There is an apostrophe (') missing before UnKnown.

  host = socket.gethostbyaddr( os.environ['REMOTE_ADDR'] )[0] or 'UnKnown Host'
-- 
Piet van Oostrum 
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]
-- 
https://mail.python.org/mailman/listinfo/python-list

Extracting lines from text files - script with a couple of 'side effects'

[mstagliamonte]

Dear All,

Here I am, with another newbie question. I am trying to extract some lines from 
a fasta (text) file which match the headers in another file. i.e:
Fasta file:
>header1|info1:info2_info3
general text
>header2|info1:info2_info3
general text

headers file:
header1|info1:info2_info3
header2|info1:info2_info3

I want to create a third file, similar to the first one, but only containing 
headers and text of what is listed in the second file. Also, I want to print 
out how many headers were actually found from the second file to match the 
first.

I have done a script which seems to work, but with a couple of 'side effects'
Here is my script:
---
import re
class Extractor():

def __init__(self,headers_file, fasta_file,output_file):
with open(headers_file,'r') as inp0:
counter0=0
container=''
inp0_bis=inp0.read().split('\n')
for x in inp0_bis:
container+=x.replace(':','_').replace('|','_')
with open(fasta_file,'r') as inp1:
inp1_bis=inp1.read().split('>')
for i in inp1_bis:
i_bis= i.split('\n')   
match = 
re.search(i_bis[0].replace(':','_').replace('|','_'),container)
if match:
counter0+=1
with open(output_file,'at') as out0:
out0.write('>'+i)
 print '{} sequences were found'.format(counter0)

---
Side effects:
1) The very first header is written as >>header1 rather than >header1
2) the number of sequences found is 1 more than the ones actually found!

Have you got any thoughts about causes/solutions?

Thanks for your time!
P.S.: I think I have removed the double posting... not sure...
Max
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extracting lines from text files - script with a couple of 'side effects'

[Dave Angel]

On 25/9/2013 16:06, mstagliamonte wrote:

> Dear All,
>
> Here I am, with another newbie question. I am trying to extract some lines 
> from a fasta (text) file which match the headers in another file. i.e:
> Fasta file:
>>header1|info1:info2_info3
> general text
>>header2|info1:info2_info3
> general text
>
> headers file:
> header1|info1:info2_info3
> header2|info1:info2_info3
>
> I want to create a third file, similar to the first one, but only containing 
> headers and text of what is listed in the second file. Also, I want to print 
> out how many headers were actually found from the second file to match the 
> first.
>
> I have done a script which seems to work, but with a couple of 'side effects'
> Here is my script:
> ---
> import re
> class Extractor():
> 
> def __init__(self,headers_file, fasta_file,output_file):
> with open(headers_file,'r') as inp0:
> counter0=0
> container=''
> inp0_bis=inp0.read().split('\n')
> for x in inp0_bis:
> container+=x.replace(':','_').replace('|','_')
> with open(fasta_file,'r') as inp1:
> inp1_bis=inp1.read().split('>')
> for i in inp1_bis:
> i_bis= i.split('\n')  
>  
> match = 
> re.search(i_bis[0].replace(':','_').replace('|','_'),container)
> if match:
> counter0+=1
> with open(output_file,'at') as out0:
> out0.write('>'+i)
>  print '{} sequences were found'.format(counter0)
>
> ---
> Side effects:
> 1) The very first header is written as >>header1 rather than >header1
> 2) the number of sequences found is 1 more than the ones actually found!
>
> Have you got any thoughts about causes/solutions?
>

The cause is the same.  The first line in the file starts with a "<",
and you're splitting on the same.  So the first item of inp1_bis is the
empty string.  That string is certainly contained within container, so
it matches, and produces a result of ">"

You can "fix" this by adding a line after the "for i in inp1_bis" line
if not i: continue

But it also seems to me you're doing the search backwards.  if the Fasta
file has a line like: >der

it would be considered a match!  Seems to me you'd want to only match
lines which contain an entire header.


-- 
DaveA


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Denis McMahon]

On Wed, 25 Sep 2013 18:16:39 +0300, Νίκος wrote:

> how caom the http_referer thing works ok now but when i just print all
> the key listing of .os.environ ket the http_referer key isnt inside?

That you do not understand this is caused by your failure to understand 
the HTTP protocol. You have been told before, this NG is not networking 
and / or tcp/ip and / or internet protocols 101.

Please go and learn about the network protocols you're trying to interact 
with before you start writing code that interacts with them. The issue 
you are raising here is not a python issue, it is a network protocols 
issue that has nothing whatsoever to do with python.

-- 
Denis McMahon, denismfmcma...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extracting lines from text files - script with a couple of 'side effects'

[MRAB]

On 25/09/2013 21:06, mstagliamonte wrote:

Dear All,

Here I am, with another newbie question. I am trying to extract some lines from 
a fasta (text) file which match the headers in another file. i.e:
Fasta file:

header1|info1:info2_info3

general text

header2|info1:info2_info3

general text

headers file:
header1|info1:info2_info3
header2|info1:info2_info3

I want to create a third file, similar to the first one, but only containing 
headers and text of what is listed in the second file. Also, I want to print 
out how many headers were actually found from the second file to match the 
first.

I have done a script which seems to work, but with a couple of 'side effects'
Here is my script:
---
import re
class Extractor():

 def __init__(self,headers_file, fasta_file,output_file):
 with open(headers_file,'r') as inp0:
 counter0=0
 container=''
 inp0_bis=inp0.read().split('\n')
 for x in inp0_bis:
 container+=x.replace(':','_').replace('|','_')
 with open(fasta_file,'r') as inp1:
 inp1_bis=inp1.read().split('>')
 for i in inp1_bis:
 i_bis= i.split('\n')
 match = 
re.search(i_bis[0].replace(':','_').replace('|','_'),container)
 if match:
 counter0+=1
 with open(output_file,'at') as out0:
 out0.write('>'+i)
  print '{} sequences were found'.format(counter0)

---
Side effects:
1) The very first header is written as >>header1 rather than >header1
2) the number of sequences found is 1 more than the ones actually found!

Have you got any thoughts about causes/solutions?

Thanks for your time!


Here's my version:

class Extractor():
def __init__(self, headers_file, fasta_file, output_file):
with open(headers_file) as inp:
headers = set('>' + line for line in inp)

counter = 0
accept = False

with open(fasta_file) as inp, open(output_file, 'w') as out:
for line in inp:
if line.startswith('>'):
accept = line in headers
if accept:
counter += 1

if accept:
out.write(line)

print '{} sequences were found'.format(counter)

--
https://mail.python.org/mailman/listinfo/python-list

Convert namedtuple to dictionary

[tripsvt]

Need suggestions. 

Say, I have a namedtuple like this:

{'a': brucelee(x=123, y=321), 'b': brucelee('x'=123, 'y'=321)

I need to convert it to:

{'a': {'x':123, 'y': 321},'b': {'x':123, 'y': 321}}

Follow-up question --

Which would be easier to work with if I had to later extract/manipulate the 
'x', 'y' values? The format (dicts) above or a list of values like this:

{'a': ['x':123, 'y': 321],'b': ['x':123, 'y': 321]}
-- 
https://mail.python.org/mailman/listinfo/python-list

Understanding how is a function evaluated using recursion

[Arturo B]

Hi, I'm doing Python exercises and I need to write a function to flat nested 
lists
as this one: 

[[1,2,3],4,5,[6,[7,8]]]

To the result:

[1,2,3,4,5,6,7,8]

So I searched for example code and I found this one that uses recursion (that I 
don't understand):

def flatten(l):
ret = []
for i in l:
if isinstance(i, list) or isinstance(i, tuple):
ret.extend(flatten(i)) #How is flatten(i) evaluated?
else:
ret.append(i)
return ret

So I know what recursion is, but I don't know how is 

   flatten(i)
 
evaluated, what value does it returns?

Thank you

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[Tim Chase]

On 2013-09-25 15:45, trip...@gmail.com wrote:
> Say, I have a namedtuple like this:
> 
> {'a': brucelee(x=123, y=321), 'b': brucelee('x'=123, 'y'=321)
> 
> I need to convert it to:
> 
> {'a': {'x':123, 'y': 321},'b': {'x':123, 'y': 321}}

While it uses the "private" member-variable "_fields", you can do

>>> brucelee = namedtuple("brucelee", "x y")
>>> d = {'a': brucelee(x=123,y=321), 'b': brucelee(x=234,y=432)}
>>> dict((k, dict((s, getattr(v, s)) for s in v._fields)) for k,v in
>>> d.iteritems())
{'a': {'y': 321, 'x': 123}, 'b': {'y': 432, 'x': 234}}

which can be made a bit more readable with a helper function:

>>> def dictify(some_named_tuple):
... return dict((s, getattr(some_named_tuple, s)) for s in 
some_named_tuple._fields)
... 
>>> dict((k, dictify(v)) for k,v in d.iteritems())
{'a': {'y': 321, 'x': 123}, 'b': {'y': 432, 'x': 234}}

This would also make it easier to change/choose in the event
"_fields" ever changes.

-tkc




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[MRAB]

On 25/09/2013 23:45, trip...@gmail.com wrote:

Need suggestions.

Say, I have a namedtuple like this:

{'a': brucelee(x=123, y=321), 'b': brucelee('x'=123, 'y'=321)


I assume you mean:

{'a': brucelee(x=123, y=321), 'b': brucelee(x=123, y=321)}



I need to convert it to:

{'a': {'x':123, 'y': 321},'b': {'x':123, 'y': 321}}


You can get the field names using ._fields and the values by using
list(...):

>>> n = brucelee(x=123, y=321)
>>> n._fields
('x', 'y')
>>> list(n)
[123, 321]

Zip then together and pass the result to dict:

>>> dict(zip(n._fields, list(n)))
{'x': 123, 'y': 321}

And, finally, putting that in a dict comprehension:

>>> n = {'a': brucelee(x=123, y=321), 'b': brucelee(x=123, y=321)}
>>> {k: dict(zip(v._fields, list(v))) for k, v in n.items()}
{'a': {'x': 123, 'y': 321}, 'b': {'x': 123, 'y': 321}}


Follow-up question --

Which would be easier to work with if I had to later extract/manipulate the 
'x', 'y' values? The format (dicts) above or a list of values like this:

{'a': ['x':123, 'y': 321],'b': ['x':123, 'y': 321]}


That's not valid Python!

--
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding how is a function evaluated using recursion

[Josh English]

On Wednesday, September 25, 2013 4:24:22 PM UTC-7, Arturo B wrote:
> Hi, I'm doing Python exercises and I need to write a function to flat nested 
> lists


> So I know what recursion is, but I don't know how is 
>
>flatten(i)
>  
> evaluated, what value does it returns?
> 

In this case, flatten always returns a list. When it hits the recursion, it 
calls itself to get another list, that it uses to extend the current list.

Josh

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding how is a function evaluated using recursion

[MRAB]

On 26/09/2013 00:24, Arturo B wrote:

Hi, I'm doing Python exercises and I need to write a function to flat nested 
lists
as this one:

[[1,2,3],4,5,[6,[7,8]]]

To the result:

[1,2,3,4,5,6,7,8]

So I searched for example code and I found this one that uses recursion (that I 
don't understand):

def flatten(l):
 ret = []
 for i in l:
 if isinstance(i, list) or isinstance(i, tuple):
 ret.extend(flatten(i)) #How is flatten(i) evaluated?
 else:
 ret.append(i)
 return ret

So I know what recursion is, but I don't know how is

flatten(i)

evaluated, what value does it returns?


Try a simpler version first:

def flatten(l):
ret = []
for i in l:
if isinstance(i, list) or isinstance(i, tuple):
# Append the contents of the item.
ret.extend(i)
else:
# Append the item itself.
ret.append(i)
return ret

In this example, flatten([[1,2,3],4,5,[6,[7,8]]]) returns [1,2,3,4,5,6,
[7,8]].

The problem here is that a sublist can itself contain a list.

It would be nice if there were a function which, when given [6,[7,8]],
would return [6,7,8] so that you could append those items.

But that's exactly what flatten does!

Try adding prints to tell you what was passed in and what is returned.

--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Steven D'Aprano]

On Wed, 25 Sep 2013 15:18:41 +, Grant Edwards wrote:

>> The Referer is not an environment variable.
> 
> It is when you're writing a CGI app.
> 
>> How would your shell know what URL you were just browsing?
> 
> Because the HTTP server sets those environment variables before invoking
> the CGI app.


I stand corrected.


That's a pretty shitty design though, isn't it? Communicating via 
environment variables. What is this, 1998? :-)

Mind you, I'm not sure what other alternatives exist.


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding how is a function evaluated using recursion

[Steven D'Aprano]

On Wed, 25 Sep 2013 16:24:22 -0700, Arturo B wrote:

> Hi, I'm doing Python exercises and I need to write a function to flat
> nested lists as this one:
> 
> [[1,2,3],4,5,[6,[7,8]]]
> 
> To the result:
> 
> [1,2,3,4,5,6,7,8]
> 
> So I searched for example code and I found this one that uses recursion
> (that I don't understand):
> 
> def flatten(l):
> ret = []
> for i in l:
> if isinstance(i, list) or isinstance(i, tuple):
> ret.extend(flatten(i)) #How is flatten(i) evaluated?
> else:
> ret.append(i)
> return ret
> 
> So I know what recursion is, but I don't know how is
> 
>flatten(i)
>  
> evaluated, what value does it returns?

You have the source code to flatten right there in front of you. The very 
last line says:

return ret


The first line of the function sets:

ret = []

and it is a list which accumulates all the items seen. If you follow the 
code with a simple non-nested list:

flatten([1, 2, 3])

flatten sets the return result "ret" to the empty list, then walks the 
argument list extracting each item in turn, which results in this 
sequence of calls:

ret = []
ret.append(1)
ret.append(2)
ret.append(3)
return ret  # [1, 2, 3, 4]


Now if we do the same thing with a nested list:

flatten([1, [2, 3], 4])

the call to flatten does the same thing, walks the input list, only this 
time it has a recursive call:

# outer call
ret = []
ret.append(1)

At this point, the item found is a list, so flatten([2, 3]) is called, so 
Python drops into into a new call, building a new list called "ret",  
leaving the old one untouched:

# inner call
ret = []
ret.append(2)
ret.append(3)
return ret  # [2, 3]

At this point Python drops back to the first call again:

# outer call
ret.extend([2, 3])
ret.append(4)
return ret  # [1, 2, 3, 4]

But it all together in one chunk, without the commentary:



flatten([1, [2, 3], 4]) => 
# outer call
ret = []
ret.append(1)
flatten([2, 3]) =>
# inner call
ret = []
ret.append(2)
ret.append(3)
return ret  # [2, 3]
# outer call
ret.extend([2, 3])
ret.append(4)
return ret  # [1, 2, 3, 4]


In principle, you can nest as many function calls as needed. In practice, 
Python will stop after 1000 nested calls by default, although you can 
tune it up and down.


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[Ned Batchelder]

On 9/25/13 6:45 PM, trip...@gmail.com wrote:

Need suggestions.

Say, I have a namedtuple like this:

{'a': brucelee(x=123, y=321), 'b': brucelee('x'=123, 'y'=321)

I need to convert it to:

{'a': {'x':123, 'y': 321},'b': {'x':123, 'y': 321}}


Namedtuples have a ._asdict() method. You can convert your dictionary of 
namedtuples (let's call it tupledict) like this:


dictdict = { k: nt._asdict() for k, nt in tupledict }

--Ned.

Follow-up question --

Which would be easier to work with if I had to later extract/manipulate the 
'x', 'y' values? The format (dicts) above or a list of values like this:

{'a': ['x':123, 'y': 321],'b': ['x':123, 'y': 321]}


--
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding how is a function evaluated using recursion

[Dave Angel]

On 25/9/2013 19:24, Arturo B wrote:

> Hi, I'm doing Python exercises and I need to write a function to flat nested 
> lists
> as this one: 
>
> [[1,2,3],4,5,[6,[7,8]]]
>
> To the result:
>
> [1,2,3,4,5,6,7,8]
>
> So I searched for example code and I found this one that uses recursion (that 
> I don't understand):
>
> def flatten(l):
> ret = []
> for i in l:

I can't imagine why you'd use either i or l in this context, but
especially use them both when they look so much alike.

> if isinstance(i, list) or isinstance(i, tuple):
> ret.extend(flatten(i)) #How is flatten(i) evaluated?
> else:
> ret.append(i)
> return ret
>
> So I know what recursion is, but I don't know how is 
>
>flatten(i)
>  
> evaluated, what value does it returns?
>

flatten() returns a list, of course.  The value of 'ret' in the inner
function.

What don't you understand about recursion?  You write a function that's
valid for the simple case (a simple list, with none of the elements
being lists or tuples).  Then you use that function inside itself to
handle the more complex cases.


-- 
DaveA


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[Steven D'Aprano]

On Wed, 25 Sep 2013 18:41:25 -0500, Tim Chase wrote about namedtuple:

> While it uses the "private" member-variable "_fields", you can do

It's not actually private!

namedtuple is documented as an exception to the rule that methods 
starting with a single leading underscore are private. Named tuples 
define three public methods and one data attribute. In order to avoid 
clashing with field names, they start with a single underscore, but they 
are documented as public:

_make
_asdict
_replace
_fields


http://docs.python.org/2/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields



-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding how is a function evaluated using recursion

[Terry Reedy]

On 9/25/2013 7:24 PM, Arturo B wrote:

Hi, I'm doing Python exercises and I need to write a function to flat nested 
lists
as this one:

[[1,2,3],4,5,[6,[7,8]]]

To the result:

[1,2,3,4,5,6,7,8]

So I searched for example code and I found this one that uses recursion (that I 
don't understand):

def flatten(l):
 ret = []
 for i in l:
 if isinstance(i, list) or isinstance(i, tuple):
 ret.extend(flatten(i)) #How is flatten(i) evaluated?
 else:
 ret.append(i)
 return ret

So I know what recursion is, but I don't know how is

flatten(i)

evaluated, what value does it returns?


It is not clear what part of 'how' you do not understand this. Perhaps 
that fact that a new execution frame with a new set of locals is created 
for each call.  So calling flatten from flatten is no different than 
call flatten from anywhere else.


If a language creates just one execution frame for the function, 
attached to the function (as with original Fortran, for instance), then 
recursion is not allowed as a 2nd call would interfere with the use of 
the locals by the 1st call, etc.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[MRAB]

On 26/09/2013 02:08, Steven D'Aprano wrote:

On Wed, 25 Sep 2013 18:41:25 -0500, Tim Chase wrote about namedtuple:


While it uses the "private" member-variable "_fields", you can do


It's not actually private!

namedtuple is documented as an exception to the rule that methods
starting with a single leading underscore are private. Named tuples
define three public methods and one data attribute. In order to avoid
clashing with field names, they start with a single underscore, but they
are documented as public:

_make
_asdict
_replace
_fields


http://docs.python.org/2/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields


Wouldn't it have made more sense to have a trailing underscore instead?

--
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Terry Reedy]

On 9/25/2013 8:11 PM, Steven D'Aprano wrote:

On Wed, 25 Sep 2013 15:18:41 +, Grant Edwards wrote:


The Referer is not an environment variable.


It is when you're writing a CGI app.


How would your shell know what URL you were just browsing?


Because the HTTP server sets those environment variables before invoking
the CGI app.



I stand corrected.


That's a pretty shitty design though, isn't it? Communicating via
environment variables. What is this, 1998? :-)


1993 https://en.wikipedia.org/wiki/Common_Gateway_Interface


Mind you, I'm not sure what other alternatives exist.


Send a series of lines with the same info over an input channel, as was 
done 3 years later for FastCGI. Since CGI uses stdout for the finished 
product, it could have used stdin for the input. Using a serial channel 
does put more burden on the page server to parse the input. But is 
allows it to be on a different machine.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[Terry Reedy]

On 9/25/2013 9:15 PM, MRAB wrote:

On 26/09/2013 02:08, Steven D'Aprano wrote:

On Wed, 25 Sep 2013 18:41:25 -0500, Tim Chase wrote about namedtuple:


While it uses the "private" member-variable "_fields", you can do


It's not actually private!

namedtuple is documented as an exception to the rule that methods
starting with a single leading underscore are private. Named tuples
define three public methods and one data attribute. In order to avoid
clashing with field names, they start with a single underscore, but they
are documented as public:

_make
_asdict
_replace
_fields


http://docs.python.org/2/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields



Wouldn't it have made more sense to have a trailing underscore instead?


Users might use such to avoid clashes with keywords and builtins: 
'print_', 'id_', 'as_', and will certainly sometimes use embedded _.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Nosetests

[melwin9]

Hello,

I am trying to write a few tests for a random input number game but not too 
sure how to proceed on.

I am following the Python Game from http://inventwithpython.com/chapter4.html

Started the tests with a file test_guess.py

from unittest import TestCase
import pexpect as pe

import guess as g

class GuessTest(TestCase):
def setUp(self):
self.intro = 'I have chosen a number from 1-10'
self.request = 'Guess a number: '
self.responseHigh = "That's too high."
self.responseLow  = "That's too low."
self.responseCorrect = "That's right!"
self.goodbye = 'Goodbye and thanks for playing!'

def test_main(self):
#cannot execute main now because it will
#require user input
from guess import main

def test_guessing_hi_low_4(self):
# Conversation assuming number is 4
child = pe.spawn('python guess.py')
child.expect(self.intro,timeout=5)
child.expect(self.request,timeout=5)
child.sendline('5')
child.expect(self.responseHigh,timeout=5)
child.sendline('3')
child.expect(self.responseLow,timeout=5)
child.sendline('4')
child.expect(self.responseCorrect,timeout=5)
child.expect(self.goodbye,timeout=5)

def test_guessing_low_hi_4(self):
# Conversation assuming number is 4
child = pe.spawn('python guess.py')
child.expect(self.intro,timeout=5)
child.expect(self.request,timeout=5)
child.sendline('3')
child.expect(self.responseLow,timeout=5)
child.sendline('5')
child.expect(self.responseHigh,timeout=5)
child.sendline('4')
child.expect(self.responseCorrect,timeout=5)
child.expect(self.goodbye,timeout=5)

and the guess.py file with

intro = 'I have chosen a number from 1-10'
request = 'Guess a number: '
responseHigh = "That's too high."
responseLow  = "That's too low."
responseCorrect = "That's right!"
goodbye = 'Goodbye and thanks for playing!'


def main():
print(intro)
user_input = raw_input(request)
print(responseHigh)
print(request)
user_input = raw_input(request)
print(responseLow)
user_input = raw_input(request)
print(responseCorrect)
print(goodbye)

if __name__ == '__main__':
main()

Not sure How to proceed on with writing a few more tests with if statement to 
test if the value is low or high. I was told to try a command line switch like 
optparse to pass the number but not sure how to do that either. 

Somewhat of a new person with Python, any guidance or assistance would be 
appreciated.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Referrer key missing form os.environ dictionary?

[Chris Angelico]

On Thu, Sep 26, 2013 at 11:32 AM, Terry Reedy  wrote:
> Since CGI uses stdout for the finished product, it could have used stdin for
> the input.

Haven't used CGI in years, but I thought POST data came on stdin?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[Steven D'Aprano]

On Wed, 25 Sep 2013 15:45:49 -0700, tripsvt wrote:

> Need suggestions.
> 
> Say, I have a namedtuple like this:
> 
> {'a': brucelee(x=123, y=321), 'b': brucelee('x'=123, 'y'=321)


That's not a namedtuple, that's a dict containing two namedtuples.


> I need to convert it to:
> 
> {'a': {'x':123, 'y': 321},'b': {'x':123, 'y': 321}}


Why bother? But if you must:

for key, value in some_dict.items():
some_dict[key] = value._asdict()

ought to work.


> Follow-up question --
> 
> Which would be easier to work with if I had to later extract/manipulate
> the 'x', 'y' values? The format (dicts) above or a list of values like
> this:
> 
> {'a': ['x':123, 'y': 321],'b': ['x':123, 'y': 321]}

That's not legal Python.

The answer depends on what you mean by "extract/manipulate" the x and y 
fields. If all you are doing is looking them up, then a namedtuple is 
easiest, since that's what you've already got:

for value in some_dict.values():
print(value.x)
print(value.y)



But if you need to change those values, then a namedtuple is no good 
because it is immutable. In that case, you can either create a new 
namedtuple, or just use the dict-of-dicts version.



# Untested
for key, value in some_dict.items():
kind = type(value)  # what sort of namedtuple is it?
new = kind(value.x+1, value.y+2)
some_dict[key] = new


-- 
Steven
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Nosetests

[Roy Smith]

In article <32f7bdcb-97e5-4a1c-a2c8-ab91e4052...@googlegroups.com>,
 melw...@gmail.com wrote:

> Not sure How to proceed on with writing a few more tests with if statement to 
> test if the value is low or high. I was told to try a command line switch 
> like optparse to pass the number but not sure how to do that either. 

One thing I would recommend is refactoring your game to keep the game 
logic distinct from the I/O.

If I was designing this, I would have some sort of game engine class, 
which stores all the state for a game.  I imagine the first thing you 
need to do is pick a random number and remember it.  Then, you'll need a 
function to take a guess and tell you if it's too high, too low, or 
correct.  Something like:

class NumberGuessingGame:
   def __init__(self):
  self.number = random.randint(0, 10)

   HIGH = 1
   LOW = 2
   OK = 3

   def guess(self, number):
  if number > self.number:
 return self.HIGH
  if number < self.number:
 return self.LOW
  return self.OK

Now you've got something that's easy to test without all this pexpect 
crud getting in the way.  Then, your game application can create an 
instance of NumberGuessingGame and wrap the required I/O operations 
around that.

I'd also probably add some way to bypass the random number generator and 
set the number you're trying to guess directly.  This will make it a lot 
easier to test edge cases.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Understanding how is a function evaluated using recursion

[rusi]

On Thursday, September 26, 2013 4:54:22 AM UTC+5:30, Arturo B wrote:
> So I know what recursion is, but I don't know how is 
> 
>flatten(i)
> 
> evaluated, what value does it returns?

When you are a noob, who do you ask? The gurus.
When you are a guru who do you ask? The computer!

And its really quite easy to ask the computer directly. Here's your code with a 
extra prints

def flatten(l):
print ("Received: %s" % l)
ret = []
for i in l:
if isinstance(i, list) or isinstance(i, tuple):
ret.extend(flatten(i)) #How is flatten(i) evaluated?
else:
ret.append(i)
print ("Returning: %s" % ret)
return ret 

Now run it with a couple of different inputs (not neglecting the trivial cases) 
and see if it does not self-explain.

If still confusing, come back and ask!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline interpretation issue with MIMEApplication with binary data, Python 3.3.2

[Chris Angelico]

On Thu, Sep 26, 2013 at 2:38 AM,   wrote:
> app = MIMEApplication(b'Q\x0dQ', _encoder=encode_noop)

What is MIMEApplication? It's not a builtin, so your test case is
missing an import, at least. Is this email.mime.MIMEApplication?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline interpretation issue with MIMEApplication with binary data, Python 3.3.2

[Nils Bunger]

Chris, 

Thanks for answering. 

Yes, it's email.mime.MIMEApplication. I've pasted a snippet with the imports 
below.

I'm trying to use this to build a multi-part MIME message, with this as one 
part. 

I really can't figure out any way to attach a binary part like this to a 
multi-part MIME message without the encoding issue... any help would be greatly 
appreciate!

Nils

-

import io
from email.mime.application import MIMEApplication
from email.generator import BytesGenerator
from email.encoders import  encode_noop

app = MIMEApplication(b'Q\x0dQ', _encoder=encode_noop)
b = io.BytesIO()
g = BytesGenerator(b)
g.flatten(app)
for i in b.getvalue()[-3:]:
print("%02x " % i, end="")
print()


On Wednesday, September 25, 2013 9:11:31 PM UTC-7, Chris Angelico wrote:
> On Thu, Sep 26, 2013 at 2:38 AM,   wrote:
> 
> > app = MIMEApplication(b'Q\x0dQ', _encoder=encode_noop)
> 
> 
> 
> What is MIMEApplication? It's not a builtin, so your test case is
> 
> missing an import, at least. Is this email.mime.MIMEApplication?
> 
> 
> 
> ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Newline interpretation issue with MIMEApplication with binary data, Python 3.3.2

[Chris Angelico]

On Thu, Sep 26, 2013 at 2:23 PM, Nils Bunger  wrote:
> Yes, it's email.mime.MIMEApplication. I've pasted a snippet with the imports 
> below.
>
> I'm trying to use this to build a multi-part MIME message, with this as one 
> part.
>
> I really can't figure out any way to attach a binary part like this to a 
> multi-part MIME message without the encoding issue... any help would be 
> greatly appreciate!

I partly responded just to ping your thread, as I'm not particularly
familiar with the email.mime module. But a glance at the docs suggests
that MIMEApplication is a "subclass of MIMENonMultipart", so might it
be a problem to use that for multipart??

It's designed to handle text, so you may want to use an encoder (like
the default base64 one) rather than trying to push binary data through
it.

Random ideas, hopefully someone who actually knows the module can respond.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert namedtuple to dictionary

[Peter Otten]

trip...@gmail.com wrote:

> Need suggestions.
> 
> Say, I have a namedtuple like this:
> 
> {'a': brucelee(x=123, y=321), 'b': brucelee('x'=123, 'y'=321)
> 
> I need to convert it to:
> 
> {'a': {'x':123, 'y': 321},'b': {'x':123, 'y': 321}}
> 
> Follow-up question --
> 
> Which would be easier to work with if I had to later extract/manipulate
> the 'x', 'y' values? The format (dicts) above or a list of values like
> this:
> 
> {'a': ['x':123, 'y': 321],'b': ['x':123, 'y': 321]}

One more:

>>> from collections import namedtuple
>>> brucelee = namedtuple("brucelee", "x y")
>>> def to_dict(n):
... return dict(zip(n._fields, n))
... 
>>> data = {'a': brucelee(x=123, y=321), 'b': brucelee(x=123, y=321)}
>>> {k: to_dict(v) for k, v in data.items()}
{'a': {'y': 321, 'x': 123}, 'b': {'y': 321, 'x': 123}}


-- 
https://mail.python.org/mailman/listinfo/python-list