Ordering dependent test failures

[paul . anton . letnes]

Hi!

Fascinated by the concept of ordering dependent test failures [0], I've run the 
python test suite [1] with 256 different random seeds (took a little more than 
12 hours). The results vary a lot - for instance, the number of tests reported 
as OK varies, the number of skips varies, etc. Since I'm not sure how to report 
or interpret them, I'll just post a summary below.

The test suite was run on arch linux [2] with gcc 5.2.0, with the source code 
taken from a clone of the python repo yesterday [3].

What could I do with all this in order to make more sense of it, and could it 
be of any help what so ever to python development? I'll gladly make the full 
log files available to whomever is interested, in whatever format is 
convenient. In the meantime I'll run more random seeds, because why not.

[0] https://docs.python.org/devguide/buildbots.html#ordering-dependent-failures
[1] ./python -Wd -E -bb -m test -uall -rwW --randseed $k > testlog/${k}.log 2>&1
[2] https://www.archlinux.org/
[3] Python 3.6.0a0 (default:3704cea9fd8e, Sep 19 2015, 16:12:53)

The command "grep OK *.log", where e.g. 5.log is the output of command [1] with 
"--randseed 5":
0.log:380 tests OK.
0.log:OK (skipped=15)
0.log:OK (skipped=15)
0.log:OK (skipped=15)
0.log:OK (skipped=2)
1.log:383 tests OK.
1.log:OK (skipped=2)
2.log:380 tests OK.
2.log:OK (skipped=15)
2.log:OK (skipped=15)
2.log:OK (skipped=15)
2.log:OK (skipped=2)
3.log:380 tests OK.
3.log:OK (skipped=15)
3.log:OK (skipped=15)
3.log:OK (skipped=2)
3.log:OK (skipped=37)
4.log:381 tests OK.
4.log:OK (skipped=15)
4.log:OK (skipped=15)
4.log:OK (skipped=2)
5.log:380 tests OK.
5.log:OK (skipped=15)
5.log:OK (skipped=15)
5.log:OK (skipped=15)
5.log:OK (skipped=2)
6.log:381 tests OK.
6.log:OK (skipped=15)
6.log:OK (skipped=15)
6.log:OK (skipped=2)
7.log:380 tests OK.
7.log:OK (skipped=15)
7.log:OK (skipped=15)
7.log:OK (skipped=15)
7.log:OK (skipped=2)
8.log:381 tests OK.
8.log:OK (skipped=15)
8.log:OK (skipped=15)
8.log:OK (skipped=2)
9.log:382 tests OK.
9.log:OK (skipped=15)
9.log:OK (skipped=2)
10.log:381 tests OK.
10.log:OK (skipped=15)
10.log:OK (skipped=15)
10.log:OK (skipped=2)
11.log:380 tests OK.
11.log:OK (skipped=15)
11.log:OK (skipped=15)
11.log:OK (skipped=2)
11.log:OK (skipped=37)
12.log:380 tests OK.
12.log:OK (skipped=15)
12.log:OK (skipped=15)
12.log:OK (skipped=15)
12.log:OK (skipped=2)
13.log:383 tests OK.
13.log:OK (skipped=2)
14.log:381 tests OK.
14.log:OK (skipped=15)
14.log:OK (skipped=2)
14.log:OK (skipped=39)
15.log:383 tests OK.
15.log:OK (skipped=2)
16.log:381 tests OK.
16.log:OK (skipped=15)
16.log:OK (skipped=15)
16.log:OK (skipped=2)
17.log:381 tests OK.
17.log:OK (skipped=15)
17.log:OK (skipped=15)
17.log:OK (skipped=2)
18.log:381 tests OK.
18.log:OK (skipped=15)
18.log:OK (skipped=15)
18.log:OK (skipped=2)
19.log:381 tests OK.
19.log:OK (skipped=15)
19.log:OK (skipped=15)
19.log:OK (skipped=2)
20.log:381 tests OK.
20.log:OK (skipped=15)
20.log:OK (skipped=15)
20.log:OK (skipped=2)
21.log:382 tests OK.
21.log:OK (skipped=15)
21.log:OK (skipped=2)
22.log:383 tests OK.
22.log:OK (skipped=2)
23.log:383 tests OK.
23.log:OK (skipped=2)
24.log:382 tests OK.
24.log:OK (skipped=15)
24.log:OK (skipped=2)
25.log:381 tests OK.
25.log:OK (skipped=15)
25.log:OK (skipped=15)
25.log:OK (skipped=2)
26.log:380 tests OK.
26.log:OK (skipped=15)
26.log:OK (skipped=15)
26.log:OK (skipped=15)
26.log:OK (skipped=2)
27.log:382 tests OK.
27.log:OK (skipped=15)
27.log:OK (skipped=2)
28.log:381 tests OK.
28.log:OK (skipped=15)
28.log:OK (skipped=15)
28.log:OK (skipped=2)
29.log:383 tests OK.
29.log:OK (skipped=2)
30.log:380 tests OK.
30.log:OK (skipped=15)
30.log:OK (skipped=15)
30.log:OK (skipped=15)
30.log:OK (skipped=2)
31.log:382 tests OK.
31.log:OK (skipped=15)
31.log:OK (skipped=2)
32.log:382 tests OK.
32.log:OK (skipped=15)
32.log:OK (skipped=2)
33.log:383 tests OK.
33.log:OK (skipped=2)
34.log:383 tests OK.
34.log:OK (skipped=2)
35.log:380 tests OK.
35.log:OK (skipped=15)
35.log:OK (skipped=15)
35.log:OK (skipped=15)
35.log:OK (skipped=2)
36.log:382 tests OK.
36.log:OK (skipped=15)
36.log:OK (skipped=2)
37.log:382 tests OK.
37.log:OK (skipped=15)
37.log:OK (skipped=2)
38.log:380 tests OK.
38.log:OK (skipped=15)
38.log:OK (skipped=15)
38.log:OK (skipped=15)
38.log:OK (skipped=2)
39.log:380 tests OK.
39.log:OK (skipped=15)
39.log:OK (skipped=15)
39.log:OK (skipped=15)
39.log:OK (skipped=2)
40.log:381 tests OK.
40.log:OK (skipped=15)
40.log:OK (skipped=15)
40.log:OK (skipped=2)
41.log:383 tests OK.
41.log:OK (skipped=2)
42.log:380 tests OK.
42.log:OK (skipped=15)
42.log:OK (skipped=15)
42.log:OK (skipped=15)
42.log:OK (skipped=2)
43.log:380 tests OK.
43.log:OK (skipped=15)
43.log:OK (skipped=15)
43.log:OK (skipped=15)
43.log:OK (skipped=2)
44.log:383 tests OK.
44.log:OK (skipped=2)
45.log:383 tests OK.
45.log:OK (skipped=2)
46.log:380 tests OK.
46.log:OK (skipped=15)
46.log:OK (skipped=15)
46.log:OK (skipped=15)
46.log:OK (skipped=2)
47.log:383 tests OK.
47.log:OK (ski

for loop

[shiva upreti]

https://ideone.com/BPflPk

Please tell me why 'print s' statement is being executed inside loop, though I 
put it outside.
Please help. I am new to python.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for loop

[Chris Warrick]

On 20 September 2015 at 09:55, shiva upreti  wrote:
> https://ideone.com/BPflPk
>
> Please tell me why 'print s' statement is being executed inside loop, though 
> I put it outside.
> Please help. I am new to python.
> --
> https://mail.python.org/mailman/listinfo/python-list

You have mixed indentation. Your code contains both tabs and spaces.
Python interprets tabs as 8 spaces, while your other indentation is 4
spaces, leading to bad parsing.

Please make sure you use only spaces (reconfigure your editor to
always insert 4 spaces and reindent everything with tabs)

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for loop

[paul . anton . letnes]

On Sunday, September 20, 2015 at 9:56:06 AM UTC+2, shiva upreti wrote:
> https://ideone.com/BPflPk
> 
> Please tell me why 'print s' statement is being executed inside loop, though 
> I put it outside.
> Please help. I am new to python.

Hi!

Welcome to python, the most awesome programming language!

The code you pasted used both spaces and tabs for indentation. The thing is 
that python, by default, interprets one tab character as 8 spaces, but the 
editor you've used shows it as 4 spaces. To avoid these kinds of headaches, I 
always 1) set my editor to show tabs, so I can detect them, and 2) never use 
tabs when I write code myself. I set my editor to insert 4 spaces whenever I 
hit the "tab" key on my keyboard. If you post the name of your editor, maybe 
someone knows how to do that in yours. You can also detect mixed space/tab 
issues by running "python -t" instead of just "python".

So, your "print s" is in fact inside the loop, since the for loop is indented 
with 4 spaces, and "print s" is indented with 1 tab = 8 spaces. It just doesn't 
look like that to you.

It looks like you're coding in python 2. If you're new to python, I'd recommend 
using a python 3 version, maybe 3.4 or 3.5. You can easily pick up python 2 
later if you need to maintain old code. Of course, it's not a big deal learning 
python 3 if you know python 2 either, but why spend energy on it?

Cheers
Paul
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for loop

[shiva upreti]

On Sunday, September 20, 2015 at 1:33:57 PM UTC+5:30, Chris Warrick wrote:
> On 20 September 2015 at 09:55, shiva upreti  wrote:
> > https://ideone.com/BPflPk
> >
> > Please tell me why 'print s' statement is being executed inside loop, 
> > though I put it outside.
> > Please help. I am new to python.
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> 
> You have mixed indentation. Your code contains both tabs and spaces.
> Python interprets tabs as 8 spaces, while your other indentation is 4
> spaces, leading to bad parsing.
> 
> Please make sure you use only spaces (reconfigure your editor to
> always insert 4 spaces and reindent everything with tabs)
> 
> -- 
> Chris Warrick 
> PGP: 5EAAEA16

Thanks. It works fine now.:)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: for loop

[shiva upreti]

On Sunday, September 20, 2015 at 1:34:32 PM UTC+5:30, paul.ant...@gmail.com 
wrote:
> On Sunday, September 20, 2015 at 9:56:06 AM UTC+2, shiva upreti wrote:
> > https://ideone.com/BPflPk
> > 
> > Please tell me why 'print s' statement is being executed inside loop, 
> > though I put it outside.
> > Please help. I am new to python.
> 
> Hi!
> 
> Welcome to python, the most awesome programming language!
> 
> The code you pasted used both spaces and tabs for indentation. The thing is 
> that python, by default, interprets one tab character as 8 spaces, but the 
> editor you've used shows it as 4 spaces. To avoid these kinds of headaches, I 
> always 1) set my editor to show tabs, so I can detect them, and 2) never use 
> tabs when I write code myself. I set my editor to insert 4 spaces whenever I 
> hit the "tab" key on my keyboard. If you post the name of your editor, maybe 
> someone knows how to do that in yours. You can also detect mixed space/tab 
> issues by running "python -t" instead of just "python".
> 
> So, your "print s" is in fact inside the loop, since the for loop is indented 
> with 4 spaces, and "print s" is indented with 1 tab = 8 spaces. It just 
> doesn't look like that to you.
> 
> It looks like you're coding in python 2. If you're new to python, I'd 
> recommend using a python 3 version, maybe 3.4 or 3.5. You can easily pick up 
> python 2 later if you need to maintain old code. Of course, it's not a big 
> deal learning python 3 if you know python 2 either, but why spend energy on 
> it?
> 
> Cheers
> Paul

Thanks.:) I use gedit in ubuntu 1
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Ordering dependent test failures

[Peter Otten]

paul.anton.let...@gmail.com wrote:

> Fascinated by the concept of ordering dependent test failures [0], I've
> run the python test suite [1] with 256 different random seeds (took a
> little more than 12 hours). The results vary a lot - for instance, the
> number of tests reported as OK varies, the number of skips varies, etc.
> Since I'm not sure how to report or interpret them, I'll just post a
> summary below.
> 
> The test suite was run on arch linux [2] with gcc 5.2.0, with the source
> code taken from a clone of the python repo yesterday [3].
> 
> What could I do with all this in order to make more sense of it, and could
> it be of any help what so ever to python development? I'll gladly make the
> full log files available to whomever is interested, in whatever format is
> convenient. In the meantime I'll run more random seeds, because why not.

Given that the most likely interference is between two tests instead of the 
shotgun approach you could run the unit tests in such a way that for any 
pair of tests a, b the tests are run at least once with a before b and with 
a after b. Assuming that a test c does not "destroy" the interference when 
run between a and b for three tests this could be achieved with

1 abc covers ab, ac, bc
2 cba covers ba, ca, cb

If b fails in run 1 you'll assume a to be the culprit.
If c fails in run 1 you'll follow up with

1.1 ac
1.2 bc

to identify the predecessor causing the failure. Of course there are some 
complications, e. g. 

- Neither 1.1 nor 1.2 or both may fail
- The typical test suite comprises more than three tests
- Should the sequence aa be covered, i. e. a test case interfering with
  itself?



-- 
https://mail.python.org/mailman/listinfo/python-list

Lightwight socket IO wrapper

[James Harris]

I guess there have been many attempts to make socket IO easier to handle 
and a good number of those have been in Python.


The trouble with trying to improve something which is already well 
designed (and conciously left as is) is that the so-called improvement 
can become much more complex and overly elaborate. That can apply to the 
initial idea, for sure, but when writing helper or convenience functions 
perhaps it applies more to the temptation to keep adding just a little 
bit extra. The end result can be overly elaborate such as a framework 
which is fine where such is needed but is overkill for simpler 
requirements.


Do you guys have any recommendations of some *lightweight* additions to 
Python socket IO before I write any more of my own? Something built in 
to Python would be much preferred over any modules which have to be 
added. I had in the back of my mind that there was a high-level 
socket-IO library - much as threading was added as a wrapper to the 
basic thread module - but I cannot find anything above socket. Is there 
any?


A current specific to illustrate where basic socket IO is limited: it 
normally provides no guarantees over how many bytes are transferred at a 
time (AFAICS that's true for both streams and datagrams) so the 
delimiting of messages/records needs to be handled by the sender and 
receiver. I do already handle some of this myself but I wondered if 
there was a prebuilt solution that I should be using instead - to save 
me adding just a little bit extra. ;-)


James

--
https://mail.python.org/mailman/listinfo/python-list

Re: ConnectionError handling problem

[Jon Ribbens]

On 2015-09-19, Mark Lawrence  wrote:
> On 19/09/2015 07:13, shiva upreti wrote:
>>  try:
>>  r=requests.post(url, data=query_args)
>>  except:
>>  print "Connection error"
>
> Never use a bare except in Python, always handle the bare minimum number 
> of exceptions that you need to, in this case your ConnectionError.

While I entirely agree with the principle of being specific in what
exceptions you are catching (with the absolute maximum being
'Exception'), it is often not obvious which ones you need to specify.
The code above probably actually needs to catch EnvironmentError if
it is intended to intercept all network problems.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ConnectionError handling problem

[Mark Lawrence]

On 20/09/2015 13:45, Jon Ribbens wrote:

On 2015-09-19, Mark Lawrence  wrote:

On 19/09/2015 07:13, shiva upreti wrote:

try:
r=requests.post(url, data=query_args)
except:
print "Connection error"


Never use a bare except in Python, always handle the bare minimum number
of exceptions that you need to, in this case your ConnectionError.


While I entirely agree with the principle of being specific in what
exceptions you are catching (with the absolute maximum being
'Exception'), it is often not obvious which ones you need to specify.
The code above probably actually needs to catch EnvironmentError if
it is intended to intercept all network problems.



I doubt it, as from the docs "The following exceptions are kept for 
compatibility with previous versions; starting from Python 3.3, they are 
aliases of OSError.".  EnvironmentError is one of those listed.  I'd 
have thought it far more likely that you'd want to catch one or more of 
the OSError subclasses, as finer grained exceptions was part of the 
rationale behind PEP 3151 -- Reworking the OS and IO exception hierarchy.


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Akira Li]

"James Harris"  writes:

> I guess there have been many attempts to make socket IO easier to
> handle and a good number of those have been in Python.
>
> The trouble with trying to improve something which is already well
> designed (and conciously left as is) is that the so-called improvement
> can become much more complex and overly elaborate. That can apply to
> the initial idea, for sure, but when writing helper or convenience
> functions perhaps it applies more to the temptation to keep adding
> just a little bit extra. The end result can be overly elaborate such
> as a framework which is fine where such is needed but is overkill for
> simpler requirements.
>
> Do you guys have any recommendations of some *lightweight* additions
> to Python socket IO before I write any more of my own? Something built
> in to Python would be much preferred over any modules which have to be
> added. I had in the back of my mind that there was a high-level
> socket-IO library - much as threading was added as a wrapper to the
> basic thread module - but I cannot find anything above socket. Is
> there any?

Does ØMQ qualify as lightweight?

> A current specific to illustrate where basic socket IO is limited: it
> normally provides no guarantees over how many bytes are transferred at
> a time (AFAICS that's true for both streams and datagrams) so the
> delimiting of messages/records needs to be handled by the sender and
> receiver. I do already handle some of this myself but I wondered if
> there was a prebuilt solution that I should be using instead - to save
> me adding just a little bit extra. ;-)

There are already convenience functions in stdlib such as
sock.sendall(), sock.sendfile(), socket.create_connection() in addition
to BSD Sockets API.

If you want to extend this list and have specific suggestions; see
  https://docs.python.org/devguide/stdlibchanges.html

Or just describe your current specific issue in more detail here.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ConnectionError handling problem

[Chris Angelico]

On Sun, Sep 20, 2015 at 10:45 PM, Jon Ribbens
 wrote:
> On 2015-09-19, Mark Lawrence  wrote:
>> On 19/09/2015 07:13, shiva upreti wrote:
>>>  try:
>>>  r=requests.post(url, data=query_args)
>>>  except:
>>>  print "Connection error"
>>
>> Never use a bare except in Python, always handle the bare minimum number
>> of exceptions that you need to, in this case your ConnectionError.
>
> While I entirely agree with the principle of being specific in what
> exceptions you are catching (with the absolute maximum being
> 'Exception'), it is often not obvious which ones you need to specify.
> The code above probably actually needs to catch EnvironmentError if
> it is intended to intercept all network problems.

General principle: If you don't know what you should be catching,
_catch nothing_. Anything that happens will get printed to the
console. Then when you find that something's getting thrown, you check
out what its name is, and maybe what its superclasses are (in case
there's a broader one worth catching), and only THEN do you stick in a
try/except.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ConnectionError handling problem

[Jon Ribbens]

On 2015-09-20, Mark Lawrence  wrote:
> On 20/09/2015 13:45, Jon Ribbens wrote:
>> On 2015-09-19, Mark Lawrence  wrote:
>>> On 19/09/2015 07:13, shiva upreti wrote:
try:
r=requests.post(url, data=query_args)
except:
print "Connection error"
>>>
>>> Never use a bare except in Python, always handle the bare minimum number
>>> of exceptions that you need to, in this case your ConnectionError.
>>
>> While I entirely agree with the principle of being specific in what
>> exceptions you are catching (with the absolute maximum being
>> 'Exception'), it is often not obvious which ones you need to specify.
>> The code above probably actually needs to catch EnvironmentError if
>> it is intended to intercept all network problems.
>
> I doubt it, as from the docs "The following exceptions are kept for 
> compatibility with previous versions; starting from Python 3.3, they are 
> aliases of OSError.".  EnvironmentError is one of those listed.  I'd 
> have thought it far more likely that you'd want to catch one or more of 
> the OSError subclasses, as finer grained exceptions was part of the 
> rationale behind PEP 3151 -- Reworking the OS and IO exception hierarchy.

PEP 3151 has nothing to do with it, as the code in question in this
thread is running under Python 2 not Python 3. If you catch only
OSError then you will miss almost all the likely exceptions raised
by requests.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ConnectionError handling problem

[Jon Ribbens]

On 2015-09-20, Chris Angelico  wrote:
> On Sun, Sep 20, 2015 at 10:45 PM, Jon Ribbens
> wrote:
>> On 2015-09-19, Mark Lawrence  wrote:
>>> On 19/09/2015 07:13, shiva upreti wrote:
  try:
  r=requests.post(url, data=query_args)
  except:
  print "Connection error"
>>>
>>> Never use a bare except in Python, always handle the bare minimum number
>>> of exceptions that you need to, in this case your ConnectionError.
>>
>> While I entirely agree with the principle of being specific in what
>> exceptions you are catching (with the absolute maximum being
>> 'Exception'), it is often not obvious which ones you need to specify.
>> The code above probably actually needs to catch EnvironmentError if
>> it is intended to intercept all network problems.
>
> General principle: If you don't know what you should be catching,
> _catch nothing_. Anything that happens will get printed to the
> console. Then when you find that something's getting thrown, you check
> out what its name is, and maybe what its superclasses are (in case
> there's a broader one worth catching), and only THEN do you stick in a
> try/except.

I'm afraid I think that's absolutely terrible advice. I agree that
you should not be afraid to let exceptions propagate up the stack
(one of the things that's disastrously wrong about Java) but the
case at hand of "try and fetch this network resource and then do
something if it didn't succeed" is perfectly reasonable, and trying to
code that by trial and error is a perfect example of bad programming.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ConnectionError handling problem

[Laura Creighton]

The discussion about why or why not to use a bare except has gotten us
away from the problem reported, which is "why is my script hanging?"

In a message of Sat, 19 Sep 2015 17:18:12 +0100, Mark Lawrence writes:
>> I am learning python. I wrote a script using requests module.
>> The scripts runs fine for sometime, but after a while it hangs. When I press 
>> CTRL+C it shows ConnectionError even though I have included exception 
>> handling.
>> I am not sure as to why it cant handle ConnectionError when the script runs 
>> for a long time.
>>
>> This is a part(causing issues) of the script I am running:
>>
>> while(k<46656):
>>  j=res[k]
>>  url="http://172.16.68.6:8090/login.xml"; 
>>  query_args = {'mode':'191', 'username':str(i), 
>> 'password':str(j), 'a':'1442397582010', 'producttype':'0'}
>>  
>>  try:
>>  r=requests.post(url, data=query_args)
>>  except:
>>  print "Connection error"
>>  time.sleep(30)
>>  continue
>>
>>  html=r.text
>>  if(len(html) < 10):
>>  continue
>>
>>  if("The system could not log you on" not in html):
>>  print "hello"
>>  filehandle=open("ids", "a")
>>  filehandle.write(str(i)+'\n')
>>  filehandle.write(str(j)+'\n')
>>  filehandle.close()
>>  break
>>  
>>  k=k+1
>>
>> Any help will be highly appreciated.

So, when it hangs there are two main problems you can have.  One sort
is that the other side, http://172.16.68.6:8090/ it in itself
configured to only let people use a connection for a certain amount of
time.  If you are using it for longer, it will disconnect you.  Or the
other sort is that the other side disconnects people who have been
silent for a certain amount of time.  If you haven't send anything for
a certain amount of time it will log you off.  There are lots of other
things that work this way -- the system may see many attempts to login
and think you are trying to break into the system, the machine may
have crashed ... but the bottom line is that the reason your script
hangs is that there is nobody there on the other end.

The other sort of problem you can have is that the other end is
alive and well and talking to you, but you don't understand what
you are getting, and you are ignoring things you don't understand.
This can look exactly the same.

To find out what is going on you need to log what it is that you
are getting, to see if the answer is 'nothing' or 'garbage'.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .bat file trouble.

[eryksun]

On 9/18/15, Christian Gollwitzer  wrote:
> Am 18.09.15 um 11:06 schrieb bobert...@googlemail.com:
>
>> We originally thought that it was because it was missing the files:
>> process_init.py and process_global_variables.py however they are
>> right there in the same directory.
>
> Concerning that, windows usually runs a .bat file in the directory where
> it is situated, so putting the python fies there /should/ work, but you
> can also set this using the right-click menu (execute in...), if you
> make a link to the desktop.

It's fragile to depend on being started by Explorer or a shell link.
For example, if the batch is run from an existing cmd shell, then it
runs with cmd's current working directory. Or another process could
run the batch file with a different working directory. If you actually
want the batch to find the scripts relative to its own directory, then
it should use the %0 environment variable, which references the
fully-qualified path of the batch file. Getting just the directory,
i.e. its [d]rive and [p]ath, is expressed as %~dp0. Note that this
path includes a trailing backslash.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dummy Decoder Example (was Re: Parallel decoding lesson for you.)

[Skybuck Flying]

Since more people might be interested in this I will re-post this a second 
time to include more newsgroups... those two threads will need to be 
followed if all responses are to be seen ;)


Here is your dummy decoder example:

Let's turn this into a somewhat of a contest and ofcourse also teaching
lesson... now I am a true teacher... I provided you with most of the code.

The code you will need to write yourself/replace is indicated by // ***

Good luck and may the force be with you !

Ofcourse as announced earlier on 29 september 2015 I will reveal my parallel
solution to all of you !

So you have now 9 days to come up with your own solution before I publish
mine ! ;) =D

For those that missed the start/most of this thread, try googling for
comp.arch and "Parallel decoding lesson for you" by Skybuck.

// Begin of Dummy Decoder Example

program TestProgram;

{$APPTYPE CONSOLE}

{$R *.res}

uses
 System.SysUtils;

function Constrain( Para : integer ) : integer;
begin
result := Para;
if Para < 0 then Result := 0;
if Para >= 1 then Result := 1;
end;

procedure Main;
// must put variables here otherwise won't show up in debugger.
var
// information stream, input
Stream : array[0..20] of integer;

// bits representing fields of data
a1,a2,a3,a4 : integer;
b1,b2,b3 : integer;
c1 : integer;
d1,d2,d3,d4,d5,d6 : integer;

// output
RowIndex : integer;

RowCount : integer;
RowLength : array[0..5] of integer;
RowOffset : array[0..5] of integer;

DataOffset : integer;

FieldCount : integer;
FieldLength : array[0..3] of integer;

Processor : array[0..3] of integer;

// debug fields
FieldA : integer;
FieldB : integer;
FieldC : integer;
FieldD : integer;
begin
a1 := 1; a2 := 1; a3:= 1; a4 := 1;
b1 := 1; b2 := 1; b3 := 1;
c1 := 1;
d1 := 1; d2 := 1; d3 := 1; d4 := 1; d5 := 1; d6 := 1;

// compute input fields to compare it later with output fields
FieldA := (a1) or (a2 shl 1) or (a3 shl 2) or (a4 shl 3);
FieldB := (b1) or (b2 shl 1) or (b3 shl 2);
FieldC := (c1);
FieldD := (d1) or (d2 shl 1) or (d3 shl 2) or (d4 shl 3) or (d5 shl 4) or
(d6 shl 5);

// print field values
writeln( 'FieldD: ', FieldD );
writeln( 'FieldA: ', FieldA );
writeln( 'FieldB: ', FieldB );
writeln( 'FieldC: ', FieldC );
writeln;

// number of rows
Stream[0] := 6;

// row lengths
Stream[1] := 4;
Stream[2] := 3;
Stream[3] := 3;
Stream[4] := 2;
Stream[5] := 1;
Stream[6] := 1;

// sorted information stream:
// d1a1b1c1d2a2b2d3a3b3d4a4d5d6

// data bits
Stream[7] := d1;
Stream[8] := a1;
Stream[9] := b1;
Stream[10] := c1;
Stream[11] := d2;
Stream[12] := a2;
Stream[13] := b2;
Stream[14] := d3;
Stream[15] := a3;
Stream[16] := b3;
Stream[17] := d4;
Stream[18] := a4;
Stream[19] := d5;
Stream[20] := d6;

// now the decoding algorithm:

// determine number of rows
RowCount := Stream[0];

// extract row lengths
RowLength[0] := Stream[1];
RowLength[1] := Stream[2];
RowLength[2] := Stream[3];
RowLength[3] := Stream[4];
RowLength[4] := Stream[5];
RowLength[5] := Stream[6];

// determine field count
FieldCount := RowLength[0]; // row[0] indicates number of fields.

// I will help out a bit... by leaving this code in ! ;) seems somewhat
obvious ;)
// first determine data offset properly ! ;) :) 1 for the row count +
RowCount to skip over row lengths.
DataOffset := 1 + RowCount;

RowOffset[0] := DataOffset;
RowOffset[1] := RowOffset[0] + RowLength[0];
RowOffset[2] := RowOffset[1] + RowLength[1];
RowOffset[3] := RowOffset[2] + RowLength[2];
RowOffset[4] := RowOffset[3] + RowLength[3];
RowOffset[5] := RowOffset[4] + RowLength[4];


// some how the data bits from the stream needs to end up in these 4
processors so that it produces the same values
// as below:
// fields may be processed in a different order though.

// *** You will need to replace this code with your own code... and
preferably it should be parallel, fast and somewhat general/scalable. ***
Processor[0] := FieldD;
Processor[1] := FieldA;
Processor[2] := FieldB;
Processor[3] := FieldC;

// print processor values.
writeln( 'Processor[0]: ', Processor[0] );
writeln( 'Processor[1]: ', Processor[1] );
writeln( 'Processor[2]: ', Processor[2] );
writeln( 'Processor[3]: ', Processor[3] );
writeln;
end;

begin
try
 Main;
except
 on E: Exception do
  Writeln(E.ClassName, ': ', E.Message);
end;
ReadLn;
end.

// End of Dummy Decoder Example

Oh one last thing when you think you have a valid solution and/or need to
test more,

replace the initialization of the test data bits with this initialization
for example:

a1 := 1; a2 := 0; a3:= 0; a4 := 1;
b1 := 1; b2 := 1; b3 := 0;
c1 := 1;
d1 := 1; d2 := 1; d3 := 0; d4 := 0; d5 := 1; d6 := 0;

in the original posting (the one before this they were all set to 1).

This helped a bit during the development just to see if it was reading
anything at all at the somewhat correct places.

But this may later put you off if the values are not matching input vs
output.

So this test data in this posting is better to spot any inconsistencies/bugs
in your solution(s).

Bye,
 Skybuck. 


--

Re: .bat file trouble.

[Ian Kelly]

On Fri, Sep 18, 2015 at 7:45 AM, Christian Gollwitzer  wrote:
> The first error indicates, that you are running Python 3, and the script was
> made for Python 2. In Python 3, print is a function so you need parentheses
> around that print("Initializing...") - either fix that
> or install Python 2.
>
> The second error "TabError" also relates to it, it means that the file mixes
> tabs and spaces for indentation. Fix it by expanding all tabs to spaces in
> an editor.

Note that even if you fix these, there could be other less obvious
compatibility issues with running this script in Python 3. The safest
and most straightforward way to proceed is to install Python 2 and use
that to run the script. However, in the long term you may want to
consider upgrading the script to Python 3 anyway. Python 2.7 is only
guaranteed to be supported through the year 2020.
-- 
https://mail.python.org/mailman/listinfo/python-list

Started to work with SQLite3 in Python3

[Cecil Westerhof]

I started a little project in Python3 with SQLite3:
https://github.com/CecilWesterhof/CarApplication

I do not mind to get some feedback on it.

I was wondering about two things:
- Would it be better to split the code into several parts. (For
  example the SQL statements.)
- I understood that with
for fuel in fuel_cursor:
  a fetchall will be executed.
  At the moment I do not see this as a problem, but if the table would
  become very big it could. Would it be better to rewrite it with a
  fetchone?
  But that is then needed inside and before the loop. What would be a
  little ugly.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Postscript to pdf

[Bala Ji]

Hello,

I'm doing a software to make an id card for a school club so i used TKINTER to 
make this software.
So i can enter details like name, student number etc..

So finally i got a Postscript file the problem is that i want to make a pdf 
file do you have any ideas? "convert ps to pdf"

Thank u
regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Laura Creighton]

In a message of Sun, 20 Sep 2015 09:20:10 -0700, Bala Ji writes:
>Hello,
>
>I'm doing a software to make an id card for a school club so i used TKINTER to 
>make this software.
>So i can enter details like name, student number etc..
>
>So finally i got a Postscript file the problem is that i want to make a pdf 
>file do you have any ideas? "convert ps to pdf"
>
>Thank u
>regards

There is a free program that does this called ps2pdf  which is
available for linux, windows and mac os.

Laura 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Started to work with SQLite3 in Python3

[Chris Angelico]

On Mon, Sep 21, 2015 at 1:49 AM, Cecil Westerhof  wrote:
> - I understood that with
> for fuel in fuel_cursor:
>   a fetchall will be executed.
>   At the moment I do not see this as a problem, but if the table would
>   become very big it could. Would it be better to rewrite it with a
>   fetchone?
>   But that is then needed inside and before the loop. What would be a
>   little ugly.

It iterates over the query, but it doesn't fetchall() into a list. So
it's as good as using fetchone().

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Bala Ji]

Thank u laura,
I tired it but the problem when i use it there is only half of the image 
its like the software cut the image
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dummy Decoder Example (was Re: Parallel decoding lesson for you.)

[Skybuck Flying]

Or point your newsgroup reader to newsgroup: comp.arch

Then see thread/topic: "Parallel decoding lesson for you" by Skybuck.

Bye,
 Skybuck.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Laura Creighton]

In a message of Sun, 20 Sep 2015 09:50:02 -0700, Bala Ji writes:
>Thank u laura,
>I tired it but the problem when i use it there is only half of the image 
>its like the software cut the image
>-- 
>https://mail.python.org/mailman/listinfo/python-list

Ok.  We need more information.
What OS are you using?
What version of ps2pdf did you use, and where did you get it?

The next thing we need to find out is whether your ps file contained
all the infomation, or whether you lost data before you tried to
convert it to a pdf.  Do you know?  Have you seen your ps file
and are sure it is ok?

We will get this to work.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Started to work with SQLite3 in Python3

[Cecil Westerhof]

On Sunday 20 Sep 2015 18:44 CEST, Chris Angelico wrote:

> On Mon, Sep 21, 2015 at 1:49 AM, Cecil Westerhof  wrote:
>> - I understood that with for fuel in fuel_cursor: a fetchall will
>> be executed. At the moment I do not see this as a problem, but if
>> the table would become very big it could. Would it be better to
>> rewrite it with a fetchone? But that is then needed inside and
>> before the loop. What would be a little ugly.
>
> It iterates over the query, but it doesn't fetchall() into a list.
> So it's as good as using fetchone().

You are right. I found this:
http://pythoncentral.io/introduction-to-sqlite-in-python/
where at:
Retrieving Data (SELECT) with SQLite
it says:
The cursor object works as an iterator, invoking fetchall()
automatically:
but all other places (I did a more extended search now) say the same
as you. So this problem is solved. :-)

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Started to work with SQLite3 in Python3

[Cecil Westerhof]

On Sunday 20 Sep 2015 17:49 CEST, Cecil Westerhof wrote:

> I started a little project in Python3 with SQLite3:
> https://github.com/CecilWesterhof/CarApplication
>
> I do not mind to get some feedback on it.

I think I found a bug. In init I should put a:
conn.commit()
after:
fill_tables()

At the moment it is not a problem, but if the program gets some
functionality and would crash after a long run, the changes would be
lost.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Laura Creighton]

In a message of Sun, 20 Sep 2015 20:27:48 +0200, Baladjy KICHENASSAMY writes:
>Hello,
>
>I'm using macosx, ps2pdf version i don't know :/ sorry
>ok actually i found what is the problem...
>
>There is no problem with the ps file every thing is fine =)
>
>Can u please just tell me how to change paper settings ?
>i want to go from portait to landscape ?
>

Aha!  Great to know.
Do you have a real printer there, in which case is this a printer problem?
If so, I need the name of the printer and its model number to help look up
the way to make it do landscape.

If you just want to have ps2pdf produce landscape files, that is
surprisingly difficult, for the ps2pdf that uses ghostscript to
get the work done.  ps2pdf 'guesses' what is the correct orientation
and, stupidly, there is no way to tell it 'stop guessing, I know
what I want'.  It's very bad at guessing things that don't have
text -- or enough text -- in them.

Most of the time ps2pdf is just this:

gs \
  -o output.pdf \
  -sDEVICE=pdfwrite \
  -dPDFSETTINGS=/prepress \
  -dHaveTrueTypes=true \
  -dEmbedAllFonts=true \
  -dSubsetFonts=false \
  -c ".setpdfwrite <> setdistillerparams" \
  -f input.ps

where input.ps is the file you have and output.pdf is the pdf you want.
and we can make gs orient itself as you want with

-c "<> setpagedevice"

Orientation 3 is landscape.  Orientation 0 is portrait.

If this is your problem, then see if you have gs (ghostscript)
and in that case see if this horrible long command works.

Laura

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Started to work with SQLite3 in Python3

[Cecil Westerhof]

On Sunday 20 Sep 2015 17:49 CEST, Cecil Westerhof wrote:

> I started a little project in Python3 with SQLite3:
> https://github.com/CecilWesterhof/CarApplication
>
> I do not mind to get some feedback on it.
>
> I was wondering about two things:
> - Would it be better to split the code into several parts. (For
> example the SQL statements.)

I changed this part. I put the initialisation of the variables with
SQL statements into sqlStatements.py and use in the program:
from sqlStatements import *

Is this the correct way of doing this?

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

error while installing using pip.

[OmPs]

Hi all,

I am getting the below error while I am trying to install atfork package
from pip repositories. I have done a thorough google search but am not able
to find and appropriate solution for it.Installation of SSL packages and
ssl package from python too do not solve the mystry.

# pip install --allow-external atfork atforkCollecting atfork
  Could not find a version that satisfies the requirement atfork (from
versions: )
  Some insecure and unverifiable files were ignored (use --allow-unverified
atfork to allow).
No matching distribution found for atfork


It says no matching distribution, but at pypi repo I am able to see it.

https://pypi.python.org/pypi/atfork/0.1.2
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error while installing using pip.

[Laura Creighton]

In a message of Mon, 21 Sep 2015 01:00:01 +0530, OmPs writes:
>Hi all,
>
>I am getting the below error while I am trying to install atfork package
>from pip repositories. I have done a thorough google search but am not able
>to find and appropriate solution for it.Installation of SSL packages and
>ssl package from python too do not solve the mystry.
>
># pip install --allow-external atfork atforkCollecting atfork
>  Could not find a version that satisfies the requirement atfork (from
>versions: )
>  Some insecure and unverifiable files were ignored (use --allow-unverified
>atfork to allow).
>No matching distribution found for atfork
>
>
>It says no matching distribution, but at pypi repo I am able to see it.
>
>https://pypi.python.org/pypi/atfork/0.1.2

If you get no useful answers here, try
distutils-...@python.org

https://mail.python.org/mailman/listinfo/distutils-sig

If you have a problem with pip, these are the people to talk to.
And having them verify that pip is fine, the package is nuts
would at least simplify your problem.

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Baladjy KICHENASSAMY]

o ok i got it
actually it's very easy the commande is :
ps2pdf -dEPSCrop image.ps

sorry but i'm new to python  my last question is how to integrate this
to python... i want that the output file must be a pdf ?

1) i created a button which i'll save my id card as "ps" file

def save():
Canevas.update()
Canevas.postscript(file=tkFileDialog.asksaveasfilename(), colormode='color')


2) so now i want to create a button to convert this "ps" file into "pdf" 

def convert():
   help :/

Regards




2015-09-20 21:09 GMT+02:00 Laura Creighton :
> In a message of Sun, 20 Sep 2015 20:27:48 +0200, Baladjy KICHENASSAMY writes:
>>Hello,
>>
>>I'm using macosx, ps2pdf version i don't know :/ sorry
>>ok actually i found what is the problem...
>>
>>There is no problem with the ps file every thing is fine =)
>>
>>Can u please just tell me how to change paper settings ?
>>i want to go from portait to landscape ?
>>
>
> Aha!  Great to know.
> Do you have a real printer there, in which case is this a printer problem?
> If so, I need the name of the printer and its model number to help look up
> the way to make it do landscape.
>
> If you just want to have ps2pdf produce landscape files, that is
> surprisingly difficult, for the ps2pdf that uses ghostscript to
> get the work done.  ps2pdf 'guesses' what is the correct orientation
> and, stupidly, there is no way to tell it 'stop guessing, I know
> what I want'.  It's very bad at guessing things that don't have
> text -- or enough text -- in them.
>
> Most of the time ps2pdf is just this:
>
> gs \
>   -o output.pdf \
>   -sDEVICE=pdfwrite \
>   -dPDFSETTINGS=/prepress \
>   -dHaveTrueTypes=true \
>   -dEmbedAllFonts=true \
>   -dSubsetFonts=false \
>   -c ".setpdfwrite <> setdistillerparams" \
>   -f input.ps
>
> where input.ps is the file you have and output.pdf is the pdf you want.
> and we can make gs orient itself as you want with
>
> -c "<> setpagedevice"
>
> Orientation 3 is landscape.  Orientation 0 is portrait.
>
> If this is your problem, then see if you have gs (ghostscript)
> and in that case see if this horrible long command works.
>
> Laura
>



-- 
KICHENASSAMY Baladjy
Ingénieur en Génie Mécanique
Spécialiste Contrôle Non Destructif et Qualification des procédés spéciaux
COSAC CND Niveau 2 RT et PT
Aircelle SAFRAN
Tel:06.03.72.53.12
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Baladjy KICHENASSAMY]

Hello,

I'm using macosx, ps2pdf version i don't know :/ sorry
ok actually i found what is the problem...

There is no problem with the ps file every thing is fine =)

Can u please just tell me how to change paper settings ?
i want to go from portait to landscape ?



2015-09-20 20:12 GMT+02:00 Laura Creighton :
> In a message of Sun, 20 Sep 2015 09:50:02 -0700, Bala Ji writes:
>>Thank u laura,
>>I tired it but the problem when i use it there is only half of the image
>>its like the software cut the image
>>--
>>https://mail.python.org/mailman/listinfo/python-list
>
> Ok.  We need more information.
> What OS are you using?
> What version of ps2pdf did you use, and where did you get it?
>
> The next thing we need to find out is whether your ps file contained
> all the infomation, or whether you lost data before you tried to
> convert it to a pdf.  Do you know?  Have you seen your ps file
> and are sure it is ok?
>
> We will get this to work.
>
> Laura



-- 
KICHENASSAMY Baladjy
Ingénieur en Génie Mécanique
Spécialiste Contrôle Non Destructif et Qualification des procédés spéciaux
COSAC CND Niveau 2 RT et PT
Aircelle SAFRAN
Tel:06.03.72.53.12
-- 
https://mail.python.org/mailman/listinfo/python-list

Python 2.7 Configuration

[Ward Kingkade]

Hello All,

I'm new to Python.  I downloaded the 64-bit build of Python 2.7.10 from
www.python.org/downloads/release/python-2710.  I run Windows 7 Pro on a Dell
PC.

I find that the installation package creates a folder called "Python 2.7" in
my Start menu, with both a command prompt and IDLE GUI options.  

I hold my scripts in another directory that is parallel to but not under the
one where Python 2.7 resides, so I set the Environment Variable PYTHONPATH
to include the directory where my scripts reside.  Unfortunately, I am
unable to reach them in either the IDLE GUI or the "Python command line" in
that folder in my start menu.  The only way I have managed to run Python
scripts is to open an ordinary command prompt from Accessories, navigate to
the directory with the scripts, and run python from the command prompt in
that directory.

Can anyone let me know what I'm not doing that I should be doing to run
Python and my scripts from the parallel directory under the GUI or the
command line from the start menu folder ?

Thanks.

Ward Kingkade 

1201 Belle View Boulevard

Apartment B1

Alexandria, VA 22307

ward_kingk...@msn.com

 

 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Laura Creighton]

In a message of Sun, 20 Sep 2015 21:32:34 +0200, Baladjy KICHENASSAMY writes:
>o ok i got it
>actually it's very easy the commande is :
>ps2pdf -dEPSCrop image.ps
>
>sorry but i'm new to python  my last question is how to integrate this
>to python... i want that the output file must be a pdf ?
>
>1) i created a button which i'll save my id card as "ps" file
>
>def save():
>Canevas.update()
>Canevas.postscript(file=tkFileDialog.asksaveasfilename(), 
> colormode='color')

>2) so now i want to create a button to convert this "ps" file into "pdf" 
>
>def convert():
>   help :/

You need to run the subprocess module to run your command.
https://docs.python.org/2/library/subprocess.html  (for python 2)
https://docs.python.org/3.4/library/subprocess.html (for python 3)

is this enough or do you need more help getting it to work?

Laura

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error while installing using pip.

[Mark Lawrence]

On 20/09/2015 20:30, OmPs wrote:

Hi all,

I am getting the below error while I am trying to install atfork package
from pip repositories. I have done a thorough google search but am not
able to find and appropriate solution for it.Installation of SSL
packages and ssl package from python too do not solve the mystry.

# pip install --allow-external atfork atforkCollecting atfork
   Could not find a version that satisfies the requirement atfork (from
versions: )
   Some insecure and unverifiable files were ignored (use
--allow-unverified atfork to allow).
No matching distribution found for atfork

It says no matching distribution, but at pypi repo I am able to see it.

https://pypi.python.org/pypi/atfork/0.1.2



I tried several combinations of things and each one failed dismally.  I 
then went to https://github.com/google/python-atfork and saw that 
nothing has been committed for six years.  The readme states "Work on 
this project has been abandoned."  Are you sure that you still want this 
package?


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Re: error while installing using pip.

[Laura Creighton]

There is an atfork patch supplied as part of this issue
https://bugs.python.org/issue16500

Maybe it can do what you want?

Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: error while installing using pip.

[Chris Angelico]

On Mon, Sep 21, 2015 at 5:30 AM, OmPs  wrote:
> It says no matching distribution, but at pypi repo I am able to see it.
>
> https://pypi.python.org/pypi/atfork/0.1.2

That has a version number of zero. I think that might count as a
pre-release version, so you need the --pre option to pip or it won't
install.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[James Harris]

"Akira Li" <4kir4...@gmail.com> wrote in message 
news:mailman.37.1442754893.21674.python-l...@python.org...

"James Harris"  writes:


I guess there have been many attempts to make socket IO easier to
handle and a good number of those have been in Python.

The trouble with trying to improve something which is already well
designed (and conciously left as is) is that the so-called 
improvement

can become much more complex and overly elaborate. That can apply to
the initial idea, for sure, but when writing helper or convenience
functions perhaps it applies more to the temptation to keep adding
just a little bit extra. The end result can be overly elaborate such
as a framework which is fine where such is needed but is overkill for
simpler requirements.

Do you guys have any recommendations of some *lightweight* additions
to Python socket IO before I write any more of my own? Something 
built
in to Python would be much preferred over any modules which have to 
be

added. I had in the back of my mind that there was a high-level
socket-IO library - much as threading was added as a wrapper to the
basic thread module - but I cannot find anything above socket. Is
there any?


Does ØMQ qualify as lightweight?


It's certainly interesting. It's puzzling, too. For example,

 http://zguide.zeromq.org/py:hwserver

The Python code there includes

 message = socket.recv()

but given that this is a TCP socket it doesn't look like there is any 
way for the stack to know how many bytes to return. Either ZeroMQ layers 
another end-to-end protocol on top of TCP (which would be no good) or it 
will be guessing (which would not be good either).


There are probably answers to that query but there is a lot of 
documentation, including on reliable communication, and that in itself 
makes ZeroMQ seem overkill, even if it can be persuaded to do what I 
want.


I am impressed that they show code in many languages. I may come back to 
it but for the moment it doesn't seem to be what I was looking for. And 
it is not built in.



A current specific to illustrate where basic socket IO is limited: it
normally provides no guarantees over how many bytes are transferred 
at

a time (AFAICS that's true for both streams and datagrams) so the
delimiting of messages/records needs to be handled by the sender and
receiver. I do already handle some of this myself but I wondered if
there was a prebuilt solution that I should be using instead - to 
save

me adding just a little bit extra. ;-)


There are already convenience functions in stdlib such as
sock.sendall(), sock.sendfile(), socket.create_connection() in 
addition

to BSD Sockets API.

If you want to extend this list and have specific suggestions; see
 https://docs.python.org/devguide/stdlibchanges.html


That may be a bit overkill just now but it's a good suggestion.


Or just describe your current specific issue in more detail here.


There are a few things and more crop up as time goes on. For example, 
over TCP it would be helpful to have a function to receive a specific 
number of bytes or one to read bytes until reaching a certain delimiter 
such as newline or zero or space etc. Even better would be to be able to 
use the iteration protocol so you could just code next() and get the 
next such chunk of read in a for loop. When sending it would be good to 
just say to send a bunch of bytes but know that you will get told how 
many were sent (or didn't get sent) if it fails. Sock.sendall() doesn't 
do that.


I thought UDP would deliver (or drop) a whole datagram but cannot find 
anything in the Python documentaiton to guarantee that. In fact 
documentation for the send() call says that apps are responsible for 
checking that all data has been sent. They may mean that to apply to 
stream protocols only but it doesn't state that. (Of course, UDP 
datagrams are limited in size so the call may validly indicate 
incomplete transmission even when the first part of a big message is 
sent successfully.)


Receiving no bytes is taken as indicating the end of the communication. 
That's OK for TCP but not for UDP so there should be a way to 
distinguish between the end of data and receiving an empty datagram.


The recv calls require a buffer size to be supplied which is a technical 
detail. A Python wrapper could save the programmer dealing with that.


Reminder to self: encoding issues.

None of the above is difficult to write and I have written the bits I 
need myself but, basically, there are things that would make socket IO 
easier and yet still compatible with more long-winded code. So I 
wondered if there were already some Python modules which were more 
convenient than what I found in the documentation.


James

--
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Laura Creighton]

In a message of Sun, 20 Sep 2015 23:11:20 +0200, Baladjy KICHENASSAMY writes:
>well one more question :/
>
>i tried this
>
>def save():
> Canevas.update()
> Canevas.postscript(file=tkFileDialog.asksaveasfilename(),
>colormode='color')
> subprocess.call(["ps2pdf", "-dEPSCrop", "test.ps", "test.pdf"])
>
>
>i got the ps file but i didn't get the pdf file :/

It's 00:41 here in Sweden, and I have an 08:00 meeting.  I am off to bed.
Perhaps somebody else can help now, otherwise I will be back
tomorrow.

I've never tried to get subprocess to run as part of a savefile dialog.
Try building a tkinter app that takes a filename argument (somepostscript
file you have lying around) and then push the button, in the same dir
makes a pdf of your ps using subprocess.

If that works, then see about getting save to run that subprocess as
part of the save process.  I am very curious about whether that can
be done -- but first check that your subprocess is working, period.
The wretched thing is slightly different between windows and linux,
and even between different linuxes, so needing to fiddle a bit here is
normal (and frustrating).

So right now I cannot tell if your problem is 'save doesn't like
it like that' or 'your subprocess isn't working'.

But nothing in Tkinter likes 'Canevas' -- (which is great French
but rotten English) you must have written Canvas in your code.

Paste the real thing in next time, ok?

Sorry I am off now,
Laura
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Chris Angelico]

On Mon, Sep 21, 2015 at 10:19 AM, Dennis Lee Bieber
 wrote:
> Even if the IP layer has to fragment a UDP packet to meet limits of 
> the
> transport media, it should put them back together on the other end before
> passing it up to the UDP layer. To my knowledge, UDP does not have a size
> limit on the message (well -- a 16-bit length field in the UDP header). But
> since it /is/ "got it all" or "dropped" with no inherent confirmation, one
> would have to embed their own protocol within it -- sequence numbers with
> ACK/NAK, for example. Problem: if using LARGE UDP packets, this protocol
> would mean having LARGE resends should packets be dropped or arrive out of
> sequence (and since the ACK/NAK could be dropped too, you may have to
> handle the case of a duplicated packet -- also large).
>

If you're going to add sequencing and acknowledgements to UDP,
wouldn't it be easier to use TCP and simply prefix every message with
a two-byte length?

UDP is great when order doesn't matter and each packet stands entirely
alone. DNS is a well-known example - the question "What is the IP
address for www.rosuav.com?" doesn't in any way affect the question
"What is the mail server for gmail.com?", so you fire off UDP packets
for each one, and get responses whenever you get them. UDP's also
perfect for a heartbeat system - you send out a packet every
however-often, and if the monitor hasn't heard from you in X seconds,
it starts alerting people. No need for responses of any kind there.
But for working with a stream, I usually find it's a lot easier to
build on top of TCP than UDP.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Cameron Simpson]

On 21Sep2015 10:34, Chris Angelico  wrote:

If you're going to add sequencing and acknowledgements to UDP,
wouldn't it be easier to use TCP and simply prefix every message with
a two-byte length?


Frankly, often yes. That's what I do. (different length encoding, but 
otherwise...)


UDP's neat if you do not care if a packet fails to arrive and if you can 
guarentee that your data fits in a packet in the face of different MTUs. 

I like TCP myself, most of the time. Another nice thing about TCP is that wil a 
little effort you get to pack multiple data packets (or partial data packets) 
into a network packet, etc.


Cheers,
Cameron Simpson 

If you lie to the compiler, it will get its revenge.- Henry Spencer
--
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Chris Angelico]

On Mon, Sep 21, 2015 at 11:55 AM, Cameron Simpson  wrote:
> On 21Sep2015 10:34, Chris Angelico  wrote:
>>
>> If you're going to add sequencing and acknowledgements to UDP,
>> wouldn't it be easier to use TCP and simply prefix every message with
>> a two-byte length?
>
>
> Frankly, often yes. That's what I do. (different length encoding, but
> otherwise...)

Out of interest, what encoding? With most protocols, I would prefer to
encode in ASCII digits terminated by end-of-line, but for arbitrary
content you're packaging up, it's usually easier to read 2 bytes (or 4
or whatever you want to specify), then read that many bytes, and
that's your content. No buffering required - you'll never read past
the end of a packet.

> UDP's neat if you do not care if a packet fails to arrive and if you can
> guarentee that your data fits in a packet in the face of different MTUs.
> I like TCP myself, most of the time. Another nice thing about TCP is that
> wil a little effort you get to pack multiple data packets (or partial data
> packets) into a network packet, etc.

Emphatically - a little effort sometimes, and other times no effort at
all! If you write a packet of data, then write another one, and
another, and another, and another, without waiting for responses,
Nagling should combine them automatically. And even if they're not
deliberately queued by Nagle's Algorithm, packets can get combined for
other reasons. So, yeah! Definitely can help a lot with packet counts
on small writes.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fixing Python install on the Mac after running 'CleanMyMac' (fwd)

[kacyjones]

On Friday, May 29, 2015 at 12:56:49 PM UTC-7, Laura Creighton wrote:
> Good news, we are getting closer to understanding what to do.
> This in from Ned.  I will continue after the message:
> 
> --- Forwarded Message
> 
> Return-Path: 
> From: Ned Deily 
> Subject: Re: Fixing Python install on the Mac after running 'CleanMyMac'
> Date: Fri, 29 May 2015 10:28:19 -0700
> Lines: 63
> 
> > Might that appear to work at first, but leave some subtle issues which are
> > not at first apparent? I don't know how intertwined Apple's Python instance
> > is in the day-to-day operation of the operating system (it certainly seems
> > to be on some Linux distros), but it's possible that some Apple-specific
> > package isn't available at part of the stock Python 2.7 distribution.
> 
> Installing a python.org Python (/usr/local/bin/python*) is independent 
> of and does not interfere with use of the Apple-supplied system Pythons 
> (/usr/bin/python*).  Apple does ship various third-party Python packages 
> ("distributions") with its system Python, like numpy, but they tend to 
> be old and outdated versions and there are a few Apple-only packages.  
> But, should that issue arise, that can be resolved by choosing the right 
> path (/usr/local/bin vs /usr/bin) or removing /usr/local/bin from $PATH.
> 
> The thing is the original message in this thread had this:
> > They are getting:
> >  Utility has encountered a fatal error, and will now terminate.  A
> >  Python runtime could not be located. You may need to install a
> >  framework build of Python or edit the PyRuntimeLocations array in this
> >  applications info.plist file.  Then there are two oblong circles. One
> >  says Open Console. The other says Terminate.
> 
> But ... I just did what I should have done earlier: googled for that 
> message.  And I find that the message is coming from a py2app-built 
> application (and it seems I answered the question 3 years ago!):
> 
> http://stackoverflow.com/questions/10184974/py2app-is-not-copying-the-pyt
> hon-framework-to-the-new-app-while-using-virutalenv
> 
> The py2app glue code could be looking for Pythons in various spots 
> including the system Python.  So, let's make sure the system Python is 
> still working.  On the most up-to-date 10.7 Lion system (10.7.5), typing 
> the following two commands should give results the same as those shown 
> (->):
> 
> /usr/bin/python2.7 -c 'import sys;print(sys.version)'
> - -> 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
> - -> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
> 
> /usr/bin/python2.7 -c 'import numpy;print(numpy.__file__)'
> - -> 
> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/pytho
> n/numpy/__init__.py
> 
> If not, then it really may be necessary to restore system files which, 
> as I noted before, is most safely and accurately done by following 
> Apple's directions to restore the system from the recovery partition and 
> a good backup of user files.
> 
> - -- 
>  Ned Deily,
>  n...@acm.org
> 
> - -- 
> https://mail.python.org/mailman/listinfo/python-list
> 
> --- End of Forwarded Message
> 
> So please type
> 
> /usr/bin/python2.7 -c 'import sys;print(sys.version)'
> 
> and tell me what you get as an answer.
> 
> Then type
> 
> /usr/bin/python2.7 -c 'import numpy;print(numpy.__file__)'
> 
> and again tell me what you get.
> 
> But if it isn't as Ned expects, then you may have to go back to
> your backups.
> 
> Laura


I was having this same problem, entered those commands and got this as a 
response:

MacBook-Pro:~ kacyjones$ /usr/bin/python2.7 -c 'import 
numpy;print(numpy.__file__)' 
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/__init__.pyc
MacBook-Pro:~ kacyjones$ 
MacBook-Pro:~ kacyjones$ /usr/bin/python2.7 -c 'import sys;print(sys.version)' 
2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]

Any idea what that means for my system?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Akira Li]

"James Harris"  writes:
...
> There are a few things and more crop up as time goes on. For example,
> over TCP it would be helpful to have a function to receive a specific
> number of bytes or one to read bytes until reaching a certain
> delimiter such as newline or zero or space etc. 

The answer is sock.makefile('rb') then `file.read(nbytes)` returns a
specific number of bytes.

`file.readline()` reads until newline (b'\n') There is Python Issue:
"Add support for reading records with arbitrary separators to the
standard IO stack"
  http://bugs.python.org/issue1152248
See also
  http://bugs.python.org/issue17083

Perhaps, it is easier to implement read_until(sep) that is best suited
for a particular case.

> Even better would be to be able to use the iteration protocol so you
> could just code next() and get the next such chunk of read in a for
> loop.

file is an iterator over lines i.e., next(file) works.

> When sending it would be good to just say to send a bunch of bytes but
> know that you will get told how many were sent (or didn't get sent) if
> it fails. Sock.sendall() doesn't do that.

sock.send() returns the number of bytes sent that may be less than given.
You could reimplement sock.sendall() to include the number of bytes
successfully sent in case of an error.

> I thought UDP would deliver (or drop) a whole datagram but cannot find
> anything in the Python documentaiton to guarantee that. In fact
> documentation for the send() call says that apps are responsible for
> checking that all data has been sent. They may mean that to apply to
> stream protocols only but it doesn't state that. (Of course, UDP
> datagrams are limited in size so the call may validly indicate
> incomplete transmission even when the first part of a big message is
> sent successfully.)
>
> Receiving no bytes is taken as indicating the end of the
> communication. That's OK for TCP but not for UDP so there should be a
> way to distinguish between the end of data and receiving an empty
> datagram.

There is no end of communication in UDP and therefore there is no end of
data. If you've got a zero bytes in return then it means that you've
received a zero length datagram.

sock.recvfrom() is a thin wrapper around the corresponding C
function. You could read any docs you like about UDP sockets.
  
http://stackoverflow.com/questions/5307031/how-to-detect-receipt-of-a-0-length-udp-datagram

> The recv calls require a buffer size to be supplied which is a
> technical detail. A Python wrapper could save the programmer dealing
> with that.

It is not just a buffer size. It is the maximum amount of data to be
received at once i.e., sock.recv() may return less but never more.
You could use makefile() and read() if recv() is too low-level.

> Reminder to self: encoding issues.
>
> None of the above is difficult to write and I have written the bits I
> need myself but, basically, there are things that would make socket IO
> easier and yet still compatible with more long-winded code. So I
> wondered if there were already some Python modules which were more
> convenient than what I found in the documentation.
>
> James

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Postscript to pdf

[Terry Reedy]

On 9/20/2015 12:20 PM, Bala Ji wrote:

Hello,

I'm doing a software to make an id card for a school club so i used
TKINTER to make this software. So i can enter details like name,
student number etc..

So finally i got a Postscript file the problem is that i want to make
a pdf file do you have any ideas? "convert ps to pdf"


Where does the .ps file come from?  tkinter canvas to .ps?  If so, I 
suggest not to do that.  Get reportlab or another python pdf generator 
(search pypi.python.org for 'pdf' or web for 'python generate pdf'.)


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Marko Rauhamaa]

Chris Angelico :

> On Mon, Sep 21, 2015 at 11:55 AM, Cameron Simpson  wrote:
>> Another nice thing about TCP is that wil a little effort you get to
>> pack multiple data packets (or partial data packets) into a network
>> packet, etc.
>
> Emphatically - a little effort sometimes, and other times no effort at
> all! If you write a packet of data, then write another one, and
> another, and another, and another, without waiting for responses,
> Nagling should combine them automatically. And even if they're not
> deliberately queued by Nagle's Algorithm, packets can get combined for
> other reasons. So, yeah! Definitely can help a lot with packet counts
> on small writes.

Unfortunately, Nagle and delayed ACK, which are both defaults, don't go
well together (you get nasty 200-millisecond hickups).

I recommend using socket.TCP_CORK with socket.TCP_NODELAY where they are
available (Linux). They give you Nagle without delayed ACK. See

   http://linux.die.net/man/7/tcp>


As for the topic, TCP doesn't need wrappers to abstract away the
difficult bits. That's a superficially good idea that leads to trouble.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Chris Angelico]

On Mon, Sep 21, 2015 at 2:39 PM, Marko Rauhamaa  wrote:
> Chris Angelico :
>
>> On Mon, Sep 21, 2015 at 11:55 AM, Cameron Simpson  wrote:
>>> Another nice thing about TCP is that wil a little effort you get to
>>> pack multiple data packets (or partial data packets) into a network
>>> packet, etc.
>>
>> Emphatically - a little effort sometimes, and other times no effort at
>> all! If you write a packet of data, then write another one, and
>> another, and another, and another, without waiting for responses,
>> Nagling should combine them automatically. And even if they're not
>> deliberately queued by Nagle's Algorithm, packets can get combined for
>> other reasons. So, yeah! Definitely can help a lot with packet counts
>> on small writes.
>
> Unfortunately, Nagle and delayed ACK, which are both defaults, don't go
> well together (you get nasty 200-millisecond hickups).

Only in the write-write-read scenario. If you write-read-write-read,
or if your reads don't depend on your writes, then Nagle + delayed ACK
works just fine. But if you write a bunch of stuff, then block waiting
for the other end to respond, and then write multiple times, and wait
for a response, _then_ the pair work badly together, yes.

> As for the topic, TCP doesn't need wrappers to abstract away the
> difficult bits. That's a superficially good idea that leads to trouble.

Depends what you're doing - if you're working with a higher level
protocol like HTTP, then abstracting away the difficult bits of TCP is
part of abstracting away the difficult bits of HTTP, and something
like 'requests' is superb. But if you're inventing your own protocol,
directly on top of a BSD socket, then I would agree - just call socket
functions directly. Otherwise you risk nasty surprises when your
file-like object has ridiculous performance problems.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Gregory Ewing]

Dennis Lee Bieber wrote:

worst case: each TCP packet is broken up to fit Hollerith
cards;


Or printed on strips of paper and tied to pigeons:

https://en.wikipedia.org/wiki/IP_over_Avian_Carriers

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Lightwight socket IO wrapper

[Cameron Simpson]

On 21Sep2015 12:40, Chris Angelico  wrote:

On Mon, Sep 21, 2015 at 11:55 AM, Cameron Simpson  wrote:

On 21Sep2015 10:34, Chris Angelico  wrote:

If you're going to add sequencing and acknowledgements to UDP,
wouldn't it be easier to use TCP and simply prefix every message with
a two-byte length?


Frankly, often yes. That's what I do. (different length encoding, but
otherwise...)


Out of interest, what encoding?


NB: this is for binary protocols.

I don't like embedding arbitrary size limits in protocols or data formats if I 
can easily avoid it. So (for my home grown binary protocols) I encode unsigned 
integers as big endian octets with the top bit meaning "another octet follows" 
and the bottom 7 bits going to the value. So my packets look like:


 encoded(length)data

For sizes below 128, one byte of length. For sizes 128-16383, two bytes. And so 
on. Compact yet unbounded.


My new protocols ar probably going to derive from the scheme implemented  in 
the code cited below. "New" means as of some weeks ago, when I completely 
rewrote a painful ad hoc protocol of mine and pulled out the general features 
into what follows.


The actual packet format is implemented by the Packet class at the bottom of 
this:


 https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/serialise.py

Simple and flexible.

As for using that data format multiplexed with multiple channels, see the 
PacketConnection class here:


 https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/stream.py

Broadly, the packets are length[tag,flags[,channel#],payload] and one 
implements whatever semantics one needs on top of that.


You can see this exercised over UNIX pipes and TCP streams in the unit tests 
here:


 https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/stream_tests.py

On the subject of packet stuffing, my preferred loop for that is visible in the 
PacketConnection._send worker thread method, which goes:


   fp = self._send_fp
   Q = self._sendQ
   for P in Q:
 sig = (P.channel, P.tag, P.is_request)
 if sig in self.__sent:
   raise RuntimeError("second send of %s" % (P,))
 self.__sent.add(sig)
 write_Packet(fp, P)
 if Q.empty():
   fp.flush()
   fp.close()

In short: get packets from the queue and write them to the stream buffer. If 
the queue gets empty, _only then_ flush the buffer. This assures synchronicity 
in comms while giving the IO library a chance to fill a buffer with several 
packets.


Cheers,
Cameron Simpson 

ERROR 155 - You can't do that.  - Data General S200 Fortran error code list
--
https://mail.python.org/mailman/listinfo/python-list