calling locale.setlocale repeatedly
Hi, From http://www.pyzine.com/Issue008/Section_Articles/article_Encodings.html#guessing-the-encoding: > The way to access the information about the "normal" encoding used on the > current computer is through the locale module. Before using locale to > retrieve the information you want, you need to call > locale.setlocale(locale.LC_ALL, ''). Because of the sensitivity of the > underlying C locale module on some platforms, this should only be done once. Why should the call locale.setlocale(locale.LC_ALL, '') only be made once? What exactly is the "sensitivity of the underlying C locale module"? Thanks, Michael -- http://mail.python.org/mailman/listinfo/python-list
converting to and from octal escaped UTF--8
Hi,
I am writing unicode stings into a special text file that requires to
have non-ascii characters as as octal-escaped UTF-8 codes.
For example, the letter "Í" (latin capital I with acute, code point 205)
would come out as "\303\215".
I will also have to read back from the file later on and convert the
escaped characters back into a unicode string.
Does anyone have any suggestions on how to go from "Í" to "\303\215" and
vice versa?
I know I can get the code point by doing
>>> "Í".decode('utf-8').encode('unicode_escape')
but there doesn't seem to be any similar method for getting the octal
escaped version.
Thanks,
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Re: converting to and from octal escaped UTF--8
Michael Goerz wrote:
> Hi,
>
> I am writing unicode stings into a special text file that requires to
> have non-ascii characters as as octal-escaped UTF-8 codes.
>
> For example, the letter "Í" (latin capital I with acute, code point 205)
> would come out as "\303\215".
>
> I will also have to read back from the file later on and convert the
> escaped characters back into a unicode string.
>
> Does anyone have any suggestions on how to go from "Í" to "\303\215" and
> vice versa?
>
> I know I can get the code point by doing
>>>> "Í".decode('utf-8').encode('unicode_escape')
> but there doesn't seem to be any similar method for getting the octal
> escaped version.
>
> Thanks,
> Michael
I've come up with the following solution. It's not very pretty, but it
works (no bugs, I hope). Can anyone think of a better way to do it?
Michael
_
import binascii
def escape(s):
hexstring = binascii.b2a_hex(s)
result = ""
while len(hexstring) > 0:
(hexbyte, hexstring) = (hexstring[:2], hexstring[2:])
octbyte = oct(int(hexbyte, 16)).zfill(3)
result += "\\" + octbyte[-3:]
return result
def unescape(s):
result = ""
while len(s) > 0:
if s[0] == "\\":
(octbyte, s) = (s[1:4], s[4:])
try:
result += chr(int(octbyte, 8))
except ValueError:
result += "\\"
s = octbyte + s
else:
result += s[0]
s = s[1:]
return result
print escape("\303\215")
print unescape('adf\\303\\215adf')
--
http://mail.python.org/mailman/listinfo/python-list
Re: converting to and from octal escaped UTF--8
MonkeeSage wrote:
> Looks like escape() can be a bit simpler...
>
> def escape(s):
> result = []
> for char in s:
> result.append("\%o" % ord(char))
> return ''.join(result)
>
> Regards,
> Jordan
Very neat! Thanks a lot...
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Re: converting to and from octal escaped UTF--8
MonkeeSage wrote:
> On Dec 3, 1:31 am, MonkeeSage <[EMAIL PROTECTED]> wrote:
>> On Dec 2, 11:46 pm, Michael Spencer <[EMAIL PROTECTED]> wrote:
>>
>>
>>
>>> Michael Goerz wrote:
>>>> Hi,
>>>> I am writing unicode stings into a special text file that requires to
>>>> have non-ascii characters as as octal-escaped UTF-8 codes.
>>>> For example, the letter "Í" (latin capital I with acute, code point 205)
>>>> would come out as "\303\215".
>>>> I will also have to read back from the file later on and convert the
>>>> escaped characters back into a unicode string.
>>>> Does anyone have any suggestions on how to go from "Í" to "\303\215" and
>>>> vice versa?
>>> Perhaps something along the lines of:
>>> >>> def encode(source):
>>> ... return "".join("\%o" % ord(c) for c in source.encode('utf8'))
>>> ...
>>> >>> def decode(encoded):
>>> ... bytes = "".join(chr(int(c, 8)) for c in encoded.split('\\')[1:])
>>> ... return bytes.decode('utf8')
>>> ...
>>> >>> encode(u"Í")
>>> '\\303\\215'
>>> >>> print decode(_)
>>> Í
>>> HTH
>>> Michael
>> Nice one. :) If I might suggest a slight variation to handle cases
>> where the "encoded" string contains plain text as well as octal
>> escapes...
>>
>> def decode(encoded):
>> for octc in (c for c in re.findall(r'\\(\d{3})', encoded)):
>> encoded = encoded.replace(r'\%s' % octc, chr(int(octc, 8)))
>> return encoded.decode('utf8')
>>
>> This way it can handle both "\\141\\144\\146\\303\\215\\141\\144\\146"
>> as well as "adf\\303\\215adf".
>>
>> Regards,
>> Jordan
>
> err...
>
> def decode(encoded):
> for octc in re.findall(r'\\(\d{3})', encoded):
> encoded = encoded.replace(r'\%s' % octc, chr(int(octc, 8)))
> return encoded.decode('utf8')
Great suggestions from both of you! I came up with my "final" solution
based on them. It encodes only non-ascii and non-printables, and stays
in unicode strings for both input and output. Also, low ascii values now
encode into a 3-digit octal sequence also, so that decode can catch them
properly.
Thanks a lot,
Michael
import re
def encode(source):
encoded = ""
for character in source:
if (ord(character) < 32) or (ord(character) > 128):
for byte in character.encode('utf8'):
encoded += ("\%03o" % ord(byte))
else:
encoded += character
return encoded.decode('utf-8')
def decode(encoded):
decoded = encoded.encode('utf-8')
for octc in re.findall(r'\\(\d{3})', decoded):
decoded = decoded.replace(r'\%s' % octc, chr(int(octc, 8)))
return decoded.decode('utf8')
orig = u"blaÍblub" + chr(10)
enc = encode(orig)
dec = decode(enc)
print orig
print enc
print dec
--
http://mail.python.org/mailman/listinfo/python-list
Catching a non-Exception object (KeyboardInterrupt)
Hi, when I try to catch ctrl+c with except KeyboardInterrupt: pychecker tells me Catching a non-Exception object (KeyboardInterrupt) It works fine, but the message indicates that it's not completely clean. How should I write the exception correctly? Thanks, Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: Unicode char replace
DiMar wrote, on 02/12/2008 09:54 PM:
> Hi all,
>
> I have this unicode string:
>
> string = u'Macworld » Jobs 1 - Twitter 0'
>
> and I want to replace the '»' (aka \xbb) char to '»'.
> I've tried 2 ways:
>
> 1.
string2 = string.replace('\\xbb','»')
> u'Macworld \xbb Jobs 1 - Twitter 0'
How about this?
string.replace(u'\xbb', u'»')
--
http://mail.python.org/mailman/listinfo/python-list
Exception on keypress
Hi,
I'm writing a command line program that watches a file, and recompiles
it when it changes. However, there should also be a possibility of doing
a complete clean restart (cleaning up temp files, compiling some
dependencies, etc.).
Since the program is in an infinite loop, there are limited means of
interacting with it. Right now, I'm using the Keyboard Interrupt: if the
user presses CTRL+C once, a clean restart is done, if he presses it
twice within a second, the program terminates. The stripped down code
looks like this:
while True:
try:
time.sleep(1)
if watched_file_has_changed():
compile_the_changed_file()
except KeyboardInterrupt: # user hits CTRL+C
try:
print("Hit Ctrl+C again to quit")
time.sleep(1)
clean_restart()
except KeyboardInterrupt:
do_some_cleanup()
sys.exit(0)
Is there another way of doing this? Ideally, there would be an exception
every time any key at all is pressed while the code in the try block is
being executed. That way, the user could just hit the 'R' key to do a
clean restart, and the 'E' key to exit the program. Is there any way to
implement something like that?
Right now, the CTRL+C solution works, but isn't very extensible (It
wouldn't be easy to add another command, for example). Any suggestions?
Thanks,
Michael
--
http://mail.python.org/mailman/listinfo/python-list
KeyboardInterrupt should not kill subprocess
Hi,
I'm using subprocess.Popen() to run some background processes. However,
the program is also supposed to catch CTRL+C keyboard interrupts for
refreshs (i.e. a keyboard interrupt doesn't shut down the program).
But as it seems, a keyboard interrupt will automatically pass down to
the subprocesses, causing them to abort. Is there a way that I can
prevent the subprocesses from being canceled by a keyboard interrupt?
To clarify my question, here is a minimal example:
import subprocess
import os
import time
import sys
# is 'sleep' available on Win machines?
# use another dummy program if it isn't
p = subprocess.Popen(['sleep', '10'])
while True:
try:
time.sleep(1)
pass # normal program procedure
print "subprocess poll: " + str(p.poll())
except KeyboardInterrupt:
try:
print("Hit Ctrl+C again to quit")
time.sleep(1)
print "Refreshing"
pass # do some refresh stuff here
except KeyboardInterrupt:
sys.exit(0)
As you can see, after the refresh, p.poll() is '-2'. I'd want the
subprocess to continue undisturbed, i.e. p.poll() would still return 'None'.
Thanks,
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Raising exception on STDIN read
Hi, I would like to raise an exception any time a subprocess tries to read from STDIN: latexprocess = subprocess.Popen( \ 'pdflatex' + " " \ + 'test' + " 2>&1", \ shell=True, \ cwd=os.getcwd(), \ env=os.environ, \ stdin=StdinCatcher() # any ideas here? ) An exception should be raised whenever the pdflatex process reads from STDIN... and I have no idea how to do it. Any suggestions? Thanks, Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: Raising exception on STDIN read
Ian Clark wrote, on 02/27/2008 12:06 PM:
> On 2008-02-27, Michael Goerz <[EMAIL PROTECTED]> wrote:
>> I would like to raise an exception any time a subprocess tries to read
>> from STDIN:
>> latexprocess = subprocess.Popen( \
>> 'pdflatex' + " " \
>> + 'test' + " 2>&1", \
>> shell=True, \
>> cwd=os.getcwd(), \
>> env=os.environ, \
>> stdin=StdinCatcher() # any ideas here?
>> )
> How about with a file-like object? I haven't tested this with subprocess
> so you might want to read the manual on files if it doesn't work[1].
>
> import sys
> class ErrorFile(object):
> def _error(self, *args, **kwargs):
> raise AssertionError("Illegal Access")
>
> def _noop(self, *args, **kwargs):
> pass
>
> close = flush = seek = tell = _noop
> next = read = readline = readlines = xreadlines = tuncate =
> _error
> truncate = write = writelines = _error
>
> sys.stdin = ErrorFile()
> print raw_input("? ")
I was already trying to do that sort of thing. While it works in your
example code, for some reason it doesn't work in the subprocess. The
first problem is that ErrorFile needs a fileno() methode (otherwise I
get AttributeError: 'ErrorFile' object has no attribute 'fileno'). So,
when I add
def fileno(self):
return 0
to ErrorFile, it runs, but it still doesn't work. The exception is never
raised.
My specific example was running pdflatex on a file test.tex which looks
like this:
\documentclass[a4paper,10pt]{article}
\usepackage{bogus}
\begin{document}
test
\end{document}
When compiled, this should fail because of a missing 'bogus' package.
The compiler will prompt for a replacement on STDIN, which I want to
catch with an exception.
So, when I run my python script
#!/usr/bin/python
import subprocess
import os
import sys
class ErrorFile(object):
def _error(self, *args, **kwargs):
raise AssertionError("Illegal Access")
def _noop(self, *args, **kwargs):
pass
def fileno(self):
return 0
close = flush = seek = tell = _noop
next = read = readline = readlines \
= xreadlines = tuncate = _error
truncate = write = writelines = _error
latexprocess = subprocess.Popen( \
'pdflatex' + " " \
+ 'test' + " 2>&1", \
shell=True, \
cwd=os.getcwd(), \
env=os.environ, \
stdin=ErrorFile()
)
the compiler notices that it doesn't get any input:
Enter file name:
! Emergency stop.
but no exception is raised, and for some reason I have to press Enter to
finish.
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Re: Raising exception on STDIN read
Gabriel Genellina wrote, on 02/27/2008 03:26 PM:
> En Wed, 27 Feb 2008 15:06:36 -0200, Ian Clark <[EMAIL PROTECTED]>
> escribi�:
>
>> On 2008-02-27, Michael Goerz <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>> I would like to raise an exception any time a subprocess tries to read
>>> from STDIN:
>>>
>>> latexprocess = subprocess.Popen( \
>>> 'pdflatex' + " " \
>>> + 'test' + " 2>&1", \
>>> shell=True, \
>>> cwd=os.getcwd(), \
>>> env=os.environ, \
>>> stdin=StdinCatcher() # any ideas here?
>>> )
>>>
>>> An exception should be raised whenever the pdflatex process
>>> reads from STDIN... and I have no idea how to do it. Any suggestions?
>
>> How about with a file-like object? I haven't tested this with subprocess
>> so you might want to read the manual on files if it doesn't work[1].
>
> Won't work for an external process, as pdflatex (and the OS) knows
> nothing about Python objects. The arguments to subprocess.Popen must be
> actual files having real OS file descriptors.
>
> Try with stdin=open("/dev/full") or stdin=open("/dev/null")
using /dev/null works in my specific case (in my posted minimal example
I still have to press Enter, but in the real program it causes pdflatex
to fail, like I want it to). However, the solution is limited do Linux,
as on Windows there's no /dev/null. Is there a platform independent
solution?
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Re: Raising exception on STDIN read
Grant Edwards wrote, on 02/27/2008 04:34 PM:
> On 2008-02-27, Michael Goerz <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> I would like to raise an exception any time a subprocess tries to read
>> from STDIN:
>>
>> latexprocess = subprocess.Popen( \
>> 'pdflatex' + " " \
>> + 'test' + " 2>&1", \
>> shell=True, \
>> cwd=os.getcwd(), \
>> env=os.environ, \
>> stdin=StdinCatcher() # any ideas here?
>> )
>>
>> An exception should be raised whenever the pdflatex process
>> reads from STDIN... and I have no idea how to do it. Any suggestions?
>
> If I were you, I'd just run LaTeX in batch mode. That's what I
> always used to do when automating the running of LaTeX using a
> shell script or makefile. IIRC, you do something like this:
>
> ret = os.system("latex \\batchmode\\input %s" % filename)
Yeah, I'm doing that by default. The problem in my actual program is
that the user can change the latex command and the compiler options, so
I can't guarantee that the latex compiler is being run in batchmode or
nonstopmode (The user might screw up). So, ideally, I want to enforce
that the subprocess is non-interactive by trowing an exception if it
does ask for input.
I have to use subprocess instead of os.system because I want to filter
the output (colorize it, parse for warnings, etc.)
Thanks,
Michael
--
http://mail.python.org/mailman/listinfo/python-list
'normal' shell with curses
Hi,
I'm trying to print out text in color. As far as I know, curses is the
only way to do that (or not?). So, what I ultimately want is a curses
terminal that behaves as closely as possible as a normal terminal, i.e.
it breaks lines and scrolls automatically, so that I can implement a
function myprint(color, text) that does what print() does, only in color.
So, my first tests brought up some problems:
#!/usr/bin/python
from time import sleep
import curses
import sys
stdscr = curses.initscr()
stdscr.addstr("Initialized\n")
stdscr.refresh()
(maxlines, maxcolumns) = stdscr.getmaxyx()
try:
for counter in xrange(24):
stdscr.addstr("Hello world %s\n" % counter)
stdscr.refresh()
if counter >= maxlines:
stdscr.scroll()
stdscr.refresh()
except Exception, data:
sys.stderr.write("Exception: \n");
sys.stderr.write(str(data));
finally:
sleep(5)
curses.endwin()
Instead of scrolling, the program throws an exception. Any hints?
Also, is there a way leave the curses output on the screen after
curses.endwin(), instead of returning to the normal terminal without a
trace of curses.
Thanks,
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Re: 'normal' shell with curses
Miki wrote, on 03/03/2008 11:14 PM: > Hello Michael, > >> I'm trying to print out text in color. As far as I know, curses is the >> only way to do that (or not?). > On unix, every XTerm compatible terminal will be able to display color > using escape sequence. > (Like the one you see in the output of 'grep --color') > > See the shameless plug in > http://pythonwise.blogspot.com/2008/03/ansiprint.html > The escape sequence approach is definitely interesting and I'd prefer it to curses if I can make it work. However, I ultimately would like to support other terminals than xterm (especially Windows), that I assume have different ANSI codes. So, two questions: What are the most common terminals that differ in the ANSI codes they use, and how can I detect which of those terminals the program is running on? Where can I find out what codes those terminals use for color? Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: 'normal' shell with curses
Thynnus wrote, on 03/04/2008 08:48 AM:
> On 3/3/2008 9:57 PM, Michael Goerz wrote:
>> Hi,
>>
>> I'm trying to print out text in color. As far as I know, curses is the
>> only way to do that (or not?). So, what I ultimately want is a curses
>> terminal that behaves as closely as possible as a normal terminal,
>> i.e. it breaks lines and scrolls automatically, so that I can
>> implement a function myprint(color, text) that does what print() does,
>> only in color.
>
> You might find the below helpful. Let us know how you make out?
>
>
>
> Python Cookbook
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475116
>
> Title:
> Using terminfo for portable color output & cursor control
>
> Description:
> The curses module defines several functions (based on terminfo) that can
> be used to perform lightweight cursor control & output formatting
> (color, bold, etc). These can be used without invoking curses mode
> (curses.initwin) or using any of the more heavy-weight curses
> functionality. This recipe defines a TerminalController class, which can
> make portable output formatting very simple. Formatting modes that are
> not supported by the terminal are simply omitted.
>
>
>
That looks *extremely* interesting. From a very brief test, it seems to
do exactly what I want!
Now, Windows seems very problematic for color output. I was using the
following as a test, based on the above recipe:
term = TerminalController()
while True:
print term.render('${YELLOW}Warning:${NORMAL}'), 'paper is crinkled'
print term.render('${RED}Error:${NORMAL}'), 'paper is ripped'
On Linux, it works fine, on Windows, it just prints white on black
(which is of course what it should do if the terminal doesn't support
color). Can anyone get the Windows cmd.exe terminal to do color? I
already tried to add device=%SystemRoot%\system32\ansi.sys to config.nt,
but that doesn't seem to do anything (neither in what I tried yesterday
with the ANSI escape codes, nor with the recipe code now). I also very
briefly tried running it on the winbash shell
(http://win-bash.sourceforge.net/), it didn't have any color either...
So, any way to get color in Windows?
Michael
--
http://mail.python.org/mailman/listinfo/python-list
Re: 'normal' shell with curses
Michael Goerz wrote, on 03/04/2008 12:05 PM:
> Thynnus wrote, on 03/04/2008 08:48 AM:
>> On 3/3/2008 9:57 PM, Michael Goerz wrote:
>>> Hi,
>>>
>>> I'm trying to print out text in color. As far as I know, curses is
>>> the only way to do that (or not?). So, what I ultimately want is a
>>> curses terminal that behaves as closely as possible as a normal
>>> terminal, i.e. it breaks lines and scrolls automatically, so that I
>>> can implement a function myprint(color, text) that does what print()
>>> does, only in color.
>>
>> You might find the below helpful. Let us know how you make out?
>>
>>
>>
>> Python Cookbook
>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475116
>>
>> Title:
>> Using terminfo for portable color output & cursor control
>>
>> Description:
>> The curses module defines several functions (based on terminfo) that
>> can be used to perform lightweight cursor control & output formatting
>> (color, bold, etc). These can be used without invoking curses mode
>> (curses.initwin) or using any of the more heavy-weight curses
>> functionality. This recipe defines a TerminalController class, which
>> can make portable output formatting very simple. Formatting modes that
>> are not supported by the terminal are simply omitted.
>>
>>
>>
>
> That looks *extremely* interesting. From a very brief test, it seems to
> do exactly what I want!
>
> Now, Windows seems very problematic for color output. I was using the
> following as a test, based on the above recipe:
>
> term = TerminalController()
> while True:
> print term.render('${YELLOW}Warning:${NORMAL}'), 'paper is crinkled'
> print term.render('${RED}Error:${NORMAL}'), 'paper is ripped'
>
> On Linux, it works fine, on Windows, it just prints white on black
> (which is of course what it should do if the terminal doesn't support
> color). Can anyone get the Windows cmd.exe terminal to do color? I
> already tried to add device=%SystemRoot%\system32\ansi.sys to config.nt,
> but that doesn't seem to do anything (neither in what I tried yesterday
> with the ANSI escape codes, nor with the recipe code now). I also very
> briefly tried running it on the winbash shell
> (http://win-bash.sourceforge.net/), it didn't have any color either...
> So, any way to get color in Windows?
>
> Michael
This recipe seems to work very well on WinXP:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496901
So now, I'll just have to combine the two methods, which shouldn't be
too hard.
Michael
--
http://mail.python.org/mailman/listinfo/python-list
putting text through pager
Hi, I'm trying to print some variable through a pager (i.e. 'less') on a linux system. My attempt was this: == snip here == import subprocess def put_through_pager(displaystring): less_pipe = subprocess.Popen(\ 'less', shell=True, \ stdin=subprocess.PIPE).stdin less_pipe.write(displaystring) less_pipe.close() def main(): put_through_pager(longstring) longstring = """ Lorem ipsum dolor sit amet,... http://www.lipsum.com/ """ main() == snip here == That doesn't work however: first of all, it only flashes the text for a fraction of a second, and secondly, after I run the program my terminal is broken, not echoing whatever I type back to me. Any suggestions for putting text through a pager from Python? This is strictly on a linux system, of course. Thanks, Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: putting text through pager
Michael Goerz wrote, on 03/20/2008 04:43 PM:
> Hi,
>
> I'm trying to print some variable through a pager (i.e. 'less') on a
> linux system. My attempt was this:
>
>
> == snip here ==
> import subprocess
>
> def put_through_pager(displaystring):
> less_pipe = subprocess.Popen(\
> 'less', shell=True, \
> stdin=subprocess.PIPE).stdin
> less_pipe.write(displaystring)
> less_pipe.close()
>
> def main():
> put_through_pager(longstring)
>
>
> longstring = """
> Lorem ipsum dolor sit amet,...
> http://www.lipsum.com/
> """
>
> main()
>
> == snip here ==
>
> That doesn't work however: first of all, it only flashes the text for a
> fraction of a second, and secondly, after I run the program my terminal
> is broken, not echoing whatever I type back to me.
>
> Any suggestions for putting text through a pager from Python? This is
> strictly on a linux system, of course.
>
> Thanks,
> Michael
Using a tempfile seems to be a good solution:
def put_through_pager(displaystring):
(temp_fd, tempname) = tempfile.mkstemp(".mail")
temp_fh = os.fdopen(temp_fd, "w")
temp_fh.write(displaystring)
temp_fh.close()
os.system("less %s" % tempname)
os.unlink(tempname)
--
http://mail.python.org/mailman/listinfo/python-list
