Re: [pyxl] xlrd-0.8.0 .xlsx formatting_info=True not implemented

[Albert-Jan Roskam]

Hi,

As a work-around, you could use the CRAN R package XLConnect, using RPy or 
RPy2, to do what you want. IIRC it's based on Java, so it's not extremely fast.
http://cran.r-project.org/web/packages/XLConnect/vignettes/XLConnect.pdf
This is another package I just saw for the first time

http://cran.r-project.org/web/packages/xlsx/xlsx.pdf

 
Regards,
Albert-Jan


~~
All right, but apart from the sanitation, the medicine, education, wine, public 
order, irrigation, roads, a 
fresh water system, and public health, what have the Romans ever done for us?
~~ 


>
> From: "python-ex...@raf.org" 
>To: python-ex...@googlegroups.com; Python List  
>Sent: Thursday, August 30, 2012 4:57 AM
>Subject: Re: [pyxl] xlrd-0.8.0 .xlsx formatting_info=True not implemented
> 
>John Yeung wrote:
>
>> > is there any other way to tell how many digits excel would round to
>> > when displaying a floating point number? that's my only reason for
>> > needing formatting_info=True.
>> 
>> I have not personally used it, but OpenPyXL is another option for
>> working with .xlsx files, and it might provide the formatting info you
>> need:
>> 
>>  http://packages.python.org/openpyxl/index.html
>>  http://pypi.python.org/pypi/openpyxl/1.5.8
>> 
>> John Y.
>
>thanks but openpyxl doesn't work well enough.
>most of the spreadsheets i need to read contain
>dropdown lists with data validation using a named
>formula like: OFFSET(Data!$K$2,0,0,COUNTA(Data!$K:$K),1)
>which causes openpyxl to throw a NamedRangeException.
>i don't even care about the named objects. i just want
>to know what's in the cell, not what other possible
>values the cell might have had. :-)
>
>apart from that, it does give access to number formats
>so your suggestion would work for simpler spreadsheets.
>
>hopefully the intention that xlrd not support formats in xlsx
>files will change one day into an intention to support them. :-)
>
>until then my users can keep manually saving xlsx files they
>receive as xls before importing them. :-(
>
>maybe i need to investigate some perl modules or pyuno instead.
>perl's Spreadsheet::XSLX module handles formats. it gets the
>date formats a bit wrong but it's workaroundable.
>
>cheers,
>raf
>
>-- 
>You received this message because you are subscribed to the Google Groups 
>"python-excel" group.
>To post to this group, send an email to python-ex...@googlegroups.com.
>To unsubscribe from this group, send email to 
>python-excel+unsubscr...@googlegroups.com.
>For more options, visit this group at 
>http://groups.google.com/group/python-excel?hl=en-GB.
>
>
>
>-- 
http://mail.python.org/mailman/listinfo/python-list

Book recommendation for Spark/Pyspark?

[Albert-Jan Roskam]

Hi,


Can anybody recommend a good, preferably recent, book about Spark and Pyspark? 
I am using Pyspark now, but I am looking for a book that also gives a thorough 
background about Spark itself. I've been looking around on e.g. Amazon but, as 
the saying goes, one can't judge a book by its cover.


Thanks!


Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Numpy and Terabyte data

[Albert-Jan Roskam]

On Jan 2, 2018 18:27, Rustom Mody  wrote:
>
> Someone who works in hadoop asked me:
>
> If our data is in terabytes can we do statistical (ie numpy pandas etc)
> analysis on it?
>
> I said: No (I dont think so at least!) ie I expect numpy (pandas etc)
> to not work if the data does not fit in memory
>
> Well sure *python* can handle (streams of) terabyte data I guess
> *numpy* cannot
>
> Is there a more sophisticated answer?
>
> ["Terabyte" is a just a figure of speech for "too large for main memory"]

Have a look at Pyspark and pyspark.ml. Pyspark has its own kind of DataFrame. 
Very, very cool stuff.

Dask DataFrames have been mentioned already.

numpy has memmapped arrays: 
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tuple unpacking inside lambda expressions

[Albert-Jan Roskam]

   On Apr 20, 2022 13:01, Sam Ezeh  wrote:

 I went back to the code recently and I remembered what the problem was.

 I was using multiprocessing.Pool.pmap which takes a callable (the
 lambda here) so I wasn't able to use comprehensions or starmap

 Is there anything for situations like these?

   =
   Could it simply be:
   multiprocessing.Pool.pmap(lambda job: result.process(*job), jobs)
-- 
https://mail.python.org/mailman/listinfo/python-list

Register multiple excepthooks?

[Albert-Jan Roskam]

   Hi,
   I have a function init_logging.log_uncaught_errors() that I use for
   sys.excepthook. Now I also want to call another function (ffi.dlclose())
   upon abnormal termination. Is it possible to register multiple
   excepthooks, like with atexit.register? Or should I rename/redefine
   log_uncaught_errors() so it does both things?
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Register multiple excepthooks?

[Albert-Jan Roskam]

   On Aug 1, 2022 19:34, Dieter Maurer  wrote:

 Albert-Jan Roskam wrote at 2022-7-31 11:39 +0200:
 >   I have a function init_logging.log_uncaught_errors() that I use for
 >   sys.excepthook. Now I also want to call another function
 (ffi.dlclose())
 >   upon abnormal termination. Is it possible to register multiple
 >   excepthooks, like with atexit.register? Or should I rename/redefine
 >   log_uncaught_errors() so it does both things?

 `sys.excepthook` is a single function (not a list of them).
 This means: at any moment a single `excepthook` is effective.

 If you need a modular design, use a dispatcher function
 as your `excepthook` associated with a registry (e.g. a `list`).
 The dispatcher can then call all registered function.

   
   Thank you both. I'll give this a try. I think it would be nice if the
   standard library function atexit.register would be improved, such that the
   registered functions would not only be called upon (a) normal program
   termination, but that one could also register functions that are called
   (b) upon error (c) unconditionally. Much like (a) try - (b) except - (c)
   finally.
-- 
https://mail.python.org/mailman/listinfo/python-list

Book/resource recommendation about Celery?

[Albert-Jan Roskam]

   Hi,
   I'm using Flask + Celery + RabbitMQ. Can anyone recommend a good book or
   other resource about Celery? 
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Find the path of a shell command

[Albert-Jan Roskam]

   On Oct 14, 2022 18:19, "Peter J. Holzer"  wrote:

 On 2022-10-14 07:40:14 -0700, Dan Stromberg wrote:
 > Alternatively, you can "ps axfwwe" (on Linux) to see environment
 > variables, and check what the environment of cron (or similar) is.  It
 > is this environment (mostly) that cronjobs will inherit.

 The simplest (and IMHO also most reliable) way to find out the
 environment a cronjob has is to write a cronjob which just dumps the
 environment.

   

   =
   Lately I've been using systemd timers instead of cronjobs. They are easier
   to debug (journalctl) but require a bit more work to write. Systemd is
   available on Fedora & friends and Debian based systems, maybe more. It has
   no builtin MAILTO. I use an OnFailure stanza to send a Slack message with
   curl instead.
   https://www.freedesktop.org/software/systemd/man/systemd.timer.html
   https://unix.stackexchange.com/questions/278564/cron-vs-systemd-timers
-- 
https://mail.python.org/mailman/listinfo/python-list

Yaml.unsafe_load error

[Albert-Jan Roskam]

   Hi,
   I am trying to create a celery.schedules.crontab object from an external
   yaml file. I can successfully create an instance from a dummy class "Bar",
   but the crontab class seems call __setstate__ prior to __init__. I have no
   idea how to solve this. Any ideas? See code below.
   Thanks!
   Albert-Jan

   Python 3.6.8 (default, Nov 16 2020, 16:55:22)

   [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux

   Type "help", "copyright", "credits" or "license" for more information.

   >>> import yaml

   >>> from celery.schedules import crontab

   >>> crontab(hour=3, minute=0)

   

   >>> yaml.unsafe_load('!!python/name:celery.schedules.crontab')

   

   >>> yaml.safe_load('celery.schedules.crontab:\n   hour: 3\n   minute:
   0\n')

   {'celery.schedules.crontab': {'hour': 3, 'minute': 0}}

   >>> class Bar:

   ... def __init__(self, x, y):

   ... pass

   ...

   >>> bar = yaml.unsafe_load('!!python/object:__main__.Bar\n   x: 42\n   y:
   666')

   >>> bar

   <__main__.Bar object at 0x7f43b464bb38>

   >>> bar.x

   42

    

   # what is the correct way for the next line?

   >>> yaml.unsafe_load('!!python/object:celery.schedules.crontab\n   hour:
   3\n   minute: 30')

   Traceback (most recent call last):

     File "", line 1, in 

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/__init__.py",
   line 182, in unsafe_load

       return load(stream, UnsafeLoader)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/__init__.py",
   line 114, in load

       return loader.get_single_data()

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 51, in get_single_data

       return self.construct_document(node)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 60, in construct_document

       for dummy in generator:

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 621, in construct_python_object

       self.set_python_instance_state(instance, state)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 727, in set_python_instance_state

       instance, state, unsafe=True)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib64/python3.6/site-packages/yaml/constructor.py",
   line 597, in set_python_instance_state

       instance.__setstate__(state)

     File
   
"/home/albertjan@mycompany/envs/myenv/lib/python3.6/site-packages/celery/schedules.py",
   line 541, in __setstate__

       super().__init__(**state)

   TypeError: __init__() got an unexpected keyword argument 'hour'
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Yaml.unsafe_load error

[Albert-Jan Roskam]

   On Oct 19, 2022 13:02, Albert-Jan Roskam  wrote:

    Hi,
    I am trying to create a celery.schedules.crontab object from an
 external
    yaml file. I can successfully create an instance from a dummy class
 "Bar",
    but the crontab class seems call __setstate__ prior to __init__. I
 have no
    idea how to solve this. Any ideas? See code below.
    Thanks!
    Albert-Jan

    

     

    # what is the correct way for the next line?

    >>> yaml.unsafe_load('!!python/object:celery.schedules.crontab\n  
 hour:
    3\n   minute: 30')

   

   
   Reading the source a bit more, me thinks it might be:
   yaml.unsafe_load('!!python/object/apply:celery.schedules.crontab\nkwds:\n 
    hour: 3\n   minute: 30')
   I did not yet test this, though.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Keeping a list of records with named fields that can be updated

[Albert-Jan Roskam]

   On Dec 15, 2022 10:21, Peter Otten <__pete...@web.de> wrote:

 >>> from collections import namedtuple
 >>> Row = namedtuple("Row", "foo bar baz")
 >>> row = Row(1, 2, 3)
 >>> row._replace(bar=42)
 Row(foo=1, bar=42, baz=3)

   
   Ahh, I always thought these are undocumented methods, but: "In addition to
   the methods inherited from tuples, named tuples support three additional
   methods and two attributes. To prevent conflicts with field names, the
   method and attribute names start with an underscore."
   
https://docs.python.org/3/library/collections.html#collections.somenamedtuple._make
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to enter escape character in a positional string argument from the command line?

[Albert-Jan Roskam]

   On Dec 21, 2022 06:01, Chris Angelico  wrote:

 On Wed, 21 Dec 2022 at 15:28, Jach Feng  wrote:
 > That's what I am taking this path under Windows now, the ultimate
 solution before Windows has shell similar to bash:-)

 Technically, Windows DOES have a shell similar to bash. It's called
 bash. :) The trouble is, most people use cmd.exe instead.

   =
   I use Git Bash quite a lot: https://gitforwindows.org/
   Is that the one you're referring to?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fast lookup of bulky "table"

[Albert-Jan Roskam]

   On Jan 15, 2023 05:26, Dino  wrote:

 Hello, I have built a PoC service in Python Flask for my work, and - now
 that the point is made - I need to make it a little more performant (to
 be honest, chances are that someone else will pick up from where I left
 off, and implement the same service from scratch in a different language
 (GoLang? .Net? Java?) but I am digressing).

   ===
   Hi,
   * I'd start by measuring where your program spends its
   time: https://docs.python.org/3/library/profile.html
   * It might be useful try DuckDB instead of
   Sqlite. https://duckdb.org/why_duckdb.html
   Best wishes,
   AJ
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: LRU cache

[Albert-Jan Roskam]

   I sometimes use this trick, which I learnt from a book by Martelli.
   Instead of try/except, membership testing with "in" (__contains__) might
   be faster. Probably "depends". Matter of measuring.
   def somefunc(arg, _cache={}):
       if len(_cache) > 10 ** 5:
           _cache.pop()
       try:
           return _cache[arg]
       except KeyError:
           result = expensivefunc(arg)
           _cache[arg] = result
           return result
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: LRU cache

[Albert-Jan Roskam]

   On Feb 18, 2023 17:28, Rob Cliffe via Python-list 
   wrote:

 On 18/02/2023 15:29, Thomas Passin wrote:
 > On 2/18/2023 5:38 AM, Albert-Jan Roskam wrote:
 >>     I sometimes use this trick, which I learnt from a book by
 Martelli.
 >>     Instead of try/except, membership testing with "in"
 >> (__contains__) might
 >>     be faster. Probably "depends". Matter of measuring.
 >>     def somefunc(arg, _cache={}):
 >>         if len(_cache) > 10 ** 5:
 >>             _cache.pop()
 >>         try:
 >>             return _cache[arg]
 >>         except KeyError:
 >>             result = expensivefunc(arg)
 >>             _cache[arg] = result
 >>             return result
 >>     Albert-Jan
 >
 > _cache.get(arg) should be a little faster and use slightly fewer
 > resources than the try/except.
 >
 Provided that you can provide a default value to get() which will never
 be a genuine "result".

   =
   This might be better than None:
   _cache.get(arg, Ellipsis)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Packaging/MANIFEST.in: Incude All, Exclude .gitignore

[Albert-Jan Roskam]

   you could call a simple bash script in a git hook that syncs your
   MANIFEST.in with your .gitignore. Something like:
   echo -n "exclude " > MANIFEST.in
   cat .gitignore | tr '\n' ' ' >> MANIFEST.in
   echo "graft $(readlink -f ./keep/this)" >> MANIFEST.in
   https://docs.python.org/2/distutils/sourcedist.html#commands
   On 1 Mar 2021 06:05, Abdur-Rahmaan Janhangeer 
   wrote:

 Greetings list,

 SInce i have a .gitignore, how do i exclude
 all files and folders listed by my gitignore?
 How do i include everything by default?

 Kind Regards,

 Abdur-Rahmaan Janhangeer
 about  | blog
 
 github 
 Mauritius
 --
 https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Albert-Jan Roskam]

   On 20 Mar 2021 23:47, Cameron Simpson  wrote:

 On 20Mar2021 12:53, Sibylle Koczian  wrote:
 >Am 20.03.2021 um 09:34 schrieb Alan Bawden:
 >>The real reason Python strings support a .title() method is surely
 >>because Unicode supports upper, lower, _and_ title case letters, and
 >>tells you how to map between them. [...]
 >>
 >But that's exactly what he's doing, with a result which is documented,
 >but not really satisfactory.

   
   This would be a good
   start: 
https://apastyle.apa.org/style-grammar-guidelines/capitalization/title-case
   It could be locale-dependent. What I also don't like about .title() is
   that it messes up abbreviations ("Oecd")
-- 
https://mail.python.org/mailman/listinfo/python-list

Async requests library with NTLM auth support?

[Albert-Jan Roskam]

   Hi,
   I need to make thousands of requests that require ntlm authentication so I
   was hoping to do them asynchronously. With synchronous requests I use
   requests/requests_ntlm. Asyncio and httpx [1] look promising but don't
   seem to support ntlm. Any tips?
   Cheers!
   Albert-Jan
   [1] https://www.python-httpx.org/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Async requests library with NTLM auth support?

[Albert-Jan Roskam]

   > Asyncio and httpx [1] look promising but don't seem to support ntlm. Any
   tips?
   ==> https://pypi.org/project/httpx-ntlm/
   Not sure how I missed this in the first place. :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Async code across Python versions

[Albert-Jan Roskam]

   Hi,
   I just started using async functions and I was wondering if there are any
   best practices to deal with developments in this area for Python 3.6 and
   up. I'm currently using Python 3.6 but I hope I can upgrade to 3.8 soon.
   Do I have to worry that my code might break? If so, is it better to use
   3rd party libraries? It seems that things get a little easier with newer
   Python versions, so it might also a reason to simplify the code.
   Cheers!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: argparse support of/by argparse

[Albert-Jan Roskam]

 >>> [1] https://pypi.org/project/clize/

   I use and like docopt (https://github.com/docopt/docopt). Is clize a
   better choice?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Tracing in a Flask application

[Albert-Jan Roskam]

   Hi,
   logging.basicConfig(level="DEBUG")
   ..in e.g  __init__.py
   AJ
   On 4 Aug 2021 23:26, Javi D R  wrote:

 Hi

 I would like to do some tracing in a flask. I have been able to trace
 request in plain python requests using sys.settrace(), but this doesnt
 work
 with Flask.

 Moreover, what i want to trace is in a flask application, when an
 endpoint
 is called, what was the request, which parts of the code was executed (i
 think i can still do it with settrace) and what is the response sent by
 the
 application

 Can you help me to understand which services i can use to do this?

 Thanks
 --
 https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Ansible, pip and virtualenv

[Albert-Jan Roskam]

   Hi
   I wrote an Ansible .yml to deploy a Flask webapp. I use python 3.6 for the
   ansible-playbook executable. The yml starts with some yum installs,
   amongst which python-pip. That installs an ancient pip version (v9). Then
   I create a virtualenv where I use a requirements.txt for pip install -r.
   Should I precede that step with a separate --upgrade pip step? Or should
   pip just be in my requirements.txt?
   Separate question: what locations do I need to specify in my pip
   --trusted-host list? Could it be that this has recently changed? I
   suddenly got SSL errors.
   Thanks in advance!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to apply a self defined function in Pandas

[Albert-Jan Roskam]

 > df['URL'] = df.apply(lambda x:  connect(df['URL']), axis=1)

   I think you need axis=0. Or use the Series, df['URL'] =
   df.URL.apply(connect)
-- 
https://mail.python.org/mailman/listinfo/python-list

Call julia from Python: which package?

[Albert-Jan Roskam]

Hi,

I have a Python program that uses Tkinter for its GUI. It's rather slow so I 
hope to replace many or all of the non-GUI parts by Julia code. Has anybody 
experience with this? Any packages you can recommend? I found three 
alternatives:

* https://pyjulia.readthedocs.io/en/latest/usage.html#
* https://pypi.org/project/juliacall/
* https://github.com/JuliaPy/PyCall.jl

Thanks in advance!

Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Call julia from Python: which package?

[Albert-Jan Roskam]

   Hi all,
   Thank you very much for your valuable replies! I will definitely do some
   tracing to see where the bottlenecks really are. It's good to know that
   pypy is still alive and kicking, I thought it was stuck in py2.7. I will
   also write a mini program during the holiday to see how this Julia/Python
   interaction might work. The little bit of experience with Julia more or
   less coincides with what Oscar mentioned: a lot of "warm up" time. This is
   actually a py2.7 project that I inherited. I was asked to convert it to
   py3.8.
   Thanks and merry xmas!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Gunicorn - HTTP and HTTPS in the same instance?

[Albert-Jan Roskam]

   I always use NGINX for this. Run Flask/Gunicorn on localhost:5000 and have
   NGINX rewrite https requests to localhost requests. In nginx.conf I
   automatically redirect every http request to https. Static files are
   served by NGINX, not by Gunicorn, which is faster. NGINX also allows you
   to easily set the transfer encoding to gzip
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Waht do you think about my repeated_timer class

[Albert-Jan Roskam]

   On Feb 2, 2022 23:31, Barry  wrote:

 > On 2 Feb 2022, at 21:12, Marco Sulla 
 wrote:
 >
 > You could add a __del__ that calls stop :)

 Didn't python3 make this non deterministic when del is called?

 I thought the recommendation is to not rely on __del__ in python3 code.

   ==>
   Adding __del__ also poses chalenges is you would like to support pypy:
   "There are a few extra implications from the difference in the GC. Most
   notably, if an object has a __del__, the __del__ is never called more than
   once in PyPy; but CPython will call the same __del__ several times if the
   object is resurrected and dies again (at least it is reliably so in older
   CPythons; newer CPythons try to call destructors not more than once, but
   there are counter-examples). The __del__ methods are called in "the right"
   order if they are on objects pointing to each other, as in CPython, but
   unlike CPython, if there is a dead cycle of objects referencing each
   other, their __del__ methods are called anyway; CPython would instead put
   them into the list garbage of the gc module."
   https://doc.pypy.org/en/latest/cpython_differences.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Pypy with Cython

[Albert-Jan Roskam]

   Hi,
   I inherited a fairly large codebase that I need to port to Python 3. Since
   the program was running quite slow I am also running the unittests against
   pypy3.8. It's a long running program that does lots of pairwise
   comparisons of string values in two files. Some parts of the program (e.g
   a modulo 11 digit check) are implemented in Cython. Should I use pure
   Python instead when using Pypy? I compiled the Cython modules for pypy and
   they work, but I'm afraid they might just slow things down.
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pypy with Cython

[Albert-Jan Roskam]

   On Feb 3, 2022 17:01, Dan Stromberg  wrote:

 > The best answer to "is this slower on

 > Pypy" is probably to measure.
 > Sometimes it makes sense to rewrite C

 > extension modules in pure python for pypy.

   
   Hi Dan, thanks. What profiler do you recommend I normally use cProfile,
   but I was thinking about this
   one: https://pyinstrument.readthedocs.io/en/latest/index.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error installing requirements

[Albert-Jan Roskam]

   On Feb 18, 2022 08:23, Saruni David  wrote:
   >> Christian Gohlke's site has a Pillow .whl for python
   2.7: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pillow
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Long running process - how to speed up?

[Albert-Jan Roskam]

   On Feb 19, 2022 12:28, Shaozhong SHI  wrote:

 I have a cvs file of 932956 row and have to have time.sleep in a Python
 script.  It takes a long time to process.

 How can I speed up the processing?  Can I do multi-processing?

   
   Perhaps a dask df: 
   https://docs.dask.org/en/latest/generated/dask.dataframe.read_csv.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: One-liner to merge lists?

[Albert-Jan Roskam]

 If you don't like the idea of 'adding' strings you can 'concat'enate:

 >>> items = [[1,2,3], [4,5], [6]]
 >>> functools.reduce(operator.concat, items)
 [1, 2, 3, 4, 5, 6]
 >>> functools.reduce(operator.iconcat, items, [])
 [1, 2, 3, 4, 5, 6]

 The latter is the functional way to spell your for... extend() loop.
 Don't forget to provide the initial value in that case lest you modify
 the input:

 >> functools.reduce(operator.iconcat, items)  # wrong
 [1, 2, 3, 4, 5, 6]
 >>> items
 [[1, 2, 3, 4, 5, 6], [4, 5], [6]]  # oops

 --
 https://mail.python.org/mailman/listinfo/python-list

   
   Was also thinking about reduce, though this one uses a dunder method:
   from functools import reduce
   d = {1: ['aaa', 'bbb', 'ccc'], 2: ['fff', 'ggg']}
   print(reduce(list.__add__, list(d.values(
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: JSON vs. PickleType vs. raw string for serialised data

[Albert-Jan Roskam]

   On Feb 28, 2022 10:11, Loris Bennett  wrote:

 Hi,

 I have an SQLAlchemy class for an event:

   class UserEvent(Base):
   __tablename__ = "user_events"

   id = Column('id', Integer, primary_key=True)
   date = Column('date', Date, nullable=False)
   uid = Column('gid', String(64), ForeignKey('users.uid'),
 nullable=False)
   info = ??

 The event may have arbitrary, but dict-like data associated with it,
 which I want to add in the field 'info'.  This data never needs to be
 modified, once the event has been inserted into the DB.

 What type should the info field have?  JSON, PickleType, String, or
 something else?

 I couldn't find any really reliable sounding information about the
 relative
 pros and cons, apart from a Reddit thread claiming that pickled dicts
 are larger than dicts converted to JSON or String.

 Cheers,

 Loris

   
   I think you need a
   BLOB. 
https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.LargeBinary
-- 
https://mail.python.org/mailman/listinfo/python-list

Marshmallow: json-to-schema helper?

[Albert-Jan Roskam]

   Hi,
   I'm looking for a convenience function to convert a Marshmallow schema
   into a valid Python class definition. That is, I want to generate python
   code (class MySchema.. etc) that I could write to a .py file. Does this
   exist? I tried the code below, but that is not the intended use of
   marshmallow.Schema.from_dict or inspect.getsource.
   from inspect import getsource
   from marshmallow import Schema
   d = dict(something='potentially', very='complicated')
   schema = Schema.from_dict(d)
   python_class_def_as_str = getsource(schema())  # OSError
   
https://marshmallow.readthedocs.io/en/stable/api_reference.html#marshmallow.Schema.from_dict
   https://docs.python.org/3/library/inspect.html#inspect.getsource
   Thanks!
   Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Fwd: dict.get_deep()

[Albert-Jan Roskam]

   -- Forwarded message --
   From: Marco Sulla 
   Date: Apr 2, 2022 22:44
   Subject: dict.get_deep()
   To: Python List <>
   Cc:

 A proposal. Very often dict are used as a deeply nested carrier of
 data, usually decoded from JSON. 

 data["users"][0]["address"]["street"]

   I have been using jsonpath expressions for that kind of stuff lately.
   Works really well.
   https://pypi.org/project/jsonpath-ng/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: flask app convert sql query to python plotly.

[Albert-Jan Roskam]

   On Apr 2, 2022 20:50, Abdellah ALAOUI ISMAILI 
   wrote:

 i would like to convert in my flask app an SQL query to an plotly pie
 chart using pandas. this is my code :

 def query_tickets_status() :
     query_result = pd.read_sql ("""
     SELECT COUNT(*)count_status, status
     FROM tickets
     GROUP BY status""", con = mydc_db)
     return query_result

 labels_statut = query_tickets_status['status']
 values_statut = query_tickets_status['count_status']

   ==
   I think you mean:
   labels_statut = query_tickets_status()['status']
   values_statut = query_tickets_status()['count_status']
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is that forwards first or backwards first? (Re: unintuitive for-loop behavior)

[Albert-Jan Roskam]

(Sorry for top-posting)

Yep, part of the baby's hardware. Also, the interface is not limited to visual 
and auditory information:
http://www.scientificamerican.com/article/pheromones-sex-lives/

From: Python-list  on 
behalf of Steven D'Aprano 
Sent: Tuesday, October 4, 2016 7:00:48 AM
To: python-list@python.org
Subject: Re: Is that forwards first or backwards first? (Re: unintuitive 
for-loop behavior)

On Tuesday 04 October 2016 14:51, Michael Torrie wrote:

> On 10/03/2016 08:21 PM, Steve D'Aprano wrote:
>> On Tue, 4 Oct 2016 05:48 am, Michael Torrie wrote:
>>
>>> There is that old, but false, saying that the only intuitive interface
>>> is the nipple.  Turns out everything, even that, is learned
>>
>> Citation required.
>
> Sure, just ask a nursing woman.
[...]
> Sucking seems to be instinctive but the actual interface is learned
> (albeit very quickly) by experience.

You say tomahto, I say tomarto. It sounds like we're using different language
to describe the same thing.

Babies do have an instinct to suck if you stick a nipple (or a finger) in their
mouth, or even if you merely touch them on the lips or cheek:

http://www.medicinenet.com/script/main/art.asp?articlekey=5392

https://en.wikipedia.org/wiki/Primitive_reflexes#Rooting_reflex

The rooting instinct, together with the sucking instinct, is present in all
healthy babies. I would expect that only the most severe development
abnormalities would prevent instincts as important for survival as these two.

Between the rooting and sucking instincts, I consider "the only intuitive
interface is the nipple" is correct. However, that doesn't necessarily mean
that babies will suckle well: there are all sorts of reasons why babies don't
breast-feed well, e.g.:

http://www.cyh.com/HealthTopics/HealthTopicDetails.aspx?p=114&np=302&id=1960


> Babies don't just see a nipple and know what it's for.

Of course the instinct isn't *visual* -- babies may not open their eyes for
many minutes after birth (one study found that about 3% of babies hadn't opened
their eyes within 20 minutes, the maximum time allotted) which may be long
after their first suckle.

Nevertheless, there are senses other than sight.



--
Steven
git gets easier once you get the basic idea that branches are homeomorphic
endofunctors mapping submanifolds of a Hilbert space.

--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Adding colormaps?

[Albert-Jan Roskam]

(sorry for top-posting)
I does not appear to be possible in matplolibrc (1). But you can use 
matplotlib.cm.register_cmap to register new cmaps (2) such as these (3).

(Note: I did not try this)

(1)http://matplotlib.org/1.4.0/users/customizing.html
(2)http://matplotlib.org/api/cm_api.html
(3)https://github.com/BIDS/colormap/blob/master/colormaps.py

From: Python-list  on 
behalf of Martin Schöön 
Sent: Saturday, January 21, 2017 8:42:29 PM
To: python-list@python.org
Subject: Re: Adding colormaps?

Den 2017-01-21 skrev Gilmeh Serda :
> On Wed, 18 Jan 2017 21:41:34 +, Martin Schöön wrote:
>
>> What I would like to do is to add the perceptually uniform sequential
>> colormaps introduced in version 1.5.something. I would like to do this
>> without breaking my Debian system in which Matplotlib version 1.4.2 is
>> the newest version available in the repo.
>
> Haven't checked, but I assume you can get the source. Compile it but
> don't install it and then use the result in virtualenv, maybe?
>
I am hoping for directions to a config file to download and place
somewhere...

/Martin
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How coding in Python is bad for you

[Albert-Jan Roskam]

sola dosis facit venenum ~ Paracelsus (1493-1541)

From: Python-list  on 
behalf of alister 
Sent: Monday, January 23, 2017 8:32:49 PM
To: python-list@python.org
Subject: Re: How coding in Python is bad for you

On Tue, 24 Jan 2017 07:19:42 +1100, Chris Angelico wrote:

> On Tue, Jan 24, 2017 at 6:59 AM, Grant Edwards
>  wrote:
>> On 2017-01-23, breamore...@gmail.com  wrote:
>>
>>> The article is here http://lenkaspace.net/index.php/blog/show/111
>>
>> I don't really think any of his points are valid, but one way that
>> programming in Python is bad for you:
>>
>>  * It reduces your tolerance for progamming in PHP zero.  If you end
>>up assigned to a PHP project, you end up miserable and unpleasant to
>>work with or just plain unemployed.
>
> I believe that's "bad for you" in the sense that chocolate is bad for
> you.
>
> It isn't.
>
> ChrisA

chocolate is a poison (lethal dose for a human approx 22lb)



--
Make headway at work.  Continue to let things deteriorate at home.
--
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Pandas, create new column if previous column(s) are not in [None, '', np.nan]

[Albert-Jan Roskam]

On Apr 11, 2018 20:52, zljubi...@gmail.com wrote:
>
> I have a dataframe:
>
> import pandas as pd
> import numpy as np
>
> df = pd.DataFrame( { 'A' : ['a', 'b', '', None, np.nan],
>  'B'  : [None, np.nan, 'a', 'b', '']})
>
>   A B
> 0 a  None
> 1 b   NaN
> 2   a
> 3  None b
> 4   NaN
>
>
> I would like to create column C in the following way:
> column C = column B if column B is not in [None, '', np.nan]
> else column A
>
> How to do that?
>
> I tried:
>
> df['C'] = df[['A', 'B']].apply(lambda x: x[1] if x[1] in [None, '', np.nan] 
> else x[0])
>
> but I got all np.nan's.

This is another approach:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to write partial of a buffer which was returned from a C function to a file?

[Albert-Jan Roskam]

On Apr 12, 2018 09:39, jf...@ms4.hinet.net wrote:
>
> Chris Angelico於 2018年4月12日星期四 UTC+8下午1時31分35秒寫道：
> > On Thu, Apr 12, 2018 at 2:16 PM,   wrote:
> > > This C function returns a buffer which I declared it as a 
> > > ctypes.c_char_p. The buffer has size 0x1 bytes long and the valid 
> > > data may vary from a few bytes to the whole size.
> > >
> > > In every call I know how much the valid data size is, but I suppose I 
> > > can't use slice to get it because there may be zero byte in it. What to 
> > > do?
> > >
> >
> > You suppose? Or have you tested it?
> >
> > ChrisA
>
> Yes, I had test it once before. Now, I re-do it again to make sure. After a 
> call which returns 3 bytes of data, I use len(buf) to check the length and 
> get the number 24. I can see the first 24 bytes of data by using buf[:30] but 
> buf[24] will cause an "index out of range" error. I don't know how to see 
> what the buf[24] exactly is but I suppose it might be a zero byte.

Aren't you looking for the .value or the .raw property?
-- 
https://mail.python.org/mailman/listinfo/python-list

Flask test generator code review?

[Albert-Jan Roskam]

Hi,

I am writing my first unittests for a Flask app. First modest goal is to test 
whether a selected subset of the templates return the expected status 200. 
I am using a nose test generator in a class for this. Is the code below the 
correct way to do this? And is there a way to dynamically set the docstring of 
test_generator? This would make the nosetests output a bit more understandable. 

Thanks!
Albert-Jan

import os
import sys
from os.path import splitext
from http import HTTPStatus as status

import nose

from MyFabulousApp import app

app.testing = True
template_folder = app.config['TEMPLATE_FOLDER']


class Test_MyFabulousApp_HTTP_Status_OK:

def __init__(self):
self.setup()   # with unittest, setUp is called automatically, but not 
with nose

def setup(self):
self.client = app.test_client()
self.client.post('/login', follow_redirects=True)

def teardown(self):
self.client.post('/logout', follow_redirects=True)

def test_generator(self):
"""Does template return HTTP Status 200?"""
def the_test(self, template):
# the following line throws an error: AttributeError: attribute 
'__doc__' of 'method' objects is not writable
#self.test_generator.__doc__ = 'Does template "%s" return HTTP 
Status 200?' % template
respons = self.client.get('/' + template)
actual = respons.status_code
desired = status.OK.value
assert actual == desired, \
   'Template "%s" returns status code %d' % (template, actual)
templates = [splitext(item)[0] for item in os.listdir(template_folder)]
for template in templates:
yield the_test, self, template


if __name__ == '__main__':
nose.run(defaultTest=__name__, argv=[sys.argv[0], '__main__', 
'--verbosity=2'])
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: RE newbie question

[Albert-Jan Roskam]

On Apr 18, 2018 21:42, TUA  wrote:
>
> import re
>
> compval = 'A123456_8'
> regex = '[a-zA-Z]\w{0,7}'
>
> if re.match(regex, compval):
>print('Yes')
> else:
>print('No')
>
>
> My intention is to implement a max. length of 8 for an input string. The 
> above works well in all other respects, but does allow for strings that are 
> too long.
>
> What is the proper way to fix this?

Use a $ sign at the end of the regex
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: The basics of the logging module mystify me

[Albert-Jan Roskam]

On Apr 19, 2018 03:03, Skip Montanaro  wrote:
>
>
> I really don't like the logging module, but it looks like I'm stuck
> with it. Why aren't simple/obvious things either simple or obvious?

Agreed. One thing that, in my opinion, ought to be added to the docs is sample 
code to log uncaught exceptions using an excepthook, with correctly formatted 
traceback.

Another thing I always do is to define a custom log level named 'message', 
which is always logged. Basically it's logging.FATAL + 1, but with a friendlier 
label. Would be a nice enhancement of the logging module IMHO.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extract data

[Albert-Jan Roskam]

On May 15, 2018 08:54, Steven D'Aprano  
wrote:

On Tue, 15 May 2018 11:53:47 +0530, mahesh d wrote:

> Hii.
>
>  I have folder.in that folder some files .txt and some files .msg files.
>  .
> My requirement is reading those file contents . Extract data in that
> files .

Reading .msg can be done with win32com or 
https://github.com/mattgwwalker/msg-extractor

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extract data from multiple text files

[Albert-Jan Roskam]

On May 15, 2018 14:12, mahesh d  wrote:

import glob,os

import errno

path = 'C:/Users/A-7993\Desktop/task11/sample emails/'

files = glob.glob(path)

'''for name in files:

print(str(name))

if name.endswith(".txt"):

   print(name)'''

for file in os.listdir(path):

print(file)

if file.endswith(".txt"):

print(os.path.join(path, file))

print(file)

try:

with open(file) as f:

msg = f.read()

print(msg)

except IOError as exc:

if exc.errno != errno.EISDIR:

raise


In the above program . Getting lot of errors . My intention is read the
list of the text files in a folder . Print them


How can resolve those error
--
https://mail.python.org/mailman/listinfo/python-list

Try:
path = 'C:/Users/A-7993/Desktop/task11/sample emails/*.msg'
msg_files = glob.glob(path)
print(msg_files)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: user defined modules

[Albert-Jan Roskam]

On 5 Jun 2018 09:32, Steven D'Aprano  
wrote:

On Mon, 04 Jun 2018 20:13:32 -0700, Sharan Basappa wrote:

> Is there a specific location where user defined modules need to be kept?
> If not, do we need to specify search location so that Python interpreter
> can find it?

Python modules used as scripts can be run from anywhere, by pointing the
interpreter at the script:

python /path/to/my/script.py


But Python modules uses as libraries, to be imported by other modules,
have to be on the Python search path. You can add extra  paths to the
Python search path from the shell by setting the environment variable
PYTHONPATH to a colon-separated list of paths. On Linux, I do this in
my .bashrc config file:

export PYTHONPATH="paths:to:add"

In the Python interpreter, you can query and modify the search path by
importing sys and looking at sys.path. (But you should not do so unless
you really know what you are doing.)

The default search path is set by the site module:

https://docs.python.org/3/library/site.html

but again, you should not mess with this unless you know what you are
doing.

There are some per-user directories which are automatically added to the
search path. I can't find the existing documentation for them, but a good
place to start is the PEP that introduced the feature:

https://www.python.org/dev/peps/pep-0370/


Apart from setting the PYTHONPATH environment variable, the best way to
add extra paths to is to install a .pth file.

If you run both Python 2 and 3, than .pth might be a better choice. With 
Pythonpath, you run the risk of e.g. importing python3-incompatible code.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python 3.7 - I try to close the thread without closing the GUI is it possible?

[Albert-Jan Roskam]

 > I try to close the thread without closing the GUI is it possible?


Qthread seems to be worth investigating:
https://medium.com/@webmamoffice/getting-started-gui-s-with-python-pyqt-qthread-class-1b796203c18c
-- 
https://mail.python.org/mailman/listinfo/python-list

[OT] master/slave debate in Python

[Albert-Jan Roskam]

*sigh*. I'm with Hettinger on this.

https://www.theregister.co.uk/2018/09/11/python_purges_master_and_slave_in_political_pogrom/
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: A tool to add diagrams to sphinx docs

[Albert-Jan Roskam]

> Subject: Re: A tool to add diagrams to sphinx docs
> From: irmen.nos...@xs4all.nl
> Date: Fri, 1 Apr 2016 18:26:48 +0200
> To: python-list@python.org
> 
> On 1-4-2016 17:59, George Trojan - NOAA Federal wrote:
> > What graphics editor would you recommend to create diagrams that can be
> > included in sphinx made documentation? In the past I used xfig, but was not
> > happy with font quality. My understanding is the diagrams would be saved in
> > a .png file and I should use an image directive in the relevant .rst file.
> > 
> > George
> > 
> 
> I've used .png successfully for a few images in my sphinx docs.
> However if you're concerned with font quality, a better option is perhaps to 
> use a
> vector format like SVG instead. You can create them using a tool like 
> Inkskape or
> Illustrator. I haven't tried to use .svg in sphinx myself yet but I guess it 
> simply
> embeds/links it into the resulting html so a recent web browser should 
> display them just
> fine.
I used .svg for dependency graphs made with snakefood. Looked good in html, 
though they essily get too detailed. The .rst:

.. figure:: images/my_image.svg:width: 100% 
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: extract rar

[Albert-Jan Roskam]

> Date: Fri, 1 Apr 2016 13:22:12 -0600
> Subject: extract rar
> From: fanjianl...@gmail.com
> To: python-list@python.org
> 
> Hello everyone,
> 
> I am wondering is there any way to extract rar files by python without
> WinRAR software?
> 
> I tried Archive() and patool, but seems they required the WinRAR software.

Perhaps 7-zip in a Python 
subprocess:http://superuser.com/questions/458643/unzip-rar-from-command-line-with-7-zip/464128
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: read datas from sensors and plotting

[Albert-Jan Roskam]

> From: ran...@nospam.it
> Subject: read datas from sensors and plotting
> Date: Sun, 17 Apr 2016 18:46:25 +0200
> To: python-list@python.org
> 
> I'm reading in python some values from some sensors and I write them in 
> a csv file.
> My problem now is to use this datas to plot a realtime graph for a 
> example in a web server.
> Is it possible to read in the same time the values, writing in the file 
> and plot them in a webpage with python?


tail -F data.log | python myprogram.py
http://stackoverflow.com/questions/1712276/tail-read-a-growing-dynamic-file-and-extract-two-columns-and-then-print-a-graphhttp://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Remove directory tree without following symlinks

[Albert-Jan Roskam]

> From: st...@pearwood.info
> Subject: Re: Remove directory tree without following symlinks
> Date: Sat, 23 Apr 2016 03:14:12 +1000
> To: python-list@python.org
> 
> On Sat, 23 Apr 2016 01:09 am, Random832 wrote:
> 
> > On Fri, Apr 22, 2016, at 10:56, Steven D'Aprano wrote:
> >> What should I use for "remove_tree"? Do I have to write my own, or does a
> >> solution already exist?
> > 
> > In the os.walk documentation it provides a simple recipe and also
> > mentions shutil.rmtree
> 
> Thanks for that.

FYI, Just today I found out that shutil.rmtree raises a WindowsError if the dir 
is read-only (or its contents). Using 'ignore_errors', won't help. Sure, no 
error is raised, but the dir is not deleted either! A 'force' option would be a 
nice improvement.



> The os.walk recipe is described as a simple version of shutil.rmtree. The
> documentation for rmtree seems lacking to me, but after testing it, it
> appears to work as I want it: it removes symbolic links, it does not follow
> them.
>
> Is anyone else able to confirm that my understanding is correct? If so, the
> documentation should probably be a bit clearer.
> 
> 
> 
> -- 
> Steven
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Remove directory tree without following symlinks

[Albert-Jan Roskam]

> From: eryk...@gmail.com
> Date: Fri, 22 Apr 2016 13:28:01 -0500
> Subject: Re: Remove directory tree without following symlinks
> To: python-list@python.org
> 
> On Fri, Apr 22, 2016 at 12:39 PM, Albert-Jan Roskam
>  wrote:
> > FYI, Just today I found out that shutil.rmtree raises a WindowsError if the 
> > dir is read-
> > only (or its contents). Using 'ignore_errors', won't help. Sure, no error 
> > is raised, but the
> > dir is not deleted either! A 'force' option would be a nice improvement.
> 
> Use the onerror handler to call os.chmod(path, stat.S_IWRITE). For
> example, see pip's rmtree_errorhandler:
> 
> https://github.com/pypa/pip/blob/8.1.1/pip/utils/__init__.py#L105

Thanks, that looks useful indeed. I thought about os.chmod, but with os.walk. 
That seemed expensive. So I used subprocess.call('rmdir "%s" /s /q' % dirname). 
That's Windows only, of course, but aside of that, is using subprocess less 
preferable?Fun fact: I used it to remove .svn dirs, just like what is mentioned 
in the pip comments :-)

  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Remove directory tree without following symlinks

[Albert-Jan Roskam]

> From: eryk...@gmail.com
> Date: Sat, 23 Apr 2016 15:22:35 -0500
> Subject: Re: Remove directory tree without following symlinks
> To: python-list@python.org
> 
> On Sat, Apr 23, 2016 at 4:34 AM, Albert-Jan Roskam
>  wrote:
>>
>>> From: eryk...@gmail.com
>>> Date: Fri, 22 Apr 2016 13:28:01 -0500
>>> On Fri, Apr 22, 2016 at 12:39 PM, Albert-Jan Roskam
>>>  wrote:
>>>> FYI, Just today I found out that shutil.rmtree raises a WindowsError if
>>>> the dir is read-only (or its contents). Using 'ignore_errors', won't help.
>>>> Sure, no error is raised, but the dir is not deleted either! A 'force' 
>>>> option
>>>> would be a nice improvement.
>>>
>>> Use the onerror handler to call os.chmod(path, stat.S_IWRITE). For
>>> example, see pip's rmtree_errorhandler:
>>>
>>> https://github.com/pypa/pip/blob/8.1.1/pip/utils/__init__.py#L105
>>
>> Thanks, that looks useful indeed. I thought about os.chmod, but with
>> os.walk. That seemed expensive. So I used subprocess.call('rmdir "%s" /s /q'
>> % dirname). That's Windows only, of course, but aside of that, is using
>> subprocess less preferable?
> 
> I assume you used shell=True in the above call, and not an external
> rmdir.exe. There are security concerns with using the shell if you're
> not in complete control of the command line.
> 
> As to performance, cmd's rmdir wins without question, not only because
> it's implemented in C, but also because it uses the stat data from the
> WIN32_FIND_DATA returned by FindFirstFile/FindNextFile to check for
> FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_READONLY.
> 
> On the other hand, Python wins when it comes to working with deeply
> nested directories. Paths in cmd are limited to MAX_PATH characters.
> rmdir uses DOS 8.3 short names (i.e. cAlternateFileName in
> WIN32_FIND_DATA), but that could still exceed MAX_PATH for a deeply
> nested tree, or the volume may not even have 8.3 DOS filenames.
> shutil.rmtree allows you to work around the DOS limit by prefixing the
> path with "\\?\". For example:
> 
>>>> subprocess.call(r'rmdir /q/s Z:\Temp\long', shell=True)
> The path Z:\Temp\long\aa
> 
> 
> 
> a is too long.
> 0
> 
>>>> shutil.rmtree(r'\\?\Z:\Temp\long')
>>>> os.path.exists(r'Z:\Temp\long')
> False
> 
> Using "\\?\" requires a path that's fully qualified, normalized
> (backslash only), and unicode (i.e. decode a Python 2 str).

Aww, I kinda forgot about that already, but I came across this last year [1]. 
Apparently, 
shutil.rmtree(very_long_path) failed under Win 7, even with the "silly prefix". 
I believe very_long_path was a Python2-str.
It seems useful if shutil or os.path would automatically prefix paths with 
"\\?\". It is rarely really needed, though.
(in my case it was needed to copy a bunch of MS Outlook .msg files, which 
automatically get the subject line as the filename, and perhaps
the first sentence of the mail of the mail has no subject).

[1] https://mail.python.org/pipermail/python-list/2015-June/693156.html

  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: re.search - Pattern matching review ( Apologies re sending)

[Albert-Jan Roskam]

> Date: Sat, 28 May 2016 23:48:16 +0530
> Subject: re.search - Pattern matching review ( Apologies re sending)
> From: ganesh1...@gmail.com
> To: python-list@python.org
> 
> Dear Python friends,
> 
> I am  on Python 2.7 and Linux . I am trying to extract the address
> "1,5,147456:8192" from the below stdout using re.search
> 
> (Pdb) stdout
> 'linux-host-machine-1: Block Address for 1,5,27320320:8192 (block
> 1,5,147456:8192) --\nlinux-host-machine-1: magic
> 0xdeaff2fe mark_cookie 0x300a\n'
> (Pdb) type(stdout)
> 
> 
> Here is the code I have come up with, this looks buggy please review
> the same and suggest any better ways  to code.
> 
> Could we use splitlines() or re.complie() etc , my intention is to
> match 1,5,147456:8192 and return the same.
> 
> 
> #Sample code
> 
> import re
> import subprocess_run
> 
> def get_block():
> try:
> cmd = "get_block_info -l"
> # stdout is the output retrieved by subprocess.Popen()
> stdout, stderr, exitcode = subprocess_run(cmd)
> search_pat = 'Block Address.* \(block (\d+),(\d+),(\d+):(\d+)'
> matched = re.search(search_pat, stdout)
> block = (int(matched.group(1)),
>int(matched.group(2)),
>int(matched.group(3)),
>int(matched.group(4)),
>   )
Perhaps:map(int,  re.search(search_pat, stdout).groups())
Or re.findall
> except IOError, e:
> logging.warning('Error reading lines from "%s" (%s).'
> % (cmd, e))
> 
> if block is None:
>logging.error("block not found")
>return False
> logging.info("block not found")
> return block
> 
> Regards,
> 
> Ganesh
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What Python related git pre-commit hooks are you using?

[Albert-Jan Roskam]

On 18 Nov 2018 20:33, Malcolm Greene  wrote:

>Curious to learn what Python related git >pre-commit hooks people are using? 
>What >hooks have you found useful and which >hooks have you tried

I use Python to reject large commits (pre-commit hook): 
http://code.activestate.com/recipes/578883-git-pre-commit-hook-to-reject-large-files-using-py/

I've also used hooks to trigger a Sphinx doc build, and for tox.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Most "pythonic" syntax to use for an API client library

[Albert-Jan Roskam]

On 29 Apr 2019 07:18, DL Neil  wrote:

On 29/04/19 4:52 PM, Chris Angelico wrote:
> On Mon, Apr 29, 2019 at 2:43 PM DL Neil  
> wrote:
>>
>> On 29/04/19 3:55 PM, Chris Angelico wrote:
>>> On Mon, Apr 29, 2019 at 1:43 PM DL Neil  
>>> wrote:
 Well, seeing you ask: a more HTTP-ish approach *might* be:

 api.update.customer( 1, name='Bob' )

 ie
 api.verb.subject( adjectives and adverbs )

 Thus:
 api_label/intro/ID.what_we're_going_to_do.who/what_we'll_do_it_to(
 customerID, support_data)

 Yet, it doesn't really *look right* does it?
 (and someone might complain about mixing the 'different' 
 variable-values...)

>>>
>>> The point here is not to make an HTTP-like interface, but a
>>> Python-like interface :)
>>

I recently used Python properties (getter, setter, deleter) for GET, POST/PUT, 
DELETE, respectively. I didn't need any other http methods.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Creating time stamps

[Albert-Jan Roskam]

On 22 Jul 2019 23:12, Skip Montanaro  wrote:

Assuming you're using Python 3, why not use an f-string?

>>> dt = datetime.datetime.now()
>>> dt.strftime("%Y-%m-%d %H:%M")
'2019-07-22 16:10'
>>> f"{dt:%Y-%m-%d %H:%M}"
'2019-07-22 16:10'

===》》 Or if you're running < Python 3.6 (no f strings): format(datetime.now(), 
"%Y-%m-%d %H:%M")
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Document Entire Apps

[Albert-Jan Roskam]

On 15 Sep 2019 07:00, Sinardy Gmail  wrote:

I understand that we can use pydoc to document procedures how about the 
relationship between packages and dependencies ?



==》 Check out snakefood to generate dependency graphs: 
http://furius.ca/snakefood/. Also, did you discover sphinx already?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Tutor] Most efficient way to replace ", " with "." in a array and/or dataframe

[Albert-Jan Roskam]

On 22 Sep 2019 04:27, Cameron Simpson  wrote:

On 21Sep2019 20:42, Markos  wrote:
>I have a table.csv file with the following structure:
>
>, Polyarene conc ,, mg L-1 ,,,
>Spectrum, Py, Ace, Anth,
>1, "0,456", "0,120", "0,168"
>2, "0,456", "0,040", "0,280"
>3, "0,152", "0,200", "0,280"
>
>I open as dataframe with the command:
>data = pd.read_csv ('table.csv', sep = ',', skiprows = 1)
[...]
>And the data_array variable gets the fields in string format:
>[['0,456' '0,120' '0,168']
[...]

>Please see the documentation for the >read_csv function here:

> https://pandas.pydata.org/pandas

>docs/stable/reference/api/pandas.read_cs> 
>v.html?highlight=read_csv#pandas.read_csv

Do you think it's a deliberate design choice that decimal and thousands where 
used here as params, and not a 'locale' param? It seems nice to be able to 
specify e.g. locale='dutch' and then all the right lc_numeric, lc_monetary, 
lc_time where used. Or even locale='nl_NL.1252' and you also wouldn't need 
'encoding' as a separate param. Or might that be bad on windows where there's 
no locale-gen? Just wondering...

Albert-Jan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Funny code

[Albert-Jan Roskam]

On 26 Sep 2019 10:28, Christian Gollwitzer  wrote:

Am 26.09.19 um 08:34 schrieb ast:
> Hello
>
> A line of code which produce itself when executed
>
>  >>> s='s=%r;print(s%%s)';print(s%s)
> s='s=%r;print(s%%s)';print(s%s)
>
> Thats funny !

==> Also impressive, a 128-language quine:
https://github.com/mame/quine-relay

-- 
https://mail.python.org/mailman/listinfo/python-list

sqlalchemy & #temp tables

[Albert-Jan Roskam]

Hi,

I am using sqlalchemy (SA) to access a MS SQL Server database (python 3.5, Win 
10). I would like to use a temporary table (preferably #local, but ##global 
would also be an option) to store results of a time-consuming query. In other 
queries I'd like to access the temporary table again in various places in my 
Flask app. How do I do that, given that SA closes the connection after each 
request?

I can do:
with engine.connect() as con:
con.execute('select * into #tmp from tbl')
con.execute('select  * from #tmp')

... but that's limited to the scope of the context manager.

Oh, I don't have rights to create a 'real' table. :-(

Thanks!

Albert-Jan

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: sqlalchemy & #temp tables

[Albert-Jan Roskam]

On 8 Oct 2019 07:49, Frank Millman  wrote:

On 2019-10-07 5:30 PM, Albert-Jan Roskam wrote:
> Hi,
>
> I am using sqlalchemy (SA) to access a MS SQL Server database (python 3.5, 
> Win 10). I would like to use a temporary table (preferably #local, but 
> ##global would also be an option) to store results of a time-consuming query. 
> In other queries I'd like to access the temporary table again in various 
> places in my Flask app. How do I do that, given that SA closes the connection 
> after each request?
>
> I can do:
> with engine.connect() as con:
>  con.execute('select * into #tmp from tbl')
>  con.execute('select  * from #tmp')
>
> ... but that's limited to the scope of the context manager.
>
> Oh, I don't have rights to create a 'real' table. :-(
>
> Thanks!
>
> Albert-Jan
>


>I do not use SA, but I have written my app to >support Sql Server,
>PostgreSQL and sqlite3 as backend >databases. However, no matter which
>one is in use, I also use sqlite3 as an in->memory database to store
>temporary information.


Hi,

I tried your approach today but I ran into problems due to differences between 
the MS SQL and Sqlite dialect. However, I just came across this page: 
https://docs.sqlalchemy.org/en/13/dialects/sqlite.html#using-temporary-tables-with-sqlite.
 I haven't tried it yey, but using a StaticPool might work.


# maintain the same connection across all threads
from sqlalchemy.pool import StaticPool
engine = create_engine('sqlite:///mydb.db',
poolclass=StaticPool)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python2 vs python3

[Albert-Jan Roskam]

On 18 Oct 2019 20:36, Chris Angelico  wrote:

On Sat, Oct 19, 2019 at 5:29 AM Jagga Soorma  wrote:
>
> Hello,
>
> I am writing my second python script and got it to work using
> python2.x.  However, realized that I should be using python3 and it
> seems to fail with the following message:
>
> --
> Traceback (most recent call last):
>   File "test_script.py", line 29, in 
> test_cmd = ("diskcmd -u " + x + " | grep -v '\*' | awk '{print $1,
> $3, $4, $9, $10}'" )
> TypeError: Can't convert 'bytes' object to str implicitly
> --
>
> I then run this command and save the output like this:
>
> --
> test_info = (subprocess.check_output( test_cmd,
> stderr=subprocess.STDOUT, shell=True )).splitlines()
> --
>
> Looks like the command output is in bytes and I can't simply wrap that
> around str().  Thanks in advance for your help with this.

>That's correct. The output of the command >is, by default, given to you
>in bytes.

Do you happen to know why this is the default?  And is there a reliable way to 
figure out the encoding? On posix, it's probably utf8, but on windows I usually 
use cp437, but knowing windows, it could be any codepage (you can even change 
it with chcp.exe)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Win32api problems

[Albert-Jan Roskam]

On 22 Oct 2019 11:23, GerritM  wrote:

> ImportError: DLL load failed: The specified > procedure could not be found.

I've had the same error before and I solved it by adding the location where the 
win32 dlls live to PATH. Maybe PATH gets messed up during the installation of 
something.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

[Albert-Jan Roskam]

On Mon, 11/25/13, Jurko Gospodnetić  wrote:

 Subject: Parallel Python x.y.A and x.y.B installations on a single Windows 
machine
 To: python-list@python.org
 Date: Monday, November 25, 2013, 1:32 PM
 
   Hi all.
 
   I was wondering what is the best way to install
 multiple Python installations on a single Windows machine.
 
   Regular Windows installer works great as long as all
 your installations have a separate major.minor version
 identifier. However, if you want to have let's say 2.4.3
 & 2.4.4 installed at the same time it does not seem to
 work.
 
   I have not been able to find any prepackaged Python
 installation or really any solution to this. Most of the
 advice seems to boil down to 'do not use such versions
 together, use only the latest'.
 
   We would like to run automated tests on one of our
 projects (packaged as a Python library) with different
 Python versions, and since our code contains workarounds for
 several problems with specific Python patch versions, we'd
 really like to be able to run the tests with those specific
 versions and with as little fuss as possible.
 
   Looking at what the Python installer does, the only
 problematic part for working around this manually seems to
 be the registry entries under
 'Software\Python\PythonCore\M.m' where 'M.n' is the
 major.minor version identifier. If Python interpreter
 expects to always find its entries there, then I guess there
 is no way to do what we need without building customized
 Python executables. Is there a way to force a specific
 Python interpreter to not read in this information, read it
 from an .ini file or something similar?
 
HI Jurko,

Check out the following packages: virtualenv, virtualenvwrapper, tox
virtualenv + wrapper make it very easy to switch from one python version to 
another. Stricly speaking you don't need virtualenvwrapper, but it makes 
working with virtualenv a whole lot easier.Tox also uses virtualenv. You can 
configure it to sdist your package under different python versions. Also, you 
can make it run nosetests for each python version and/or implementation (pypy 
and jython are supported)

Albert-Jan 


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: cx_Oracle throws: ImportError: DLL load failed: This application has failed to start ...

[Albert-Jan Roskam]

On Sun, 11/24/13, MRAB  wrote:

 Subject: Re: cx_Oracle throws: ImportError: DLL load failed: This application 
has failed to start ...
 To: python-list@python.org
 Date: Sunday, November 24, 2013, 7:17 PM
 
 On 24/11/2013 17:12, Ruben van den
 Berg wrote:
 > I'm on Windows XP SP3, Python 2.7.1. On running
 >
 > import cx_Oracle
 >
 > I got the error
 >
 > ImportError: DLL load failed: This application has
 failed to start because the application configuration is
 incorrect. Reinstalling the application may fix this
 problem.
 >
 > I then ran Dependency Walker on cx_Oracle.pyd. Its
 first complaint was about msvcr80.dll. However, this file is
 present in
 
C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.91_x-ww_0de56c07.
 (I believe it's part of the MS Visual Studio C++ 2008
 package which I installed.)
 >
 > I obviously uninstalled and reinstalled the cx_Oracle a
 couple of times but so far to no avail.
 >
 > Does anybody have a clue what to try next?
 >
 > For a screenshot of Dependency Walker, please see: 
 > https://dl.dropboxusercontent.com/u/116120595/dep_walker_orac.jpg
 >
 It looks like it's a path issue.
 
 You say that msvcr80.dll is in 
 
C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.91_x-ww_0de56c07,
 
 but is that folder listed as part of the search path?
 
 Have a look at the Windows' PATH environment variable.


===> Unlike in Linux with LD_LIBRARY_PATH, you can change PATH at runtime in 
Windows, e.g
import os, sys, ctypes
if sys.platform.startswith("win"):
os.environ["PATH"] += (os.pathsep + r"c:\your\new\path")
ctypes.WinDLL("msvcr80.dll")
 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Parallel Python x.y.A and x.y.B installations on a single Windows machine

[Albert-Jan Roskam]

On Mon, 11/25/13, Jurko Gospodnetić  wrote:

 Subject: Re: Parallel Python x.y.A and x.y.B installations on a single Windows 
machine
 To: python-list@python.org
 Date: Monday, November 25, 2013, 2:57 PM
 
   Hi.
 
 On 25.11.2013. 14:20, Albert-Jan Roskam wrote:
 > Check out the following packages: virtualenv,
 virtualenvwrapper, tox
 > virtualenv + wrapper make it very easy to switch from
 one python
 > version to another. Stricly speaking you don't need
 > virtualenvwrapper, but it makes working with virtualenv
 a whole lot
 > easier.Tox also uses virtualenv. You can configure it
 to sdist your
 > package under different python versions. Also, you can
 make it run
 > nosetests for each python version and/or implementation
 (pypy and
 > jython are supported)
 
   I'll look into using virtualenv and possibly tox once
 I get into issues with mismatched installed Python package
 versions, but for now I'm dealing with installing different
 Python interpreter versions and, unless I'm overlooking
 something here, virtualenv does not help with that. :-(
 
 > Are you sure? 
http://stackoverflow.com/questions/1534210/use-different-python-version-with-virtualenv

Below is a little terminal session.  I often switch between python 3.3 and 
python 2.7. My virtualenv for python 3.3 is called "python33". "workon" is a 
virtualenv wrapper command. And check out the envlist in tox.ini on 
http://tox.readthedocs.org/en/latest/example/basic.html

antonia@antonia-HP-2133 ~ $ workon python3.3
ERROR: Environment 'python3.3' does not exist. Create it with 'mkvirtualenv 
python3.3'.
antonia@antonia-HP-2133 ~ $ workon python33
(python33)antonia@antonia-HP-2133 ~ $ python
Python 3.3.2 (default, Sep  1 2013, 22:59:57) 
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
(python33)antonia@antonia-HP-2133 ~ $ deactivate
antonia@antonia-HP-2133 ~ $ python
Python 2.7.3 (default, Sep 26 2013, 16:38:10) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()



-- 
https://mail.python.org/mailman/listinfo/python-list

L[:]

[Albert-Jan Roskam]

In Python Cookbook, one of the authors (I forgot who) consistently used the 
"L[:]" idiom like below. If the second line simply starts with "L =" (so no 
"[:]") only the name "L" would be rebound, not the underlying object. That was 
the authorś explanation as far as I can remember. I do not get that. Why is the 
"L[:]" idiom more memory-efficient here? How could the increased efficiency be 
demonstrated?

#Python 2.7.3 (default, Sep 26 2013, 16:38:10) [GCC 4.7.2] on linux2
>>> L = [x ** 2 for x in range(10)]
>>> L[:] = ["foo_" + str(x) for x in L]


Thanks!


Regards,

Albert-Jan



~~

All right, but apart from the sanitation, the medicine, education, wine, public 
order, irrigation, roads, a 

fresh water system, and public health, what have the Romans ever done for us?

~~

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problem writing some strings (UnicodeEncodeError)

[Albert-Jan Roskam]

On Sun, 1/12/14, Paulo da Silva  wrote:

 Subject: Problem writing some strings (UnicodeEncodeError)
 To: python-list@python.org
 Date: Sunday, January 12, 2014, 4:36 PM
 
 Hi!
 
 I am using a python3 script to produce a bash script from
 lots of
 filenames got using os.walk.
 
 I have a template string for each bash command in which I
 replace a
 special string with the filename and then write the command
 to the bash
 script file.
 
 Something like this:
 
 shf=open(bashfilename,'w')
 filenames=getfilenames() # uses os.walk
 for fn in filenames:
     ...
     cmd=templ.replace("",fn)
     shf.write(cmd)
 
 For certain filenames I got a UnicodeEncodeError exception
 at
 shf.write(cmd)!
 I use utf-8 and have # -*- coding: utf-8 -*- in the source
 .py.
 
 How can I fix this?
 
 Thanks for any help/comments.
 

==> what is the output of locale.getpreferredencoding(False)? That is the 
default value of the "encoding" parameter of the open function.
 shf=open(bashfilename,'w', encoding='utf-8') might work, though on my Linux 
macine  locale.getpreferredencoding(False) returns utf-8.
help(open)
...
   In text mode, if encoding is not specified the encoding used is platform
dependent: locale.getpreferredencoding(False) is called to get the
current locale encoding. (For reading and writing raw bytes use binary
mode and leave encoding unspecified.)
...


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: L[:]

[Albert-Jan Roskam]

 On 1/13/2014 4:00 AM, Laszlo Nagy
 wrote:
 > 
 >> Unless L is aliased, this is silly code.
 > There is another use case. If you intend to modify a
 list within a for
 > loop that goes over the same list, then you need to
 iterate over a copy.
 > And this cannot be called an "alias" because it has no
 name:
 
 for i in somelist: creates a second reference to somelist
 that somewhere in the loop code has a name, so it is
 effectively an 'alias'. The essential point is that there
 are two access paths to the same object.
 
 > for idx,item in enumerate(L[:]):
 >     # do something with L here,
 including modification
 
 The copy is only needed in the above if one inserts or
 deletes. But if one inserts or deletes more than one item,
 one nearly always does better to iterate through the
 original and construct a new list with new items added and
 old items not copied.
 

> Hi, first, thank you all for your replies -much appreciated!
Terry, this would be making a shallow copy, right? If so, then "list(L)" is 
slightly nicer IMHO, but that's probably a matter of taste (however, I don't 
like copy.copy, even though that's perhaps most clear --oh well nitpicking ;-)

I also found that item assignment ([1] below) is much faster than using the 
more standard (I think) .append ([2]).
# [1]
for idx,item in enumerate(L[:]):
   if some_condition: 
  L[idx] = foobarify(item)
# [2]
L2 = [] 
for idx,item in enumerate(L):
   if some_condition: 
   L2.append(foobarify(item))
   else:
   L2.append(item)


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it possible to get string from function?

[Albert-Jan Roskam]

On Thu, 1/16/14, Peter Otten <__pete...@web.de> wrote:

 Subject: Re: Is it possible to get string from function?
 To: python-list@python.org
 Date: Thursday, January 16, 2014, 9:52 AM
 
 Roy Smith wrote:
 
 > I realize the subject line is kind of meaningless, so
 let me explain :-)
 > 
 > I've got some unit tests that look like:
 > 
 > class Foo(TestCase):
 >   def test_t1(self):
 >     RECEIPT = "some string"
 > 
 >   def test_t2(self):
 >     RECEIPT = "some other string"
 > 
 >   def test_t3(self):
 >     RECEIPT = "yet a third string"
 > 
 > and so on.  It's important that the strings be
 mutually unique.  In the
 > example above, it's trivial to look at them and observe
 that they're all
 > different, but in real life, the strings are about 2500
 characters long,
 > hex-encoded.  It even turns out that a couple of
 the strings are
 > identical in the first 1000 or so characters, so it's
 not trivial to do
 > by visual inspection.
 > 
 > So, I figured I would write a meta-test, which used
 introspection to
 > find all the methods in the class, extract the strings
 from them (they
 > are all assigned to a variable named RECEIPT), and
 check to make sure
 > they're all different.
 > 
 > Is it possible to do that?  It is straight-forward
 using the inspect
 > module to discover the methods, but I don't see any way
 to find what
 > strings are assigned to a variable with a given
 name.  Of course, that
 > assignment doesn't even happen until the function is
 executed, so
 > perhaps what I want just isn't possible?
 > 
 > It turns out, I solved the problem with more mundane
 tools:
 > 
 > grep 'RECEIPT = ' test.py | sort | uniq -c
 > 
 > and I could have also solved the problem by putting all
 the strings in a
 > dict and having the functions pull them out of
 there.  But, I'm still
 > interested in exploring if there is any way to do this
 with
 > introspection, as an academic exercise.
 
 Instead of using introspection you could make it explicit
 with a decorator:
 
 $ cat unique_receipt.py 
 import functools
 import sys
 import unittest
 
 _receipts = {}
 def unique_receipt(receipt):
     def deco(f):
         if receipt in _receipts:
             raise ValueError(
                
 "Duplicate receipt {!r} in \n    {} and \n 
   {}".format(
                
     receipt, _receipts[receipt], f))
         _receipts[receipt] = f
         @functools.wraps(f)
         def g(self):
             return f(self,
 receipt)
         return g
     return deco
 
 class Foo(unittest.TestCase):
     @unique_receipt("foo")
     def test_t1(self, RECEIPT):
         pass
 
     @unique_receipt("bar")
     def test_t2(self, RECEIPT):
         pass
 
     @unique_receipt("foo")
     def test_t3(self, RECEIPT):
         pass
 
 if __name__ == "__main__":
     unittest.main()
 $ python unique_receipt.py 
 Traceback (most recent call last):
   File "unique_receipt.py", line 19, in 
     class Foo(unittest.TestCase):
   File "unique_receipt.py", line 28, in Foo
     @unique_receipt("foo")
   File "unique_receipt.py", line 11, in deco
     receipt, _receipts[receipt], f))
 ValueError: Duplicate receipt 'foo' in 
      and
 
     
 

 > Very cool approach. Question, though: what would be wrong with 
the following approach:


import unittest

class Test(unittest.TestCase):

receipts = {}

def unique_value(self, k, v):
assert Test.receipts.get(k) is None, "Duplicate: %s" % v
Test.receipts[k] = v

def test_a(self):
self.unique_value("large_value", "foo")

def test_b(self):
self.unique_value("large_value", "bar")  # oh no, a duplicate! 

def test_c(self):
self.unique_value("another_large_value", "blah")

unittest.main()



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Guessing the encoding from a BOM

[Albert-Jan Roskam]

On Thu, 1/16/14, Chris Angelico  wrote:

 Subject: Re: Guessing the encoding from a BOM
 To: 
 Cc: "python-list@python.org" 
 Date: Thursday, January 16, 2014, 7:06 PM
 
 On Fri, Jan 17, 2014 at 5:01 AM,
 Björn Lindqvist 
 wrote:
 > 2014/1/16 Steven D'Aprano :
 >> def guess_encoding_from_bom(filename, default):
 >>     with open(filename, 'rb')
 as f:
 >>         sig =
 f.read(4)
 >>     if
 sig.startswith((b'\xFE\xFF', b'\xFF\xFE')):
 >>         return
 'utf_16'
 >>     elif
 sig.startswith((b'\x00\x00\xFE\xFF', b'\xFF\xFE\x00\x00')):
 >>         return
 'utf_32'
 >>     else:
 >>         return
 default
 >
 > You might want to add the utf8 bom too:
 '\xEF\xBB\xBF'.
 
 I'd actually rather not. It would tempt people to pollute
 UTF-8 files
 with a BOM, which is not necessary unless you are MS
 Notepad.
 

 ===> Can you elaborate on that? Unless your utf-8 files will only contain 
ascii characters I do not understand why you would not want a bom utf-8.

Btw, isn't "read_encoding_from_bom" a better function name than 
"guess_encoding_from_bom"? I thought the point of BOMs was that there would be 
no more need to guess?

Thanks!

Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: doctests compatibility for python 2 & python 3

[Albert-Jan Roskam]

On Fri, 1/17/14, Terry Reedy  wrote:

 Subject: Re: doctests compatibility for python 2 & python 3
 To: python-list@python.org
 Date: Friday, January 17, 2014, 10:10 PM
 
 On 1/17/2014 7:14 AM, Robin Becker
 wrote:
 
 > I tried this approach with a few more complicated
 outcomes and they fail
 > in python2 or 3 depending on how I try to render the
 result in the doctest.
 
 I never got how you are using doctests. There were certainly
 not meant for heavy-duty unit testing, but for testing
 combined with explanation. Section 26.2.3.7. (in 3.3)
 Warnings warns that they are fragile to even single char
 changes and suggests == as a workaround, as 'True' and
 'False' will not change. So I would not reject that option.
 

=> I used doctests in .txt files and I converted ALL of them when I wanted 
to make my code work for both Python 2 and 3. I tried to fix something like a 
dozen of them so they'd work in Python 2.7 and 3,3. but I found it just too 
cumbersome and time consuming. The idea of doctest is super elegant, but it is 
really only meant for testable documentation (maybe with sphinx). If you'd put 
all the (often boring, e.g. edge cases) test cases in docstrings, the .py file 
will look very cluttered. One thing that I missed in unittest was Ellipsis, 
but: https://pypi.python.org/pypi/gocept.testing/1.6.0 offers assertEllipsis 
and other useful stuff.

Albert-Jan


-- 
https://mail.python.org/mailman/listinfo/python-list

RE: counting unique numpy subarrays

[Albert-Jan Roskam]

Hi

(Sorry for topposting)

numpy.ravel is faster than numpy.flatten (no copy)
numpy.empty is faster than numpy.zeros
numpy.fromiter might be useful to avoid the loop (just a hunch)

Albert-Jan

> From: duncan@invalid.invalid
> Subject: counting unique numpy subarrays
> Date: Fri, 4 Dec 2015 19:43:35 +
> To: python-list@python.org
> 
> Hello,
>   I'm trying to find a computationally efficient way of identifying
> unique subarrays, counting them and returning an array containing only
> the unique subarrays and a corresponding 1D array of counts. The
> following code works, but is a bit slow.
> 
> ###
> 
> from collections import Counter
> import numpy
> 
> def bag_data(data):
> # data (a numpy array) is bagged along axis 0
> # returns concatenated array and corresponding array of counts
> vec_shape = data.shape[1:]
> counts = Counter(tuple(arr.flatten()) for arr in data)
> data_out = numpy.zeros((len(counts),) + vec_shape)
> cnts = numpy.zeros((len(counts,)))
> for i, (tup, cnt) in enumerate(counts.iteritems()):
> data_out[i] = numpy.array(tup).reshape(vec_shape)
> cnts[i] =  cnt
> return data_out, cnts
> 
> ###
> 
> I've been looking through the numpy docs, but don't seem to be able to
> come up with a clean solution that avoids Python loops. TIA for any
> useful pointers. Cheers.
> 
> Duncan
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Unicode failure

[Albert-Jan Roskam]

I think you need to use a raw unicode string, ur

>>> unicodedata.name(ur'\u2122')
'TRADE MARK SIGN'

> Date: Fri, 4 Dec 2015 13:07:38 -0500
> From: da...@vybenetworks.com
> To: python-list@python.org
> Subject: Unicode failure
> 
> I thought that going to Python 3.4 would solve my Unicode issues but it
> seems I still don't understand this stuff.  Here is my script.
> 
> #! /usr/bin/python3
> # -*- coding: UTF-8 -*-
> import sys 
> print(sys.getdefaultencoding()) 
> print(u"\N{TRADE MARK SIGN}") 
> 
> And here is my output.
> 
> utf-8
> Traceback (most recent call last):
>   File "./g", line 5, in 
> print(u"\N{TRADE MARK SIGN}")
> UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in
> position 0: ordinal not in range(128)
> 
> What am I missing?
> 
> TIA.
> 
> -- 
> D'Arcy J.M. Cain
> Vybe Networks Inc.
> http://www.VybeNetworks.com/
> IM:da...@vex.net VoIP: sip:da...@vybenetworks.com
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

Screenshots in Sphinx docs

[Albert-Jan Roskam]

Hello,

I'd like to include up-to-date screenshots (of a tkinter app) into my Sphinx 
documentation. This looks ok:
https://pypi.python.org/pypi/sphinxcontrib-programscreenshot
BUT I need something that works on Windows (Python 2.7). Can any recommend an 
approach? I thought about using PIL: 
http://www.varesano.net/blog/fabio/capturing%20screen%20image%20python%20and%20pil%20windows

Thanks!

Albert-Jan
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Screenshots in Sphinx docs

[Albert-Jan Roskam]

> To: python-list@python.org
> From: tjre...@udel.edu
> Subject: Re: Screenshots in Sphinx docs
> Date: Mon, 14 Dec 2015 14:01:03 -0500
> 
> On 12/14/2015 11:31 AM, Albert-Jan Roskam wrote:
> 
> > I'd like to include up-to-date screenshots (of a tkinter app)
>  > into my Sphinx documentation.
> 
> If you manually take screenshots with *any* screen grabber and save in 
> an appropriate format, this is apparently trivial -- use the ..image 
> directive.  From the below, it appears that what you want is to have a 
> literally up-to-date screenshot taken automatically during the doc build.
> 
> This requires that one be able to automate getting the application into 
> the exact display state one wants to capture.  You can probably do that 
> with a tkinter app if you write it with that possibility in mind.  In 
> particular, you must keep a Python reference to every widget you want to 
> manipulate, even if not needed for normal program operation.
> 
> There is also an issue with grabbing the whole screen versus only a 
> program-specific window.

I need only a few screens. I think I will call my tkinter app with 
subprocess.Popen, wait until it's loaded,
grab the image, then kill it. Then I indeed wanted to use the ..image directive.

>  > This looks ok:
> > https://pypi.python.org/pypi/sphinxcontrib-programscreenshot
> 
> This (automatically) takes 'screenshots' on a no-screen (headless) *nix 
> system (during doc build) by redirecting X-windows output to a 
> pseudo-screen program.  Rather clever, and system-specific.
> 
> > BUT I need something that works on Windows (Python 2.7).
>  > Can any recommend an approach? I thought about using PIL:
> 
> Get the pillow fork/upgrade on pypi.

Thanks for the tip! So good ol' PIL is no longer maintained?

 
> > http://www.varesano.net/blog/fabio/capturing%20screen%20image%20python%20and%20pil%20windows
> 
> Or look into Windows screen grabber programs, of which there are many.
> 
> -- 
> Terry Jan Reedy
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: libre office

[Albert-Jan Roskam]

> From: ji...@frontier.com
> To: python-list@python.org
> Subject: libre office
> Date: Tue, 19 Jan 2016 17:01:40 -0600
> 
> How do I get data from libre office using python?

Does this help?http://www.openoffice.org/udk/python/python-bridge.html  
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: SQLite

[Albert-Jan Roskam]

(Sorry for top posting)

IIRC, you have to do
sudo apt-get install build-essential python-dev
... then re-compile python

> To: python-list@python.org
> From: k.d.jant...@mailbox.org
> Subject: SQLite
> Date: Sun, 21 Feb 2016 18:11:18 +0100
> 
>Hello,
> 
>I have downloaded Python3.5.1 as .targz, compiled it(configure, make,...)
>and it works
>(under Debian Wheezy AMD64) up to the moment I wanted to use SQLite.
> 
>I get the following message:
>===
>jantzen@PC4:~$ python
>Python 3.5.0 (default, Dec  2 2015, 14:16:16)
>[GCC 4.7.2] on linux
>Type "help", "copyright", "credits" or "license" for more information.
>>>> import sqlite3
>Traceback (most recent call last):
>  File "", line 1, in 
>  File "/usr/local/lib/python3.5/sqlite3/__init__.py", line 23, in
>
>from sqlite3.dbapi2 import *
>  File "/usr/local/lib/python3.5/sqlite3/dbapi2.py", line 27, in 
>from _sqlite3 import *
>ImportError: No module named '_sqlite3'
>===
> 
>Obviously something is missing.
>How do I solve the problem? Where do I find this module?
> 
>Thanks for a hint.
>--
> 
>K.D.J.
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: reversed(zip(...)) not working as intended

[Albert-Jan Roskam]

(Sorry for top-posting)

 No TypeError here:

Python 2.7.2 (default, Nov  2 2015, 01:07:37) [GCC 4.9 20140827 (prerelease)] 
on linux4
Type "help", "copyright", "credits" or "license" for more information.
>>> ten = range(10)
>>> reversed(zip(ten, ten))

>>> list(reversed(zip(ten, ten)))
[(9, 9), (8, 8), (7, 7), (6, 6), (5, 5), (4, 4), (3, 3), (2, 2), (1, 1), (0, 0)]
>>>

> To: python-list@python.org
> From: srku...@mail.de
> Subject: reversed(zip(...)) not working as intended
> Date: Sun, 6 Mar 2016 19:29:59 +0100
> 
> Hi,
> 
> what's the reason that reversed(zip(...)) raises as a TypeError?
> 
> Would allowing reversed to handle zip and related functions lead to 
> strange errors?
> 
> Best,
> Sven
> -- 
> https://mail.python.org/mailman/listinfo/python-list
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: looping and searching in numpy array

[Albert-Jan Roskam]

> Date: Thu, 10 Mar 2016 08:48:48 -0800
> Subject: Re: looping and searching in numpy array
> From: heml...@gmail.com
> To: python-list@python.org
> 
> On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> > Heli wrote:
> > 
> > > Dear all,
> > > 
> > > I need to loop over a numpy array and then do the following search. The
> > > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > > the example below) with around 300K values.
> > > 
> > > 
> > > for id in np.nditer(npArray1):
> > >   
> > >newId=(np.where(npArray2==id))[0][0]
> > > 
> > > 
> > > Is there anyway I can make the above faster? I need to run the script
> > > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > > the lines above, npArray1 and npArray2  are not necessarily the same size,
> > > but they are both 1d.
> > 
> > You mean you are looking for the index of the first occurence in npArray2 
> > for every value of npArray1?
> > 
> > I don't know how to do this in numpy (I'm not an expert), but even basic 
> > Python might be acceptable:
> > 
> > lookup = {}
> > for i, v in enumerate(npArray2):
> > if v not in lookup:
> > lookup[v] = i
> > 
> > for v in npArray1:
> > print(lookup.get(v, ""))
> > 
> > That way you iterate once (in Python) instead of 2*len(npArray1) times (in 
> > C) over npArray2.
> 
> Dear Peter, 
> 
> Thanks for your reply. This really helped. It reduces the script time from 
> 61(s) to 2(s). 
> 
> I am still very interested in knowing the correct numpy way to do this, but 
> till then your fix works great. 


Hi, I suppose you have seen this already (in particular the first link): 
http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI
 don't thonk it's part of numpy yet.
Albert-Jan
  
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: looping and searching in numpy array

[Albert-Jan Roskam]

> From: sjeik_ap...@hotmail.com
> To: heml...@gmail.com; python-list@python.org
> Subject: RE: looping and searching in numpy array
> Date: Sun, 13 Mar 2016 13:51:23 +



> 
> Hi, I suppose you have seen this already (in particular the first link): 
> http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI
>  don't thonk it's part of numpy yet.
> Albert-Jan

sorry, the correct url is: 
http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.html

-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Effects of caching frequently used objects, was Re: Explaining names vs variables in Python

[Albert-Jan Roskam]

> To: python-list@python.org
> From: __pete...@web.de
> Subject: Effects of caching frequently used objects, was Re: Explaining  
> names  vs variables  in Python
> Date: Wed, 2 Mar 2016 10:12:48 +0100
> 
> Salvatore DI DIO wrote:
> 
> > Hello,
> > 
> > I know Python does not have variables, but names.
> > Multiple names cant then be bound to the same objects.
> > 
> > So this behavior
> > 
>  b = 234
>  v = 234
>  b is v
> > True
> > 
> > according to the above that is ok
> > 
> > 
> > 
> > But where is the consistency ? if I try :
> > 
>  v = 890
>  w = 890
>  v is w
> > False
> > 
> > It is a little difficult to explain this behavior to a newcommer in Python
> > 
> > Can someone give me the right argument to expose ?
> 
> You should not bother with object identity for objects other than None.


A little late to the party, but: how about Ellipsis? Shouldn't "is" also be 
used for that one? (It's rare, I know :))
Albert-Jan

  
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I am out of trial and error again Lists

[Albert-Jan Roskam]

-
On Fri, Oct 24, 2014 5:56 PM CEST Rustom Mody wrote:

>On Friday, October 24, 2014 8:11:12 PM UTC+5:30, Seymore4Head wrote:
>> On Thu, 23 Oct 2014 21:56:31 -0700 (PDT), Rustom Mody wrote:
>> 
>> >On Thursday, October 23, 2014 10:33:57 PM UTC+5:30, Seymore4Head wrote:
>> > On Thu, 23 Oct 2014 15:55:35 + (UTC), Denis McMahon wrote:
>> > 
>> > >On Thu, 23 Oct 2014 10:04:56 -0400, Seymore4Head wrote:
>> > >
>> > > On Thu, 23 Oct 2014 09:15:16 + (UTC), Denis McMahon wrote:
>> > >
>> > >>Try the following 3 commands at the console:
>> > >
>> > >You obviously didn't, so I'll try again. Try each of the following three 
>> > >commands in the python console at the ">>" prompt.
>> > >
>> > >1) 10
>> > 10
>> > 
>> > >2) range(10)
>> > range(0, 10)
>> > 
>> > >3) str(range(10))
>> > 'range(0, 10)'
>> > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>> > >
>> > >Show *and* describe the output in each case. Describing the output that 
>> > >you see is actually the key here, as it will allow us to assess whether 
>> > >you understand what you are actually seeing or not, and if you don't 
>> > >understand the output you see in the console, then we need to fix that 
>> > >very fundamental and basic issue before moving on to more complex stuff!
>> > >
>> > > Ok Thanks
>> > >
>> > >You were expected to answer the question in the original. I have now set 
>> > >it as a clearer and more specific task.
>> > >
>> > >If you're not going to do these things that are intended to help you 
>> > >learn some of the basic features of the language, then I and everyone 
>> > >else here that has so far been attempting to help you are wasting our 
>> > >time.
>> > 
>> > I did try them.  I may have missed replying your to your specific
>> > comment, but I tried them.
>> > 
>> > BTW str(range (10)) does work with Python 2 which is where I may have
>> > got the idea.  I happened to be using Python 3 at the time I tried to
>> > implement it.  It is a little confusing jumping back and forth, but
>> > for the moment, I am going to tough it out.


u0_a100@condor_umts:/ $ python
Python 3.2.2 (default, Jun 23 2014, 00:13:13)
[GCC 4.8] on linux-armv7l
Type "help", "copyright", "credits" or "license" for more information.
>>> list(range(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>


>> > I do appreciate all the help too.
>> >
>> >Hi Seymore!
>> >
>> >Happy to see that you are moving on from
>> >"reading much; understanding nothing; thrashing"
>> >
>> >to
>> >
>> >"reading a bit; understanding a bit"
>> >[And thanks to Denis to getting you out of your confusion-hole]
>> >
>> >So heres a small additional question set that I promise will more than repay
>> >you your time.
>> >
>> >Better done in python 2. But if you use python3, below replace
>> >range(10)
>> >with
>> >list(range(10))
>
>
>
>
>1. You are reading too much
>2. Trying to hard
>
>Think of riding a bicycle.
>Cant do it by reading many books on cycling -- thats 1.
>Nor by holding the handle so hard you tremble -- thats 2.
>
>Just relax a bit...
>And take small steps
>
>Chill... as Chris joked, no monster in the computer (or on this list!)

+1 for that remark. Talking about chill: grab a couple of beers (suggest: 
sixpack) and enjoy an evening of Python!



>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python script that does batch find and replace in txt files

[Albert-Jan Roskam]

- Original Message -
> From: Syed Khalid 
> To: python-list@python.org
> Cc: 
> Sent: Sunday, November 9, 2014 8:58 PM
> Subject: Python script that does batch find and replace in txt files
> 
> Python script that does batch find and replace in txt files Need a python 
> script 
> that opens all .txt files in a folder find replace/delete text and save files.
> 
> I have text files and I need to perform below steps for each file. 
> 
> Step 1: Put cursor at start of file and Search for "Contact's 
> Name:". Delete all the rows before it.
> Step 2: Put cursor at end of file, Search for "Contact's Name:" 
> select option UP.
> Step 3: Search for "Photo of the" Replace with blanks
> Step 4: Search for "Contact is" Replace with blanks
> Step 5: Search for "Contact's Name:" Replace with blanks
> Step 6: Search for "Age:" Replace with blanks
> Step 7: Search for "Sex:" Replace with blanks
> Step 8: Search for "House No:" Replace with blanks
> Step 9: Search for "available" Replace with blanks
> Step 10: Remove Empty Lines Containing Blank Characters from file
> Step 11: Trim Leading Space for each line
> Step 12: Trim Trailing Space after each line
> Step 13: Search for - (hyphen) Replace with _ (underscore)

> Step 14: Save file.

something like (untested)


import glob, codecs, re, os

regex = re.compile(r"Age: |Sex: |House No: ") # etc etc

for txt in glob.glob("/some/path/*.txt"):
with codecs.open(txt, encoding="utf-8") as f:
oldlines = f.readlines()
for i, line in enumerate(oldlines):
if "Contact's Name: " in line:
break
newlines = [regex.sub("", line).strip().replace("-", "_") for line in 
oldlines[i:]
with codecs.open(txt + "_out.txt", "wb", encoding="utf-8") as w:
w.write(os.linesep.join(newlines))




> 
> Currently I have recorded a macro in Notepad++.
> I open each file, run macro and save file.
> As there are many files I was looking for a program to automate the process. 
> 
> I posted the same query in Notepad++ forum. I got a reply that it can be done 
> by 
> using Python script.
> 
> Kindly do the needful. 
> 
> Thank you.
> khalidness
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python script that does batch find and replace in txt files

[Albert-Jan Roskam]

On Sun, Nov 9, 2014 10:51 PM CET Syed Khalid wrote:

>Albert,
>
>Thanks a million for script,
>
>It worked fine after I closed the bracket.
>
>
>import glob, codecs, re, os
>
>regex = re.compile(r"Age: |Sex: |House No:  ") # etc etc
>
>for txt in glob.glob("D:/Python/source/*.txt"):
>with codecs.open(txt, encoding="utf-8") as f:
>oldlines = f.readlines()
>for i, line in enumerate(oldlines):
>if "Elector's Name:" in line:
>break
>newlines = [regex.sub(", line).strip().replace("-", "_") for line in
>oldlines[i:]]

You're welcome

newlines = [regex.sub("", line).strip().replace("-", "_") for line in
oldlines[i:] if line.strip()]
-- 
https://mail.python.org/mailman/listinfo/python-list

locale.getlocale() in cmd.exe vs. Idle

[Albert-Jan Roskam]

Hi,

Why do I get different output for locale.getlocale() in Idle vs. cmd.exe?

# IDLE
Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on 
win32
Type "copyright", "credits" or "license()" for more information.
>>> import locale
>>> locale.getdefaultlocale()
('nl_NL', 'cp1252')
>>> locale.getlocale()
('Dutch_Netherlands', '1252')  # I need this specific notation
>>>

# cmd.exe or Ipython
C:\Users\albertjan>python
Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
('nl_NL', 'cp1252')
>>> locale.getlocale()
(None, None)

# using setlocale does work (one of these instances when I answer my own 
question while writing to the Python list)
C:\Users\albertjan>python
Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, "")
'Dutch_Netherlands.1252'
>>> locale.getlocale()
('Dutch_Netherlands', '1252') 
 
Thank you!

Regards,

Albert-Jan




~~

All right, but apart from the sanitation, the medicine, education, wine, public 
order, irrigation, roads, a 

fresh water system, and public health, what have the Romans ever done for us?

~~ 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: locale.getlocale() in cmd.exe vs. Idle

[Albert-Jan Roskam]

- Original Message -
> From: Terry Reedy 
> To: python-list@python.org
> Cc: 
> Sent: Monday, November 10, 2014 9:31 PM
> Subject: Re: locale.getlocale() in cmd.exe vs. Idle
> 
> On 11/10/2014 4:22 AM, Albert-Jan Roskam wrote:
>>  Hi,
>> 
>>  Why do I get different output for locale.getlocale() in Idle vs. cmd.exe?



> 
> Idle runs code in an environment that is slightly altered from the 
> standard python startup environment'.  idlelib.IOBinding has this
> '''
> # Try setting the locale, so that we can find out
> # what encoding to use
> try:
>  import locale
>  locale.setlocale(locale.LC_CTYPE, "")
> '''
 
Hi Terry,
 
Thank you. Any idea why setlocale (a *setter*) returns something other than 
None? (this question is not related to the (None, None) thing of getlocale, 
just curious). Would it be a good idea to put this setlocale line in site.py? 
Or should it be in __init__.py to make the code more portable?

> idlelib.run, which runs in the user-code subprocess, imports IOBinding. 
> Setting LC_CTYPE is sufficient for getlocale() to not return null values.

So then I would have all the locale categories of the 'bare' locale (sorry, I 
don't know what else I should call it), except for LC_CTYPE, which is derived 
from my system. So in LC_NUMERIC I'd still have the en_US period/comma for 
decimal/thousand grouping, respectively, but I switch to the nl_NL LC_CTYPE. I 
doubt if it matters, but still: will this not introduce an ueber hard-to-find 
possible bug when I use re.LOCALE? 
 
> C:\Users\Terry>python -c "import locale; 
> print(locale.getlocale())"
> 
> (None, None)
> 
> C:\Users\Terry>python -c "import locale; 
> locale.setlocale(locale.LC_CTYPE, ''); print(locale.getlocale())"
> ('English_United States', '1252')
 
What is the difference between getlocale and getdefaultlocale anyway? The 
docstrings are even partially the same. The notatation of getlocale appears to 
be OS-specific ("English_United States" in Windows) and not Unix-like (cf. 
getdefaultlocale: en_US)
 
regards,
Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What is \1 here?

[Albert-Jan Roskam]

- Original Message -
> From: Ned Batchelder 
> To: python-list@python.org
> Cc: 
> Sent: Tuesday, November 11, 2014 12:52 PM
> Subject: Re: What is \1 here?

 

 
> You need to learn how to find this stuff out for yourself. Ben Finney 
> even gave you a pointer to a helpful site for experimenting with 
> regexes: http://pythex.org/
 
Cool. I also like this one, which hardly anybody appears to use:

python C:\Python27\Tools\Scripts\redemo.py
 
The functionality is virtually the same.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I love assert

[Albert-Jan Roskam]

- Original Message -

> From: Ethan Furman 
> To: python-list@python.org
> Cc: 
> Sent: Tuesday, November 11, 2014 9:08 PM
> Subject: Re: I love assert
> 
> On 11/11/2014 11:40 AM, Peter Cacioppi wrote:
>> 
>>  I get the impression that most Pythonistas aren't as habituated with 
> assert statements
>>   as I am. Is that just a misimpression on my part? If not, is there a good 
> reason to
>>   assert less with Python than other languages?
>> 
>>  As far as I can tell, Python supports assert perfectly well. When run with 
> the optimization
>>   flagging, the asserts are truly removed.
>> 
>>  I think one needs to take care with some basic assert coding - it's not 
> a substitute for
>>   unit tests, it doesn't absolve you of normal exception 
> responsibilities, and, most of all,
>>   it should be used for passive inspection and not action. But given these 
> guidelines, I
>>   still find it very useful as "active comments".
> 
> asserts are a specialized tool, easily abused.  Sounds like you are using 
> them 
> exactly as intended.


Would you say that assert is baaadly abused in nose?*) I never tried it, but 
probably all tests pass when Python is run with -O or -OO.

*) Actually, I love that package. Much cleaner than unittest.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: I love assert

[Albert-Jan Roskam]

- Original Message -

> From: Ethan Furman 
> To: Albert-Jan Roskam 
> Cc: "python-list@python.org" 
> Sent: Tuesday, November 11, 2014 10:15 PM
> Subject: Re: I love assert
> 
> On 11/11/2014 01:09 PM, Albert-Jan Roskam wrote:
>>  Ethan Furman wrote:
>>> 
>>>  asserts are a specialized tool, easily abused.  Sounds like you are 
> using them
>>>  exactly as intended.
>> 
>>  Would you say that assert is baaadly abused in nose?*) I never tried it, 
> but
>>   probably all tests pass when Python is run with -O or -OO.
> 
> I don't know, haven't used it nor read the code.  It would certainly not 
> be good if it failed in optimized mode.


antonia@antonia-HP-2133 /tmp $ cat test.py
def test_func_1():
assert 1 == 2

def test_func_2():
x = 1; y = 2
assert x == y

if __name__ == "__main__":
import nose
nose.main()


antonia@antonia-HP-2133 /tmp $ python -O test.py
..
--
Ran 2 tests in 0.015s

OK

antonia@antonia-HP-2133 /tmp $ python -O -m nose test.py
..
--
Ran 2 tests in 0.003s

OK


antonia@antonia-HP-2133 /tmp $ python test.py
FF
==
FAIL: test.test_func_1
--
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/tmp/test.py", line 2, in test_func_1
assert 1 == 2
AssertionError

==
FAIL: test.test_func_2
--
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/tmp/test.py", line 6, in test_func_2
assert x == y
AssertionError

--
Ran 2 tests in 0.044s

FAILED (failures=2)

antonia@antonia-HP-2133 /tmp $ python -m nose test.py
... (same as previous)


antonia@antonia-HP-2133 /tmp $ nosetests -O -v test.py   
Usage: nosetests [options]

nosetests: error: no such option: -O   # phweeew
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: jitpy - Library to embed PyPy into CPython

[Albert-Jan Roskam]

On Fri, Dec 5, 2014 8:54 PM CET Mark Lawrence wrote:

>For those who haven't heard thought this might be of interest 
>https://github.com/fijal/jitpy

Interesting, but it is not clear to me when you would use jitpy instead of 
pypy. Too bad pypy alone was not included in the benchmarks (cython would have 
also been nice).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: jitpy - Library to embed PyPy into CPython

[Albert-Jan Roskam]

On Sun, Dec 7, 2014 11:06 AM CET Stefan Behnel wrote:

>Albert-Jan Roskam schrieb am 06.12.2014 um 21:28:
>> On Fri, Dec 5, 2014 8:54 PM CET Mark Lawrence wrote:
>> For those who haven't heard thought this might be of interest
>> https://github.com/fijal/jitpy
>> 
>> Interesting, but it is not clear to me when you would use jitpy instead
>> of pypy.
>
>I think this is trying to position PyPy more in the same corner as other
>JIT compilers for CPython, as opposed to keeping it a completely separate
>thing which suffers from being "not CPython". It's a huge dependency, but
>so are others.

You mean like psyco? Well, if implementation differences between cpython and 
pypy are a problem, it might be useful. I've only come across a few unimportant 
ones. Bu then, I never reimplement __del__. 
http://pypy.readthedocs.org/en/latest/cpython_differences.html

>Being able to choose tools at this level is great, so if PyPy becomes yet
>another way to speed up the critical 5% of a CPython application, that's a
>good thing.

Totally agree, provided that 5 % makes a practical difference ("wow, it runs 5 
ns faster now" :-))

-- 
https://mail.python.org/mailman/listinfo/python-list

numpy question (fairly basic, I think)

[Albert-Jan Roskam]

Hi,

I am new to numpy. I am reading binary data one record at a time (I have to) 
and I would like to store all the records in a numpy array which I 
pre-allocate. Below I try to fill the empty array with exactly one record, but 
it is filled with as many rows as there are columns. Why is this? It is 
probably something simple, but I am stuck! It is like the original record is 
not unpacked *as in tuple unpacking) into the array, so it remains one chunk, 
not an (nrows, ncols) structure.


from __future__ import print_function
import numpy as np


# one binary record
s = 
'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00;@\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x80U\xe1@\x00\x00\x00\x00\x80\xd9\xe4@\x00\x00\x00\x00@\xa7\xe3@\xab\xaa\xaa\xaajG\xe3@\x00\x00\x00\x00\x80\xd9\xe4@\x00\x00\x00\x00\x00\x00;@\x00\x00\x00\x00\x00\x00;@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\xa4DI\tBx
   qwertyuiopasdfghjklzxcvbnm,./
   
\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00p\x9f@\x00\x00\x00\x00\x00\x00\x10@\x00\x00\x00\x00\x00\x00(@DEC
 2012'


# read it into a structured array
formats = ['https://mail.python.org/mailman/listinfo/python-list

Re: numpy question (fairly basic, I think)

[Albert-Jan Roskam]

- Original Message -

> From: Steven D'Aprano 
> To: python-list@python.org
> Cc: 
> Sent: Sunday, December 14, 2014 12:52 AM
> Subject: Re: numpy question (fairly basic, I think)
> 
> Albert-Jan Roskam wrote:
> 
>>  Hi,
>> 
>>  I am new to numpy. I am reading binary data one record at a time (I have
>>  to) and I would like to store all the records in a numpy array which I
>>  pre-allocate. Below I try to fill the empty array with exactly one record,
>>  but it is filled with as many rows as there are columns. Why is this? It
>>  is probably something simple, but I am stuck! It is like the original
>>  record is not unpacked *as in tuple unpacking) into the array, so it
>>  remains one chunk, not an (nrows, ncols) structure.
> 
> Can you simplify the example to something shorter that focuses on the issue
> at hand? It isn't clear to me which bits of the code you show are behaving
> the way you expect and which bits are not.


Hi Steven,

Thanks for replying. My code was so elaborate because I did not know which part 
made it go wrong. I think I have got it already. Numpy arrays (ndarrays) must 
be homogeneous wrt their datatype (dtype). 

Probably to make vectorization work (?). However, a structured array (which I 
was using) *can* contain multiple dtypes, but it can only be one-dimensional. 
Its records are tuples (or arrays). In this sense, even a structured array is 
homogeneous. I was trying to change the one-dim array into a two-dim array so I 
could easily retrieve columns. I now use a pandas DataFrame to do that. If my 
sample data would have contained *only* floats (or ints, or ...), my original 
approach would have worked. 


Thanks!

Albert-Jan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to import sqlite3 in my python3.4 successfully?

[Albert-Jan Roskam]

---
On Sun, Dec 14, 2014 4:06 PM CET sir wrote:

>There are two python version in my debian7, one is python2.7 the system 
>default version, the other is python3.4 which compiled to install this way.
>
>|  apt-get  update
> apt-get  upgrade
> apt-get  install build-essential
> wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz
> tar-zxvfPython-3.4.0.tgz
> cdPython-3.4.0
> mkdir/usr/local/python3.4
> ./configure--prefix=/usr/local/python3.4
> make
> make install
> ln-s/usr/local/python3.4/bin/python3.4/usr/bin/python3.4
> ln-s/usr/local/python3.4/bin/pip3.4/usr/bin/pip3.4|
>
>I have installed sqlite this way on my debian.
>
>|sudo apt-get  install sqlite3 libsqlite3-dev|
>
>In python2.7
>
>|root@rebuild:~#  python
>Python  2.7.3  (default,  Mar  14  2014,  11:57:14)
>[GCC4.7.2]  on linux2
>Type  "help",  "copyright",  "credits"  or  "license"  for  more information.
>>>  import  sqlite3|
>
>In python3.4
>
>|root@rebuild:~#  python3.4
>Python  3.4.0  (default,  Nov  27  2014,  13:54:17)
>[GCC4.7.2]  on linux
>Type  "help",  "copyright",  "credits"  or  "license"  for  more information.
>>>  import  sqlite3
>Traceback  (most recent calllast):
>  File  "",  line1,  in  
>  File  "/usr/local/python3.4/lib/python3.4/sqlite3/__init__.py",  line23,  in 
>  
>from  sqlite3.dbapi2import  *
>  File  "/usr/local/python3.4/lib/python3.4/sqlite3/dbapi2.py",  line26,  in  
> 
>from  _sqlite3import  *
>ImportError:  No  module  named'_sqlite3'|
>
>How can i import sqlite3 in my python3.4 successfully?
>

Did you get any errors after compiling? One of the things to do BEFORE you 
install python is sudo apt-get install libsqlite3-dev, and probably a couple 
more (but not for sqlite)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Searching through more than one file.

[Albert-Jan Roskam]

-
On Sun, Dec 28, 2014 8:12 PM CET Dave Angel wrote:

>On 12/28/2014 12:27 PM, Seymore4Head wrote:
>> I need to search through a directory of text files for a string.
>> Here is a short program I made in the past to search through a single
>> text file for a line of text.
>> 
>> How can I modify the code to search through a directory of files that
>> have different filenames, but the same extension?
>> 
>
>You have two other replies to your specific question, glob and os.listdir.  I 
>would also mention the module fileinput:
>
>https://docs.python.org/2/library/fileinput.html


Ah, I was just about to say that. I found out about this gem after reading 
Dough Helmann's book. Here are some usage examples: 
http://pymotw.com/2/fileinput/


>import fileinput
>from glob import glob
>
>fnames = glob('*.txt')
>for line in fileinput.input(fnames):
>pass # do whatever
>
>If you're not on Windows, I'd mention that the shell will expand the wildcards 
>for you, so you could get the filenames from argv even simpler.  See first 
>example on the above web page.
>
>
>I'm more concerned that you think the following code you supplied does a 
>search for a string.  It does something entirely different, involving making a 
>crude dictionary.  But it could be reduced to just a few lines, and probably 
>take much less memory, if this is really the code you're working on.
>
>> fname = raw_input("Enter file name: ")  #"*.txt"
>> fh = open(fname)
>> lst = list()
>> biglst=[]
>> for line in fh:
>>  line=line.rstrip()
>>  line=line.split()
>>  biglst+=line
>> final=[]
>> for out in biglst:
>>  if out not in final:
>>  final.append(out)
>> final.sort()
>> print (final)
>> 
>
>Something like the following:
>
>import fileinput
>from glob import glob
>
>res = set()
>fnames = glob('*.txt')
>for line in fileinput.input(fnames):
>res.update(line.rstrip().split())
>print sorted(res)
>
>
>
>
>-- DaveA
>-- https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list