Re: Help with implementing callback functions using ctypes

[dieter]

jamadagni  writes:
> ...
I cannot help you with "ctypes". But, if you might be able to use
"cython", then calling callbacks is not too difficult
(you can find an example in e.g. my "dm.xmlsec.binding").

Note, however, that properly handling the GIL ("Global Interpreter Lock")
may be of great importance when the Python/C boundary is crossed --
at least when you intend to use your code in a multi thread environment.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Debugging difficulty in python with __getattr__, decorated properties and AttributeError.

[dieter]

"Mr. Joe"  writes:
> ...
> Sorry for digging this old topic back. I see that my "'property' does not
> play well with polymorphic code" comment generated some controversy. So
> here's something in my defense:

I did not intend to "attack" you.

> ...
> Yes, I like decorators and descriptors. I also like the ability to create a
> "virtual property" in a python class by binding a bunch of methods as
> setter/getter. But I find the implementation of this "virtual property
> feature" a bit awkward sometimes - every time I need to override a
> getter/setter in a child class, I need to decorate them again. Some of you
> may like this "explicitness", but I don't.

True, I have been hit by this, too - not with "property" but with
other decorators (those for caching).
But, after reflection, I came to the conclusion that I should be
happy with this feature:

  If Python would automatically redecorate overridden methods in a derived
  class, I would have no control over the process. What if I need
  the undecorated method or a differently decorated method (an
  uncached or differently cached method, in my case)?

Your getter/setter use case can quite easily be solved
with a class "DelayedMethodAccessor":

  class DelayedMethodAccess(object):
"""
def __init__(self, name): self.__name = name
def __call__(self, inst, *args, **kw): 
  return getattr(inst, self.__name)(*args, **kw)

You can then use:

   prop = property(DelayedMethodAccess(""), ...)

Or define a new decorator "property_by_name" and then
use

   prop = property_by_name("", ...)

Of course, this requires that the property name and the getter/setter names
differ, but this should be naturally enough.


Python's current behavior is very natural once you know that
decorators are just syntactic sugar and

   @
   def ...

simply means:

   def ...
   f = (f)

True, this is no perfect fit for all use cases - but
such a fit (if it existed at all) would have to be so complex
that only experts could understand it.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PDF generator decision

[dieter]

Christian Jurk  writes:

> ...
> So far I'd like to ask which is the (probably) best way to create PDFs in 
> Python (3)? It is important for me that I am able to specify not only 
> background graphics, paragaphs, tables and so on but also to specify page 
> headers/footers. The reason is that I have a bunch of documents to be 
> generated (including Invoice templates, Quotes - stuff like that).

High quality layouts, especially those containing (potentially complex
tables), are very difficult to generate automatically.

I do not suppose that you will find a generic solution - one applicable
to a wide range of document classes and garanteeing high quality
layout for almost all of its document instances.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Generating multi-part emails

[dieter]

Steven D'Aprano  writes:

> I wish to generate some test data for a program that deals with emails. I 
> need something that can produce multi-part emails, including "broken" 
> emails that violate email standards, especially when it comes to Unicode.

I would start producing legal messages (e.g. with the "email"
package) and then put in common standard violations.

I am using the XEmacs "vm" email package and it has
difficulties with the quoting of non-ascii characters
in header lines (it performes the quoting based on non-ascii
character sequences rather than on words, as it should).


Maybe, there is an email message catalog around for the test
of email servers/clients. I would check for this in corresponding
open source projects.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: so if zodb has no index or search, how run fast query?

[dieter]

visphatesj...@gmail.com writes:
> ...
The only service of the ZODB is storing (persistent) objects
persistently and providing (almost) transparent acces to them.
Persistent objects have a (usually) implicitely
managed attribute which contains the object's "object id"
and the ZODB can (quite) efficiently access the objects based
on its "object id" (it is using caches and indexes for this).

The objects managed by the ZODB form a (directed) graph with
a distinguished root node - where all nodes are (persistent)
Python objects. You essentially work with those persistent objects
as with normal Python objects: the ZODB transparently accesses
the persistent storage, when necessary, to give you
the feeling as if those objects were normal Python objects.
There are a few caveats with respect to modifications -
but most modifications work as you would expect for
"normal" Python objects (however, you must call "transaction.commit()",
in order to persist the modifications).

That is essentially all, the ZODB does for you.
When you need searches, you must use appropriate datatypes --
in the same way, you would need to do this in a "normal" Python program.

The ZODB comes with a collection of datatypes (so called "BTrees")
which may form the base for searches.
"BTrees" are essentially persistent dicts - however, they
are not build from a fixed number of persistent subobjects but
the persistent substructure is changed automatically to accommodate
efficiently for various tree sizes.

Someone else has already mentioned some higher level packages which 
can provide search capabilities.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: serialize a class to XML and back

[dieter]

Schneider  writes:

> how can I serialize a python class to XML? Plus a way to get the class
> back from the XML?
>
> My aim is to store instances of this class in a database.

In case you want to describe the XML data via an XML-schema
(e.g. to exchange it with other applications; maybe via
WebServices), you may have a look at "PyXB".


The approach of "PyXB" may be a bit different from yours:

  It starts with an XML-schema description and from
  it generates Python classes corresponding to the types
  mentioned in the schema.

  Instances of those classes can then be easily serialized
  to XML and XML documents corresponding to types defined
  in the schema can easily be converted into corresponding
  class instances.

  It is not too difficult to customize the classes
  used for a given type - e.g. to give them special methods
  related to your application.


You may want to start with your (arbitrary) Python classes
and get their instances serialized into an adequate XML document.

This will not work in all cases: some things are very difficult
to serialize (maybe even not serializable at all - e.g. locks).

If you plan to use anything already existing, then almost
surely, this will impose restrictions of your classes.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: authentication with python-ldap

[dieter]

avazqu...@grm.uci.cu writes:

> import ldap
> conn = ldap.initialize("ldap://ldap.uci.cu";)
> conn.protocol_version = ldap.VERSION3
> conn.simple_bind_s( "uid=xxx,dc=uci,dc=cu", "xxx" )
>
> Result:
>
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line
> 207, in simple_bind_s
> return self.result(msgid,all=1,timeout=self.timeout)
>   File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line
> 422, in result
> res_type,res_data,res_msgid = self.result2(msgid,all,timeout)
>   File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line
> 426, in result2
> res_type, res_data, res_msgid, srv_ctrls = self.result3(msgid,all,timeout)
>   File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line
> 432, in result3
> ldap_result = self._ldap_call(self._l.result3,msgid,all,timeout)
>   File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line
> 96, in _ldap_call
> result = func(*args,**kwargs)
> INVALID_CREDENTIALS: {'desc': 'Invalid credentials'}


You are accessing a protected operation of the LDAP server
and it (the server) rejects it due to invalid credentials.
You may have forgotten to pass on credentials (e.g. a password)
or the credentials do not fit to the specified user
(maybe the user does not exist at all).

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Ldap module and base64 oncoding

[dieter]

"Joseph L. Casale"  writes:
> ...
> After parsing the data for a user I am simply taking a value from the ldif 
> file and writing
> it back out to another which fails, the value parsed is:
>
> officestreetaddress:: T3R0by1NZcOfbWVyLVN0cmHDn2UgMQ==
>
>
>   File "C:\Python27\lib\site-packages\ldif.py", line 202, in unparse
> self._unparseChangeRecord(record)
>   File "C:\Python27\lib\site-packages\ldif.py", line 181, in 
> _unparseChangeRecord
> self._unparseAttrTypeandValue(mod_type,mod_val)
>   File "C:\Python27\lib\site-packages\ldif.py", line 142, in 
> _unparseAttrTypeandValue
> self._unfoldLDIFLine(':: 
> '.join([attr_type,base64.encodestring(attr_value).replace('\n','')]))
>   File "C:\Python27\lib\base64.py", line 315, in encodestring
> pieces.append(binascii.b2a_base64(chunk))
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 
> 7: ordinal not in range(128)
>
>> c:\python27\lib\base64.py(315)encodestring()
> -> pieces.append(binascii.b2a_base64(chunk))

This looks like a coding bug: "chunk" seems to be a unicode string;
"b2a_base64" expects an encoded string ("str/bytes"); as
a consequence, Python tries to convert the unicode to "str" by
encoding with its "default enconding" ("ascii" by default) - and
fails.

You could try to find out, why "chunk" is unicode (rather than "str").
That will probably bring you to the real problem.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Debugging memory leaks

[dieter]

writeson  writes:

> ...
> Anyway, my real question is how to go about debugging memory leak problems in 
> Python, particularly for a long running server process written with Twisted. 
> I'm not sure how to use heapy or guppy, and objgraph doesn't tell me enough 
> to locate the problem.

Analysing memory leaks is really difficult: huge amounts of data
is involved and usually, it is almost impossible to determine which
of the mentioned objects are leaked and which are rightfully in use.
In addition, long running Python processes usually have degrading
memory use - due to memory fragmentation. There is nothing you
can do against this.

Therefore: if the leak seems to be small, it may be much more
advicable to restart your process periodically (during times
where a restart does not hurt much) rather than try to find
(and fix) the leaks. Only when the leak is large enough that
it would force you to too frequent restarts, a deeper
analysis may be advicable (large leaks are easier to locate as well).


I have analysed memory leaks several times for Zope applications.

Zope helped me much by its "Top Refcount" functionality.
This uses the fact that a class/type instance (in many cases)
holds a reference to the corresponding class/type instance
(it seems not to work for all elementary types).
Thus looking at the refcount of the class/type gives you
an indication how many instances of this class/type are around.
Zope presents this information sorted by number.
Then you send requests against Zope and reexamine the information:
You get something like:

 Class  June 13, 2013 8:18 am   June 13, 2013 8:18 am   Delta
...ApplicationManager   15  22  +7
...DebugManager 9   12  +3 

In the case above, my requests have created 7 additional
"ApplicationManager" and 3 additional "DebugManager" instances.

If Zope objects for which this functionality works leak,
then this is a powerful tool to detect those object classes.

You could implement something similar for your server.


As mentioned, the approach does not work for (many; all?) elementary
Python types. Thus, if the leak involves only those instances, it
cannot be detected this way.

Memory leaks are often introduced by C extensions - and do not
involve Python objects (but leak C level memory). Those, too,
cannot be analysed by Python level approaches.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Any speech to text conversation python library for Linux and mac box

[dieter]

Ranjith Kumar  writes:

> I'm looking for speech to text conversation python library for linux and
> mac box, I found few libraries but non of them supports any of these
> platform.
> I found following libraries speech, dragonfly and pyspeech supports only
> windows and sphinx for linux.

You will not do the speech analysis directly in Python - you
always will do that in an external library.
Thus, your first (not Python related) task is to find
such a library available for the platforms you are concerned about.

Then you find out how this library can be used by applications.
Should the library provide a C level interface, then
it is not too difficult to use "Cython" to realize a Python bridge
for it.
Should the library provide a process based interface, then
you can use Python's "subprocess" module.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Debugging memory leaks

[dieter]

Chris Angelico  writes:

> ...
> It's terrible advice in generality, because it encourages a sloppiness
> of thinking: "Memory usage doesn't matter, we'll just instruct people
> to reset everything now and then".

"Memory usage" may matter. But if you loose 1 kb a day, your process
can run 3 years before you have lost 1 MB. Compare this to the
485 MB used when you start "firefox". The situation looks different
when you loose 10 MB a day.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Debugging memory leaks

[dieter]

Chris Angelico  writes:
> ...
> Right. Everything needs to be scaled. Everything needs to be in
> perspective. Losing 1 kilobit per day is indeed trivial; even losing
> one kilobyte per day, which is what I assume you meant :), isn't
> significant. But it's not usually per day, it's per leaking action.
> Suppose your web browser leaks 1024 usable bytes of RAM every HTTP
> request. Do you know how much that'll waste per day? CAN you know?

What I suggested to the original poster was that *he* checks
whether *his* server leaks a really significant amount of memory
-- and starts to try a (difficult) memory leak analysis only in this
case. If he can restart his server periodically, this may make
the analysis unnecessary.

I also reported that I have undertaken such an analysis several times and
what helped me in these cases. I know - by experience - how difficult 
those analysis are. And there have been cases, where I failed despite
much effort: the systems I work with are huge, consisting of
thousands of components, developed by various independent groups,
using different languages (Python, C, Java); each of those components
may leak memory; most components are "foreign" to me.
Surely, you understand that in such a context a server restart
in the night of a week end (leading to a service disruption of a
few seconds) seems an attractive alternative to trying to locate the leaks.

Things would change drastically if the leak is big enough to force a restart
every few hours. But big leaks are *much* easier to detect
and locate than small leaks.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Memory usage steadily going up while pickling objects

[dieter]

Giorgos Tzampanakis  writes:
> ...
> So it seems that the pickle module does keep some internal cache or
> something like that.

This is highly unlikely: the "ZODB" (Zope object database)
uses pickle (actually, it is "cPickle", the "C" implementation
of the "pickle" module) for serialization. The "ZODB" is
used in long running Zope processes. Should pickling cause
significant memory leackage, this would have been observed
(and reported).

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Finding all instances of a string in an XML file

[dieter]

Jason Friedman  writes:

> I have XML which looks like:
>
> 
> 
> 
>   
> 
>   
>   
> 
> 
>   
> 
> 
>   
> 
>
> The "Property X" string appears twice times and I want to output the "path"
> that leads  to all such appearances.

You could use "lxml" and its "xpath" support.

This is a high end approach: you would use a powerful (and big)
infrastructure (but one which could also be of use for other
XML applications). There are more elementary approaches as well
(e.g. parse the XML into a DOM and provide your own visitor
to find the elements you are interested in).

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Dealing with ' (suds)

[dieter]

Ombongi Moraa Fe  writes:

> I'm working with suds to send send messages to smsc. I've notices the
> messages with ' are really not processed on my side (which could mean
> an error occurs somewhere - most probably in my php script). SMSC has also
> asked that I 'take care of the '' as it's the reason my subscribers
> may not be receiving the responses.
>
> How can i 'take care of ' ' so it doesn't give me problems in future?

"'" is one of five so called entity references predefined
by the XML specification. The XML parser (that used by "suds")
should automatically convert it to "'". There should be no need
for your application to do anything special for "'".

Of course, there might be a bug somewhere (in "suds" or its parser).
A deeper analysis would be necessary to verify this assumption
and to locate/fix the bug. Even if this looks more difficult,
I recommend to go this route rather than to try something
special for "'" at the application level.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: UnpicklingError: NEWOBJ class argument isn't a type object

[dieter]

skunkwerk  writes:

> Hi,
>   I'm using a custom pickler that replaces any un-pickleable objects (such as 
> sockets or files) with a string representation of them, based on the code 
> from Shane Hathaway here:
> http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-some-unpicklable-items
>
> It works most of the time, but when I try to unpickle a Django HttpResponse, 
> I get the following error:
> UnpicklingError: NEWOBJ class argument isn't a type object
>
> I have no clue what the error actually means.

The pickling protocol uses a form of bytecode which is executed
during the unpickling to reconstruct the python objects based
on their state found in the pickle alongside the bytecode.

"NEWOBJ" is executed in response to such a bytecode operation.
It expects to get a type as a parameter but in your case,
it gets something else.


> If it pickles okay, why should it not be able to unpickle?  Any ideas?

It is by principle impossible for the pickler to garantee
that an unpickler will later succeed: the pickler does not know
which classes/types are available for the unpickler.

In your special case, the pickler could probably
detect that unpickling will fail - but when
an aim cannot be achieved completely this may provide
motivation to abandon it as a whole - and not put much effort
into a partial achievement.
I have seen many cases where pickling succeeded but
unpickling failed and in principle the pickler could have
already predicted the failure (under the assumption that
the unpickler sees the same classes/types as the pickler).


If it is important for you to get Django HttpResponses
successfully unpickled then you likely need to guide
their pickling process better.

Maybe (as an alternative), you can extract the relevant information
from the "HttpResponse" and pickle that instead of
the response itself?

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: AMQP listening and user-facing daemon

[dieter]

Justin Chiu  writes:

> What is the best approach to writing a concurrent daemon that can
> execute callbacks for different types of events (AMQP messages, parsed
> output of a subprocess, HTTP requests)?

I fear your question is too specific (which means you must investigate
yourself).

First of all, it is always very difficult to answer questions
like "the best way" -- many many details are involved there.

You provided details but to appreciate them one has to know
"AMQP", "Openstack Notifications", "twisted", "greenlets", ...
There will not be a lot of people with this knowledge (I do not
belong to them).


I believe that you will be able to realize a solution
based on (Python) threads. It gives you the more freedom
than "twisted" or "greenlets". But a different approach
might be "better" (in some respect).

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is PyArg_ParseTuple necessary to parse arguments?

[dieter]

rahulgar...@gmail.com writes:

> Or can I just loop through the argument tuple manually by using something 
> like PyTuple_GET_ITEM(args,i), then putting manual code to convert the 
> objects to appropriate C type?

If you like you can do the "parsing" yourself.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: XML validation / exception.

[dieter]

Andrew Robinson  writes:
> On xml.etree,
> When I scan in a handwritten XML file, and there are mismatched tags -- 
> it will throw an exception.
> and the exception will contain a line number of the closing tag which
> does not have a mate of the same kind.
>
> Is there a way to get the line number of the earlier tag which caused
> the XML parser to know the closing tag was mismatched, so I can narrow
> down the location of the mismatches for a manual repair?

This is parser dependent -- and likely not the case for the
standard parsers.

In order to check for the correspondence between opening and
closing tags, that parser must maintain a stack of open tags.
Your request can be fullfilled when the parser keeps associated
line numbers in this stack. I expect that most parser will not do that.

Python's "xml" framework is highly modularied - with each component
having only a minimal task. Especially, the parser is responsible
for parsing only: it parses and generated events for what is sees
(opening tag, closing tag, text, comment, entity, error, ...).
The events are consumend by a separate component (when I remember
right, a so called "handler"). Such a component is responsible
to create the "ETree" during the parsing.
You might be able to provide an alternative for this component
which captures (in addition) line information for opening tags.
Alternatively, you could provide an alternative parser.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Cancel threads after timeout

[dieter]

hyperboreean  writes:

> Here's the use case I want to implement - I have to generate a report
> from multiple database servers. This report should be generated every 2
> hours. Sometimes it happens that a query on one of the database servers
> takes longer than expected and impedes the generation of this report
> (that's right, the queries are ran sequential). What I am trying to
> achieve is to parallelize the queries on each database server and to be
> able to cancel one of them if it takes longer than X minutes.
> threading.Thread doesn't support this and seems that in
> general programming languages don't implement a way to cancel threads
> from the outside.

And it is difficult in general.

The problem lies in the interaction between Python and (unknown by Python)
C extensions. Many C extensions would not work properly when simply
aborted in between (loose memory, fail to release resources,
leave an inconsistent state). You need something like "try: ... finally: ..."
or "try: ... except: ..." to have the ability to handle asynchronous
aborts - but C lacks such support.

In the light of this difficulty, the Python developpers decided not
to expose the possibility of many threading packages to abort a thread.

I was told that their is an approximative function at the Python
C interface (in Python 2, not sure whether it is also available in
Python 3): a function that allows you register the wish for a thread
aborting. This wish is recognized only when the thread is on
Python (not C) level (thus, not applicable in your case) and it causes
an exception that normally leads to the thread abortion but
plays well with Python's "try: ... finally: ..." and
"try: ... except: ...": thus, the thread can properly clean up.


You best approach is probably to ask the databases in parallel,
wait for some time and simply ignore answers that do not arrive
within that time (without any aborting trial).

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: CamelCase vs. all-lowercase package names

[dieter]

Rhubarb Sin  writes:

> PEP-8 calls for "short, all-lowercase names" for packages:
>
> http://www.python.org/dev/peps/pep-0008/#package-and-module-names

This is mainly to support case insensitive file systems (and
file systems with quite limited path length).
With mixed case, some packages/modules may not conflict on
a case sensitive file system but happen to conflict on a
case insensitive file system.

-- 
http://mail.python.org/mailman/listinfo/python-list

read() does not read new content on FreeBSD/OpenBSD/OSX

[Dieter]

Hi group,

I have a problem while reading from a file on BSD-like systems.
I have a writer process which continuously appends data to a file
and a reader (a data logger, something like tail -f), which should
read and analyse date from the file.
It works on Linux, but on BSD-like systems, it only reads one time,
then nothing more, although fstat shows that the file is growing.
And there's also a diff between tell() and the filelen.

If I write a reader in C, then it works, so the problem seems to be
in the python layer.

The writer:

while [ 1 ]; do
echo `date` >> ./file
sleep 1
done

The reader:

import time
import os

f=open("./file", "r")
while True:
print "filelen %d, tell %d, read: %d" % (
  os.fstat(f.fileno()).st_size,
  f.tell(),
  len(f.read()))
time.sleep(1.0)


On Linux:
dieter@linuxbox$ python reader.py 
filelen 15215, tell 0, read: 15215
filelen 15215, tell 15215, read: 0
filelen 15251, tell 15215, read: 36
filelen 15251, tell 15251, read: 0
filelen 15285, tell 15251, read: 34
filelen 15285, tell 15285, read: 0

On FreeBSD/OpenBSD/MacOS:
dieter@osx$ python reader.py 
filelen 183147, tell 0, read: 183147
filelen 183180, tell 183147, read: 33
filelen 183180, tell 183180, read: 0
filelen 183606, tell 183180, read: 0
filelen 183606, tell 183180, read: 0

I began to analyse it with strace/ktrace, but maybe I am missing something.
Are there any special switches/flags for BSD?

dieter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: read() does not read new content on FreeBSD/OpenBSD/OSX

[Dieter]

oh, I forgot the versions:
Mac OS X 10.6.7, python 2.6.1
OpenBSD 4.8, python 2.6.5
FreeBSD 8.0, python 2.6.4

dieter
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: non-standard glibc location

[dieter]

"Fetchinson . via Python-list"  writes:

> I'm trying to install a binary package (tensorflow) which contains
> some binary C extensions. Now my system glibc is 2.15 but the binaries
> in the C extensions were created (apparently) with glibc 2.17. So I
> thought no problemo I installed glibc 2.17 to a custom location, built
> python2.7 from source (hopefully using my custom glibc) and installed
> pip and everything else using this custom built python. But still when
> I try to import tensorflow I get:
>
> ImportError: /lib64/libc.so.6: version `GLIBC_2.17' not found
> (required by 
> /home/nogradi/fetch/custom/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
>
> So apparently it's trying to use my system glibc, not the custom one.
>
> How do I tell this extension to use the custom glibc? Is it even possible?

When you import a C extension, a variant of the linker ("ld")
is used to make the linkage between the open references in
the C extension and its (now) embedding environment.
You can use the envvar "LD_LIBRARY_PATH" (using the typical path
syntax) to tell the linker where it can look for general
shared objects (such as "glibc.so").

There is also a linker command line option to tell such information
when the shared object (corresponding to the C extension) is
build.


> But maybe I have an even more basic issue: how do I link python not
> with the system glibc but with my custom glibc?

Again, you could use the "LD_LIBRARY_PATH" envvar.

There may be a way to specify the information also for the
"configure" (generating the makefiles for the Python build process).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why do we nned both - __init__() and __new__()

[dieter]

Andrej Viktorovich  writes:

> For my understanding both - __init__() and __new__() works like constructors. 
> And __new__() looks is closer to constructor. __init__() is more for variable 
> initialization. Why I can't just initialize in __init__() ?
>
> class ExampleClass(object):
> def __new__(cls,value):
> print("creating new instance with val %s" % (value,) )
> instance = super(ExampleClass,cls).__new__(cls)
> return instance
> def __init__(self, value):
> print("Initialising instance... with val %s" % (value,))
> self.payload = value
>
> exampleInstance = ExampleClass(42)
> print(exampleInstance.payload)

In this special case (as others already explained, it is quite common),
you do not need "__new__".

In the general case, constructing an object can be split into two
subtasks: obtain a raw piece of storage able to manage the object's state;
initialize the object's state. The first subtask is handled by "__new__",
the second by "__init__".

Python has defaults for both subtasks -- and as others already pointed
out, the default "__new__" is almost always sufficient.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Merge pdf files using information from two files

[dieter]

accessnew...@gmail.com writes:
> I have two files (right now they are spreadsheets but I can export them to 
> any format). 
>
> File1 has StoreID (unique) in one column and a pdf map location in the second 
> column. (Names not really sequenced numerically)
>
> 1 C:/maps/map1.pdf
> 2 C:/maps/map2.pdf
> 3 C:/maps/map3.pdf
> 4 C:/maps/map4.pdf
>
> File2 has 3 columns. Column1 is the County name (unique), Column2 are the 
> store IDs that fall in that county separated by commas, and Column3 is 
> warehouse that services the store.
>
> County11,2Warehouse1
> County21,3Warehouse1
> County33  Warehouse4
> County42,4Warehouse3
>
> Is it possible to compare both files and append the maps that belong in each 
> county and naming it by the county_warehouse.pdf?

This will not be easy: PDF is a page layout oriented format, not
a format to facilitate the processing of general structural
data (such as e.g. XML).

You could use a package like "pdfminer" to get at the text content
of a PDF file. You will then need specialized code (developed by yourself)
to reconstruct the column information.
You could then use a PDF generating package such as "reportlab"
to generate a new PDF file.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: mutiprocessing gui

[dieter]

Antoon Pardon  writes:
> When one wants to combine multithreading and gui programming
> all sorts of issues arise. So I wonder how one might combine
> multiprocessing with gui programming.
>
> gui libraries typically have some registration mechanisme,
> where you can register a call back for when data is available
> on a network connection. Could one write a wrapper so that
> the multiprocessing.Queue or multiprocessing.Pipe could be
> usable like that?

The Zope application server has a similar problem.
Though not a GUI application framework, it is using a similar
architecture: a main thread with an event loop and a set of "worker"s.
when a worker has finished its work, it usually must wake up
the event loop (in this case to distribute the result).
It does this by using short messages on a socket (the event loop
in this case is an IO based event loop; thus, has means to recognize
if IO is available).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: windows 8 versus urllib2 certificate verify

[dieter]

Robin Becker  writes:

> I have an application built on 32 bit windows 7 with python 2.7.10.
> The application runs fine on windows 7 and older windows machines, but
> it is failing to connect properly using urllib2 when run on windows 8.
>
> The error CERTIFICATE_VERIFY_FAILED indcates this is some issue with
> urllib2 not being able to verify the remote certificate.
>
> This is pyinstaller so the python and python library seem to be
> constant. I thought I understood that python uses its own cert path,
> but obviously I am wrong.
>
> Googling sort of implies I might need certifi to be installed, but is that 
> true?
>
> Why does this fail only on windows 8?

Certificate verification generally depends on local configuration:
specifically, the set of installed trusted root certificates.

I do not know about Windows, but often the root certificates installed
for other packages, e.g. the browser, are used. And different
versions of the same package may install different sets of trusted
certificates. It might also be that your new environment lacks
one of the packages from your old environment - and it has provided
the required root certificate.


Of course, the problem might also be caused by a general problem.
Try to access a "common" https site via urllib2, one you do not have
any problems to connect with a browser (or another "http" utility
like "wget" or "curl") from the same machines.
If this works, the problem is assiciated with the specific certificate.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Creating a python client for an external service advice

[dieter]

ivan77  writes:
> I would like to create a Python module/client library for a data 
> visualization service that I use (and will be using more) as my first larger 
> contribution to open source Python.

What kind of service is this "data visualization service"?
Is it a library, destined to be linked to an application?
Is it a web service, accessed via an internet protocol? Which one?

> I have not come across any best practices for this, and am wondering whether 
> there are some resources that you may know of.  

This highly depends on the kind of service and how its API
is described.

If it is a library, you make be able to use one of the many
tools that create a binding from C/C++ header files.
If it is a WSDL described web service, you can use a SOAP/WSDL client
library (e.g. "suds") to directly access the service.
etc...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: speech_to_text python command not working

[dieter]

"pizza python"  writes:
>I'm on Linux Mint 18.2 Cinnamon 64-bit.
>
>I am trying to get IBM Watson BlueMix Speech-To-Text to transcribe my
>spoken-word audio files. Because I'm not a coder, I tried to find the
>simplest way to use BlueMix Speech-to-Text. And what I found
>is [1]https://github.com/rmotr/speech-to-text
> ...
Traceback (most recent call last):
> ...
>  speech_to_text formatted_output = FormatterClass().format(result) File
>  "/usr/local/lib/python2.7/dist-packages/speech_to_text/formatters.py", line 
> 36,
>  in format for obj in self._parse(data)) File
>  "/usr/local/lib/python2.7/dist-packages/speech_to_text/formatters.py", line 
> 10,
>  in _parse for obj in data[`results']) KeyError: `results'
> ...
>__
>
>I was expecting an html with the transcript. So why did I get the errors
>above?

Speech to text conversion is not a pure Python solution. It must be based
on some kind of "external service". From the details you have provided,
it looks like some web service.

Your error occurs because what the "external service" has delivered
it not what "speech-to-text" has expected. More precisely,
"speech-to-text" has excepted as result a dict with a "results" key --
but this is missing (likely because some input is wrong or there
is a version mismatch between your "speech-to-text" and the "external service").

I expect that the "data" mentioned in the traceback above contains
some clues for what went wrong. I would use the Python debugger
to investigate along these lines. As you are not yourself a Python
programmer, find one in your region to support you.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to share class relationship representations?

[dieter]

Leam Hall  writes:
> On 09/19/2017 11:16 AM, Stefan Ram wrote:
>> leam hall  writes:
>>> I'm working on designing the classes, sub-classes, and relationships in my
>>> code. What is a good visual way to represent it so it can be stored in git
>>> and shared on the list without large images or attachments?
>>Code /is/ design.
>
>
> I tried that with the small bit of code I have and was told it was too
> confusing. Still working on this.

A standard to describe such relationships is
UML (= "Universal Modeling Language").
There are (partially expensive) tools to visualize UML models.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Sharing code between different projects?

[dieter]

> On Monday, August 13, 2012 at 7:53:32 PM UTC+3, andrea crotti wrote:
>> I am in the situation where I am working on different projects that
>> might potentially share a lot of code.
>> 
>> I started to work on project A, then switched completely to project B
>> and in the transiction I copied over a lot of code with the
>> corresponding tests, and I started to modify it.
>> 
>> Now it's time to work again on project A, but I don't want to copy
>> things over again.

I use so called "namespace packages" for this. Those are "virtual" packages
"containing" a set of related "real" packages -  individually manageable
on PyPI.  The "real" packages describe in their dependencies on which
other packages they depend, among them (potentially) "friend" packages.

An example is "dm.zope.rpc" and "dm.zope.rpc.wsdl_suds". In this case,
"dm.zope.rpc" provides general rpc infrastructure for the web
application framework Zope and implementations for some elementary
rpc protocols and "dm.zipe.rpc.wsdl_suds" uses this infrastructure
for the implementation of a WSDL based rpc protocol by means of "suds".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Distributing multiple packages with on setup.py

[dieter]

Jimmy Thrasibule  writes:
> ...
> Is it possible, still keeping one unique ``setup.py`` file, to create
> 3 independent packages?
>
> * ``myproj.common``
> * ``myproj.subpackage1``
> * ``myproj.subpackage2``
>
> Also I'd like to specify that when installing ``myproj.subpackage1``,
> ``myproj.common`` is required or that ``myproj.subpackage2`` will
> require both ``myproj.common`` and ``myproj.subpackage1``.

Yes - in principal - but ...
the "setup.py" essentially contains the "setup" function call
and this call needs package specific parameter values (e.g.
the package name, the package dependencies, ...).
If you want a single "setup.py", this "setup.py" must use
some (potentially sophisticated) logic to determine those
package specific parameters and select (depending on the context)
which parameter set to pass on to the "setup" function call.

Personally, I never want interested in this approach.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python multiprocessing

[dieter]

Xristos Xristoou  writes:

> I have three functions in the python that each one puts an image (image path) 
> as input and makes a simple image processing and creates a new image (image 
> path) as output.

In order to make effective use of multiprocessing, you need to split
your complete task into (mostly) independent subtasks and let them be processed
by subprocesses. For this to be effective, the subtasks must need
significant processing. If dependencies remain, the processes must
communicate which makes things more complicated and less efficient.

I agree with Stefan Ram: if you have essentially a single image
which gets processed in order by different alorithms, then you
have a sequential process which can not profit from multiprocessing.

However, if you have a set of input images, then you can process
each image in a separate process -- and potentially gain significantly
in speed.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Return str to a callback raise a segfault if used in string formating

[dieter]

Thomas Jollans  writes:

> On 2017-10-13 11:07, Vincent Vande Vyvre wrote:
>> Le 13/10/17 à 09:23, Chris Angelico a écrit :
>>> On Fri, Oct 13, 2017 at 4:46 PM, Vincent Vande Vyvre
> ...
> If you have custom C code, it's likely that there is a memory management
> problem there. It's not unusual for incorrect memory management to cause
> problems in a completely different part of the code, where something
> tries to access freed memory or something.

It is very easy to get something wrong in custom C code.
Therefore, I typically use "cython" (a compiler translating extended
Python to C) to realize such code. It takes care of almost all
subleties.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Return str to a callback raise a segfault if used in string formating

[dieter]

Vincent Vande Vyvre  writes:
> Le 14/10/17 à 15:59, Stefan Behnel a écrit :
> ...
> Thanks, I know Cython but the code is already in C.
>
> This is a lib released as a frontend.  Then usable only in command line.

This does not prevent the use of "Cython". In fact, one of
its use cases is the creation of bindings - a thin wrapper around
a C library to make it easily usable from Python.

An example is "lxml", a binding for the C library "libxml2".
Another one is "dm.xmlsec.binding", a binding for the C library
"libxmlsec".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[dieter]

Karsten Hilbert  writes:
> ...
> When run under "normal" py2.7 it runs all the way through but
> upon shutdown (*after* sys.exit(0)) faulthandler shows a
> problem (and returns 134 which made me think of SIGABRT):
>
>   *** Error in `python': free(): invalid pointer: 0x00770b14 ***

This indicates some form of memory corruption, usually caused by
some C extension. Unfortunately, memory corruption is rarely noticed
locally; therefore, the localization is typically complex.

Maybe, you are lucky and the "mxdatetime" is to blame.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[dieter]

Karsten Hilbert  writes:
> ...
> So here's the final console output of that:
> ...
>   Debug memory block at address p=0x717b7c: API ''
>   0 bytes originally requested
>   The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>   at p-3: 0x03 *** OUCH
>   at p-2: 0x4e *** OUCH
>   at p-1: 0x00 *** OUCH
>   Because memory is corrupted at the start, the count of bytes 
> requested
>  may be bogus, and checking the trailing pad bytes may segfault.
>   The 4 pad bytes at tail=0x717b7c are not all FORBIDDENBYTE (0xfb):
>   at tail+0: 0x00 *** OUCH
>   at tail+1: 0x00 *** OUCH
>   at tail+2: 0x00 *** OUCH
>   at tail+3: 0x00 *** OUCH
>   The block was made by call #0 to debug malloc/realloc.
>   Fatal Python error: bad ID: Allocated using API '', verified using API 
> 'o'
> ...
> Can anyone give more guidance on what the above python debug
> output might vaguely point to ?

It points to a memory corruption.

I would approach the problem by means of debugging: put a write
breakpoint at the corrupted address "0x717b7c" and check what part
of the system accesses it (this assumes you are using a CPU
supporting write breakpoints).
It may be very tedious as the address might be accessed very often
legally before it gets corrupted.

Another approach may be to use a tool designed for memory debugging,
e.g. "valgrind".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: choice of web-framework

[dieter]

Patrick Vrijlandt  writes:
> ...
> The project is completely new, there are no histories to take into
> account (current solutions are paper-based). The website involves
> questionnaires that will be developed, filled out and stored. Users
> are not programmers or developers. They should be
> authenticated. Version control is required. Internationalization is
> not an issue. I expect that the project will add additional
> requirements and complexity later on that I can not foresee yet. I'm
> targeting a single deployment (maybe a second on a development
> machine). I usually work on Windows, but Linux can be considered.

I am using Plone (a CMS (= Content Management System) build on top of
Zope) for something like this. It has a standard extension
"CMFEditions" for version control of its content.

The content is managed in the Zope Object Database (= "ZODB"). This is
also valid for the revisions. Thus, you do not get
"git/mercurial/..."-style version control (based on files) but
you see when and by whom a version was created, can revert to a
previous version and see differences between versions. There is no
merge support, though.

The standard Plone content types are likely not sufficient to
implement your questionnaires; you will probably define one of
more specific for your task. But, this is not too complex.

Questions about Plone (and the underlying Zope) can be asked at
"https://community.plone.org/";.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Modern website

[dieter]

Andrew Z  writes:
> I realize the following has little todo with python per se. But i hope to
> get a guidance on how these types of tasks are done nowadays.
>
> The task:
> Ive been asked to create  an integration process. That is a few webpages
> with questioneer with the submission to a "mother" company using its API .
> Ideally this process will be used by a few 3rd party "child" companies -
> they would just point a link from their site to mine. Prospective users
> will fill out the pages and "mother company" will create accounts
> associated with the referal "child" company.
>
> The problem:
>   how do i create a modern , slick  website. I also see , that each of
> "child" would want to extend the look and feel of their site onto this
> process. So, my process should have at least a few templates i can offer ,
> that would match their site...
> Im not too excited to use weebly or squarespace - id rather host everything

If the liberty of the "client" sites is of primary concern, then
you can use a so called "service oriented architecture".
Such an architecture sonsists of components, most of which providing
a services via a standard API, destined for programmatic use.

You can have services on different levels of abstraction, e.g.
a base level using only structural data without any presentation
and on top of this some standard presentation (which can be used
by clients with more standard presentation requirements).

There are several API-frameworks: e.g. frameworks supporting
WSDL (= "Web Services Description language", an XML-based technology),
JSON-RPC, XML-RPC, ReST, ...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[dieter]

Karsten Hilbert  writes:

> On Sat, Oct 21, 2017 at 09:31:54AM +0200, dieter wrote:
>> It points to a memory corruption.
> While running valgrind and friends is beyond my capabilitis I
> have run the problem through gdb to at least obtain a stack
> trace to see what gives:

The i386/x64 architecture supports memory access breakpoints
and GDB, too, has support for this. You know the address which
gets corrupted. Thus, the following apporach could have a chance
to succeed:

   Put a "memory write" breakpoint on the address which gets corrupted.
   this should stop the program each time this address is written;
   Check then the backtrace. As the address forms part of the
   address block prologue, it should only be accessed from
   Python's "malloc" (and "free") implementation. Any other access
   indicates bad behaviour.

   Should your program get stopped too often (because the memory
   block is reused too often), you must find a good point in your
   program to activate the memory access breakpoint in a delayed way.
   You could use the Python debugger for this: execute significant
   parts of your program in the Python debugger; switching to
   GDB, check whether the address is already corrupted - if
   so, restart and refine the checks above in the portion of your program
   which caused the corruption. If the part in your program
   is sufficiently small, you can activate the memory access breakpoint.
   This approach may also be feasible, should you use a CPU
   without memory access breakpoints.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Objects with __name__ attribute

[dieter]

"ast"  writes:
> I know two Python's objects which have an intrinsic name, classes and
> functions.
> ...
> Are there others objects with a __name__ attribute
> and what is it used for ?

Some Python objects naturally have a name: functions, classes, modules, ...;
others don't: tuples, lists, integers, ...

If a Python object natureally has a name, there is a good chance that
it is accessible in a standard way: "obj.__name__".

Sometimes, the name is a bit special. An example is the "filename"
(potentially) associated with a "file" like object. In those cases,
the name can be accessed in a slightly different way, e.g. "file.filename".


I recommend that you do not try to enumerate all cases where an object
has a name accessible via "obj.__name__". Instead, start with
your concrete task. Select the objects appropriate for your task.
Use the "buildin" functions "dir" and "help" on a sample object
to find out (in an interactive Python sesssion)
which attributes and methods it supports. If this does not give
enough information, consult the documentation.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Let's talk about debuggers!

[dieter]

Thomas Jollans  writes:
> I just wanted to know what tools everyone used for debugging Python
> applications - scripts / backend / desktop apps / notebooks / whatever.
> Apart from the usual dance with log files and strategically inserted
> print() calls, that is.
>
> Of course we all know and mildly dislike pdb.

I mostly develop backend components for internet applications
based on the web application framework "Zope". There,
the extension "Products.PDBDebugMode" is highly helpfull: it enters
the Python debugger ("pdb"), whenever an uncatched exception is raised
or an error is logged; "pdb" can then be used to examine the state
and check what caused the problem.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[dieter]

Karsten Hilbert  writes:
>> >> It points to a memory corruption.
>> 
>> The i386/x64 architecture supports memory access breakpoints
>> and GDB, too, has support for this. You know the address which
>> gets corrupted. Thus, the following apporach could have a chance
>> to succeed:
>> 
>>Put a "memory write" breakpoint on the address which gets corrupted.
>>this should stop the program each time this address is written;
>>Check then the backtrace. As the address forms part of the
>>address block prologue, it should only be accessed from
>>Python's "malloc" (and "free") implementation. Any other access
>>indicates bad behaviour.
>
> I understand. Thank you for the explanation. This may seem
> like a dumb question: the actual address that gets corrupted
> varies from run to run (it may be the same "place" in the
> code but that place gets put at a different address each
> run).

That's sad.

It is a long time ago (more than 10 years)
that I had to analyse such a kind of memory corruption. Fortunately,
in my case, the address was stable accross runs.
Likely, ASLR was not yet used by that time on a standard Linux platform.

Maybe, you find a way to disable ASLR.

If ASLR is the cause of the randomness, it might also be possible to
compute the new address. More on this later.


In another message, you reported how you tried to obtain an invariant
for the affected address by using "info symbol". I have not much
hope that this will succeed:
It is most likely, that the corrupted memory block is part of the
heap (in may also be a stack block, wrongly freed; this would be
a local error - and easily detectable from the traceback).
If you use "info symbol" on a heap address, you get not very
reliable information - especially, if ASLR is in effect (which
randomizes the various process segments and the heap blocks
independently from one another).


Back to an approach how to maybe compute the corrupted address
for a new run. The approach assumes a heap address and
uses that "mallog" (and friends) request large (which means hopefully few)
memory blocks from the OS which are then split into smaller blocks internally.
You can then catalog the large memory block requests and determine
the index of the block and the corrupted offset. In a following run,
you determine the new base address of this block and apply the
same offset to find the corrupted address.

Of cause, this assumes that your application is totally deterministic
(apart from maybe ASLR).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: right list for SIGABRT python binary question ?

[dieter]

Karsten Hilbert  writes:
> ...
> I have posted backtraces taken from the address being
> watched. Does that help any at all ?

Only in the case that the error is "local", i.e. detected
(quite) immediately.

You might be in this case as you have observed that the address
is stable after library preload. Thus, it might not be a heap
address but one associated with one of the libraries. Such
a memory block should never be "freed". The backtrace would allow
you to determine the library affected. Obtain its source. Recompile
with symbols and try to find out where this memory block comes from.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Share unpickleable object across processes

[dieter]

Israel Brewster  writes:

> I have a Flask/UWSGI web app that serves up web socket connections. When a 
> web socket connection is created, I want to store a reference to said web 
> socket so I can do things like write messages to every connected 
> socket/disconnect various sockets/etc. UWSGI, however, launches multiple 
> child processes which handle incoming connections, so the data structure that 
> stores the socket connections needs to be shared across all said processes. 
> How can I do this?

At low (= Posix) level, socket objects are represented as
ints, representing an "open communication object".

Usually, a child process shares the parent's "open communication object"s
*AT THE TINE OF THE CHILD CREATION*. However, often, this sharing
is broken very quickly: the parent may have set up those objects
to close automatically in the child; the parent may close its
"open communication object" after the spawning of the child.

If two processes have not inherited the same "open communication object"
from a common ancestor, there is no way that they can share
the communication object.


I am quite convinced that UWSGI will
be set up to close "open communication object"s passed on to
child processes. Thus, I see little chance that your current approach
can succeed.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to maintain same permissions on python dist-package after upgrade?

[dieter]

Debraj Manna  writes:

> I am using a python package patroni version 1.0 on Ubuntu 14. On doing ls
> -lrt /usr/local/lib/python2.7/dist-packages/ I am seeing the permission
> like below
>
> drwxr-sr-x4   root   staff 4096 Nov 6 14:29 patroni
> drwxr-sr-x2   root   staff 4096 Nov 6 14:29 patroni-1.0-py2.7.egg-info
>
> But once I upgrade the patroni via the command sudo pip install patroni
> --upgrade I am seeing the permission of the patroni change after upgrade
>
> drwxr-s---4   root   staff 4096 Nov 6 15:29 patroni
> drwxr-s---2   root   staff 4096 Nov 6 15:29 patroni-1.3.6.dist-info
>
> The installation output is attached (sudo-pip-install.txt).
>
> If I don't use sudo and just do pip install patroni --upgrade it fails. The
> output is attached (pip-install.txt).
>
> Can someone let me know :-

Verify the "umask" effective for your "sudo" user, i.e.

   sudo bash
   umask

Should the "umask" result end in "7",
then this may be responsible for the permissions you observe.

On POSIX systems, the "umask" setting effects the permissions of newly
created directories and files.

Use "umask 0002" to change this setting temporarily.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: reading text in pdf, some working sample code

[dieter]

Daniel Gross  writes:
> I am new to python and jumped right into trying to read out (english) text
> from PDF files.
>
> I tried various libraries (including slate)

You could give "pdfminer" a try.

Note, however, that it may not be possible to extract the text:
PDF is a generic format which works by mapping character codes to glyphs
(i.e. visual symbols); if your PDF uses a special map for this
(especially with non standard glyph collections (aka "font"s)),
then the text extraction (which in fact extracts sequences
of character codes) can give unusable results.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Save and load initialized class

[dieter]

Bob van der Poel  writes:

> I'm trying to something simple (yeah, right). Mainly I want to have a bunch
> of variables which my program needs and uses to be in a saveable/loadable
> block. Currently I have then all as a bunch of globals which works, but
> trying to keep track of them and the correct spellings, etc becomes a bit
> of maintenance nightmare. So, demonstrating all my cleverness I came up
> with:
>
> class Opts():
>  var1 = 123
>  var2 = "hello world"
> 
> Further, and importantly, I can save the lot with:
>
>   json.dump(Opts.__dict__, open("mybuffer", "w"))
>
> But, I can't figure out how to load the saved data back into my program.
> Doing
>
>   z=json.load (open("mybuffer", "r"))
>
> loads a dictionary ... which makes sense since that is what I saved. So,
> can I now reset the values in Opts from a saved dictionary?

You could use:

  for k, v in z.iteritems: setattr(Opts, k, v)


Of course, this means that you already have a class "Opts" (maybe
without or with different parameter values).


This poses the question why you want to dump/load the class in the first
place. Usually, you would want to have the values in code, not stored/loaded.


If you really want make parameter values persistent, you could use
the following approach:

class ParameterCollection(object):
  def __init__(self, **kwargs):
for k, v in kwargs.iteritems(): setattr(self, k, v)

my_parameters = ParameterCollection(a=1, b=2, )

from pickle import load, dump
dump(my_parameters, open(..., "wb"), -1)
...
my_parameters_reloaded = load(open(..., "rb"))

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Can't install wrapt on Windows

[dieter]

Cutter  writes:
> ...
> I have trouble installing pylint on Windows 10, Python 3.6. There's a
> problem during the installation of wrapt, which is a dependency of
> pylint.
>
> Here are the contents of the commandline:
>
>> C:\WINDOWS\system32>python -m pip install pylint
>> ...
>> Collecting wrapt (from astroid>=1.5.1->pylint)
>>   Using cached wrapt-1.10.11.tar.gz
>> Installing collected packages: colorama, wrapt, astroid, pylint
>>   Running setup.py install for wrapt ... error
>> Exception:
>> Traceback (most recent call last):
>>   File 
>> "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py",
>>  line 73, in console_to_str
>> return s.decode(sys.__stdout__.encoding)
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: 
>> invalid start byte

Obviously, something causes some "s" to be decoded using the
"stdout" encoding (which usually is the system encoding).
In your case, this encoding is "utf-8", but "s" does not seem to
be utf-8 encoded.

I would use debugging to find out what "s" is, where is comes from
and why it does not use the "stdout" encoding.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: request fails on wikipedia (https) - certificate verify failed (_ssl.c:748)

[dieter]

F Massion  writes:
> ...
> I would like to get information from Wikipedia articles and I am testing the 
> connection to Wikipedia. 
>
> I am running Python 3.6.2 on Windows 10.
>
> I get certificate errors for all pages with https. 
> Any suggestions are welcome!
> ...
> self._sslobj.do_handshake()
>   File "c:\Program Files (x86)\Python36-32\lib\ssl.py", line 683, in 
> do_handshake
> self._sslobj.do_handshake()
> ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 
> (_ssl.c:748)

There are too possibilities:
  * the certificate is really invalid
  * Python does not know the root certificate[s] necessary for the verification.

To differenciate between the possibilities, connect (on the same computer)
Wikipedia via a browser. If it, too, reports certificate problems,
you likely have the first possibility; otherwise, the second.

Python typically looks for root certificate information at the same
(platform dependent) place as the browsers. Thus, normally, there
is no need to tell Python where to look for them; but, you can do it
in the case that the root certificates are installed strangely. Consult
the documentation.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol

[dieter]

Piyush Verma <114piy...@gmail.com> writes:

> Getting SSL error while connecting from httplib.HTTPSConnection.
>
> Any help would be appreciated.
> ...
> line 579, in __init__
> self.do_handshake()
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py",
> line 808, in do_handshake
> self._sslobj.do_handshake()
> ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:590)

Are you sure, you try to connect to an HTTPS port?

The error message tells you that the "ssl" handshake failed because
the protocol was unknown (likely because the message was not understood).
The most natural reason is to try to connect to something which is not
prepared for an "ssl" handshake.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol

[dieter]

Piyush Verma <114piy...@gmail.com> writes:

> Yes Dieter, I see that it is connecting with 443 port number and service is
> running. Is this related to python version or mac?

It might be.

Python does not perform the SSL handling itself but delegates it to
an external SSL library ("OpenSSL" on a *nix like platform).
SSL can use different protocols, e.g. for the encryption.
The error message might mean that the server side uses a protocol
which is not supported by the client side. In this case, you would need
to use a more up-to-date SSL library.


I have had no problems to acces "https://www.facebook.com/piyushkv1";
with "urllib2.urlopen" of Python 2.7.12 (on Ubuntu 16.4).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Problems with imports on multiple threads, with embedded Python

[dieter]

geoff.ba...@gmail.com writes:

> I have a multithreaded application using an embedded Python 3.6.4 (upgraded 
> from 3.6.2 today in the hope that the problem was now solved: it doesn't seem 
> to be). The standard library is in a zip file. So long as only one thread is 
> running Python at a time it seems to work fine. But there seems to be a 
> problem with the module importing when several Python threads are active.
>
> I get a variety of errors indeterministically, usually indicating that some 
> symbol hasn't been imported. This occurs both in my own code and in the 
> standard library. The most frequent is probably this line:

I do not know your specific problem (up to now, I used only Python 2),
but I know about import problems in multi threaded applications.
The problem is easily understood: import affects the global 
object "sys.modules". To protect this object, Python must do something.
In former Python versions (somewhere before Python 3), it was using a 
a thread lock: this could cause a deadlock in the case of some import
dependencies. The recoomendation to avoid the problem has been to
avoid module level imports for modules imported by threads.

Python 3 may have changed the way to protect "sys.modules".


In case of recursive import dependencies, you may see that some symbols are not
defined even in single threaded applications -- however then usually
deterministically.


At your place, I would start to check whether your application
has recursive import dependancies -- and in this case, remove them.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Regarding the error: TypeError: can�t pickle _thread.lock objects

[dieter]

Winston Manuel Vijay  writes:

> It would be of immense help, if someone could provide a suitable solution or 
> related information that helps to sort out the below stated issue-
>
>
> Ø  I had installed the Python version 3.6.4
>
> Ø  Then I installed the package: Tensorflow
>
> Ø  Installed g2p.exe by downloading from GitHub
>
> Ø  Then tried running the below command-
>
> g2p-seq2seq --interactive --model  (model_folder_path: is 
> the path to an English model 2-layer LSTM with 512 hidden units CMU Sphinx 
> dictionary downloaded from the CMU Sphinx website)
>
> Following the above procedure, I encountered the following error: TypeError: 
> can.t pickle _thread.lock objects-please find the attached screenshot for 
> your reference.

This looks like a programming error in "g2p-seq2sec": contact its
authors or try a different version.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python server socket file transfer

[dieter]

bingbong3...@gmail.com writes:
> how much client can i handel whit this code what the amount of client that i 
> can handel
> the size of the file is 716 kb
> ...
> self.sock.send(l)

Please read the documentation for *send* in the "socket" module:
it tells you that "send" (in contrast to "sendall") is *not* guarantied
to send the complete *l* (there is no guarantee how much is sent);
the return value of "send" tells you how many bytes have been sent.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python server socket file transfer

[dieter]

bingbong3...@gmail.com writes:

> On Wednesday, January 10, 2018 at 9:07:33 AM UTC+2, dieter wrote:
>> bingbong3...@gmail.com writes:
>> > how much client can i handel whit this code what the amount of client that 
>> > i can handel
>> > the size of the file is 716 kb
>> > ...
>> > self.sock.send(l)
>> 
>> Please read the documentation for *send* in the "socket" module:
>> it tells you that "send" (in contrast to "sendall") is *not* guarantied
>> to send the complete *l* (there is no guarantee how much is sent);
>> the return value of "send" tells you how many bytes have been sent.
>
>
> bro i dont ask about if it works or not or what you say is not right i ask 
> how much he can handel...

This depends on implementation details of your network package (and maybe
other even less stable things). Thus, there is no reliable anwser
to your question.

And you do not need an anwser: use "sendall" or look on the "send"
return value to learn how much has been transmitted (and use subsequent
"send"'s to transmit the rest).


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: exec and traceback

[dieter]

ken...@gameofy.com writes:

> I'm using exec() to run a (multi-line) string of python code. If an
> exception occurs, I get a traceback containing a stack frame for the
> string. I've labeled the code object with a "file name" so I can
> identify it easily, and when I debug, I find that I can interact with
> the context of that stack frame, which is pretty handy.
>
> What I would like to also be able to do is make the code string
> visible to the debugger so I can look at and step through the code in
> the string as if it were from a python file.

The debugger may not be prepared for this kind of source; thus, you
might need a much more intelligent debugger.

The "exec" itself works with arbitary strings, not only with string
constants coming from a source file. It has no way to reliably
associate a filename and a line number in the respective file with
the code lines. The debugger (at least currently) expects to locate
the source code in a file (maybe an archive) given a line number in that
file.

Based on this, I expect that your wish will be difficult to fulfill.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: xpath prob, was Re: Why is there no functional xml?

[dieter]

> Rustom Mody wrote:
>
>> With
>> # Read above xml
> with open('soap_response.xml') as f: inp = etree.parse(f)
>> # namespace dict
> nsd = {'soap': "http://schemas.xmlsoap.org/soap/envelope/";, 'locns':
> "http://example.com/"}
>> 
>> The following behavior is observed $(G!7(B actual responses elided in the
>> interest of brevity
>> 
> inp.xpath('//soap:Body', namespaces = nsd)
>> finds/reaches the node
>> 
> inp.xpath('//locns:blobRetrieveResponse', namespaces = nsd)
>> finds
>> 
> inp.xpath('//locns:dtCreationDate', namespaces = nsd)
>> does not find
>> 
> inp.xpath('//dtCreationDate', namespaces = nsd)
>> finds
>> 
> inp.xpath('//dtCreationDate')
>> also finds
>> 
>> 
>> Doesnt this contradict the fact that dtCreationDate is under the locns
>> namespace??

Apparently, "dtCreationDate" is not associated with the
namespace corresponding to the "locns" namespace.

Note, that the namespace association is not by default inherited
by child elements -- as least not with stand alone XML. Thus,
if you have e.g.

   

then the element "child" does not belong to the namespace indicated
by "nspref" but to the "default namespace".


An XML schema can change this default. However, to get such an
XML schema effective, you must specify this wish when you are
parsing your XML document. Otherwise, your XML document is parsed
as a "stand alone" XML and the rules of the XML-namespace standard
apply -- which means, that the namespace association is not inherited
to child elements.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Package containing C sources

[dieter]

Victor Porton  writes:

> I am going to create a Python wrapper around a generally useful C library. 
> So the wrapper needs to contain some C code to glue them together.
>
> Can I upload a package containing C sources to PyPi?

You can.

This is documented in "https://docs.python.org/2/extending/building.html";
(for Python 2; likely, it works in the same way for Python 3).


Note: I use "setuptools" (a "distutils" extension)
rather than the base "distutils" -- essentially, because it facilitates
the inclusion of non Python files in a distribution.
I do not know whether the base "distutils" automatically includes
the sources of C extensions (likely it does); "setuptools" provides
a way to include everything under control of a versioning system
(which I rely upon to not miss important files).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dependency injection: overriding defaults

[dieter]

Victor Porton  writes:

> I am writing a library, a command line utility which uses the library, and a 
> I am going to use dependency_injector package.
>
> Consider loggers:
>
> For the core library the logger should default to stderr.
>
> For the command line utility, we use the default logger of the library.
>
> For the server, the log should go to a file (not to stderr).
>
> Question: How to profoundly make my software to use the appropriate logger, 
> dependently on whether it is a command line utility or the daemon?

I would distinguish between the common library and distinct
applications (command line utility, daemon). The applications
configure the logging system (differently) while the library uses
uniform logging calls.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Dependency injection: overriding defaults

[dieter]

Kushal Kumaran  writes:

> Victor Porton  writes:
>> dieter wrote:
> ...
>>> I would distinguish between the common library and distinct
>>> applications (command line utility, daemon). The applications
>>> configure the logging system (differently) while the library uses
>>> uniform logging calls.
>>
>> You have essentially just repeated my requirements.
>>
>> But HOW to do this (using dependency_injector module)?
>>
>
> No idea how dependency_injector affects this, but have you read the
> logging howto? The section
> https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library
> would be appropriate.

In fact, it was something like this I had in mind.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: auto-correct a speech-to-text output and relate to of the words based on syllables

[dieter]

nav...@emagevisionpl.com writes:

> I have to make an application in which, 
>
> The user asks a question, Google's API is used to convert the speech to text. 
> But the problem is due to different accent the translator misunderstands the 
> words. 
> I want my application to guess the word to the nearest word spoken by the 
> user. 
> ...
> But sometimes for more than a word it makes the same guess.
> example:
> The user speaks "Light". The system translates it as "Bright"
> The user speaks "White" The system translates it as "Bright"

As those words are phonetically quite apart (they have
very different first consonants), some step in your
processing chain does something seriously wrong.
First thing to do: identify the responsible step. After that,
one can try to find a way to improve on it.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: This newsgroup (comp.lang.python) may soon be blocked by Google Groups

[dieter]

jlada...@itu.edu writes:
> ...
> Let me ask those of you who are not using Google Groups: how do you search?  
> In my experience, searching through mailing list archives has been poor.

I am using "gmane.org" which has quite a good search (in my view).
Not sure, whether "gmane.org" hosts "your" newsgroup.

Of course, the service may go away in the future.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: auto-correct a speech-to-text output and relate to of the words based on syllables

[dieter]

Steven D'Aprano  writes:

> On Fri, 02 Feb 2018 08:14:03 +0100, dieter wrote:
>
>>> The user speaks "Light". The system translates it as "Bright" The user
>>> speaks "White" The system translates it as "Bright"
>> 
>> As those words are phonetically quite apart (they have very different
>> first consonants), some step in your processing chain does something
>> seriously wrong.
>
> I disagree: Light, Bright and White sound very similar. They're identical 
> except for the first consonant:

We have here an example that the first consonant can significantly
influence the meaning.
As a consequence, it will in general be spoken and affect the sound.
And obviously, I should not be ignored when one is interested in
a narrow match.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: auto-correct a speech-to-text output and relate to of the words based on syllables

[dieter]

"Peter J. Holzer"  writes:
> On 2018-02-03 09:34:57 +0100, dieter wrote:
> ...
> The difficulty is to *recognise* it correctly. Was that tangle of sound
> waves an "l" or an "r"? This not as unambiguous as you seem to think.
> So a speech-to-text program may hear "right" when the speaker was really
> saying "light". If you have only the output from that program you must
> determine whether "right" is correct or must be corrected to "light".

The primary recommendation from my first response has been:
determine the step in the process chain that made the error.
If already the first step ("speech-to-text") has wrongly interpreted
"light" as "right", then obviously context is necessary to determine
that this was wrong.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Help on PyImport_Import(pNAme)

[dieter]

Jason Qian via Python-list  writes:

>   This only works when loading  modules from the current directory.
>   Is there a way I can load from somewhere else ?

Have a look at the module `importlib` and especially its function
`import_module`.

Python knows about two forms of "import": "absolute import"
and "relative import".
An "absolute import" tries (in the usual case) to import from a list
of directories, maintained in "sys.path"; a "relative import"
(recognized via the module name starting with a .") relative to
a package.

Section 31.5.6.3 of
"https://docs.python.org/3/library/importlib.html#module-importlib.util";
tells you how to import a source file directly (independent from
any context).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: libxml2 installation/binding issue with Python 3.6.4

[dieter]

"Priest, Matt"  writes:
> ...
> I've successfully (?) installed Python 3.6.4 and libxml2, with the ultimate 
> goal of installing GTK+ 3.22.0.

You might also try "lxml" - which is an alternative for "libxml2".

> However, I'm running into this error:
> ...
> import libxml2mod
> ImportError: 
> /nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/lib/python3.6/site-packages/libxml2mod.so:
>  undefined symbol: _PyVerify_fd

This indicates a version mismatch between the C part of
"libxml2" and something in your system - maybe the "libxml2" C library
but more likely Python (given the name).

Are you sure that "libxml2" is Python 3 ready?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to work on a package

[dieter]

Roel Schroeven  writes:
> I'm fairly comfortable writing Python code, but I only have experience
> writing scripts with perhaps a few supporting modules. Now I want to
> start writing a package, and I'm feeling a bit helpless: I'm not sure
> how to organize my work.

You may have a look at
"https://packaging.python.org/"; and
"https://packaging.python.org/tutorials/distributing-packages/#requirements-for-packaging-and-distributing";.

> In my way of thinking, I would have a working tree for the package (or
> even more than one), and also an installed version (once version 0.1
> or so is ready).
>
> For example, let's say I'm working on luaparser
> (https://github.com/boolangery/py-lua-parser). There are tests in
> directory luaparser/tests, and I want to execute those tests. So I
> execute, for example:
>
> $ python3 -m unittest test_ast.py

Likely, there are many ways to execute tests for your package.

I am using "setuptools" for packaging (an extension
of Python's standard "disutils"). Its "setup.py" supports the "test"
command. This means, properly set up, I can run tests
with "python setup.py test".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to set/update value in a xml file using requests in python

[dieter]

Sum J  writes:
> My xml file is located in local network : 
> http://192.168.43.109/DevMgmt/NetAppsDyn.xml
>
> Below is a part content of above xml I want to update :
>
> 
> 
> 
> off
> 
>
> I want to set value for 'ResourceUI' and 'Port' field in above xml.
> I have used below code :
>
>  data = {
>   'ResourceURI':'web-proxy.xxx.yy.com',
>   'Port':8080
> }
>
> URL = 'http://192.168.75.165/DevMgmt/NetAppsDyn.xml'
>
> # content type
> head = {'Content-type': 'text/xml'}
> # sending get request
> gr= requests.get(url=URL)
> print gr
>
> # sending put request
> r = requests.put(url=URL, data=data,headers=head)
> print r.status_code
> # extracting response text
> output_xml = r.text
> print("The op xml is:%s" % output_xml)
>
> Issue : The fields are not getting updated in xml using put request. I am 
> able to see the response for get (request) , but for put request it is 
> throwing errror code : 301 , resource has been moved permanently.

This may not be a Python question.

You are sending your data
("{'ResourceURI':'web-proxy.xxx.yy.com', 'Port':8080})
to the web service under "http://192.168.75.165/DevMgmt/NetAppsDyn.xml";.
Apparently, this web service does not interpret the data in the way you
expect.

Note, that you do not send an XML document to "NetAppsDyn.xml"
(despite your `Content-type="text/xml"`). The typical (WevDAV)
"put" would require that "data" contains a complete replacement
for the object to be replaced, not just some modifications
(of course, the object might have a specialized "put" which could
support partial updates -- but apparently, in your current
situation, this is either not the case or the "put" involved
expects to get the replacement information in a different way).

I would approach your task as follows:

 1. Fetch the original XML

 2. Use one of Python's XML libraries (I would use "lxml")
to create the modified XML

 3. "put" the modified XML to the server

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to work on a package

[dieter]

Roel Schroeven  writes:
> dieter schreef op 7/02/2018 8:21:
> ...
>> Likely, there are many ways to execute tests for your package.
>
>> I am using "setuptools" for packaging (an extension
>> of Python's standard "disutils"). Its "setup.py" supports the "test"
>> command. This means, properly set up, I can run tests
>> with "python setup.py test".
>
> That can solve the testing issue, I guess, even though it feels weird
> to me that the most straightforward way doesn't work.
>
> But testing is not the only issue. Often I'd like to start the Python
> interpreter to load one of the modules in the package to try some
> things out, or write a little script to do the same. These things are
> very natural to me when writing Python code, so it seems very strange
> to me that there's no easy way when working on a packages.

I use a "virtualenv" for this: it is like a private Python environment
with its own set of packages.

The "setup" from "setuptools" not only supports a "test" command
but also a "develop" command. This "links" the package (source)
into Python's (the one you used to run "setup.py"; i.e. usually
the one of a "virtualenv") library and thereby makes it available
for this Python.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Extracting data from ython dictionary object

[dieter]

Stanley Denman  writes:

> I am new to Python. I am trying to extract text from the bookmarks in a PDF 
> file that would provide the data for a Word template merge. I have gotten 
> down to a string of text pulled out of the list object that I got from using 
> PyPDF2 module.  I am stuck on now to get the data out of the string that I 
> need.  I am calling it a string, but Python is recognizing as a dictionary 
> object.  
>
> Here is the string: 
>
> {'/Title': '1F:  Progress Notes  Src.:  MILANI, JOHN C Tmt. Dt.:  05/12/2014 
> - 05/28/2014 (9 pages)', '/Page': IndirectObject(465, 0), '/Type': '/FitB'}
>
> What a want is the following to end up as fields on my Word template merge:
> MedSourceFirstName: "John"
> MedSourceLastName: "Milani"
> MedSourceLastTreatment: "05/28/2014"
>
> If I use keys() on the dictionary I get this:
> ['/Title', '/Page', '/Type']I was hoping "Src" and Tmt Dt." would be treated 
> as keys.  Seems like the key/value pair of a dictionary would translate 
> nicely to fieldname and fielddata for a Word document merge.  Here is my  
> code so far. 

A Python "dict" is a mapping of keys to values. Its "keys" method
gives you the keys (as you have used above).
The subscription syntax ("[]"; e.g.
"pdf_info['/Title']") allows you to access the value associated with
"".

In your case, relevant information is coded inside the values themselves.
You will need to extract this information yourself. Python's "re" module
might be of help (see the "library reference", for details).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: atws

[dieter]

Larry Martell  writes:
> ...
> I had 2.2.1. I updated requests to 2.18.4 and now when I import atws I get:
>
> No handlers could be found for logger "atws.connection"

This is a warning (only), telling you that the "atws" package wants
to log a message but there is not corresponding logging handler
defined -- likely because you did not configure Python's logging
infrastructure (--> "logging" package in Python's standard documentation).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Questions about `locals` builtin

[dieter]

Kirill Balunov  writes:
> I am a little bit confused with `locals` builtin in these moments:
>
> 1. The documentation says that _free varaibles_ are returned, which seems
> incorrect description. In my mind the term free variable refers to
> variables used in a function that are not local variables nor parameters of
> that function.
>
> In docs: "Update and return a dictionary representing the current local
> symbol table. Free variables are returned by `locals()` when it is called
> in function blocks, but not in class blocks."

I agree with your definition of "free variable" and I think that
the documentation partially speaks of it.

In a function body, you have two kinds of "free variables":
implicitly used free variables and explicitly declared (via "global",
"nonlocal") free variables.

I think the documentation wants to make clear, that "locals"
does not treat "free variables" in a completely consistent way
(there is a difference between class and function blocks).

I assume that "locals" may return some (but not necessary all)
free variables in a function block -- depending on the actual
implementation.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Questions about `locals` builtin

[dieter]

Kirill Balunov  writes:
>  2018-02-27 2:57 GMT+03:00 Terry Reedy :
>
>> The point of point 3 is that terminology and details would likely be
>> different if Python were freshly designed more or less as it is today, and
>> some things only make more or less sense in historical context. Learn what
>> you need to know to write code that works.
>>
>
> Thank you, I'm fairly familiar with the scope and namespace concepts in
> Python 3, and they are also very well described in the "Execution model"
> https://docs.python.org/3/reference/executionmodel.html#execution-model,
> for this special thanks to the person who wrote it ;-)
>
> I started using Python with СPython 3.5 and I'm not familiar with the
> Python 2 features. But since Python 2 got into our discussion, I still have
> a question:
>
> a.  Is this restriction for locals desirable in the implementation of
> CPython in Python 3?
> b.  Or is it the result of temporary fixes for Python 2?

I think it is an implementation artefact: for efficiency reasons,
local variables in a function are handled differently than "local"
variables elsewhere. A side effect of the concrete implementation
causes that truely local variables cannot be changed by updating
the result of "locals()". The "locals" documentation tries to
document the restrictions.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Questions about `locals` builtin

[dieter]

Ned Batchelder  writes:
> On 2/27/18 3:52 AM, Kirill Balunov wrote:
>> a.  Is this restriction for locals desirable in the implementation of
>> CPython in Python 3?
>> b.  Or is it the result of temporary fixes for Python 2?
>
> My understanding is that the behavior of locals() is determined mostly
> by what is convenient for the implementors, so that they can keep
> regular code running as quickly as possible.  The answer to the
> question, "why can't we make locals() work more like I expect?" is,
> "because that would make things slower."
>>
>> Personally, I find the convenient functionality to update the local symbol
>> table inside a function, similar to `globals`.
>
> Can you show us an example of why you would want to update locals
> through locals()?  There might be more natural ways to solve your
> problem.

I am still working with Python 2 (Python 3 may behave differently).
There, during debugging, I would sometimes like to change the value
of variables (I know that the variable has got a wrong value
and would like to fix it to continue senseful debugging without a restart).
This works for variables not yet known inside the function but does
not work for true local variables.
I assume that this is one effect of the "locals()" restriction.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: In Python2, does it need to wrap imp.find/load_module with imp_acquire/release_lock?

[dieter]

Xiang Zhang  writes:

> Just like the title. It seems to me it is needed from the source code but 
> codes in stdlib all doesn't do that.

The "import" machinery uses locks of its own (to protect "sys.modules").

I assume that "load_module" will use those locks internally.
"find" might be considered to usually see quite a static situation;
should your file system be modified by concurrent threads, it might
be necessary that you use your own locks to protect against this.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: RFC: Proposal: Deterministic Object Destruction

[dieter]

Rick Johnson  writes:
> ...
> Can you provide a real world example in which you need an
> object which circularly references _itself_?

Circular structures occur in real life.

One example are tracebacks (which are very helpful for debugging
reasons). Keeping a traceback in a variable can easily introduce
a cycle.

Another example is a tree structure where you need both efficient
access to children and ancestors of a node (see the DOM standard).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Strange problem with pip2

[dieter]

Cecil Westerhof  writes:

> There are three pip2 packages that should be updated:
> apsw (3.13.0.post1) - Latest: 3.9.2.post1 [sdist]
> mysql-utilities (1.6.4) - Latest: 1.4.3 [sdist]
> pygobject (3.22.0) - Latest: 3.27.4 [sdist]
>
> But the strange thing is that the installed version is newer as the
> new version. And when I try:
> pip2 install --upgrade apsw
>
> I get:
> Requirement already up-to-date: apsw in /usr/lib/python2.7/dist-packages

Playing recently with a daily Ubuntu 18.4 preview, I have
seen something similar: I was told that a Python package must
be updated but Python's installation tools reported: current version
already installed.

I have not investigated (thinking it was a fault of the early
Ubuntu version).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Send Msg To Specific Tcp Client from Tcp Server Python

[dieter]

Anil reddy reddy M  writes:

> I was written a simple tcp server in python, which is nicely communicating 
> with multiple tcp clients. My Tcp Server can accept multiple clients at time, 
> each client as a new thread. I want send message to specific tcp client after 
> some time.
> I can client Address from   connction, addr = socket.accept(), but i am not 
> able send msg to client with client  IP.POrT. I would be thankfull for any 
> help i get

TCP channels are bidirectional: you can send messages in both
directions. Thus, writing to the server's socket (for a specific client)
sends the written data to the client.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Knowing which thread had the GIL before

[dieter]

Julien Kauffmann  writes:
> ...
> Is there a way to know which thread had the GIL right before my
> profiling thread acquired it ? Perhaps through a C extension ? I've
> seen such profilers for Linux that take advantage of an ITIMER signal
> to do that. Sadly this is not an option on Windows.

As you have already suspected, you will likely need a C extension:
Python does (almost surely) not maintain a GIL history (not even a short
one containing only the previous GIL holder).

If you implement your monitoring thread (mostly) in C, there definitely
is a global interpreter variable which contains a thread identifier
for the thread currently holding the GIL.


> Any feedback or remark concerning my technique is also welcome.

It would be best if you could get informed when the GIL switches.
Almost surely, this is unsupported on Python level. But perhaps
on C level. I would check the (C) code for Python's GIL handling.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: importing down in code rather than at top of file.

[dieter]

Tobiah  writes:

> Is it  worth while to defer the import of a large module that seldom
> gets used in the script?
>
>
>   import sys
>   import os
>
>   if hardly_ever_happens():
>   
>   import large_module
>   large_module.do_task()

I have used delayed import for different reasons:

 * to avoid cyclical imports

 * to avoid import deadlocks in multi-tasking programs
   (Python 2 (at least) used to protect the import machinery with
   a lock; which under some conditions could lead to deadlocks
   in a multi-tasking program).

Typically, the delayed import was then in a function - relying
on the fact that importing an already imported module is fast
(thus, we do not lose much even if the function is called multiple times).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Variables visibility for methods

[dieter]

"ast"  writes:
> ...
> So it seems that when an object's méthod is executed, variables
> in the scope outside the object's class can be read (2nd example),
> but not variables inside the class (1st example).
>
> For 1st example, I know that print(MyClass.a) or print(self.a)
> would have work.
>
> Any comments are welcome.

You are right. And it is documented this way.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: C Python extension to export an Function

[dieter]

Ganesh Pal  writes:

> Iam pretty new to C Python extension , I was able to export few simple
> modules to python and it look like the cool thing to do ...

Maybe, it is a good idea to have a look at "cython".

"cython" is a compiler. It translates Python code enhanced with
special annotations into C. The annotations mostly tell the compiler
that something ("object", "method", "function", ...) should be at "C"
rather than "Python" level, thus avoiding much of Python's overhead
and allows to do things possible in "C" but not in "Python".


Developing safe "C" extensions for Python is difficult. You
need some quite deep understanding of the Python-C interface
and must be very careful to observe all requirements (especially
those related to proper reference management).

Developing "C" extensions with "cython" is much easier as
"cython" hides many of the complexities and takes care of most
requirements.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: C Python extension to export an Function

[dieter]

Ganesh Pal  writes:
> ...
> Thanks stefan and  Gollwitzer  , good to know there are many ways to do this
> i.e via cython or SWIG   but  the C/Python API
>  is probably the most widely used method
> - not for it’s simplicity but for the fact that you can manipulate python
> objects in your C code.
>
> I want to use C/Python API 

I have used both Python's C API (to implement "dm.incrementalsearch")
and (later) "cython" (to implement "dm.xmlsec.binding"). I can tell
you from personal experience
that the "cython" approach is much easier (and even more straight forward).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: Fwd: Python freeze fails with warning: the use of `tempnam' is dangerous, better use `mkstemp'

[dieter]

"Alexander N. Moibenko"  writes:
> when I build a binary with python 2.6 it builds without any problem.
> When I build with python 2.7 the build fails after warnings like:
> /opt/python/Python-2.7.12/Modules/posixmodule.o: In function
> `posix_tempnam':
> /opt/python/Python-2.7.12/./Modules/posixmodule.c:7578: warning: the
> use of `tempnam' is dangerous, better use `mkstemp'
> collect2: ld returned 1 exit status
>
> How this can be fixed?

I doubt that the (compiler) warning is responsible for the (linker) failure.

When I am faced with problems like this (usually during extension building;
I have never encountered a problem during a Python built), I try
to find out what (OS level) commands are executed by the build machinery
and then issue them manually on the command line: the build machinery
often hides error details revealed by executing the commands
directly on the command line.


What I can imagine is that your (OS level installed) C libraries (i.e. "libc"
and friends) no longer contain a "tempnam", letting the linker fail.
Thus, maybe, the compiler still warns about "tempnam" but the linker
already considers it as an error.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Fwd: Fwd: Python freeze fails with warning: the use of `tempnam' is dangerous, better use `mkstemp'

[dieter]

"Alexander N. Moibenko"  writes:
> In fact I tried issuing commands manually, but they did not give me
> any hint more than I already had.

When I remember right, then "gcc" has an option to show details
about the calling of the phase commands ("gcc" is a coordinating
programm delegating the real work to subcommands ("cpp", "cc1", "ld", ...)).


Key to your problem seems to understand why "ld/collect2" fails.
Usually, you should get a clear error message corresponding to this
failure (e.g. "symbol  undefined"). Getting only
"collect2 exited with error code 1" is unusual.

I would approach the problem by isolating the "collect2/ld" command
and then try hard to determine what the command is doing and why it
fails (e.g. by "strace"ing it).


"collect2" is usually used for "C++" programs (collecting code
for the initialization of static C++ objects). Maybe, the difference
between Python 2.7 and earlier Python versions is that it includes C++
parts - and maybe, "collect2" has some (still not understood) problems
in your environment.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to diagnose CERTIFICATE_VERIFY_FAILED on Windows for websites working in browsers?

[dieter]

Chi Hsuan Yen  writes:
> ...
> Apparently OpenSSL verifies the whole certificate chain and
> report an error as soon as it finds an invalid certificate in the chain.

As it must, if you require verification.

When I remember right, you can disable the verification altogether
(though you should not for security reasons).

> My
> questions is: how to tell from several possible causes to
> CERTIFICATE_VERIFY_FAILED? Currently both expired self signed certificate
> leads to CERTIFICATE_VERIFY_FAILED. Thanks for any help or advice.

I would try to find (or write) a utility that follows the certificate
chain and provides details information about its state.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to diagnose CERTIFICATE_VERIFY_FAILED on Windows for websites working in browsers?

[dieter]

Chi Hsuan Yen  writes:
> ...
> Thanks a lot! I just lost my direction when trying to understand how
> certificate verification works in Python.

It sets things up for "OpenSSL" and then delegates all details
to it.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to extend a tuple of tuples?

[dieter]

"Frank Millman"  writes:

> Hi all
>
> This should be easy, but I cannot figure it out.
>
> Assume you have a tuple of tuples -
>
> a = ((1, 2), (3, 4))
>
> You want to add a new tuple to it, so that it becomes -
>
>((1, 2), (3, 4), (5, 6))
>
> The obvious way does not work -
>
> a += (5, 6)
>
>((1, 2), (3, 4), 5, 6)

You could use:

  a += (5, 6),

or (more clearly written):

  a += ((5, 6),)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to diagnose CERTIFICATE_VERIFY_FAILED on Windows for websites working in browsers?

[dieter]

Chi Hsuan Yen  writes:
> ...
> I found that OpenSSL provides an X509 callback hook that allows clients to
> know why the verification process fails.

For a long time, Python 2 (unlike Python 3) did not perform
certificate validation at all. As a consequence, some developpers
provided the functionality in external packages. Maybe, one of those
exposes the callback you have found.
I never used one of those
packages and cannot provide more information that what I said above.

You might also look at "PyOpenSSL" (--> "PyPI"). Apparently, it supports
callbacks written in Python.
Again, I have not myself used this package.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Mysterious Logging Handlers

[dieter]

Josh English  writes:
> I have a Python script that imports a utility script. Both scripts use 
> logging, but the logs don't work as advertised. I'm getting logging output 
> from the utility script but none from the main file. Worse, the format of the 
> utility script's logs don't match anything I define.
>
> The utility script is called xlreader.py. It has the following lines:
>
> -- begin snippet --
> import logging
> XLOGGER = logging.getLogger('XLREADER')
> -- end snippet --
>
> And it calls XLOGGER.debug(), XLOGGER.error() and XLOGGER.info() but it never 
> defines a handler.
> ...

Python's logging system is quite complex. Among others, it has a logger
hierarchy where loggers on a lower level can delegate to their parent.
Usually, most configuration happens at the so called "root logger"
and is used by all loggers which have not overridden certains aspects.

In your special case, the handlers likely come from the "root logger".

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: collect2: ld returned 1 exit status when building from source

[dieter]

Steve D'Aprano  writes:
> ...
> but the build still fails, with the same errors:
>
>
> Python/dtrace_stubs.o: In function `PyDTrace_LINE':
> /home/steve/python/python-dev/cpython/Include/pydtrace.h:25: multiple
> definition of `PyDTrace_LINE'
> Python/ceval.o:/home/steve/python/python-dev/cpython/Include/pydtrace.h:25:
> first defined here
>   [ ... many, many, many more similar errors ... ]
>
> collect2: ld returned 1 exit status
> make: *** [Programs/_freeze_importlib] Error 1

Looks as if some error has slipped in the sources: you should not have multiple
conflicting definitions for the same symbol (e.g. "PyDTrace_LINE").
Alternatively, your built might fetch wrong headers (e.g. headers
for a different Python version).

I would look at the places from which the "multiple definition"s come
and try to find out why there are multiple of them.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Global variable is undefined at the module level

[dieter]

Daiyue Weng  writes:

> Hi, I defined a global variable in some function like this,
>
> def some_function(self):
>
>  global global_var
>
> PyCharm inspection gave me,
>
> Global variable is undefined at the module level
>
> How to fix this?

You define the global variable at the module level.

"global VVV" (in a function) actually does not define "VVV"
as a global variable. Instead, it tells the interpreter that
"VVV" should not be treated as a local variable but be looked up
in the "global" (usually module) namespace.

In order to define "VVV", you must assign a value to it -- either
directly in the "global" (i.e. module) namespace or in your function
(together with a "global VVV" declaration).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ‘pip2 install cryptography’ does not work

[dieter]

Cecil Westerhof  writes:

> I try to do a:
> pip2 install cryptography
>
> But this give:
> gcc -pthread -shared 
> build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o 
> -L/usr/lib64 -lssl -lcrypto -lpython2.7 -o 
> build/lib.linux-x86_64-2.7/cryptography/hazmat/bindings/_openssl.so
> 
> /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: 
> /usr/lib64/libpython2.7.a(abstract.o): relocation R_X86_64_32S against 
> `_Py_NotImplementedStruct' can not be used when making a shared object; 
> recompile with -fPIC
> /usr/lib64/libpython2.7.a: error adding symbols: Bad value
> collect2: error: ld returned 1 exit status
> error: command 'gcc' failed with exit status 1

Apparently, the linker requires use of the "-fPIC" option
("PIC" stands for "position independent code") for linking a shared
library.

The problem might come from your system's Python installation
(apparently, that one is used). Usually, you would have
a "libpython*.so" (i.e. a shared object) and it would contain
modules compiled with "-fPIC". In your case, the
static library "libpython*.a" seems to be used and (apparently)
contains not position idependent modules.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Functions Of Functions Returning Functions

[dieter]

Lawrence D’Oliveiro  writes:
> The less code you have to write, the better. Less code means less
> maintenance, and fewer opportunities for bugs.

While I agree with you in general, sometimes less code can be harder
to maintain (when it is more difficult to understand).

Some time ago, we had a (quite heated) discussion here about
a one line signature transform involving a triply nested lambda construction.
It was not difficult for me to understand what the construction did;
however, the original poster found it very difficult to grasp.

Often, functions returning functions are more difficult to understand
than "first order" functions (those returning simple values only) --
at least for people not familiar with higher abstraction levels.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.5.1 C API, the global available available is not destroyed when delete the module

[dieter]

dl l  writes:
> I found the problem is resolved if call PyGC_Collect() after
> PyDict_DelItemString(). Is it expected to call PyGC_Collect() here for
> Python 3.5 which is not needed for Python 3.3?

Usually, Python uses reference counting to determine when an object
can be deleted (and then reclaims its memory).
However, reference counting cannot determine an object's obsoleteness
when it is involved in a reference cycle. To address most of these
cases, Python has an alternative way to determine object obsoleteness
based on marking the accessible objects and garbage collecting
what is inaccessible. The alternative way is what "PyGC_collect" starts.

If part of the objects where involved in a reference cycle,
you would have to call "PyGC_collect" also in Python 3.3 in order
to get a (mostly) clean memory state.
It might be, that Python 3.5 introduces more cycles than Python 3.3.
However, more likely, your code was responsible for the cycle creation
(and not Python itself). Likely, you just did not notice the problem
back for Python 3.3.

Note that "PyGC_collect", too, is not garanteed to give a memory clean
state. It does not release object cycles involving objects with
an explicite destructor ("__del__" method). Those cycles are
made available to the application (consult the documentation of the
"gc" module) which can then decide what to do about them
(e.g. break the cycles and release the associated objects).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.5.1 C API, the global available available is not destroyed when delete the module

[dieter]

dl l  writes:
> I understood call PyGC_collect to release object cycles. But in my python
> test code, seems there is NO reference cycle on the Simple object in the
> python test code. Why needs to call PyGC_Collect() here?
>
> =
>
> class Simple:
>  def __init__( self ):
>  print('Simple__init__')
>  def __del__( self ):
>  print('Simple__del__')
>
> simple = None
>
> def run():
> global simple
> simple = Simple()
>
> if __name__ == '__main__':
>   run()

I do not see a cycle in this example. Thus, maybe, Python itself
has introduced one (maybe via "__builtins__").

To further analyse the case, I would start by checking the "gc" module's
documentation to find out, whether it provides a way to get information
about the cycles it has found (and would have deleted if it had
not generated references for this analysis).

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PyThreadState_Get

[dieter]

Bharadwaj Srivatsa  writes:

> Which ever project I am trying to install using python setup.py install 
> command, i am getting the following error..
>
> python -mtrace --trace setup.py install
> Fatal Python error: PyThreadState_Get: no current thread
> ABORT instruction (core dumped)
>
> How to get rid of this error and whats the cause for this

What happens when you omit "-mtrace --trace"?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python C API: How to debug reference leak?

[dieter]

dl l  writes:
> I want to check the references of an object. Any way to get the references
> of an object with Python C API? Like: gc.get_referrs(), is there similar
> API in C lib?

"gc" is a module. You can import and access modules from the C API.
Thus, you can use "gc.get_referers" from "C" code.

-- 
https://mail.python.org/mailman/listinfo/python-list