Re: making executables smaller

2016-07-27 Thread Thomas 'PointedEars' Lahn
Carter Temm wrote:

> I’m writing a couple different projects at the moment, and when I compile
> it into a single executable using pyinstaller, it becomes extremely large.
> I’m guessing this is because of the modules used. Because I’m not that
> skilled at python, I put stuff like for example, import sys. I imagine the
> final project could be made smaller by specifying from something import
> something_else. but the thing is, I don’t know what smaller I could import
> with these set of modules. Is there a program that could tell me this.

I recommend to comment out all “import” statements (for later reference) and 
then use a Python editor like PyDev to generate step by step “from … import 
…” statements for all used symbols that are not yet defined.

If it is a rather simple program where you can easily make no-ops out of 
modifying statements by defining mockups, you can just run it before you 
compile it to see where you get errors with commented-out “import” 
statements.

In theory, you could look for code of the form “foo.bar” using the regular 
expression "(?:"")?(?:[^"\\]|\\"(?:"")?)+"(?:"")?|'(?:'')?(?:[^'\\]|
\\'(?:'')?)+'(?:'')?)|((?:\w+\.)+)(\w+)¹.  If $1 (e.g., “foo”) is not 
defined in the file, then it is likely that $2 (e.g., “bar”) is a symbol 
from module $1 and you could write

#-
from $1 import $2
…
$2
#-

instead of

#-
import $1
…
$1.$2
#-

But there can be false positives and duplicate symbols this way.

_
¹  the first two alternatives exclude Python string literals
-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python Print Error

2016-07-27 Thread Steven D'Aprano
On Wednesday 27 July 2016 13:45, Cai Gengyang wrote:

> How to debug this error message ?

Start by reading the message:

invalid literal for int() with base 10: ''

Now try to experiment at the interactive interpreter:

int('45')  # works
int('xyz')  # ValueError: invalid literal for int() with base 10: 'xyz'
int('QWE')  # ValueError: invalid literal for int() with base 10: 'QWE'
int('!@#')  # ValueError: invalid literal for int() with base 10: '!@#'

What do you think int('') will do? Try it and see. Were you right?


> print('You will be ' + str(int(myAge) + 1) + ' in a year.')
> Traceback (most recent call last):
>   File "", line 1, in 
> print('You will be ' + str(int(myAge) + 1) + ' in a year.')
> ValueError: invalid literal for int() with base 10: ''

What value do you think myAge has?

Hint: you call int(myAge). That raises ValueError, and says that '' is an 
invalid literal for int. What value do you think myAge must have?


Where does myAge get its value from?


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: making executables smaller

2016-07-27 Thread Steven D'Aprano
On Wednesday 27 July 2016 14:52, Thomas 'PointedEars' Lahn wrote:

> Carter Temm wrote:
> 
>> I’m writing a couple different projects at the moment, and when I compile
>> it into a single executable using pyinstaller, it becomes extremely large.
>> I’m guessing this is because of the modules used. Because I’m not that
>> skilled at python, I put stuff like for example, import sys. I imagine the
>> final project could be made smaller by specifying from something import
>> something_else. but the thing is, I don’t know what smaller I could import
>> with these set of modules. Is there a program that could tell me this.
> 
> I recommend to comment out all “import” statements (for later reference) and
> then use a Python editor like PyDev to generate step by step “from … import
> …” statements for all used symbols that are not yet defined.

What benefit do you think you will gain from changing (let's say):

import collections
x = collections.deque(foo)

to:

from collections import deque
x = deque(foo)


as far as executable size goes? Do you think that by using from...import... the 
module, and all of its dependencies, won't need to be included?

Or are you just trying to save a handful of bytes ("collections.deque" in UTF-8 
is 17 bytes, compared to "deque" is just 5 bytes)?


Since the OP is talking about saving space, how much space do you expect to be 
able to save using this technique?


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: making executables smaller

2016-07-27 Thread Daniel Bradburn
A couple of things you can try:

* Generate a directory rather than onefile, on the directory you can apply du
-hs * | sort -h -r (or treesize if you are using windows
https://www.jam-software.com/treesize_free) to see which folders / files
are taking up a lot of space. Then once you see what is taking up a lot of
space you can try and figure out why it is being included, maybe you have a
load of unused imports? With pyinstaller you can explicitly exclude modules
you know you won't need with --exclude-module once you've optimised the
directory build you can of course switch back to onefile

* If all else fails you can use upx to compress your binary files (dlls +
exe) which can help reduce the overall size:
https://pythonhosted.org/PyInstaller/usage.html#using-upx

Hope this helps.


2016-07-27 10:20 GMT+02:00 Steven D'Aprano <
steve+comp.lang.pyt...@pearwood.info>:

> On Wednesday 27 July 2016 14:52, Thomas 'PointedEars' Lahn wrote:
>
> > Carter Temm wrote:
> >
> >> I’m writing a couple different projects at the moment, and when I
> compile
> >> it into a single executable using pyinstaller, it becomes extremely
> large.
> >> I’m guessing this is because of the modules used. Because I’m not that
> >> skilled at python, I put stuff like for example, import sys. I imagine
> the
> >> final project could be made smaller by specifying from something import
> >> something_else. but the thing is, I don’t know what smaller I could
> import
> >> with these set of modules. Is there a program that could tell me this.
> >
> > I recommend to comment out all “import” statements (for later reference)
> and
> > then use a Python editor like PyDev to generate step by step “from …
> import
> > …” statements for all used symbols that are not yet defined.
>
> What benefit do you think you will gain from changing (let's say):
>
> import collections
> x = collections.deque(foo)
>
> to:
>
> from collections import deque
> x = deque(foo)
>
>
> as far as executable size goes? Do you think that by using
> from...import... the
> module, and all of its dependencies, won't need to be included?
>
> Or are you just trying to save a handful of bytes ("collections.deque" in
> UTF-8
> is 17 bytes, compared to "deque" is just 5 bytes)?
>
>
> Since the OP is talking about saving space, how much space do you expect
> to be
> able to save using this technique?
>
>
> --
> Steve
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Two constructive reviewers sought

2016-07-27 Thread David Shi via Python-list
To promote the use of Python and formalise Python approach, I decided to 
publish a paper.
I used geodata as a showcase.
Geodata lies in the heart of geographical information science.  The management 
and processing of such data is of great importance.

I got an email from International Journal of Information Management.
I was asked to suggest two reviewers for the following article.
"Key solutions for automated data linkage, processing and generating output for 
big data analytics over the Internet".

Nitrogen input from fertilizer use data was used as a showcase.
If you like to be one of reviewers, please email drshishaozh...@gmail.com with 
your full contact details.
Regards.
David
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: pyinstaller

2016-07-27 Thread Larry Martell
On Wed, Jul 27, 2016 at 2:23 AM, Christian Gollwitzer  wrote:
> Am 27.07.16 um 03:15 schrieb Larry Martell:
>>
>> On Tue, Jul 26, 2016 at 8:49 PM, Tom Brown  wrote:
>>>
>>> I used pyinstaller quite a bit 3 years ago. I could brush off the cobwebs
>>> and see if I can help if you have not solved it already.
>>>
>>> What is the issue you are having?
>>
>>
>> If I import the requests module, then when I run the executable I get:
>>
>> ImportError: No module named 'requests.packages.chardet'
>
>
> That's a classic issue. pyinstaller does static analysis of the program,
> which modules must be included. If the code computes a module dynamically,
> it can not always succeed. The solution is to tell pyinstaller to add this
> module. In previous versions, you could add these by "pyinstaller
> --hidden-import=requests.packages.chardet" or similar. If this doesn't work,
> you need to edit the spec file. See here:
>
> https://pythonhosted.org/PyInstaller/when-things-go-wrong.html#listing-hidden-imports

Yes, I had seen that and I tried it with:

--hidden-import=requests.packages.chardet

and I got the same error at run time:

$ dist/pyi_test/pyi_test
Traceback (most recent call last):
File "pyi_test.py", line 1, in
import requests
File 
"/usr/lib/python2.7/site-packages/PyInstaller/loader/pyimod03_importers.py",
line 389, in load_module
exec(bytecode, module.dict)
File "requests/init.py", line 58, in
File 
"/usr/lib/python2.7/site-packages/PyInstaller/loader/pyimod03_importers.py",
line 389, in load_module
exec(bytecode, module.dict)
File "requests/utils.py", line 26, in
File 
"/usr/lib/python2.7/site-packages/PyInstaller/loader/pyimod03_importers.py",
line 389, in load_module
exec(bytecode, module.dict)
File "requests/compat.py", line 7, in
File "requests/packages/init.py", line 95, in load_module
ImportError: No module named 'requests.packages.chardet'
-- 
https://mail.python.org/mailman/listinfo/python-list


logging: getLogger() or getLogger(__name__)?

2016-07-27 Thread Malcolm Greene
I've read that best practice for logging is to place the following line
at the top of all modules:
 
logger = getLogger(__name__)
 
I'm curious why the following technique wouldn't be a better choice:
 
logger = getLogger()
 
Are there scenarios that favor one style over another?
 
Thank you,
Malcolm
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: reshape with xyz ordering

2016-07-27 Thread Heli
Thanks for your replies. Let me explain my problem a little bit more. I have 
the following data which i read from a file using numpy.loadtxt and then i sort 
it using np.lexsort:

x=f[:,0] # XColumn
y=f[:,1] # YColumn
z=f[:,2] # ZColumn
val=f[:,3] # Val Column
xcoord=np.sort(np.unique(f[:,0])) # XCoordinates
ycoord=np.sort(np.unique(f[:,1])) # YCoordinates
zcoord=np.sort(np.unique(f[:,2])) # ZCoordinates

ind = np.lexsort((val,z,y,x)) 
val_sorted=np.array(val[ind])

I know that the val column has data sorted first by x, then by y, then by z 
which means that column x changes slowest and column z changes fastest.

x,y,z, val 
0,0,0,val1 
0,0,1,val2 
0,0,2,val3 

0,0,zn,valn
...
xn,yn,zn,valfin

I want to reshape val_sorted in to a 3d numpy array of (nx,ny,nz). 

which of the following is the correct way and why?

#1
val_sorted_reshaped=val_sorted.reshape((xcoord.size,ycoord.size,zcoord.size))

#2
#val_sorted_reshaped=val_sorted.reshape((xcoord.size,ycoord.size,zcoord.size)).transpose()

#3
#val_sorted_reshaped=val_sorted.reshape((zcoord.size,ycoord.size,xcoord.size)) 

#4
#val_sorted_reshaped=val_sorted.reshape((zcoord.size,ycoord.size,xcoord.size)).transpose()
 

Thanks, 


-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Python environment on mac

2016-07-27 Thread Crane Ugly
Yep, I agree. virtualenv is the best way to go so far. It solves my wishes to 
use python completely without root access too.
Thank's a lot.
Leonid 
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: pyinstaller

2016-07-27 Thread Vlastimil Brom
2016-07-27 3:15 GMT+02:00 Larry Martell :
> On Tue, Jul 26, 2016 at 8:49 PM, Tom Brown  wrote:
>> I used pyinstaller quite a bit 3 years ago. I could brush off the cobwebs
>> and see if I can help if you have not solved it already.
>>
>> What is the issue you are having?
>
> If I import the requests module, then when I run the executable I get:
>
> ImportError: No module named 'requests.packages.chardet'
>
> I tried to post to the pyinstaller group, but it said my post had to
> be approved by the moderator, and it apparently never was. I have no
> idea who the moderator is, so there was no one I could contact about
> that. I posted an issue to github
> (https://github.com/pyinstaller/pyinstaller/issues/2060) and some
> suggestions were made, but none fixed the problem. I am on RHEL 7.2
> with Python 2.7.5, and it's reproducible, just by having a 1 line
> script that has "import requests". Thanks for any help you could
> provide.
>
>
>
>
>>
>> On Jun 21, 2016 16:57, "Larry Martell"  wrote:
>>>
>>> Anyone here have any experience with pyinstaller? I am trying to use
>>> it, but I'm not having much success. I tried posting to the
>>> pyinstaller ML but it said my post had to be approved first, and that
>>> hasn't happened in a while. I'll post details if someone here thinks
>>> they can help.
>>> --

Hi,
is there a direct reference to that function in your code?

On win7, python 3.5, if I use just a trivial stub source file like:

import requests
print(requests.packages.chardet.detect(b"qwe"))

and freeze it with
_path_to_\Python3\Scripts\pyinstaller.exe _path_to_\test_chardet.py
--clean --noconfirm --onedir

the resulting executable works ok, (it prints {'encoding': 'ascii',
'confidence': 1.0} just like the source version).

However, I remember, that I had problems in the past with freezing
(sub)modules or (sub)packages, that were not actually used in the
source but should be made available for interactive usage on runtime
within the app. I believe, manually referencing such objects helped in
such cases, but I can't remember the details.

Otherwise, it may be some version or platform dependent issue, of course.

regards,
 vbr
-- 
https://mail.python.org/mailman/listinfo/python-list


Behavior of tempfile temp files when scripts killed, interpreter crashes, server crashes?

2016-07-27 Thread Malcolm Greene
Can someone share their OS specific experience in working with tempfile
generated temp files under these conditions?
 
1. Script killed by another process
2. Interpreter crashes
3. Server crashes (sudden loss of power)
4. Other application termination conditions ???
 
I'm curious which scenarios result in temp files not being automatically
deleted after use and what technique you're using to cleanup temp files
left over after these conditions (without affecting legitimate temp
files present from the current session)?
 
Do any OS's support a type of temp file that truly gets automatically
deleted in all of the above scenarios?
 
Thank you,
Malcolm
-- 
https://mail.python.org/mailman/listinfo/python-list


TypeError: '_TemporaryFileWrapper' object is not an iterator

2016-07-27 Thread Antoon Pardon
I am rewriting a program so it can work in python3.

I am making progress, but now I stumble on the exception mentioned in
the subject.
The code still works in python2.

What I'm doing is to write the results to temporary file that needs to be
sorted afterwards. For as far as I understand after done writing all to
the file and seeking it to the beginning I get the following when I start
reading from it.

Traceback (most recent call last):
  File "/local/home/apardon/src/fietsroutes/pymain.py", line 132, in main
Exit_Nr = process(sys.argv)
  File "/local/home/apardon/src/fietsroutes/route.py", line 491, in program
make_routes(map)
  File "/local/home/apardon/src/fietsroutes/route.py", line 465, in make_routes
for rt in sortfile(allroutes, load, dump):
  File "/local/home/apardon/src/fietsroutes/filesort.py", line 129, in sortfile
prv = next(itr)
  File "/local/home/apardon/src/fietsroutes/filesort.py", line 67, in Queuer
buf = take(21, itr)
  File "/local/home/apardon/src/fietsroutes/filesort.py", line 21, in take
ls.append(next(iterable))
  File "/local/home/apardon/src/fietsroutes/filesort.py", line 35, in loading
ln = load(fl)
  File "/local/home/apardon/src/fietsroutes/route.py", line 366, in load
return greep(next(fl))
TypeError: '_TemporaryFileWrapper' object is not an iterator
Locals by frame, innermost last

Frame main in /local/home/apardon/src/fietsroutes/pymain.py at line 149
  pn = 'route'
 process = 

Frame program in /local/home/apardon/src/fietsroutes/route.py at line 491
argv = ['route', '-r', '45']
   index = 3
 map = {'84Bh': {'83Bh': 17, '01Wo': 42, '85Bm': 37}, '70Rb': 
{'18Rb': 26, '03Zu': 91, "43Gd'": 33}, "43Gd'": { ...

Frame make_routes in /local/home/apardon/src/fietsroutes/route.py at line 465
   Dists = {'35HS': 204, '91Rt': 208, '79Oi': 492, "61Ch'": 347, 
'78AP': 479, '33Bt': 444, '241L': 429, '54Rh': 385 ...
   MinLength = 249
  OldMin = 216
   allroutes = 
dest = '16Pl'
dist = 58
last = '86Ll'
   last_time = 1469626452.5767715
 map = {'84Bh': {'83Bh': 17, '01Wo': 42, '85Bm': 37}, '70Rb': 
{'18Rb': 26, '03Zu': 91, "43Gd'": 33}, "43Gd'": { ...
  nr = 29
nwrt = ('16Pl', '86Ll', '80Wz', '79Bs', '59Bs', '69Vs', 
"70Vs'", '70Vs"', "78Th'", '78Th"', '48Th', '45Th', '01 ...
   nwtot = 274
 prv = '80Wz'
   route = ('86Ll', '80Wz', '79Bs', '59Bs', '69Vs', "70Vs'", 
'70Vs"', "78Th'", '78Th"', '48Th', '45Th', '01Th', '04 ...
   start = '00Th'
   total = 216
   vizor = ''

Frame sortfile in /local/home/apardon/src/fietsroutes/filesort.py at line 129
  fl = 
 in1 = 
 in2 = 
 itr = 
 ou1 = 
 ou2 = 
   swaps = 0
warn = 

Frame Queuer in /local/home/apardon/src/fietsroutes/filesort.py at line 67
  fl = 
 itr = 

Frame take in /local/home/apardon/src/fietsroutes/filesort.py at line 23
   _ = 0
iterable = 
  ls = []
   n = 21

Frame loading in /local/home/apardon/src/fietsroutes/filesort.py at line 40
  fl = 

Frame load in /local/home/apardon/src/fietsroutes/route.py at line 366
  fl = 

-- 
https://mail.python.org/mailman/listinfo/python-list


python and open office

2016-07-27 Thread Crane Ugly
I try to create some scripts that will help me to open and manipulate 
OpenOffice documents. Calc in particular. But I have some problems finding 
right packages or libraries that offer such interface.
So far I was trying uno and unotools but the first step is to import them 
failed. Here is the output:

UNO tools are installed:
$ pip list | grep uno
uno (0.3.3)
unotools (0.3.3)

Try to import them:
$ python
Python 2.7.12 (default, Jun 29 2016, 12:53:15)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import uno
>>> import unotools
Traceback (most recent call last):
  File "", line 1, in 
  File 
"/Volumes/home/lshanin/Dropbox/Python/ve/accounting/lib/python2.7/site-packages/unotools/__init__.py",
 line 16
def __init__(self, name: str):
   ^
SyntaxError: invalid syntax
>>>

I would appreciate is somebody help me to find what is wrong with untools 
package.
Or point me to some other available libraries. I expect to work with OpenOffice 
(LibreOffice) files only not with MS Excel files.

Regards
Leonid
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python and open office

2016-07-27 Thread Chris Kaynor
def __init__(self, name: str):

That "name: str" syntax is called function annotations, and was added in
Python 3, and you are trying to use the module in Python 2.7.

There may be another variation of the module compatible with Python 2, or
you'll need to upgrade your Python to a version of Python 3.

Chris

On Wed, Jul 27, 2016 at 9:28 AM, Crane Ugly  wrote:

> I try to create some scripts that will help me to open and manipulate
> OpenOffice documents. Calc in particular. But I have some problems finding
> right packages or libraries that offer such interface.
> So far I was trying uno and unotools but the first step is to import them
> failed. Here is the output:
>
> UNO tools are installed:
> $ pip list | grep uno
> uno (0.3.3)
> unotools (0.3.3)
>
> Try to import them:
> $ python
> Python 2.7.12 (default, Jun 29 2016, 12:53:15)
> [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import uno
> >>> import unotools
> Traceback (most recent call last):
>   File "", line 1, in 
>   File
> "/Volumes/home/lshanin/Dropbox/Python/ve/accounting/lib/python2.7/site-packages/unotools/__init__.py",
> line 16
> def __init__(self, name: str):
>^
> SyntaxError: invalid syntax
> >>>
>
> I would appreciate is somebody help me to find what is wrong with untools
> package.
> Or point me to some other available libraries. I expect to work with
> OpenOffice (LibreOffice) files only not with MS Excel files.
>
> Regards
> Leonid
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: logging: getLogger() or getLogger(__name__)?

2016-07-27 Thread Laurent Pointal
Malcolm Greene wrote:

> I've read that best practice for logging is to place the following line
> at the top of all modules:
>  
> logger = getLogger(__name__)
>  
> I'm curious why the following technique wouldn't be a better choice:
>  
> logger = getLogger()
>  
> Are there scenarios that favor one style over another?

With __name__ you will have one logger per source file (module), with 
corresponding filtering possibilities, and organized hierarchically as are 
packages (logging use . to built its loggers hierarchy). 

Without __name__, you have one global default logger.

>  
> Thank you,
> Malcolm

-- 
https://mail.python.org/mailman/listinfo/python-list


working with OpenOffice Calc

2016-07-27 Thread id23...@gmail.com
I am looking for a library that will allow me to work with Calc documents from 
Python.
But so far I was not able to build properly working environment for that.
Here is what I already tried.
Installed uno and unotools for Python 2.7, but importing unotools gives an 
error:
UNO tools are installed: 
(accounting) $ pip list | grep uno 
uno (0.3.3) 
unotools (0.3.3) 

Try to import them: 

(accounting) $ python 
Python 2.7.12 (default, Jun 29 2016, 12:53:15) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin 
Type "help", "copyright", "credits" or "license" for more information. 
>>> import uno 
>>> import unotools 
Traceback (most recent call last): 
  File "", line 1, in  
  File 
"/Python/ve/accounting/lib/python2.7/site-packages/unotools/__init__.py", line 
16 
def __init__(self, name: str): 
   ^ 
SyntaxError: invalid syntax 
>>> 


I also tried to install pyoo, but it is also failing on import:

(accounting) $ pip list |grep pyoo
pyoo (1.1)

(accounting) $ python
Python 2.7.12 (default, Jun 29 2016, 12:53:15)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyoo
Traceback (most recent call last):
  File "", line 1, in 
  File "/Python/ve/accounting/lib/python2.7/site-packages/pyoo.py", line 26, in 

FORMAT_TEXT = 
uno.getConstantByName('com.sun.star.i18n.NumberFormatIndex.TEXT')
AttributeError: 'module' object has no attribute 'getConstantByName'
>>>


Could someone point me to the right direction to solve the issue.

Regards
Leonid

-- 
https://mail.python.org/mailman/listinfo/python-list


Re: python and open office

2016-07-27 Thread Terry Reedy

On 7/27/2016 12:37 PM, Chris Kaynor wrote:

def __init__(self, name: str):

That "name: str" syntax is called function annotations, and was added in
Python 3, and you are trying to use the module in Python 2.7.

There may be another variation of the module compatible with Python 2, or
you'll need to upgrade your Python to a version of Python 3.

Chris

On Wed, Jul 27, 2016 at 9:28 AM, Crane Ugly  wrote:


I try to create some scripts that will help me to open and manipulate
OpenOffice documents. Calc in particular. But I have some problems finding
right packages or libraries that offer such interface.
So far I was trying uno and unotools but the first step is to import them
failed. Here is the output:

UNO tools are installed:
$ pip list | grep uno
uno (0.3.3)
unotools (0.3.3)

Try to import them:
$ python
Python 2.7.12 (default, Jun 29 2016, 12:53:15)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import uno
import unotools

Traceback (most recent call last):
  File "", line 1, in 
  File
"/Volumes/home/lshanin/Dropbox/Python/ve/accounting/lib/python2.7/site-packages/unotools/__init__.py",
line 16
def __init__(self, name: str):
   ^
SyntaxError: invalid syntax




I would appreciate is somebody help me to find what is wrong with untools
package.
Or point me to some other available libraries. I expect to work with
OpenOffice (LibreOffice) files only not with MS Excel files.


Are you working with OpenOffice or LibreOffice?  There are *different 
programs*.  Last I know, current LibreOffice comes with python 3.3.3 in 
its program directory and you need at least Python 3.3.3 for its UNO 
bridge, as it used the FSR unicode representation introduced in 3.3.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list


Re: working with OpenOffice Calc

2016-07-27 Thread Terry Reedy

On 7/27/2016 1:54 PM, id23...@gmail.com wrote:

I am looking for a library that will allow me to work with Calc documents from 
Python.
But so far I was not able to build properly working environment for that.


You posted this same question 1 1/2 hours before under a different name. 
 Please don't repost, especially when a correct answer was posted 9 
minutes after the first.



I also tried to install pyoo, but it is also failing on import:


A different question, but the answer is still the same.  Use compatible 
versions of python and 3rd party packages.



(accounting) $ pip list |grep pyoo
pyoo (1.1)

(accounting) $ python
Python 2.7.12 (default, Jun 29 2016, 12:53:15)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import pyoo

Traceback (most recent call last):
  File "", line 1, in 
  File "/Python/ve/accounting/lib/python2.7/site-packages/pyoo.py", line 26, in 

FORMAT_TEXT = 
uno.getConstantByName('com.sun.star.i18n.NumberFormatIndex.TEXT')
AttributeError: 'module' object has no attribute 'getConstantByName'





Could someone point me to the right direction to solve the issue.

Regards
Leonid




--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list


Re: python and open office

2016-07-27 Thread Leonid Shanin
I use LibreOffice. Yes, it looks like I have to switch to Python 3 in this case.
I'll try v.3 then
Thank you
Leonid



> On 27-07-2016, at 21:01, Terry Reedy  wrote:
> 
> On 7/27/2016 12:37 PM, Chris Kaynor wrote:
>> def __init__(self, name: str):
>> 
>> That "name: str" syntax is called function annotations, and was added in
>> Python 3, and you are trying to use the module in Python 2.7.
>> 
>> There may be another variation of the module compatible with Python 2, or
>> you'll need to upgrade your Python to a version of Python 3.
>> 
>> Chris
>> 
>> On Wed, Jul 27, 2016 at 9:28 AM, Crane Ugly  wrote:
>> 
>>> I try to create some scripts that will help me to open and manipulate
>>> OpenOffice documents. Calc in particular. But I have some problems finding
>>> right packages or libraries that offer such interface.
>>> So far I was trying uno and unotools but the first step is to import them
>>> failed. Here is the output:
>>> 
>>> UNO tools are installed:
>>> $ pip list | grep uno
>>> uno (0.3.3)
>>> unotools (0.3.3)
>>> 
>>> Try to import them:
>>> $ python
>>> Python 2.7.12 (default, Jun 29 2016, 12:53:15)
>>> [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>> import uno
>> import unotools
>>> Traceback (most recent call last):
>>>  File "", line 1, in 
>>>  File
>>> "/Volumes/home/lshanin/Dropbox/Python/ve/accounting/lib/python2.7/site-packages/unotools/__init__.py",
>>> line 16
>>>def __init__(self, name: str):
>>>   ^
>>> SyntaxError: invalid syntax
>> 
>>> 
>>> I would appreciate is somebody help me to find what is wrong with untools
>>> package.
>>> Or point me to some other available libraries. I expect to work with
>>> OpenOffice (LibreOffice) files only not with MS Excel files.
> 
> Are you working with OpenOffice or LibreOffice?  There are *different 
> programs*.  Last I know, current LibreOffice comes with python 3.3.3 in its 
> program directory and you need at least Python 3.3.3 for its UNO bridge, as 
> it used the FSR unicode representation introduced in 3.3.
> 
> -- 
> Terry Jan Reedy
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list


Is it possible to draw a BUTTON?

2016-07-27 Thread huey . y . jiang
Hi Folks,

It is common to put a BUTTON on a canvas by the means of coding. However, in my 
application, I need to draw a circle on canvas, and then make this circle to 
work as if it is a button. When the circle is clicked, it triggers a new image 
to be displayed. Somebody can help? Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list


ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Larry Martell
When I try and unpickle an object with pickle.loads it fails with:

ImportError: Import by filename is not supported when unpickleing

I've never used pickle before. Why do I get this and how can I fix it?
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: pyinstaller

2016-07-27 Thread Larry Martell
On Wed, Jul 27, 2016 at 9:12 AM, Vlastimil Brom
 wrote:
> 2016-07-27 3:15 GMT+02:00 Larry Martell :
>> On Tue, Jul 26, 2016 at 8:49 PM, Tom Brown  wrote:
>>> I used pyinstaller quite a bit 3 years ago. I could brush off the cobwebs
>>> and see if I can help if you have not solved it already.
>>>
>>> What is the issue you are having?
>>
>> If I import the requests module, then when I run the executable I get:
>>
>> ImportError: No module named 'requests.packages.chardet'
>>
>> I tried to post to the pyinstaller group, but it said my post had to
>> be approved by the moderator, and it apparently never was. I have no
>> idea who the moderator is, so there was no one I could contact about
>> that. I posted an issue to github
>> (https://github.com/pyinstaller/pyinstaller/issues/2060) and some
>> suggestions were made, but none fixed the problem. I am on RHEL 7.2
>> with Python 2.7.5, and it's reproducible, just by having a 1 line
>> script that has "import requests". Thanks for any help you could
>> provide.
>>
>>
>>
>>
>>>
>>> On Jun 21, 2016 16:57, "Larry Martell"  wrote:

 Anyone here have any experience with pyinstaller? I am trying to use
 it, but I'm not having much success. I tried posting to the
 pyinstaller ML but it said my post had to be approved first, and that
 hasn't happened in a while. I'll post details if someone here thinks
 they can help.
 --
>
> Hi,
> is there a direct reference to that function in your code?

No there is no reference to that function. As I said, for testing I
have just a single line script with

import requests

>
> On win7, python 3.5, if I use just a trivial stub source file like:
>
> import requests
> print(requests.packages.chardet.detect(b"qwe"))
>
> and freeze it with
> _path_to_\Python3\Scripts\pyinstaller.exe _path_to_\test_chardet.py
> --clean --noconfirm --onedir
>
> the resulting executable works ok, (it prints {'encoding': 'ascii',
> 'confidence': 1.0} just like the source version).

Yes, people have reported on git that it works on Windows.

> However, I remember, that I had problems in the past with freezing
> (sub)modules or (sub)packages, that were not actually used in the
> source but should be made available for interactive usage on runtime
> within the app. I believe, manually referencing such objects helped in
> such cases, but I can't remember the details.

I will try that.

> Otherwise, it may be some version or platform dependent issue, of course.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Jason Benjamin
On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote:

> When I try and unpickle an object with pickle.loads it fails with:
> 
> ImportError: Import by filename is not supported when unpickleing
> 
> I've never used pickle before. Why do I get this and how can I fix it?

Try using *pickle.load* instead of *pickle.loads*.  pickle.loads is for 
strings.  Retrieved from documentation: help(pickle) after importing 
pickle. 



-- 
[The Computer] was the first machine man built that assisted the power of 
his brain instead of the strength of his arm. - Grace Hopper
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Is it possible to draw a BUTTON?

2016-07-27 Thread Jason Benjamin
On Wed, 27 Jul 2016 13:18:16 -0700, huey.y.jiang wrote:

> Hi Folks,
> 
> It is common to put a BUTTON on a canvas by the means of coding.
> However, in my application, I need to draw a circle on canvas, and then
> make this circle to work as if it is a button. When the circle is
> clicked, it triggers a new image to be displayed. Somebody can help?
> Thanks!

It depends on what GUI toolkit you use.  Can you give more specific 
information?  

Since a canvas is refreshed (updated) every time the function for redraw 
is called, it is as simple as changing the variable for what should be 
drawn when a click event is sent to the canvas and the area hot-spot is 
the same as the area of the button.  Buffered drawing is similar. 



-- 
[The Computer] was the first machine man built that assisted the power of 
his brain instead of the strength of his arm. - Grace Hopper
-- 
https://mail.python.org/mailman/listinfo/python-list


ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Larry Martell
On Wednesday, July 27, 2016, Jason Benjamin > wrote:

> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote:
>
> > When I try and unpickle an object with pickle.loads it fails with:
> >
> > ImportError: Import by filename is not supported when unpickleing
> >
> > I've never used pickle before. Why do I get this and how can I fix it?
>
> Try using *pickle.load* instead of *pickle.loads*.  pickle.loads is for
> strings.  Retrieved from documentation: help(pickle) after importing
> pickle.
>

I am unpickling a string created with pickle.dumps
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Is it possible to draw a BUTTON?

2016-07-27 Thread huey . y . jiang
On Wednesday, July 27, 2016 at 4:18:29 PM UTC-4, huey.y...@gmail.com wrote:
> Hi Folks,
> 
> It is common to put a BUTTON on a canvas by the means of coding. However, in 
> my application, I need to draw a circle on canvas, and then make this circle 
> to work as if it is a button. When the circle is clicked, it triggers a new 
> image to be displayed. Somebody can help? Thanks!

---> By the way, the GUI is TK.
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Larry Martell
On Wed, Jul 27, 2016 at 6:54 PM, Jason Benjamin  wrote:
> If it has and 's' on the end it will only work on strings.  *dumps* refers
> to a string too.

Yes, I know. I have an object, which I pickle with dumps, which turns
it into a string. Then I try to unpickle it with loads and I get that
error.


(Pdb) type(args.target)

(Pdb) pickle.loads(args.target)
*** ImportError: Import by filename is not supported.


> On Wed, Jul 27, 2016 at 3:44 PM, Larry Martell 
> wrote:
>
>
>
> On Wednesday, July 27, 2016, Jason Benjamin  wrote:
>>
>> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote:
>>
>> > When I try and unpickle an object with pickle.loads it fails with:
>> >
>> > ImportError: Import by filename is not supported when unpickleing
>> >
>> > I've never used pickle before. Why do I get this and how can I fix it?
>>
>> Try using *pickle.load* instead of *pickle.loads*.  pickle.loads is for
>> strings.  Retrieved from documentation: help(pickle) after importing
>> pickle.
>
>
> I am unpickling a string created with pickle.dumps
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Is it possible to draw a BUTTON?

2016-07-27 Thread MRAB

On 2016-07-28 00:13, huey.y.ji...@gmail.com wrote:

On Wednesday, July 27, 2016 at 4:18:29 PM UTC-4, huey.y...@gmail.com wrote:

Hi Folks,

It is common to put a BUTTON on a canvas by the means of coding. However, in my 
application, I need to draw a circle on canvas, and then make this circle to 
work as if it is a button. When the circle is clicked, it triggers a new image 
to be displayed. Somebody can help? Thanks!


---> By the way, the GUI is TK.


Here's a simple example:


#! python3.5
# -*- coding: utf-8 -*-
import tkinter as tk

def mouse_clicked(event):
dist_sq = (event.x - circle['x']) ** 2 + (event.y - circle['y']) ** 2

if dist_sq <= circle['radius'] ** 2:
print('Clicked inside circle')

root = tk.Tk()

canvas = tk.Canvas(root, width=400, height=200)
canvas.pack()

circle = dict(x=60, y=60, radius=20)

left = circle['x'] - circle['radius']
top = circle['y'] - circle['radius']
right = circle['x'] + circle['radius']
bottom = circle['y'] + circle['radius']

canvas.create_oval((left, top, right, bottom), outline='red', fill='red')
canvas.bind('', mouse_clicked)

root.mainloop()

--
https://mail.python.org/mailman/listinfo/python-list


Re: Is it possible to draw a BUTTON?

2016-07-27 Thread Rick Johnson
On Wednesday, July 27, 2016 at 7:15:20 PM UTC-5, MRAB wrote:
> On 2016-07-28 00:13, huey.y.ji...@gmail.com wrote:
> > On Wednesday, July 27, 2016 at 4:18:29 PM UTC-4, huey.y...@gmail.com wrote:
> >> Hi Folks,
> >>
> >> It is common to put a BUTTON on a canvas by the means of coding. However, 
> >> in my application, I need to draw a circle on canvas, and then make this 
> >> circle to work as if it is a button. When the circle is clicked, it 
> >> triggers a new image to be displayed. Somebody can help? Thanks!
> >
> > ---> By the way, the GUI is TK.
> >
> Here's a simple example:
> 
> 
> #! python3.5
> # -*- coding: utf-8 -*-
> import tkinter as tk
> 
> def mouse_clicked(event):
>  dist_sq = (event.x - circle['x']) ** 2 + (event.y - circle['y']) ** 2
> 
>  if dist_sq <= circle['radius'] ** 2:
>  print('Clicked inside circle')
> 
> root = tk.Tk()
> 
> canvas = tk.Canvas(root, width=400, height=200)
> canvas.pack()
> 
> circle = dict(x=60, y=60, radius=20)
> 
> left = circle['x'] - circle['radius']
> top = circle['y'] - circle['radius']
> right = circle['x'] + circle['radius']
> bottom = circle['y'] + circle['radius']
> 
> canvas.create_oval((left, top, right, bottom), outline='red', fill='red')
> canvas.bind('', mouse_clicked)
> 
> root.mainloop()

I didn't try your code, but you can simply it by using some of the core 
functionality provided via "canvas.tag_bind(...)"

## START CODE ##
import Tkinter as tk

def cb_canvasButton(event):
print 'You clicked me using button {0}'.format(event.num)
canvas.move('button', 10, 10) # Little extra surprise!

root = tk.Tk()
canvas = tk.Canvas(root)
canvas.create_oval(10,10,50,50, outline='red', fill='blue',
   tags=('button',)
   )
canvas.tag_bind('button', '', cb_canvasButton)
canvas.pack()

root.mainloop() 
## END CODE ##
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Is it possible to draw a BUTTON?

2016-07-27 Thread huey . y . jiang
On Wednesday, July 27, 2016 at 4:18:29 PM UTC-4, huey.y...@gmail.com wrote:
> Hi Folks,
> 
> It is common to put a BUTTON on a canvas by the means of coding. However, in 
> my application, I need to draw a circle on canvas, and then make this circle 
> to work as if it is a button. When the circle is clicked, it triggers a new 
> image to be displayed. Somebody can help? Thanks!

The example works. It is smart to bind a button to a widget, I'v learned. 
Thanks so much!
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Larry Martell
On Wed, Jul 27, 2016 at 7:45 PM, Jason Benjamin  wrote:
> Look at this: https://wiki.python.org/moin/UsingPickle
> It uses *pickle.dump* not *pickle.dumps*

Yes that uses a file. I do not want to use a file. I want to pass the
object as a string.

> If you still don't get it send me the code for the function with a working
> code example the calls that function revealing the error.
>
> I've had a lot of experience with Python and it looks like its the sequence
> of functions that causes the problem and not just one Python function call
> on its own.

I have an object of type Target:

(Pdb) type(target)


And I pickle it like this:

(Pdb) type(pickle.dumps(target))


And then it looks like this:

(Pdb) pickle.dumps(target)
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

And I pass it into a subprocess.Popen call. Then in the program that
is called, it comes in as a string:

(Pdb) type(args.target)


With the same content:

(Pdb) args.target
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

And when I unpickle it I get the error:

(Pdb) pickle.loads(args.target)
***ImportError: Import by filename is not supported.


Also, please keep the discussion on the list, and please don't top post.

>
>
>
> On Wed, Jul 27, 2016 at 4:29 PM, Larry Martell 
> wrote:
>
> On Wed, Jul 27, 2016 at 6:54 PM, Jason Benjamin 
> wrote:
>
> If it has and 's' on the end it will only work on strings. *dumps* refers to
> a string too.
>
> Yes, I know. I have an object, which I pickle with dumps, which turns it
> into a string. Then I try to unpickle it with loads and I get that error.
> (Pdb) type(args.target)  (Pdb) pickle.loads(args.target) ***
> ImportError: Import by filename is not supported.
>
> On Wed, Jul 27, 2016 at 3:44 PM, Larry Martell 
> wrote: On Wednesday, July 27, 2016, Jason Benjamin 
> wrote:
>
> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote: > When I try and
> unpickle an object with pickle.loads it fails with: > > ImportError: Import
> by filename is not supported when unpickleing > > I've never used pickle
> before. Why do I get this and how can I fix it? Try using *pickle.load*
> instead of *pickle.loads*. pickle.loads is for strings. Retrieved from
> documentation: help(pickle) after importing pickle.
>
> I am unpickling a string created with pickle.dumps
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Larry Martell
On Wed, Jul 27, 2016 at 10:39 PM, Larry Martell  wrote:
> On Wed, Jul 27, 2016 at 7:45 PM, Jason Benjamin  wrote:
>> Look at this: https://wiki.python.org/moin/UsingPickle
>> It uses *pickle.dump* not *pickle.dumps*
>
> Yes that uses a file. I do not want to use a file. I want to pass the
> object as a string.
>
>> If you still don't get it send me the code for the function with a working
>> code example the calls that function revealing the error.
>>
>> I've had a lot of experience with Python and it looks like its the sequence
>> of functions that causes the problem and not just one Python function call
>> on its own.
>
> I have an object of type Target:
>
> (Pdb) type(target)
> 
>
> And I pickle it like this:
>
> (Pdb) type(pickle.dumps(target))
> 
>
> And then it looks like this:
>
> (Pdb) pickle.dumps(target)
> "ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."
>
> And I pass it into a subprocess.Popen call. Then in the program that
> is called, it comes in as a string:
>
> (Pdb) type(args.target)
> 
>
> With the same content:
>
> (Pdb) args.target
> "ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."
>
> And when I unpickle it I get the error:
>
> (Pdb) pickle.loads(args.target)
> ***ImportError: Import by filename is not supported.
>
>
> Also, please keep the discussion on the list, and please don't top post.


Also let me add that initially I was calling Popen with shell=False
and the arguments in a list, and that was failing with:

arg 2 must contain only strings

And when I debugged I found that this was happening in the execvp call
in subprocess and arg 2 was a str. So then I changed it using
shell=False with the command line in a string, and that is when I get
the  ImportError error.

>> On Wed, Jul 27, 2016 at 4:29 PM, Larry Martell 
>> wrote:
>>
>> On Wed, Jul 27, 2016 at 6:54 PM, Jason Benjamin 
>> wrote:
>>
>> If it has and 's' on the end it will only work on strings. *dumps* refers to
>> a string too.
>>
>> Yes, I know. I have an object, which I pickle with dumps, which turns it
>> into a string. Then I try to unpickle it with loads and I get that error.
>> (Pdb) type(args.target)  (Pdb) pickle.loads(args.target) ***
>> ImportError: Import by filename is not supported.
>>
>> On Wed, Jul 27, 2016 at 3:44 PM, Larry Martell 
>> wrote: On Wednesday, July 27, 2016, Jason Benjamin 
>> wrote:
>>
>> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote: > When I try and
>> unpickle an object with pickle.loads it fails with: > > ImportError: Import
>> by filename is not supported when unpickleing > > I've never used pickle
>> before. Why do I get this and how can I fix it? Try using *pickle.load*
>> instead of *pickle.loads*. pickle.loads is for strings. Retrieved from
>> documentation: help(pickle) after importing pickle.
>>
>> I am unpickling a string created with pickle.dumps
-- 
https://mail.python.org/mailman/listinfo/python-list


Python text file fetch specific part of line

2016-07-27 Thread Arshpreet Singh
I am writing Imdb scrapper, and getting available list of titles from IMDB 
website which provide txt file in very raw format, Here is the one part of 
file(http://pastebin.com/fpMgBAjc) as the file provides tags like Distribution  
Votes,Rank,Title I want to parse title names, I tried with readlines() method 
but it returns only list which is quite heterogeneous, is it possible that I 
can parse each value comes under title section?
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: Behavior of tempfile temp files when scripts killed, interpreter crashes, server crashes?

2016-07-27 Thread eryk sun
On Wed, Jul 27, 2016 at 1:58 PM, Malcolm Greene  wrote:
> Can someone share their OS specific experience in working with tempfile
> generated temp files under these conditions?
>
> 1. Script killed by another process
> 2. Interpreter crashes
> 3. Server crashes (sudden loss of power)
> 4. Other application termination conditions ???
>
> I'm curious which scenarios result in temp files not being automatically
> deleted after use and what technique you're using to cleanup temp files
> left over after these conditions (without affecting legitimate temp
> files present from the current session)?
>
> Do any OS's support a type of temp file that truly gets automatically
> deleted in all of the above scenarios?

Linux has the O_TMPFILE open() flag [1]. This creates an anonymous
file that gets automatically deleted when the last open file
descriptor is closed. If the file isn't opened O_EXCL, then you can
make it permanent by linking it back into the filesystem. For example:

>>> fd = os.open('/tmp', os.O_TMPFILE | os.O_RDWR, 0o600)
>>> tmp_dir_fd = os.open('/tmp', os.O_DIRECTORY)
>>> os.write(fd, b'spam')
4
>>> os.link('/proc/self/fd/%d' % fd, 'tempfile', dst_dir_fd=tmp_dir_fd)
>>> os.close(fd); os.close(tmp_dir_fd)
>>> open('/tmp/tempfile').read()
'spam'

tempfile.TemporaryFile uses the O_TMPFILE flag when it's available.
Since it also uses O_EXCL, the files it creates cannot be made
permanent via os.link.

[1]: http://man7.org/linux/man-pages/man2/open.2.html

Windows CreateFile [2] has two related flags:
FILE_FLAG_DELETE_ON_CLOSE to ensure that a temporary file gets deleted
and FILE_ATTRIBUTE_TEMPORARY, which avoids writing the file to disk as
long as there's enough available cache memory. The Windows C runtime
and Python make these available as O_TEMPORARY and O_SHORT_LIVED,
respectively [3].

tempfile.NamedTemporaryFile uses O_TEMPORARY. tempfile.TemporaryFile
is an alias for NamedTemporary, since files can't be anonymous on
Windows.

You can delete the temporary file, because it's opened with delete
sharing, but it won't actually get unlinked from the directory until
the last handle is closed. The only benefit to deleting the file is
getting exclusive access, but you can already control that in Windows
via the dwShareMode parameter of CreateFile or the shflag parameter of
_wsopen_s. Unfortunately Python doesn't provide a high-level API for
setting the sharing mode, so if you need to set it you're stuck with
using _winapi.CreateFile and msvcrt.open_osfhandle, PyWin32, or
ctypes.

It's possible to work around the delete-on-close flag by opening the
file a 2nd time, closing the original handle, and then calling
SetFileInformationByHandle [4] to unset the file's delete disposition.

[2]: https://msdn.microsoft.com/en-us/library/aa363858
[3]: https://msdn.microsoft.com/en-us/library/w64k0ytk
[4]: https://msdn.microsoft.com/en-us/library/aa365539
-- 
https://mail.python.org/mailman/listinfo/python-list


Re: ImportError: Import by filename is not supported when unpickleing

2016-07-27 Thread Steven D'Aprano
On Thursday 28 July 2016 12:39, Larry Martell wrote:

> I have an object of type Target:
> 
> (Pdb) type(target)
> 
> 
> And I pickle it like this:
> 
> (Pdb) type(pickle.dumps(target))
> 
> 
> And then it looks like this:
> 
> (Pdb) pickle.dumps(target)
> 
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-
CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-
CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

Gag me with a spoon!

What happens if you do 

s = pickle.dumps(target)
pickle.loads(s)

Do you get the same error?


> And I pass it into a subprocess.Popen call. Then in the program that
> is called, it comes in as a string:
> 
> (Pdb) type(args.target)
> 
> 
> With the same content:
> 
> (Pdb) args.target
> 
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-
CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-
CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

Are you sure it's the same content, and not just the same to the naked eye?

> And when I unpickle it I get the error:
> 
> (Pdb) pickle.loads(args.target)
> ***ImportError: Import by filename is not supported.

Does the target process already have workitem imported? What happens if you 
import workitem first?




By the way, just in case this is relevant to you... pickle is an insecure 
format. If your target process is running in a context where it can receive 
pickles from untrusted clients, you're vulnerable to them running arbitrary 
code on your machine.

If you are trying to do a remote procedure call, rather than invent your own, 
you should use a reliable, trusted library. I've used both rpyc and Pyro and 
can recommend them both:

https://pypi.python.org/pypi/rpyc/
https://pypi.python.org/pypi/Pyro4/


-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list