[issue6715] xz compressor support
Dan Stromberg added the comment: On Sun, Oct 2, 2011 at 3:49 PM, Nadeem Vawda wrote: > > Nadeem Vawda added the comment: > > Thanks for investigating the Windows situation. > > > - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly > > variables declared in the middle of a block. It's doable for sure, but > it's a > > lot of work. > > I don't think that creating our own MSVC-friendly fork of liblzma is really > an > option. Over and above the work of porting it in the first place (and all > the > opportunities for bugs to creep in along the way), we'd also have to worry > about > keeping up to date with upstream changes. I believe we currently do > something > similar with libffi (for ctypes), and the impression I've gotten is that > it's > caused a lot of trouble. It's much better to contribute patches upstream. > > - The way recommended by XZ is to use a precompiled liblzma.dll; Then it > > should be easy to build an extension module, but its would be the first > time > > that we distribute an extension module which needs a non-system DLL. Is > it > > enough to copy it next to _lzma.pyd? Is there some work to do in the > > installer? > > I would guess that this is sufficient, but my knowledge of how Windows DLLs > work > is minimal. Could someone with more platform knowledge weigh in on whether > this > would work (and if there are any problems it might cause)? I've not done much with windows dll's, but I've heard they're pretty similar to AIX shared libraries which I've done some work with. AIX shared libraries don't deal with versioning well - if you have two version of the same library on a system, you have to pop them into two different loader domains, or suffer unresolved externals at runtime. Then your applications are in of the two loader domains, and if they're in the wrong one, you again suffer unresolved externals at runtime. -- Added file: http://bugs.python.org/file23300/unnamed ___ Python tracker <http://bugs.python.org/issue6715> ___On Sun, Oct 2, 2011 at 3:49 PM, Nadeem Vawda <mailto:rep...@bugs.python.org";>rep...@bugs.python.org> wrote: Nadeem Vawda <mailto:nadeem.va...@gmail.com";>nadeem.va...@gmail.com> added the comment: Thanks for investigating the Windows situation. > - liblzma can't be compiled by Visual Studio: too many C99 isms, mostly > variables declared in the middle of a block.  It's doable for sure, but it's a > lot of work. I don't think that creating our own MSVC-friendly fork of liblzma is really an option. Over and above the work of porting it in the first place (and all the opportunities for bugs to creep in along the way), we'd also have to worry about keeping up to date with upstream changes. I believe we currently do something similar with libffi (for ctypes), and the impression I've gotten is that it's caused a lot of trouble.It's much better to contribute patches upstream. > - The way recommended by XZ is to use a precompiled liblzma.dll; Then it > should be easy to build an extension module, but its would be the first time > that we distribute an extension module which needs a non-system DLL.  Is it > enough to copy it next to _lzma.pyd?  Is there some work to do in the > installer? I would guess that this is sufficient, but my knowledge of how Windows DLLs work is minimal. Could someone with more platform knowledge weigh in on whether this would work (and if there are any problems it might cause)?I've not done much with windows dll's, but I've heard they're pretty similar to AIX shared libraries which I've done some work with.  AIX shared libraries don't deal with versioning well - if you have two version of the same library on a system, you have to pop them into two different loader domains, or suffer unresolved externals at runtime.  Then your applications are in of the two loader domains, and if they're in the wrong one, you again suffer unresolved externals at runtime.  -- Dan Stromberg ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11449] tarfile tries to file_.tell() even when creating a new archive
Dan Stromberg added the comment: Yes, I just needed to use 'w|' instead of 'w'. Thanks! -- resolution: -> works for me status: open -> closed ___ Python tracker <http://bugs.python.org/issue11449> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10369] tarfile requires an actual file on disc; a file-like object is insufficient
New submission from Dan Stromberg :
The tarfile module's gettarinfo callable insists on stat'ing the file in
question, preventing one from dynamically generating file content by passing a
file-like object for addfile's fileobj argument.
I believe the attached patch fixes this issue. I generated the patch against
2.7 and tested it with 2.7, but it applies cleanly against 3.1 and "feels
innocuous". I've also included my test code at the bottom of this comment.
Why would you want to do this? Imagine you've stored a file in three smaller
files (perhaps to save the pieces on small external media, or as part of a
deduplication system), with the content divided up into thirds. To
subsequently put this file as a whole into a tar archive, it'd be nice if you
could just create a file-like object to emit the catenation, rather than having
to create a temporary file holding that catenation.
It's occurred to me that this should be done in a more object oriented style,
but that feels a bit inconsistent given that fstat is in the os module, and not
provided as an attribute of a file(-like) object. Comments?
Here's the test code:
#!/usr/local/cpython-2.7/bin/python
import os
import sys
import copy
import array
import stat_tarfile
def my_stat(filename):
class mutable_stat:
pass
readonly_statobj = os.lstat(filename)
mutable_statobj = mutable_stat()
for attribute in dir(readonly_statobj):
if not attribute.startswith('_'):
value = getattr(readonly_statobj, attribute)
setattr(mutable_statobj, attribute, value)
return mutable_statobj
class generate_file_content:
def __init__(self, number):
self._multiplier = 100
self._multipleno = 0
self._number = str(number)
self._buffer = ''
def read(self, length):
while self._multipleno < self._multiplier and len(self._buffer)
< length:
self._buffer += self._number
self._multipleno += 1
if self._buffer == '':
return ''
else:
result = self._buffer[:length]
self._buffer = self._buffer[length:]
return result
def main():
with stat_tarfile.open(fileobj = sys.stdout, mode = "w|") as tar:
for number in xrange(100):
#string = str(number) * 100
fileobj = generate_file_content(number)
statobj = my_stat('/etc/passwd')
statobj.st_size = len(str(number)) * 100
filename = 'file-%d.txt' % number
tarinfo = tar.gettarinfo(filename, statobj = statobj)
tarinfo.uid = 1000
tarinfo.gid = 1000
tarinfo.uname = "dstromberg"
tarinfo.gname = "dstromberg"
tar.addfile(tarinfo, fileobj)
main()
--
components: Library (Lib)
files: tarfile.diff
keywords: patch
messages: 120822
nosy: strombrg
priority: normal
severity: normal
status: open
title: tarfile requires an actual file on disc; a file-like object is
insufficient
versions: Python 2.7, Python 3.1
Added file: http://bugs.python.org/file19549/tarfile.diff
___
Python tracker
<http://bugs.python.org/issue10369>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11449] tarfile tries to file_.tell() even when creating a new archive
New submission from Dan Stromberg : The attached file demonstrates the problem in 2.5, 2.5, 2.7, 3.0, 3.1 and 3.2. In short, I believe when you're creating a new tar archive (say, to a pipe), there should be no need for a file_.tell() (which blows up when it's a pipe). I have a workaround, as demonstrated in the file, but probably others shouldn't have to work around this also in the future. Suggested fix: Only do the tell when you're appending or performing other operations that require subsequent seek's. -- components: Extension Modules files: tar-test messages: 130422 nosy: strombrg priority: normal severity: normal status: open title: tarfile tries to file_.tell() even when creating a new archive type: crash versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2 Added file: http://bugs.python.org/file21059/tar-test ___ Python tracker <http://bugs.python.org/issue11449> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Dan Stromberg added the comment: I don't know that much about compression, but I wonder if a threadsafe compression module would enable parallel forms of compression? If yes, then multithreaded might be a big benefit, in light of multicore taking off. EG: http://www.compression.ca/pbzip2/ It might be worthwhile to check with the authors (of bzip2 and pyliblzma modules) about why they went with thread safety. -- ___ Python tracker <http://bugs.python.org/issue6715> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Dan Stromberg added the comment: Interesting thing to consider: maybe it'd be better to add support for libarchive, which includes xz support among other things. http://code.google.com/p/libarchive/ -- ___ Python tracker <http://bugs.python.org/issue6715> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Dan Stromberg added the comment: While I realize Python extension modules are pretty much the norm in CPython, it'd be pretty cool if xz support could be written overtop of ctypes. ctypes seems to be emerging as the way of doing FFI across different python implementations. I've been working on a backup engine in my spare time, and I'd very much like to use a Python xz module in it. However, the code I have so far runs about 4x faster on pypy than CPython, even if I turn on psyco in CPython. IOW, I'd be pretty pleased to see an xz module that works equally well on CPython, pypy (seems to have pretty good ctypes), jython (has the beginnings of ctypes in 2.5.2rc*), iron python (supposedly included recently)... -- nosy: +strombrg ___ Python tracker <http://bugs.python.org/issue6715> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6715] xz compressor support
Dan Stromberg added the comment: I agree that ctypes is a bit more brittle - both ctypes and c extension modules can yield segfaults, but at least the c extension module is likely to give an error or warning when you rebuild it. However, I'm getting the impression that: 1) In pypy, the C extension module layer isn't a complete emulation 2) In pypy, the C extension module is a bit disfavored compared to ctypes 3) In pypy, there's a performance expense for CPyExt compared to ctypes 3) (less important) In pypy, the C extension module layer is still too young to do much with at this time I've been wondering if maybe there should be a way to check ctypes use against .h's at build time. That way, the brittleness should be about the same. -- ___ Python tracker <http://bugs.python.org/issue6715> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44611] CPython uses deprecated randomness API
New submission from Dan Stromberg : CPython 3.9 uses CryptGenRandom(), which has been deprecated by Microsoft. I'm told the randomness produced by CryptGenRandom() is fine, but Microsoft has introduced a newer API for getting randomness. For these reasons, Python/bootstrap_hash.c should be updated to use https://docs.microsoft.com/en-us/windows/win32/seccng/cng-por , but it is not urgent, and is not needed in older versions of CPython. Also the documentation that references CryptGenRandom() should be updated, EG: https://docs.python.org/3/library/os.html#os.urandom -- components: Windows messages: 397339 nosy: paul.moore, steve.dower, strombrg, tim.golden, zach.ware priority: normal severity: normal status: open title: CPython uses deprecated randomness API type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue44611> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44611] CPython uses deprecated randomness API
Dan Stromberg added the comment: Yes, cng-portal. On Mon, Jul 12, 2021 at 3:24 PM Thomas Grainger wrote: > > Thomas Grainger added the comment: > > https://docs.microsoft.com/en-us/windows/win32/seccng/cng-portal ? > > -- > nosy: +graingert > > ___ > Python tracker > <https://bugs.python.org/issue44611> > _______ > -- Dan Stromberg | Senior Software Engineer Mobile +1.949.342.6502 <https://keepersecurity.com/> ** This email is confidential and is intended for the recipient(s) addressed herein ** -- ___ Python tracker <https://bugs.python.org/issue44611> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5672] Implement a way to change the python process name
Dan Stromberg added the comment: Isn't this "python script doesn't show up in top correctly" issue a matter of "#!/usr/bin/env python3"? If you "#!/usr/bin/python3" (for example) instead, top seems happy. -- nosy: +strombrg ___ Python tracker <https://bugs.python.org/issue5672> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15039] module/ found before module.py when both are in the CWD
New submission from Dan Stromberg : CPython 3.3a4 appears to find treap/ before treap.py in the CWD. If I rename treap to treap-dir, all seems well for the code in question: dstromberg@zareason-limbo6000a /tmp/tt $ mv treap treap-dir dstromberg@zareason-limbo6000a /tmp/tt $ /usr/local/cpython-3.3/bin/python -c 'import sys; print(sys.path); import treap; t = treap.treap()' ['', '/usr/local/cpython-3.3/lib/python33.zip', '/usr/local/cpython-3.3/lib/python3.3', '/usr/local/cpython-3.3/lib/python3.3/plat-linux', '/usr/local/cpython-3.3/lib/python3.3/lib-dynload', '/usr/local/cpython-3.3/lib/python3.3/site-packages'] dstromberg@zareason-limbo6000a /tmp/tt $ mv treap-dir/ treap dstromberg@zareason-limbo6000a /tmp/tt $ /usr/local/cpython-3.3/bin/python -c 'import sys; print(sys.path); import treap; t = treap.treap()' ['', '/usr/local/cpython-3.3/lib/python33.zip', '/usr/local/cpython-3.3/lib/python3.3', '/usr/local/cpython-3.3/lib/python3.3/plat-linux', '/usr/local/cpython-3.3/lib/python3.3/lib-dynload', '/usr/local/cpython-3.3/lib/python3.3/site-packages'] Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'treap' dstromberg@zareason-limbo6000a /tmp/tt $ ls -l treap/__init__.py ls: cannot access treap/__init__.py: No such file or directory dstromberg@zareason-limbo6000a /tmp/tt $ /usr/local/cpython-3.3/bin/python Python 3.3.0a4 (default, Jun 8 2012, 14:14:41) [GCC 4.6.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> dstromberg@zareason-limbo6000a /tmp/tt $ -- components: Library (Lib) messages: 162552 nosy: strombrg priority: normal severity: normal status: open title: module/ found before module.py when both are in the CWD type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue15039> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23397] PEP 431 implementation
Changes by Dan Stromberg : -- nosy: +strombrg ___ Python tracker <http://bugs.python.org/issue23397> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
