On 2019/09/29 23:36, Johan Corveleyn wrote:
On Sun, Sep 29, 2019 at 2:00 PM Yasuhito FUTATSUKI <futat...@poem.co.jp> wrote:
On 2019/09/28 17:17, Yasuhito FUTATSUKI wrote:
On 2019/09/28 5:51, Johan Corveleyn wrote:
[[[
Testing Release configuration on local repository.
-- Running Swig Python tests --
..........................................................C:\Python37\lib\subprocess.py:858:
ResourceWarning: subprocess
8548 is still running
ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
C:\Python37\lib\unittest\case.py:628: ResourceWarning: unclosed file
<_io.BufferedReader name=3>
testMethod()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.C:\Python37\lib\unittest\case.py:628: ResourceWarning: unclosed file
<_io.BufferedReader name='R:\\temp\\tmpryeb61g1'>
testMethod()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
..............................................................................................
----------------------------------------------------------------------
Ran 153 tests in 66.182s
OK
]]]
On FreeBSD those warning are also produced.
[[[
futatuki@retina-alpha[15] make check-swig-py
if [ "LD_LIBRARY_PATH" = "DYLD_LIBRARY_PATH" ]; then for d in
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/../../../libsvn_*; do if [ -n
"$DYLD_LIBRARY_PATH" ]; then LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$d/.libs"; else
LD_LIBRARY_PATH="$d/.libs"; fi; done; export LD_LIBRARY_PATH; fi; cd
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python; /usr/local/bin/python3.7
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/tests/run_all.py
........................................................../usr/local/lib/python3.7/subprocess.py:858:
ResourceWarning: subprocess 81136 is still running
ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.7/unittest/case.py:615: ResourceWarning: unclosed file
<_io.BufferedReader name=4>
testMethod()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
./usr/local/lib/python3.7/unittest/case.py:615: ResourceWarning: unclosed file
<_io.BufferedReader name='/tmp/tmporh5038d'>
testMethod()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
..............................................................................................
----------------------------------------------------------------------
Ran 153 tests in 10.773s
OK
]]]
I think these warnings comes from difference of warning between Python 2
and Python 3, and not affect quality of swig Python binding itself but
affect quality of its test code.
This was partly wrong. The warning caused by subprocess comes from
class svn.fs.FileDiff.get_pipe(), not in test code but in support class.
ResourceWarning itself appeared in Python 3.2
https://docs.python.org/3/whatsnew/3.2.html
(section "Other Language Changes")
If we can allow this warning and want to hide it, we can use
environment variable PYTHONWARNINGS='ignore::ResourceWarning::',
as the document above says.
[[[
futatuki@retina-alpha[17] env PYTHONWARNINGS='ignore::ResourceWarning::' make
check-swig-py
if [ "LD_LIBRARY_PATH" = "DYLD_LIBRARY_PATH" ]; then for d in
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/../../../libsvn_*; do if [ -n
"$DYLD_LIBRARY_PATH" ]; then LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$d/.libs"; else
LD_LIBRARY_PATH="$d/.libs"; fi; done; export LD_LIBRARY_PATH; fi; cd
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python; /usr/local/bin/python3.7
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/tests/run_all.py
.........................................................................................................................................................
----------------------------------------------------------------------
Ran 153 tests in 10.757s
OK
]]]
Of course, we can also resolve those warnings by fixing clean up code
on tests.
The patch attached is expected to fix these warnings. I already tested
on FreeBSD with Python 3.7 (treat ResourceWarning as an error).
[[[
futatuki@retina-alpha[113] env PYTHONWARNINGS='error::ResourceWarning::' make
check-swig-py
if [ "LD_LIBRARY_PATH" = "DYLD_LIBRARY_PATH" ]; then for d in
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/libsvn_swig_py
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/../../../libsvn_*; do if [ -n
"$DYLD_LIBRARY_PATH" ]; then LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$d/.libs"; else
LD_LIBRARY_PATH="$d/.libs"; fi; done; export LD_LIBRARY_PATH; fi; cd
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python; /usr/local/bin/python3.7
/home/futatuki/work/subversion/vwc/branches/swig-py3/subversion/bindings/swig/python/tests/run_all.py
.........................................................................................................................................................
----------------------------------------------------------------------
Ran 153 tests in 11.040s
OK
]]]
On my Windows setup, there is still 1 ResourceWarning (before your
patch there were 3, now only 1 remains).
[[[
C:\research\svn\dev\swig-py3>runtest.bat --swig=python
'ruby' is not recognized as an internal or external command,
operable program or batch file.
Testing Release configuration on local repository.
-- Running Swig Python tests --
..........................................................C:\Python37\lib\subprocess.py:858:
ResourceWarning: subprocess
1408 is still running
ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
...............................................................................................
----------------------------------------------------------------------
Ran 153 tests in 63.597s
OK
]]]
Hmm.. It seems the diffrence comes from difference of implementation
of class subprocess.Popen.
I tweaked again clean up code for class svn.fs.FileDiff. Only if
Python >= 3.3, it wait for maximum 10 sec to terminate subprocess.
'timeout' parameter for subprocess.Popen.wait() is added on Python 3.3.
and not back ported to Python 2, so if we want to use wait() on
earlier Python, it may wait forever.... (It can be possible to implement
timeout by using poll() and sleep loop, but is it needed here?)
When I treat ResourceWarning as an error, I get a lot more output,
especially before the test runs start. I suppose those are unrelated
to that one RW above.
Yes, ... and ResourceWarnings are usually hided by default warning filter.
https://docs.python.org/3/library/warnings.html#default-warning-filter
Only while executing unittest.main(), warning filter is overrided by
its optional parameter `warning'. Its default is specified filter on
Python start up by environment variable PYTHONWARNINGS or -W option
if it is specified, otherwise "default" action for all warning
categories (i.e. show all warnings).
[[[
C:\research\svn\dev\swig-py3>set PYTHONWARNINGS=error::ResourceWarning::
C:\research\svn\dev\swig-py3>runtest.bat --swig=python
Exception ignored in: <_io.FileIO name='build.conf' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='build.conf'
mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr\\include\\apr_version.h'
mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr\\include
\\apr_version.h' mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\include\\apu_versi
on.h' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\in
clude\\apu_version.h' mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\xml/expat/lib\\exp
at.h' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\srclib\\apr-util\\xm
l/expat/lib\\expat.h' mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\zlib-1.2.11\\zlib.h' mode='rb'
closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\zlib-1.2.11\\zlib.h' mode='r' enco
ding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\sqlite-amalgamation-3.22.0.0\\sqlite3.h'
mode='rb'
closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\sqlite-amalgamation-3.22.0.0\\sqli
te3.h' mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='subversion\\libsvn_subr\\lz4\\lz4internal.h' mode='rb'
closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='subversion\\libsvn_subr\\lz4\\lz4internal.h' mode='r' encoding='
cp1252'>
Exception ignored in: <_io.FileIO
name='subversion\\libsvn_subr\\utf8proc\\utf8proc_internal.h'
mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='subversion\\libsvn_subr\\utf8proc\\utf8proc_internal.h' mode='r'
encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\include\\ap_release.h'
mode='rb' clo
sefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\httpd-2.4.29\\include\\ap_release.
h' mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\openssl-1.0.2n\\inc32/openssl/opensslv.h'
mode='rb
' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\openssl-1.0.2n\\inc32/openssl/open
sslv.h' mode='r' encoding='cp1252'>
Exception ignored in: <_io.FileIO
name='C:\\research\\svn\\dev\\deps\\serf-1.3.9\\serf.h' mode='rb'
closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper
name='C:\\research\\svn\\dev\\deps\\serf-1.3.9\\serf.h' mode='r' encod
ing='cp1252'>
Exception ignored in: <function Popen.__del__ at 0x01043420>
Traceback (most recent call last):
File "C:\Python37\lib\subprocess.py", line 858, in __del__
ResourceWarning, source=self)
ResourceWarning: subprocess 3716 is still running
Exception ignored in: <function Popen.__del__ at 0x01043420>
Traceback (most recent call last):
File "C:\Python37\lib\subprocess.py", line 858, in __del__
ResourceWarning, source=self)
ResourceWarning: subprocess 1356 is still running
'ruby' is not recognized as an internal or external command,
operable program or batch file.
Testing Release configuration on local repository.
-- Running Swig Python tests --
..........................................................Exception
ignored in: <function Popen.__del__ at 0x02D4B030>
Traceback (most recent call last):
File "C:\Python37\lib\subprocess.py", line 858, in __del__
ResourceWarning, source=self)
ResourceWarning: subprocess 2972 is still running
...............................................................................................
----------------------------------------------------------------------
Ran 153 tests in 93.884s
OK
]]]
These ResourceWarning except last one are raised in test driver
win_test.py itself and in gen_win_dependencies module which is
used by win_test.py, I guess.
Cheers,
--
Yasuhito FUTATSUKI <futat...@poem.co.jp>
On branch swig-py3: fix resource warnings on Python 3
* subversion/bindings/swig/python/svn/fs.py
(FileDiff.procs): New list variable to hold process objects
(FileDiff.get_pipe): Hold process object to self.procs.
(FileDiff.__del__): Kill processes if they are alive before delete.
* subversion/bindings/swig/python/tests/fs.py
(SubversionFSTestCase.test_diff_repos_paths_internal,
SubversionFSTestCase.test_diff_repos_paths_external): explicitly close pipe.
Index: subversion/bindings/swig/python/svn/fs.py
===================================================================
--- subversion/bindings/swig/python/svn/fs.py (revision 1867729)
+++ subversion/bindings/swig/python/svn/fs.py (working copy)
@@ -60,6 +60,7 @@
self.tempfile1 = None
self.tempfile2 = None
self.difftemp = None
+ self.procs = []
self.root1 = root1
self.path1 = path1
@@ -124,6 +125,7 @@
# open the pipe, and return the file object for reading from the child.
p = _subprocess.Popen(cmd, stdout=_subprocess.PIPE, bufsize=-1,
close_fds=_sys.platform != "win32")
+ self.procs.append(p)
return p.stdout
else:
@@ -158,3 +160,12 @@
_os.remove(tmpfile)
except OSError:
pass
+ for proc in self.procs:
+ if proc.poll() is None:
+ proc.terminate()
+ if _sys.hexversion >= 0x030300F0:
+ try:
+ proc.wait(10)
+ except _subprocess.TimeoutExired:
+ proc.kill()
+ del proc
Index: subversion/bindings/swig/python/tests/fs.py
===================================================================
--- subversion/bindings/swig/python/tests/fs.py (revision 1867729)
+++ subversion/bindings/swig/python/tests/fs.py (working copy)
@@ -96,6 +96,7 @@
diffp = fdiff.get_pipe()
diffoutput = diffp.read().decode('utf8')
+ diffp.close()
self.assertTrue(diffoutput.find(u'-' + self.unistr) > 0)
@@ -116,6 +117,7 @@
None, None, diffoptions=[])
diffp = fdiff.get_pipe()
diffoutput = diffp.read().decode('utf8')
+ diffp.close()
self.assertTrue(diffoutput.find(u'< ' + self.unistr) > 0)
