Re: AW: AW: [VOTE] Release PyLucene 4.2.1-1

[Andi Vajda]

Thank you, Thomas, for debugging this.
I'm going to add PREFIX to the built-in JCC reserved word list.
More comments inline...

On Fri, 19 Apr 2013, Thomas Koch wrote:

The PREFIX seems to be the only issue - using:
JCC=$(PYTHON) -m jcc --reserved PREFIX --shared --find-jvm-dll
'make' and 'make install' did pass.
'make test' yields the usual windows specific issues (due to some file
locking issues) - there are a total of 45 errors in "make test", as said
most due to windows not being able to remove a locked resource, e.g.

 File "C:\Python27\lib\shutil.py", line 247, in rmtree
   os.remove(fullname)
WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
sie von einem anderen Prozess verwendet wird: 'testrepo\\_0.fdt'

Most errors seem to be follow-up issues (test-repo locked) - this also
occurred in PyLucene 3.6.

If anyone is interested in details - the output of 'make test' is available
here:
https://dl.dropboxusercontent.com/u/4384120/pylucene-4.2.1-test-20130419.txt

I debugged this (once again) for test_PyLucene and found that
- there is no write.lock pending on the store (directory 'testrepo')
- the store is closed properly
- there is a windows lock on a  Field Data file that therefore cannot be
deleted: 'testrepo\_0.fdt'
- even timeout-retry on shutil.rmtree does not fix it
- even passing an onerror-handler to shutil.rmtree() that tries to force
remove this file (using os.chmod before) does not help

I'm clueless here. It could be that the JVM actually holds the lock and a
JVM shutdown would help here. I therefore tried

   def setUp(self):
       self.jvm = lucene.initVM()
                ...
and in

   def tearDown(self):
       ...
       self.jvm.detachCurrentThread()

This doesn't close the VM, it just detaches the current thread (and is a 
rather risky operation). If I remember correctly, the closing and reopening 
of a VM in a given is not supported by the JNI interface (or the interface 
exists but it's not implemented). (I could be wrong, of course).

But this did crash my python.exe ,-( Is there any other way to safely 
re-init jcc during one process loop? Here's the test_PyLucene.py with all 
my attempts: 
https://dl.dropboxusercontent.com/u/4384120/test_PyLucene-win-20130419.py

I then ended up in using a 'fresh' test-repository (i.e. new dir name) if
rmtree fails. The test_PyLucene test then passes on windows - the downside
is that you end up with some testrepo.1, testrepo.2 etc. dirs - and of
course an attempt to cleanup these dirs after unittest.main(exit=False) also
fails...

There's a patch against the pylucene_4_2 for test_PyLucene with this "fix":
https://dl.dropboxusercontent.com/u/4384120/pylucene_4_2_test_PyLucene_WinFi
x_20130419.patch

Note: this should not affect linux tests and fixes the tests for windows.
Though the test_PyLucene now passes (with the fix), still 20 tests fail ,
e.g. test_PythonDirectory  and test_FieldEnumeration. Possibly similar
workarounds could be applied here.  As the main problem seems to be mixing
up several lucene-tests in one process call, alternatively one could write a
batch script, that calls all those tests one by one (in a new windows
process) - or even split test_-files into several smaller tests (python
files). Don't know if these are better ideas though.

So in summary: if you add the PREFIX to reserved words it will build and run
on windows.

Thanks again. A trick - on Windows - might be to rename the testrepo dir to 
testrepo.<current datetime> and move to the next test. And clean them all up 
at the end. Or create a testrepos directory tree where each 
testrepo.<currentTime> is going to be created and remove the whole tree at 
the end.

GIven that Lucene 4.3 is about to be released, I think I'm going to drop the 
4.2.1 PyLucene release and move to proposing a PyLucene 4.3 release 
candidate next.

Andi..


Regards,
Thomas



-----Ursprüngliche Nachricht-----
Von: Andi Vajda [mailto:va...@apache.org]
Gesendet: Donnerstag, 18. April 2013 19:09
An: pylucene-dev@lucene.apache.org
Betreff: Re: AW: [VOTE] Release PyLucene 4.2.1-1


On Thu, 18 Apr 2013, Thomas Koch wrote:

Andi,
I now get a different error while compiling __init__.cpp:

org/apache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42
)
:
error C2059: Syntaxfehler: 'Zeichenfolge'
org/apache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42
)
:
error C2238: Unerwartete(s) Token vor ';'

The line complained about is #42

40            static CompiledAutomaton$AUTOMATON_TYPE *NONE;
41            static CompiledAutomaton$AUTOMATON_TYPE *NORMAL;
42            static CompiledAutomaton$AUTOMATON_TYPE *PREFIX;
43            static CompiledAutomaton$AUTOMATON_TYPE *SINGLE;

PREFIX seems to be another reserved word ... I could compile
__init__.cpp
after renaming PREFIX to PREFIX1.

Instead of renaming PREFIX, could you please have JCC do it for you by
adding it to the list of reserved words in the JCC invocation via the
--reserved command line flag ? and rinse and repeat until all such conficts
due to macro definitions are solved ?

Or were you able to complete the build already once PREFIX was renamed ?

I tried to google a list of reserved words used by VS C++ compiler,
but
had
no luck...

These are not reserved words but macro definitions that conflict with the
generated code. If PREFIX is, say, defined to 1, line 42 becomes:

  static CompiledAutomaton$AUTOMATON_TYPE *1;

and that doesn't compile.

There are some predefined macros -but none that match our issues
http://msdn.microsoft.com/en-us/library/b0084kay(v=vs.100).aspx

Andi..




Make output details:

C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo
/Ox
/MD /W3 /GS- /DNDEBUG -DPYTHON -DJCC_VER="2.16" -D_jcc_shared
-D_java_generics -D_dll_lucene=__declspec(dllexport) "-IC:\Program
Files\Java\jdk1.6.0_06/include" "-IC:\Program
Files\Java\jdk1.6.0_06/include/win32" -Ibuild\_lucene
-IC:\Python27\lib\site-packages\jcc-2.16-py2.7-win32.egg\jcc\sources
-IC:\Python27\include -IC:\Python27\PC /Tpbuild\_lucene\__init__.cpp
/Fobuild\temp.win32-2.7\Release\build\_lucene\__init__.obj /EHsc
/D_CRT_SECURE_NO_WARNINGS __init__.cpp

C:\Python27\lib\site-packages\jcc-2.16-py2.7-win32.egg\jcc\sources\JCCEnv.
h(
118) : warning C4251: 'JCCEnv::refs': class 'std::multimap<_Kty,_Ty>'
erfordert eine DLL-Schnittstelle, die von Clients von class 'JCCEnv'
verwendet wird
       with
       [
           _Kty=int,
           _Ty=countedRef
       ]

f:\devel\workspaces\workspace.pylucene\pylucene-4.2.1-1\build\_lucene\org/
ap
ache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42) :
error
C2059: Syntaxfehler: 'Zeichenfolge'

f:\devel\workspaces\workspace.pylucene\pylucene-4.2.1-1\build\_lucene\org/
ap
ache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42) :
error
C2238: Unerwartete(s) Token vor ';'
error: command '"C:\Program Files\Microsoft Visual Studio
9.0\VC\BIN\cl.exe"' failed with exit status 2
make: *** [compile] Error 1


regards,
Thomas

-----Ursprüngliche Nachricht-----
Von: Andi Vajda [mailto:va...@apache.org]
Gesendet: Mittwoch, 17. April 2013 22:11
An: pylucene-dev@lucene.apache.org
Cc: gene...@lucene.apache.org
Betreff: [VOTE] Release PyLucene 4.2.1-1


The PyLucene 4.2.1-0 release candidate had a number of problems
preventing
its release. A PyLucene 4.2.1-1 release candidate is now ready for
review
from:

  http://people.apache.org/~vajda/staging_area/

A list of changes in this release can be seen at:

http://svn.apache.org/repos/asf/lucene/pylucene/branches/pylucene_4_2/CHAN
GE
S

PyLucene 4.2.1 is built with JCC 2.16 included in these release
artifacts:
http://svn.apache.org/repos/asf/lucene/pylucene/trunk/jcc/CHANGES

A list of Lucene Java changes can be seen at:

http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_4_2_1/lucene/C
HA
NGES.txt

Please vote to release these artifacts as PyLucene 4.2.1-1.

Thanks !

Andi..

ps: the KEYS file for PyLucene release signing is at:
http://svn.apache.org/repos/asf/lucene/pylucene/dist/KEYS
http://people.apache.org/~vajda/staging_area/KEYS

pps: here is my +1