Re: AW: AW: AW: PyLucene use JCC shared object by default

[Andi Vajda]

 Hi Thomas,
On Wed, 2 May 2012, Thomas Koch wrote:

could you download the patch from the link?

Yes, I got your patch just fine.

I fixed a few bugs today having to do with converting sequences to JArray
and added support for auto-boxing primitive types when converting a sequence
to an object JArray. Now your collections-demo.py all works fine !

With these fixes the Python toArray() methods can return a Python sequence 
object directly, there is no need to do the JArray conversion in Python 
anymore.

I simplified the collections.py file a bit to reflect the fixes and all 
changes, including the PythonList/PythonListIterator code is now checked in.

Could you please convert collections-demo.py into a proper unit test module 
like the unit tests in pylucene/test so that it gets integrated into the 
test suite ?

Thanks !

Andi..


Just one more thing ... in the initial implementation of PythonList I did the 
toArray() method in Python and the toArray(Object[]) method in Java - just as 
was done for the PythonSet:

+    public native List subList(int fromIndex, int toIndex);
+    public native Object[] toArray();
+
+    public Object[] toArray(Object[] a)
+    {
+        Object[] array = toArray();
+
+        if (a.length < array.length)
+            a = (Object[]) Array.newInstance(a.getClass().getComponentType(),
+                                             array.length);
+
+        System.arraycopy(array, 0, a, 0, array.length);
+
+        return a;
+    }

(from patch of Feb 22nd I sent to you)

From the current patch you can see that the latter part is missing now - the 
toArray(Object[]) method is now done in Python as well, i.e. it simply calls 
toArray():

===================================================================
--- java/org/apache/pylucene/util/PythonSet.java        (revision 1332162)
+++ java/org/apache/pylucene/util/PythonSet.java        (working copy)
@@ -62,14 +62,6 @@

    public Object[] toArray(Object[] a)
    {
-        Object[] array = toArray();
-
-        if (a.length < array.length)
-            a = (Object[]) Array.newInstance(a.getClass().getComponentType(),
-                                             array.length);
-
-        System.arraycopy(array, 0, a, 0, array.length);
-
-        return a;
+        return toArray();
    }
}

As far as I remember that was part of your changes in between (I probably never 
touched PythonSet). Anyway I could imagine that this is related to the current 
problem.

However, the ArrayList never calls the 2nd toArray method. The ArrayList constructor 
actually triggers the "simple" toArray method:

   /**
    * Constructs a list containing the elements of the specified
    * collection, in the order they are returned by the collection's
    * iterator.
    *
    * @param c the collection whose elements are to be placed into this list
    * @throws NullPointerException if the specified collection is null
    */
   public ArrayList(Collection<? extends E> c) {
       elementData = c.toArray();
       size = elementData.length;
       // c.toArray might (incorrectly) not return Object[] (see 6260652)
       if (elementData.getClass() != Object[].class)
           elementData = Arrays.copyOf(elementData, size, Object[].class);
   }

The ArrayList source code is attached (from OpenJDK6 sources).

So maybe that's the wrong path ...  Anyhow I feel the mapping of 
toArray(Object[]) to toArray() does not fully comply with the Java API 
description:
http://docs.oracle.com/javase/6/docs/api/java/util/List.html#toArray(T[])


Hope that helps...


Regards,
Thomas

-----Ursprüngliche Nachricht-----
Von: Andi Vajda [mailto:va...@apache.org]
Gesendet: Montag, 30. April 2012 19:32
An: pylucene-dev@lucene.apache.org
Betreff: Re: AW: AW: PyLucene use JCC shared object by default


On Mon, 30 Apr 2012, Thomas Koch wrote:

Dear Andi, I again had a look at the patch I submitted recently and
would like to get back to it.  An updated version of the patch is
attached to this email - the patch is against the branch_3x repo
http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x

Oh, and there is no attachment in your email. Maybe it got eaten up by some
mail server. Please, make sure it's of a text mimetype or mail it to me
directly.

Thanks !

Andi..


The patch mainly
- adds two java classes:  PythonList,  PythonListIterator
- adds according Python classes   (JavaListIterator and JavaList in
collections.py)

Purpose:
- provide a Java-based List implementation in JCC/PyLucene (similar to
existing PythonSet/JavaSet)
- allow to pass python lists via Java Collections into PyLucene

Let's try summarize shortly: PythonSet /JavaSet was already existing, but
nothing similar for Lists. I made an implementation of PythonList /JavaList
and with your help this is now basically working. Except of an open issue that
affects both JavaSet and JavaList: initialization of an ArrayList with a JavaSet
(or JavaList) may cause trouble.

As you said: "There is a bug somewhere with constructing an ArrayList from
a python collection like JavaSet or JavaList."

I tried to change the toArray() method as you suggested, but that didn't
help. As far as I understood, there are two options to box python values into
a typed JArray:

1)  use the object based JArray class and box python values by wrapping
them with the corresponding Java object (e.g. type<int> -> lucene.Integer):

x =
lucene.JArray('object')([lucene.Boolean(True),lucene.Boolean(False)
])
JArray<object>[<Object: true>, <Object: false>]
type(x[0])
<type 'Object'>

2)  use the correct array type (int, float, etc.) and pass the list of Python
elements or literals) to the JArray constructur, e.g.

y = lucene.JArray('bool')([True,False])
JArray<bool>[True, False]
type(y[0])
<type 'bool'>

I tried both of them (see _pyList2JArray methods in collections.py) but
none of them did the trick. Actually the 'empty objects in ArrayList' problem
remains when handling with strings (the ArrayList object that is initialized
with a JavaSet or JavaList of string items will have a number of objects as the
original JavaSet/JavaList, but all objects are the same - ooks like an array of
empty objects). Furthermore another issue with integer lists comes into play:
here the initialization of  ArrayList with the Collection fails with a Java
stacktrace (lucene.JavaError: org.apache.jcc.PythonException).

The most simple test case is as follows:

--%< --
import lucene
lucene.initVM()
from lucene.collections import JavaList

# using strings: the ArrayList is created, but initialized with empty
objects jl = JavaList(['a','b']) al = lucene.ArrayList(jl) assert (not
al.get(0).equals(al.get(1))), "unique values"

# using ints: the ArrayList is not created,  but an error occurs instead:
# Java stacktrace: org.apache.jcc.PythonException: ('while calling
toArray') jl = JavaList(range(3)) al = lucene.ArrayList(jl) --%< --

I currently feel like having to stab around in the dark to find out
what's going on here and would welcome any suggestions. Needs some
JCC
expert I guess ,-)

Of course we can leave the patch out - but still there's the same issue with
JavaSet.


kind regards

Thomas
--
OrbiTeam Software GmbH & Co. KG, Germany http://www.orbiteam.de


-----Ursprüngliche Nachricht-----
Von: Andi Vajda [mailto:va...@apache.org]
Gesendet: Mittwoch, 18. April 2012 20:37
An: pylucene-dev@lucene.apache.org
Betreff: Re: AW: PyLucene use JCC shared object by default


Hi Thomas,
...
Lucene 3.6 just got released a few days ago. Apart from your patch, the
PyLucene 3.6 release is ready. I'm about to go offline (email only) for a
week.
Let's revisit this patch then (first week of May). It's not blocking the
release
right now as, even if I sent out a release candidate for a vote, the three
business days required for this would take this into the time I'm away.
...
Andi..