Hey,
Andi, thanks for the reply - I've created a github mirror of the pylucene
project for our own use (which I intend to keep synced with your SVN
repository as its official upstream), located at:
https://github.com/lskillen/pylucene
As suggested I have formatted (and attached) a patch of the unfinished code
that we're using for the through-layer exceptions. Alternatively the diff
can be inspected via github by diff'ing between the new
feature-thru-exception branch that I have created and the master branch, as
such:
https://github.com/lskillen/pylucene/compare/feature-thru-exception?expand=1
Although we've run the test suite without issues I realise there may still
be functionality/style/logical issues with the code. I also suspect that
there may not be specific test cases that target regression failures for
exceptions (yet), so confidence isn't high! All comments are welcome and I
realise this will likely require further changes and a repeatable test case
before it is acceptable, but that's fine.
Thanks,
Lee
On 4 July 2014 18:17, Andi Vajda <va...@apache.org> wrote:
>
> On Jul 4, 2014, at 18:33, Lee Skillen <lskil...@vulcanft.com> wrote:
>
> Hi Andi,
>
> First of all, absolutely fantastic work on the JCC project, we love it -
> Second of all, apologies for contacting you out of the blue via email, but
> I wasn't sure of the best place to contact you to ask a few questions
> (technical-level, not support-level). If it isn't acceptable to email
> please let me know and I can move the discussion somewhere else.
>
>
> Yes, please include pylucene-dev@lucene.apache.org (subscription
> required. send mail pylucene-dev-subscribe@ and follow the instructions
> in the reply).
>
> A bit of background first of all :-
>
> We've been integrating JCC for the past two weeks on a sizeable
> Python/Java/C++ project (where Python is a controller/glue layer) having
> migrated from a previous solution to integrate across the languages, and
> for the most part JCC has been a lifesaver.
>
> Technically we've only hit one major issue so far and that has been the
> translation of exceptions through the layers. Our Java application does a
> lot of double-dispatch (visitation) type of execution and we're calling
> outwards to Python to implement the concrete visitation logic.
>
> In the case of an exception being thrown it results in JCC collating a
> traceback and captured exception at every level of scope, resulting in a
> giant JavaError exception which has lost it's original PythonException
> context.
>
> I was actually able to fix this by attempting to capture the full context
> via PyErr_Fetch() when the first python object is thrown and storing it
> within PythonException, and then ensuring that this is carried through each
> layer appropriately - This seems to work very well, although admittedly I
> haven't considered regressive errors or compatibility yet, but just wanted
> to proof-of-concept it first of all then speak with a JCC maintainer.
>
> The net result is being able to catch errors in a pipeline similar to the
> following:
>
> Python
> -> Java
> -> Python
> -> Raise MyException
> <- Throw PythonException (containing MyException)
> <- Inject MyException
> Catch MyException
>
> My main questions are to ask your thoughts about :-
>
> - What are your thoughts about through-layer exceptions and a feature
> improvement like this?
>
>
> I agree that losing information during exception throwing is not good. If
> you've got an improvement in the form of a patch, do not hesitate in
> sending it in.
>
> - JCC is (I think) currently at home with pylucene, are there are plans to
> making it separate?
>
>
> No such plans at the moment.
>
> - Are you happy if I create a public github project and add the patched
> code in there for review?
>
>
> You're welcome to fork the code if you'd like but you don't have to if all
> you want is sending a patch. Your call.
>
> Thank you for the kind words !
>
> Andi..
>
>
> Hope to hear from you!
>
> Cheers,
> Lee
>
>
>
> TL;DR;
>
> We would like to :-
>
> - Improve JCC's through-layer exception support.
> - Establish a github project for JCC alone.
>
> --
> Lee Skillen
>
> Vulcan Financial Technologies
> 1st Floor, 47 Malone Road, Belfast, BT9 6RY
>
> Office: +44 (0)28 95 817888
> Mobile: +44 (0)78 41 425152
> Web: www.vulcanft.com
>
>
--
Lee Skillen
Vulcan Financial Technologies
1st Floor, 47 Malone Road, Belfast, BT9 6RY
Office: +44 (0)28 95 817888
Mobile: +44 (0)78 41 425152
Web: www.vulcanft.com
From 935d6c36fa8a547c1f0161342a2fa8bfe10be735 Mon Sep 17 00:00:00 2001
From: Lee Skillen <lskil...@vulcanft.com>
Date: Tue, 8 Jul 2014 18:14:46 +0100
Subject: [PATCH] Improve support for through-layer exceptions in which an
exception thrown within the PythonVM will be transparently carried through
the JavaVM and can be recaught within Python again.
Organization: Vulcan Financial Technologies
Signed-off-by: Lee Skillen <lskil...@vulcanft.com>
---
jcc/java/org/apache/jcc/PythonException.java | 1 +
jcc/jcc/sources/functions.cpp | 54 ++++++++++++++++++++++------
jcc/jcc/sources/jcc.cpp | 22 ++++++++++++
3 files changed, 66 insertions(+), 11 deletions(-)
diff --git a/jcc/java/org/apache/jcc/PythonException.java b/jcc/java/org/apache/jcc/PythonException.java
index 4eae86b..454240e 100644
--- a/jcc/java/org/apache/jcc/PythonException.java
+++ b/jcc/java/org/apache/jcc/PythonException.java
@@ -19,6 +19,7 @@ package org.apache.jcc;
public class PythonException extends RuntimeException {
public boolean withTrace = true;
protected String message, errorName, traceback;
+ protected Object exc, value, tb;
public PythonException(String message)
{
diff --git a/jcc/jcc/sources/functions.cpp b/jcc/jcc/sources/functions.cpp
index 7a85fdf..6211988 100644
--- a/jcc/jcc/sources/functions.cpp
+++ b/jcc/jcc/sources/functions.cpp
@@ -787,7 +787,7 @@ int _parseArgs(PyObject **args, unsigned int count, char *types, ...)
}
else
break;
- }
+ }
if (last && (arg == Py_None ||
PyString_Check(arg) || PyUnicode_Check(arg)))
@@ -1354,15 +1354,44 @@ PyObject *PyErr_SetArgsError(PyTypeObject *type, char *name, PyObject *args)
PyObject *PyErr_SetJavaError()
{
JNIEnv *vm_env = env->get_vm_env();
- jthrowable throwable = vm_env->ExceptionOccurred();
- PyObject *err;
+ jthrowable obj = vm_env->ExceptionOccurred();
+#if defined(PYTHON)
+ jclass pycls = env->getPythonExceptionClass();
+
+ /*
+ * Support through-layer exceptions by taking the active PythonException and
+ * making the enclosed exception visible to Python again.
+ */
+ if (vm_env->IsSameObject(vm_env->GetObjectClass(obj), pycls))
+ {
+ /* Retrieve the store exception from PythonException */
+ jfieldID exc_fid = vm_env->GetFieldID(pycls, "exc", "Ljava/lang/Object;");
+ jfieldID value_fid = vm_env->GetFieldID(pycls, "value", "Ljava/lang/Object;");
+ jfieldID tb_fid = vm_env->GetFieldID(pycls, "tb", "Ljava/lang/Object;");
+
+ PyObject* exc = (PyObject*) env->longValue(vm_env->GetObjectField(obj, exc_fid));
+ PyObject* value = (PyObject*) env->longValue(vm_env->GetObjectField(obj, value_fid));
+ PyObject* tb = (PyObject*) env->longValue(vm_env->GetObjectField(obj, tb_fid));
+
+ /* Make the exception visible to Python */
+ PyErr_Restore(exc, value, tb);
+ Py_XDECREF(exc);
+ Py_XDECREF(value);
+ Py_XDECREF(tb);
+ return NULL;
+ }
+#endif
+
+ /*
+ * Expose the Java exception to Python as a JavaError.
+ */
+ PyObject *err;
vm_env->ExceptionClear();
- err = t_Throwable::wrap_Object(Throwable(throwable));
+ err = t_Throwable::wrap_Object(Throwable(obj));
PyErr_SetObject(PyExc_JavaError, err);
Py_DECREF(err);
-
return NULL;
}
@@ -1411,17 +1440,20 @@ void throwPythonError(void)
return;
}
+ jclass pycls = env->getPythonExceptionClass();
+
if (exc)
{
- PyObject *name = PyObject_GetAttrString(exc, "__name__");
+ if (!env->get_vm_env()->ExceptionOccurred())
+ {
+ PyObject *name = PyObject_GetAttrString(exc, "__name__");
- env->get_vm_env()->ThrowNew(env->getPythonExceptionClass(),
- PyString_AS_STRING(name));
- Py_DECREF(name);
+ env->get_vm_env()->ThrowNew(pycls, PyString_AS_STRING(name));
+ Py_DECREF(name);
+ }
}
else
- env->get_vm_env()->ThrowNew(env->getPythonExceptionClass(),
- "python error");
+ env->get_vm_env()->ThrowNew(pycls, "python error");
}
void throwTypeError(const char *name, PyObject *object)
diff --git a/jcc/jcc/sources/jcc.cpp b/jcc/jcc/sources/jcc.cpp
index 0062bb5..620a309 100644
--- a/jcc/jcc/sources/jcc.cpp
+++ b/jcc/jcc/sources/jcc.cpp
@@ -761,6 +761,25 @@ static void JNICALL _PythonException_getErrorInfo(JNIEnv *vm_env, jobject self)
jclass jcls = vm_env->GetObjectClass(self);
PyErr_Fetch(&type, &value, &tb);
+ PyErr_NormalizeException(&type, &value, &tb);
+
+ if (value != NULL && tb != NULL)
+ {
+ /* PyErr_NormalizeException does not implicitly set the __traceback__
+ * attribute on the exception value, so it needs to be set explicitly.
+ * See: https://docs.python.org/3/c-api/exceptions.html
+ */
+ PyException_SetTraceback(value, tb);
+ }
+
+ /* Store the exception within the PythonException java class */
+ jfieldID exc_fid = vm_env->GetFieldID(jcls, "exc", "Ljava/lang/Object;");
+ jfieldID value_fid = vm_env->GetFieldID(jcls, "value", "Ljava/lang/Object;");
+ jfieldID tb_fid = vm_env->GetFieldID(jcls, "tb", "Ljava/lang/Object;");
+
+ vm_env->SetObjectField(self, exc_fid, env->boxLong((jlong) type));
+ vm_env->SetObjectField(self, value_fid, env->boxLong((jlong) value));
+ vm_env->SetObjectField(self, tb_fid, env->boxLong((jlong) tb));
errorName = PyObject_GetAttrString(type, "__name__");
if (errorName != NULL)
@@ -790,6 +809,8 @@ static void JNICALL _PythonException_getErrorInfo(JNIEnv *vm_env, jobject self)
}
}
+#if 0
+ // FIXME: Still capture the full traceback?
PyObject *module = NULL, *cls = NULL, *stringIO = NULL, *result = NULL;
PyObject *_stderr = PySys_GetObject("stderr");
if (!_stderr)
@@ -831,6 +852,7 @@ static void JNICALL _PythonException_getErrorInfo(JNIEnv *vm_env, jobject self)
PySys_SetObject("stderr", _stderr);
Py_DECREF(_stderr);
+#endif /* 0 */
return;
--
1.9.0.dirty
