Hi guys I've been trying to reproduce your bug this morning, but I'm not able to. I restored the db, and imported the xml without any issues, I tested on both 2.19 and 2.20, so I will need more information to reproduce.
This was the command I used to import: curl -X POST -d @dxfstream.xml -H "Content-Type: application/xml" -v -u admin:district http://localhost:8080/api/metadata -- Morten On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland <halvda...@gmail.com> wrote: > So. The stack size thing was just a qualified guess derived from the stack > trace. It's pretty obviously not the cause. I'll wait on feedback from > Morten before going any further on this. > On Jun 27, 2015 6:50 PM, "Calle Hedberg" <calle.hedb...@gmail.com> wrote: > >> Halvdan, >> >> I understand from your last email that the root cause of the problem is a >> 2.19 software issue - the meta data importer in 2.19 is not able to handle >> OrgUnit attribute values when present. That makes sense, and it is likely >> that the root cause of this bug / design flaw is a 2.19 change (since the >> same import worked fine in 2.18). Morten and/or Lars will hopefully know >> what change(s) caused this. >> >> With regard to JVM running out of heap space, and in relation to the >> (outdated - very!) chapter 8.2.8 you referenced: >> >> Firstly, with regard to the 150+ instances we are hosting on our data >> centre servers, we do have some constraints at the moment: Those servers >> have 32GB RAM only, and most instances are in reality running on 1-3GB. >> That will only change from next week, when we start moving to new servers >> with (initially) 128GB RAM. That should on average provide 6-8GB per >> instance (varying between 2 and 32GB). >> >> Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64 >> bits) memory allocations on my Windows notebook (32GB RAM) has been a >> nightmare - the people writing up java and tomcat documentation are >> useless, they seem unable to give any quantified advice at all where any >> normal person can find it. (In comparison, chapter 8.2.8 although outdated >> is super-clear). I am currently using the following setting for Java: >> >> JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m >> >> For some reason I/we do not understand, it seems Tomcat 8.23 does not use >> CATALINA_OPTS settings, so I ended up using the "Configure Tomcat" and then >> specify the following under the "Java" tab: >> Java options: >> -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0 >> -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0 >> -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat >> 8.0\endorsed >> -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat >> 8.0\temp >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> -Djava.util.logging.config.file=C:\Program Files\Apache Software >> Foundation\Tomcat 8.0\conf\logging.properties >> -XX:MaxPermSize=4096m >> >> (NOTE: the -XX value here is different from what I currently have in the >> JAVA_OPTS environmental variable - I cannot find any info on which of them >> takes precedence >> >> Also under that same "Java" tab on "Configure Tomcat": >> Initial memory pool: 4096MB >> Maximum memory pool: 8192MB >> Thread stack size: 4096KB (4MB) >> >> Based on your initial advice to increase the stack size, I tried to >> increase it to 8MB (GML import still crashing), 64MB (GML import still >> crashing), and 132MB (GML import went completely haywire). Which is >> understandable, given your subsequent finding that the root cause of the >> problem was not related to heap/stack settings. >> >> So I've set the thread stack size back to 4096KB (4MB) - even if I don't >> know whether that's correct or not (seems high to my layman eyes, but maybe >> DHIS2 require it?) >> >> If anybody actually KNOWS what memory/stack sizes that I optimal for a >> 32GB Windows machine, I would appreciate the advice... >> >> Regards >> Calle >> >> On 26 June 2015 at 20:26, Halvdan Grelland <halvda...@gmail.com> wrote: >> >>> Ok, I could have a look, but am preoccupied at the moment. Will get back >>> to you. >>> >>> In the meantime, it seems that you're facing a stack overflow error, >>> which could be caused by the JVM running out of heap space. Could you >>> please check the allocated heap for tomcat? If it is not being set (in the >>> CATALINA_OPTS env variable), please set it to something quite large and try >>> again. >>> >>> Under chapter 8.2.8 at the following link is a brief guide >>> https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html >>> >>> Please let me know how this works out for you. :) >>> >>> 2015-06-26 11:44 GMT+02:00 Calle Hedberg <calle.hedb...@gmail.com>: >>> >>>> Halvdan, >>>> >>>> Here's a stripped down version of the Free State instance (no records >>>> in the data value table). Restore it as za_fs_s (psql -U dhis >>>> -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis) >>>> >>>> You have got that single-record country GML file already. >>>> >>>> I would appreciate any help in figuring out what's happening. Just note >>>> that I'm using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23. >>>> >>>> Best regards >>>> Calle >>>> >>>> On 26 June 2015 at 11:17, Calle Hedberg <calle.hedb...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I ran it again (revision 19247) and pulled out the first part of the >>>>> log error: >>>>> >>>>> * INFO 2015-06-26 11:04:18,338 User 'Calle_Hedberg' started import at >>>>> Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:18,348 [Level: INFO, category: >>>>> METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing >>>>> meta-data] (InMemoryNotifier.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:18,348 Building object-bridge maps >>>>> (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 >>>>> seconds. (DefaultObjectBridge.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:21,141 [Level: INFO, category: >>>>> METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 >>>>> OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object >>>>> of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object >>>>> of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1]) >>>>> * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object >>>>> of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1]) >>>>> * ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled >>>>> task. (TaskUtils.java [taskScheduler-1]) >>>>> java.lang.StackOverflowError >>>>> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source) >>>>> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source) >>>>> at java.util.regex.Pattern$BitClass.<init>(Unknown Source) >>>>> at java.util.regex.Pattern.clazz(Unknown Source) >>>>> at java.util.regex.Pattern.sequence(Unknown Source) >>>>> at java.util.regex.Pattern.expr(Unknown Source) >>>>> at java.util.regex.Pattern.compile(Unknown Source) >>>>> at java.util.regex.Pattern.<init>(Unknown Source) >>>>> at java.util.regex.Pattern.compile(Unknown Source) >>>>> at java.lang.String.replaceAll(Unknown Source) >>>>> at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134) >>>>> at >>>>> com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95) >>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396) >>>>> at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source) >>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>> at >>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196) >>>>> at >>>>> org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at >>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at java.util.AbstractCollection.toString(Unknown Source) >>>>> at >>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at >>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359) >>>>> >>>>> Any ideas? >>>>> >>>>> Regards >>>>> Calle >>>>> >>>>> On 26 June 2015 at 10:12, Calle Hedberg <calle.hedb...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Firstly, yes we are using a custom revision - although it is SUPPOSED >>>>>> to be based on a 2.19 version from a few days ago. I will check that. >>>>>> >>>>>> BUT I have tried the import using revision 19247 too, and the >>>>>> problem/behaviour is exactly the same - it hangs. Whereas if I import >>>>>> into >>>>>> an older version of the same provincial instance that still runs on 2.18, >>>>>> it imports fine. >>>>>> >>>>>> The fact that the same GML file imports in the Sierra Leone demo - I >>>>>> cannot explain that, and the tomcat log only shows that >>>>>> Attribute.toString() looping... >>>>>> >>>>>> Can I create a stripped down version of the instance and upload it >>>>>> for you to test? >>>>>> >>>>>> Regards >>>>>> Calle >>>>>> >>>>>> >>>>>> >>>>>> On 26 June 2015 at 04:09, Morten Olav Hansen <morte...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> It seems to be looping in Attribute.toString(), I remember this was >>>>>>> an issue at some point.. but that was a long long time ago.. so maybe >>>>>>> you >>>>>>> need a revision bump >>>>>>> >>>>>>> -- >>>>>>> Morten >>>>>>> >>>>>>> On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland < >>>>>>> halvda...@gmail.com> wrote: >>>>>>> >>>>>>>> I tried your example GML on both our demo servers (dev on 2.20 and >>>>>>>> demo on 2.19, same rev as your server) and it works just fine. >>>>>>>> >>>>>>>> If you're experiencing it hanging and not finishing you might be >>>>>>>> experiencing some kind of performance and/or memory issue. The >>>>>>>> coordinate >>>>>>>> set is fairly large (though it works well on our demos, but does take >>>>>>>> some >>>>>>>> time to finish). You can verify by trying yourself to create the >>>>>>>> orgunit >>>>>>>> (with the exact name) and import the GML on the Sierra Leone demo. >>>>>>>> >>>>>>>> The fact that the dry run seems to "work" for you (as well as the >>>>>>>> stack trace you provided) also points at the issue not residing in the >>>>>>>> GML >>>>>>>> processing but rather in the actual import stage (the GML importer >>>>>>>> uses the >>>>>>>> metadata importer as it's backend). >>>>>>>> >>>>>>>> Another thing: The instance you linked to is using a revision of 16783 >>>>>>>> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). >>>>>>>> Are you using a custom build? Does it contain customizations to the >>>>>>>> core in >>>>>>>> any way? That would also be a possible source of issues, obviously. >>>>>>>> >>>>>>>> You could try deploying this war >>>>>>>> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war >>>>>>>> and replicate the issue using the za South Africa orgunit. >>>>>>>> >>>>>>>> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedb...@gmail.com>: >>>>>>>> >>>>>>>>> Halvdan, >>>>>>>>> >>>>>>>>> The GML files I'm trying to import using 2.19 is the same GML >>>>>>>>> files I've been using successfully for 2.17, 2.18 (so it's using >>>>>>>>> NAME). >>>>>>>>> Something in the import has changed: >>>>>>>>> >>>>>>>>> I have attached the small GML file I'm trying to import - it has >>>>>>>>> the international boundary of SA, so there is only ONE GIS record >>>>>>>>> (top node >>>>>>>>> in all our OrgHierarchies). It imports with no problems in 2.18. >>>>>>>>> >>>>>>>>> In 2.19, if I choose YES for dry run, it will complete with usual >>>>>>>>> import summary etc. >>>>>>>>> >>>>>>>>> If I choose NO for dry run, I get >>>>>>>>> 2015-06-25 22:18:15Importing 1 OrganisationUnits 2015-06-25 >>>>>>>>> 22:18:11Importing meta-data >>>>>>>>> and there it hangs. Nothing gets imported. >>>>>>>>> >>>>>>>>> The online instance you can try to import into is >>>>>>>>> http://fs.dhis.dhmis.org/staging/ >>>>>>>>> >>>>>>>>> When I try to import into a local instance, it seems to enter some >>>>>>>>> kind of endless loop based on the tomcat log: >>>>>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397) >>>>>>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source) >>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>>>>>>>> Source) >>>>>>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>>>>>> at >>>>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192) >>>>>>>>> at >>>>>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java) >>>>>>>>> at java.lang.String.valueOf(Unknown Source) >>>>>>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>>>>>> at >>>>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136) >>>>>>>>> at java.lang.String.valueOf(Unknown Source) >>>>>>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>>>>>> at java.util.AbstractCollection.toString(Unknown Source) >>>>>>>>> at >>>>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327) >>>>>>>>> at java.lang.String.valueOf(Unknown Source) >>>>>>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>>>>>> at >>>>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359) >>>>>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397) >>>>>>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source) >>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>>>>>>>> Source) >>>>>>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>>>>>> at >>>>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192) >>>>>>>>> at >>>>>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java) >>>>>>>>> at java.lang.String.valueOf(Unknown Source) >>>>>>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>>>>>> at >>>>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136) >>>>>>>>> at java.lang.String.valueOf(Unknown Source) >>>>>>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>>>>>> at java.util.AbstractCollection.toString(Unknown Source) >>>>>>>>> at >>>>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327) >>>>>>>>> at java.lang.String.valueOf(Unknown Source) >>>>>>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>>>>>> at >>>>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359) >>>>>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397) >>>>>>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source) >>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>>>>>>>> Source) >>>>>>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>>>>>> >>>>>>>>> On 25 June 2015 at 18:16, Halvdan Grelland <halvda...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Calle, >>>>>>>>>> >>>>>>>>>> The commit you've replied to implements (but not fully, therefore >>>>>>>>>> 'WIP', there's another commit following it pretty soon after) better >>>>>>>>>> error >>>>>>>>>> reporting feedback to the user through the interface. I.e. not >>>>>>>>>> having to >>>>>>>>>> see the stacktrace on the server to determine if and why something >>>>>>>>>> went >>>>>>>>>> wrong. I believe you have been among the people requesting this too >>>>>>>>>> :) >>>>>>>>>> >>>>>>>>>> Also, as you're referencing, we've made some other changes to GML >>>>>>>>>> import. In particular the importer will now match on uid, code or >>>>>>>>>> name, as >>>>>>>>>> oppsed to the former mode of matching only on name. This is not >>>>>>>>>> reflected >>>>>>>>>> in the UI as it is a backend enhancement only, and does not change >>>>>>>>>> the >>>>>>>>>> import options at all. It IS, however, documented and ready for you >>>>>>>>>> to read >>>>>>>>>> about. Short version: it matches on a prioritized list of uid -> >>>>>>>>>> code -> >>>>>>>>>> name. Long version: >>>>>>>>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html >>>>>>>>>> >>>>>>>>>> Be adviced that the commit you've referenced is towards >>>>>>>>>> development trunk, and is not backported to 2.19, the match on >>>>>>>>>> multiple >>>>>>>>>> identifiers, however, was introduced in 2.19 and should work in >>>>>>>>>> current >>>>>>>>>> 2.19 stable. If you're having problems with it you can send me a >>>>>>>>>> full stack >>>>>>>>>> trace and preferably the GML input data. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <calle.hedb...@gmail.com >>>>>>>>>> >: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I'm unable to import GML files using 2.19 (the same files import >>>>>>>>>>> fine in 2.18) - the import simply hangs (endless rolling progress >>>>>>>>>>> bar icon). >>>>>>>>>>> >>>>>>>>>>> Will the above bug-fix sort out that problem - I do not >>>>>>>>>>> understand what the above actually means, or if the GML format has >>>>>>>>>>> changed >>>>>>>>>>> in some way. >>>>>>>>>>> >>>>>>>>>>> Note also that the final 2.19 release was supposed to include a >>>>>>>>>>> more diversified GML import - but I cannot see WHERE to do that (I >>>>>>>>>>> still >>>>>>>>>>> see the same two buttons: choose file and import, either dry run or >>>>>>>>>>> not) >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> Calle >>>>>>>>>>> >>>>>>>>>>> On 5 June 2015 at 14:57, <nore...@launchpad.net> wrote: >>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>> revno: 19296 >>>>>>>>>>>> committer: Halvdan Hoem Grelland <halvda...@gmail.com> >>>>>>>>>>>> branch nick: dhis2 >>>>>>>>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200 >>>>>>>>>>>> message: >>>>>>>>>>>> WIP proper error reporting on failed GML parsing during import >>>>>>>>>>>> added: >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java >>>>>>>>>>>> modified: >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties >>>>>>>>>>>> >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> lp:dhis2 >>>>>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk >>>>>>>>>>>> >>>>>>>>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2. >>>>>>>>>>>> To unsubscribe from this branch go to >>>>>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java >>>>>>>>>>>> 2015-06-03 15:00:19 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -32,6 +32,8 @@ >>>>>>>>>>>> import com.google.common.base.Strings; >>>>>>>>>>>> import com.google.common.collect.Iterators; >>>>>>>>>>>> import com.google.common.collect.Maps; >>>>>>>>>>>> +import org.apache.commons.io.IOUtils; >>>>>>>>>>>> +import org.hibernate.Hibernate; >>>>>>>>>>>> import org.hisp.dhis.common.IdentifiableObjectManager; >>>>>>>>>>>> import org.hisp.dhis.common.IdentifiableProperty; >>>>>>>>>>>> import org.hisp.dhis.common.MergeStrategy; >>>>>>>>>>>> @@ -88,13 +90,91 @@ >>>>>>>>>>>> // GmlImportService implementation >>>>>>>>>>>> // >>>>>>>>>>>> ------------------------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> + @Transactional >>>>>>>>>>>> + @Override >>>>>>>>>>>> + public GmlPreProcessingResult preProcessGml( InputStream >>>>>>>>>>>> inputStream ) >>>>>>>>>>>> + { >>>>>>>>>>>> + InputStream dxfStream = null; >>>>>>>>>>>> + MetaData metaData = null; >>>>>>>>>>>> + >>>>>>>>>>>> + try >>>>>>>>>>>> + { >>>>>>>>>>>> + dxfStream = transformGml( inputStream ); >>>>>>>>>>>> + metaData = renderService.fromXml( dxfStream, >>>>>>>>>>>> MetaData.class ); >>>>>>>>>>>> + } >>>>>>>>>>>> + catch ( IOException | TransformerException e ) >>>>>>>>>>>> + { >>>>>>>>>>>> + return GmlPreProcessingResult.failure( e ); >>>>>>>>>>>> + } >>>>>>>>>>>> + finally >>>>>>>>>>>> + { >>>>>>>>>>>> + IOUtils.closeQuietly( dxfStream ); >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + Map<String, OrganisationUnit> uidMap = >>>>>>>>>>>> Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = >>>>>>>>>>>> Maps.newHashMap(); >>>>>>>>>>>> + >>>>>>>>>>>> + matchAndFilterOnIdentifiers( >>>>>>>>>>>> metaData.getOrganisationUnits(), uidMap, codeMap, nameMap ); >>>>>>>>>>>> + >>>>>>>>>>>> + Map<String, OrganisationUnit> persistedUidMap = >>>>>>>>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(), >>>>>>>>>>>> IdentifiableProperty.UID ); >>>>>>>>>>>> + Map<String, OrganisationUnit> persistedCodeMap = >>>>>>>>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), >>>>>>>>>>>> IdentifiableProperty.CODE ); >>>>>>>>>>>> + Map<String, OrganisationUnit> persistedNameMap = >>>>>>>>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), >>>>>>>>>>>> IdentifiableProperty.NAME ); >>>>>>>>>>>> + >>>>>>>>>>>> + Iterator<OrganisationUnit> persistedIterator = >>>>>>>>>>>> Iterators.concat( persistedUidMap.values().iterator(), >>>>>>>>>>>> + persistedCodeMap.values().iterator(), >>>>>>>>>>>> persistedNameMap.values().iterator() ); >>>>>>>>>>>> + >>>>>>>>>>>> + while ( persistedIterator.hasNext() ) >>>>>>>>>>>> + { >>>>>>>>>>>> + OrganisationUnit persisted = >>>>>>>>>>>> persistedIterator.next(), imported = null; >>>>>>>>>>>> + >>>>>>>>>>>> + if ( !Strings.isNullOrEmpty( persisted.getUid() ) >>>>>>>>>>>> && uidMap.containsKey( persisted.getUid() ) ) >>>>>>>>>>>> + { >>>>>>>>>>>> + imported = uidMap.get( persisted.getUid() ); >>>>>>>>>>>> + } >>>>>>>>>>>> + else if ( !Strings.isNullOrEmpty( >>>>>>>>>>>> persisted.getCode() ) && codeMap.containsKey( persisted.getCode() >>>>>>>>>>>> ) ) >>>>>>>>>>>> + { >>>>>>>>>>>> + imported = codeMap.get( persisted.getCode() ); >>>>>>>>>>>> + } >>>>>>>>>>>> + else if ( !Strings.isNullOrEmpty( >>>>>>>>>>>> persisted.getName() ) && nameMap.containsKey( persisted.getName() >>>>>>>>>>>> ) ) >>>>>>>>>>>> + { >>>>>>>>>>>> + imported = nameMap.get( persisted.getName() ); >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + if ( imported == null || imported.getCoordinates() >>>>>>>>>>>> == null || imported.getFeatureType() == null ) >>>>>>>>>>>> + { >>>>>>>>>>>> + continue; // Failed to dereference a persisted >>>>>>>>>>>> entity for this org unit or geo data incomplete/missing, therefore >>>>>>>>>>>> ignore >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + mergeNonGeoData( persisted, imported ); >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + return GmlPreProcessingResult.success( metaData ); >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> @Override >>>>>>>>>>>> public MetaData fromGml( InputStream inputStream ) >>>>>>>>>>>> throws IOException, TransformerException >>>>>>>>>>>> { >>>>>>>>>>>> - InputStream dxfStream = transformGml( inputStream ); >>>>>>>>>>>> - MetaData metaData = renderService.fromXml( dxfStream, >>>>>>>>>>>> MetaData.class ); >>>>>>>>>>>> - dxfStream.close(); >>>>>>>>>>>> + InputStream dxfStream; >>>>>>>>>>>> + MetaData metaData; >>>>>>>>>>>> + >>>>>>>>>>>> + try >>>>>>>>>>>> + { >>>>>>>>>>>> + dxfStream = transformGml( inputStream ); >>>>>>>>>>>> + } >>>>>>>>>>>> + catch (Exception e) >>>>>>>>>>>> + { >>>>>>>>>>>> + dxfStream = null; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + if(dxfStream != null) >>>>>>>>>>>> + { >>>>>>>>>>>> + metaData = renderService.fromXml( dxfStream, >>>>>>>>>>>> MetaData.class ); >>>>>>>>>>>> + dxfStream.close(); >>>>>>>>>>>> + } >>>>>>>>>>>> + else >>>>>>>>>>>> + { >>>>>>>>>>>> + return null; >>>>>>>>>>>> + } >>>>>>>>>>>> >>>>>>>>>>>> Map<String, OrganisationUnit> uidMap = >>>>>>>>>>>> Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = >>>>>>>>>>>> Maps.newHashMap(); >>>>>>>>>>>> >>>>>>>>>>>> @@ -143,6 +223,7 @@ >>>>>>>>>>>> importService.importMetaData( userUid, fromGml( >>>>>>>>>>>> inputStream ), importOptions, taskId ); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> + @Transactional >>>>>>>>>>>> @Override >>>>>>>>>>>> public void importGml( MetaData metaData, String userUid, >>>>>>>>>>>> ImportOptions importOptions, TaskId taskId ) >>>>>>>>>>>> { >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java >>>>>>>>>>>> 2015-06-03 15:00:19 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -62,6 +62,24 @@ >>>>>>>>>>>> throws IOException, TransformerException; >>>>>>>>>>>> >>>>>>>>>>>> /** >>>>>>>>>>>> + * Pre-process a GML document. The process, in short, >>>>>>>>>>>> entails the following: >>>>>>>>>>>> + * <ol> >>>>>>>>>>>> + * <li>Parse the GML payload and transform it into >>>>>>>>>>>> DXF2 format</li> >>>>>>>>>>>> + * <li>Get the given identifiers (uid, code or name) >>>>>>>>>>>> from the parsed payload and fetch >>>>>>>>>>>> + * the corresponding entities from the DB</li> >>>>>>>>>>>> + * <li>Merge the geospatial data given in the input >>>>>>>>>>>> GML into DB entities and return</li> >>>>>>>>>>>> + * </ol> >>>>>>>>>>>> + * >>>>>>>>>>>> + * The result of this process in returned in a {@link >>>>>>>>>>>> GmlPreProcessingResult} which >>>>>>>>>>>> + * encapsulates the returned {@link MetaData} object or >>>>>>>>>>>> the exception in cause of parse >>>>>>>>>>>> + * failure due to IO errors or malformed input. >>>>>>>>>>>> + * >>>>>>>>>>>> + * @param gmlInputStream the InputStream providing the GML >>>>>>>>>>>> input. >>>>>>>>>>>> + * @return a GmlPreProcessingResult representing the end >>>>>>>>>>>> result of the process. >>>>>>>>>>>> + */ >>>>>>>>>>>> + GmlPreProcessingResult preProcessGml( InputStream >>>>>>>>>>>> gmlInputStream ); >>>>>>>>>>>> + >>>>>>>>>>>> + /** >>>>>>>>>>>> * Imports GML data and merges the geospatial data updates >>>>>>>>>>>> into the database. >>>>>>>>>>>> * See {@link #fromGml(InputStream)} for details on the >>>>>>>>>>>> underlying process. >>>>>>>>>>>> * >>>>>>>>>>>> >>>>>>>>>>>> === added file >>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java >>>>>>>>>>>> 1970-01-01 00:00:00 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -0,0 +1,70 @@ >>>>>>>>>>>> +package org.hisp.dhis.dxf2.gml; >>>>>>>>>>>> + >>>>>>>>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData; >>>>>>>>>>>> + >>>>>>>>>>>> +import java.io.InputStream; >>>>>>>>>>>> + >>>>>>>>>>>> +/** >>>>>>>>>>>> + * Wraps the result of {@link >>>>>>>>>>>> GmlImportService#preProcessGml(InputStream)}. >>>>>>>>>>>> + * This is necessary when performing GML import on a context >>>>>>>>>>>> where exceptions >>>>>>>>>>>> + * due to malformed input cannot be caught, thus leaving the >>>>>>>>>>>> user uninformed of >>>>>>>>>>>> + * the error. This class will wrap the failure and provide the >>>>>>>>>>>> Throwable to the >>>>>>>>>>>> + * consuming class on error or the resulting MetaData object >>>>>>>>>>>> on success. >>>>>>>>>>>> + * >>>>>>>>>>>> + * @author Halvdan Hoem grelland <halvda...@gmail.com> >>>>>>>>>>>> + */ >>>>>>>>>>>> +public final class GmlPreProcessingResult >>>>>>>>>>>> +{ >>>>>>>>>>>> + private boolean isSuccess; >>>>>>>>>>>> + private MetaData resultMetaData; >>>>>>>>>>>> + private Throwable throwable; >>>>>>>>>>>> + >>>>>>>>>>>> + private GmlPreProcessingResult(){} >>>>>>>>>>>> + >>>>>>>>>>>> + public static GmlPreProcessingResult success( MetaData >>>>>>>>>>>> resultMetaData ) { >>>>>>>>>>>> + GmlPreProcessingResult result = new >>>>>>>>>>>> GmlPreProcessingResult(); >>>>>>>>>>>> + result.setResultMetaData( resultMetaData ); >>>>>>>>>>>> + result.setSuccess( true ); >>>>>>>>>>>> + >>>>>>>>>>>> + return result; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + public static GmlPreProcessingResult failure( Throwable >>>>>>>>>>>> throwable ) >>>>>>>>>>>> + { >>>>>>>>>>>> + GmlPreProcessingResult result = new >>>>>>>>>>>> GmlPreProcessingResult(); >>>>>>>>>>>> + result.setThrowable( throwable ); >>>>>>>>>>>> + result.setSuccess( false ); >>>>>>>>>>>> + >>>>>>>>>>>> + return result; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + private void setSuccess( boolean isSuccess ) >>>>>>>>>>>> + { >>>>>>>>>>>> + this.isSuccess = isSuccess; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + public boolean isSuccess() >>>>>>>>>>>> + { >>>>>>>>>>>> + return isSuccess; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + private void setResultMetaData( MetaData resultMetaData ) >>>>>>>>>>>> + { >>>>>>>>>>>> + this.resultMetaData = resultMetaData; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + public MetaData getResultMetaData() >>>>>>>>>>>> + { >>>>>>>>>>>> + return resultMetaData; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + private void setThrowable( Throwable throwable ) >>>>>>>>>>>> + { >>>>>>>>>>>> + this.throwable = throwable; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + public Throwable getThrowable() >>>>>>>>>>>> + { >>>>>>>>>>>> + return throwable; >>>>>>>>>>>> + } >>>>>>>>>>>> +} >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java >>>>>>>>>>>> 2015-01-17 07:41:26 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -117,4 +117,6 @@ >>>>>>>>>>>> assertEquals( 1, units.get( >>>>>>>>>>>> "Blindern").getCoordinatesAsList().get( 0 >>>>>>>>>>>> ).getNumberOfCoordinates() ); >>>>>>>>>>>> assertEquals( 76, units.get( "Forskningsparken" >>>>>>>>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() ); >>>>>>>>>>>> } >>>>>>>>>>>> + >>>>>>>>>>>> + // TODO Add test for >>>>>>>>>>>> GmlImportService#preProcessGml(InputStream) >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java >>>>>>>>>>>> 2015-05-30 13:36:07 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -77,7 +77,7 @@ >>>>>>>>>>>> { >>>>>>>>>>>> taskExecutor.execute( task ); >>>>>>>>>>>> } >>>>>>>>>>>> - >>>>>>>>>>>> + >>>>>>>>>>>> @Override >>>>>>>>>>>> public boolean scheduleTask( String key, Runnable task, >>>>>>>>>>>> String cronExpr ) >>>>>>>>>>>> { >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java >>>>>>>>>>>> 2015-05-28 16:10:07 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -187,7 +187,7 @@ >>>>>>>>>>>> } >>>>>>>>>>>> else if ( "gml".equals( importFormat ) ) >>>>>>>>>>>> { >>>>>>>>>>>> - scheduler.executeTask( new ImportMetaDataGmlTask( >>>>>>>>>>>> userId, gmlImportService, importOptions, in, taskId ) ); >>>>>>>>>>>> + scheduler.executeTask( new ImportMetaDataGmlTask( >>>>>>>>>>>> userId, gmlImportService, notifier, importOptions, in, taskId ) ); >>>>>>>>>>>> } >>>>>>>>>>>> else if ( "json".equals( importFormat ) || >>>>>>>>>>>> "xml".equals( importFormat ) ) >>>>>>>>>>>> { >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java >>>>>>>>>>>> 2015-04-11 14:06:51 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -28,14 +28,18 @@ >>>>>>>>>>>> * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >>>>>>>>>>>> */ >>>>>>>>>>>> >>>>>>>>>>>> +import org.apache.commons.lang.exception.ExceptionUtils; >>>>>>>>>>>> import org.apache.commons.logging.Log; >>>>>>>>>>>> import org.apache.commons.logging.LogFactory; >>>>>>>>>>>> import org.hisp.dhis.dxf2.common.ImportOptions; >>>>>>>>>>>> import org.hisp.dhis.dxf2.gml.GmlImportService; >>>>>>>>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult; >>>>>>>>>>>> import org.hisp.dhis.scheduling.TaskId; >>>>>>>>>>>> +import org.hisp.dhis.system.notification.NotificationLevel; >>>>>>>>>>>> +import org.hisp.dhis.system.notification.Notifier; >>>>>>>>>>>> +import org.springframework.web.util.HtmlUtils; >>>>>>>>>>>> +import org.xml.sax.SAXParseException; >>>>>>>>>>>> >>>>>>>>>>>> -import javax.xml.transform.TransformerException; >>>>>>>>>>>> -import java.io.IOException; >>>>>>>>>>>> import java.io.InputStream; >>>>>>>>>>>> >>>>>>>>>>>> /** >>>>>>>>>>>> @@ -50,12 +54,15 @@ >>>>>>>>>>>> >>>>>>>>>>>> private String userUid; >>>>>>>>>>>> >>>>>>>>>>>> + >>>>>>>>>>>> // >>>>>>>>>>>> ------------------------------------------------------------------------- >>>>>>>>>>>> // Dependencies >>>>>>>>>>>> // >>>>>>>>>>>> ------------------------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> private GmlImportService gmlImportService; >>>>>>>>>>>> >>>>>>>>>>>> + private Notifier notifier; >>>>>>>>>>>> + >>>>>>>>>>>> private ImportOptions importOptions; >>>>>>>>>>>> >>>>>>>>>>>> private InputStream inputStream; >>>>>>>>>>>> @@ -64,11 +71,12 @@ >>>>>>>>>>>> // Constructors >>>>>>>>>>>> // >>>>>>>>>>>> ------------------------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> - public ImportMetaDataGmlTask( String userUid, >>>>>>>>>>>> GmlImportService gmlImportService, >>>>>>>>>>>> + public ImportMetaDataGmlTask( String userUid, >>>>>>>>>>>> GmlImportService gmlImportService, Notifier notifier, >>>>>>>>>>>> ImportOptions importOptions, InputStream inputStream, >>>>>>>>>>>> TaskId taskId ) >>>>>>>>>>>> { >>>>>>>>>>>> this.userUid = userUid; >>>>>>>>>>>> this.gmlImportService = gmlImportService; >>>>>>>>>>>> + this.notifier = notifier; >>>>>>>>>>>> this.importOptions = importOptions; >>>>>>>>>>>> this.inputStream = inputStream; >>>>>>>>>>>> this.taskId = taskId; >>>>>>>>>>>> @@ -83,15 +91,37 @@ >>>>>>>>>>>> { >>>>>>>>>>>> importOptions.setImportStrategy( "update" ); // Force >>>>>>>>>>>> update only for GML import >>>>>>>>>>>> >>>>>>>>>>>> - try >>>>>>>>>>>> - { >>>>>>>>>>>> - gmlImportService.importGml( inputStream, userUid, >>>>>>>>>>>> importOptions, taskId ); >>>>>>>>>>>> - } >>>>>>>>>>>> - catch ( IOException | TransformerException e ) >>>>>>>>>>>> - { >>>>>>>>>>>> - log.error( "Unable to read GML data from input >>>>>>>>>>>> stream", e ); >>>>>>>>>>>> - >>>>>>>>>>>> - throw new RuntimeException( "Failed to parse GML >>>>>>>>>>>> input stream", e ); >>>>>>>>>>>> - } >>>>>>>>>>>> + GmlPreProcessingResult gmlPreProcessingResult = >>>>>>>>>>>> gmlImportService.preProcessGml( inputStream ); >>>>>>>>>>>> + >>>>>>>>>>>> + if ( !gmlPreProcessingResult.isSuccess() ) >>>>>>>>>>>> + { >>>>>>>>>>>> + Throwable throwable = >>>>>>>>>>>> gmlPreProcessingResult.getThrowable(); >>>>>>>>>>>> + String message = createErrorMessage( throwable ); >>>>>>>>>>>> + >>>>>>>>>>>> + notifier.notify( taskId, NotificationLevel.ERROR, >>>>>>>>>>>> message, false ); >>>>>>>>>>>> + log.error( "GML import failed: " + message, >>>>>>>>>>>> throwable ); >>>>>>>>>>>> + >>>>>>>>>>>> + return; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + gmlImportService.importGml( >>>>>>>>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, >>>>>>>>>>>> importOptions, taskId >>>>>>>>>>>> ); >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + private String createErrorMessage( Throwable throwable ) >>>>>>>>>>>> + { >>>>>>>>>>>> + String message = ""; >>>>>>>>>>>> + Throwable rootThrowable = ExceptionUtils.getRootCause( >>>>>>>>>>>> throwable ); >>>>>>>>>>>> + >>>>>>>>>>>> + if ( rootThrowable instanceof SAXParseException ) >>>>>>>>>>>> + { >>>>>>>>>>>> + SAXParseException e = (SAXParseException) >>>>>>>>>>>> rootThrowable; >>>>>>>>>>>> + message += "Syntax error on line " + >>>>>>>>>>>> e.getLineNumber() + ". " + e.getMessage(); >>>>>>>>>>>> + } >>>>>>>>>>>> + else >>>>>>>>>>>> + { >>>>>>>>>>>> + message += rootThrowable.getMessage(); >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + return HtmlUtils.htmlEscape( message ); >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties >>>>>>>>>>>> 2015-02-20 11:17:31 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -300,6 +300,7 @@ >>>>>>>>>>>> ignored=Ignored >>>>>>>>>>>> conflicts=Conflicts >>>>>>>>>>>> no_conflicts_found=No conflicts found >>>>>>>>>>>> +no_import_summary_available=No summary available >>>>>>>>>>>> type=Type >>>>>>>>>>>> count=Count >>>>>>>>>>>> export_as_xml=Export as XML >>>>>>>>>>>> >>>>>>>>>>>> === modified file >>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm' >>>>>>>>>>>> --- >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm >>>>>>>>>>>> 2013-03-21 09:15:24 +0000 >>>>>>>>>>>> +++ >>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm >>>>>>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>>>>>> @@ -1,4 +1,6 @@ >>>>>>>>>>>> <h3>$i18n.getString( "import_summary" )</h3> >>>>>>>>>>>> + >>>>>>>>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML >>>>>>>>>>>> import (pre-processing fails) >>>>>>>>>>>> <h4>$i18n.getString( "import_count" )</h4> >>>>>>>>>>>> >>>>>>>>>>>> $summary.importCount.imported Imported<br/> >>>>>>>>>>>> @@ -54,3 +56,6 @@ >>>>>>>>>>>> #else >>>>>>>>>>>> <p>$i18n.getString( "no_conflicts_found" )</p> >>>>>>>>>>>> #end >>>>>>>>>>>> +#else >>>>>>>>>>>> + $i18n.getString( "no_summary_available" ) >>>>>>>>>>>> +#end >>>>>>>>>>>> \ No newline at end of file >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>>>>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> ******************************************* >>>>>>>>>>> >>>>>>>>>>> Calle Hedberg >>>>>>>>>>> >>>>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>>>>>>>>> >>>>>>>>>>> Tel/fax (home): +27-21-685-6472 >>>>>>>>>>> >>>>>>>>>>> Cell: +27-82-853-5352 >>>>>>>>>>> >>>>>>>>>>> Iridium SatPhone: +8816-315-19274 >>>>>>>>>>> >>>>>>>>>>> Email: calle.hedb...@gmail.com >>>>>>>>>>> >>>>>>>>>>> Skype: calle_hedberg >>>>>>>>>>> >>>>>>>>>>> ******************************************* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>>>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> ******************************************* >>>>>>>>> >>>>>>>>> Calle Hedberg >>>>>>>>> >>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>>>>>>> >>>>>>>>> Tel/fax (home): +27-21-685-6472 >>>>>>>>> >>>>>>>>> Cell: +27-82-853-5352 >>>>>>>>> >>>>>>>>> Iridium SatPhone: +8816-315-19274 >>>>>>>>> >>>>>>>>> Email: calle.hedb...@gmail.com >>>>>>>>> >>>>>>>>> Skype: calle_hedberg >>>>>>>>> >>>>>>>>> ******************************************* >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> ******************************************* >>>>>> >>>>>> Calle Hedberg >>>>>> >>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>>>> >>>>>> Tel/fax (home): +27-21-685-6472 >>>>>> >>>>>> Cell: +27-82-853-5352 >>>>>> >>>>>> Iridium SatPhone: +8816-315-19274 >>>>>> >>>>>> Email: calle.hedb...@gmail.com >>>>>> >>>>>> Skype: calle_hedberg >>>>>> >>>>>> ******************************************* >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> ******************************************* >>>>> >>>>> Calle Hedberg >>>>> >>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>>> >>>>> Tel/fax (home): +27-21-685-6472 >>>>> >>>>> Cell: +27-82-853-5352 >>>>> >>>>> Iridium SatPhone: +8816-315-19274 >>>>> >>>>> Email: calle.hedb...@gmail.com >>>>> >>>>> Skype: calle_hedberg >>>>> >>>>> ******************************************* >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> ******************************************* >>>> >>>> Calle Hedberg >>>> >>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>> >>>> Tel/fax (home): +27-21-685-6472 >>>> >>>> Cell: +27-82-853-5352 >>>> >>>> Iridium SatPhone: +8816-315-19274 >>>> >>>> Email: calle.hedb...@gmail.com >>>> >>>> Skype: calle_hedberg >>>> >>>> ******************************************* >>>> >>>> >>> >> >> >> -- >> >> ******************************************* >> >> Calle Hedberg >> >> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >> >> Tel/fax (home): +27-21-685-6472 >> >> Cell: +27-82-853-5352 >> >> Iridium SatPhone: +8816-315-19274 >> >> Email: calle.hedb...@gmail.com >> >> Skype: calle_hedberg >> >> ******************************************* >> >>
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
