I would recommend running your shapefile through Mapshaper to simplify it before conversion to GML
On Fri, Jun 26, 2015 at 2:17 PM, Dan Cocos <d...@dancocos.com> wrote: > I’ve seen this error in the past on import of GML the cause was an Org > Unit had a lot of coordinates (as in greater than 200 it was every turn of > Lake Victoria) and this caused the importer to hang. > > On Jun 26, 2015, at 5:17 AM, Calle Hedberg <calle.hedb...@gmail.com> > wrote: > > Hi, > > I ran it again (revision 19247) and pulled out the first part of the log > error: > > * INFO 2015-06-26 11:04:18,338 User 'Calle_Hedberg' started import at Fri > Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT, > time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data] > (InMemoryNotifier.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache: > true). (DefaultObjectBridge.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79 > seconds. (DefaultObjectBridge.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT, > time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits] > (InMemoryNotifier.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of > type AttributeValue (DefaultDeletionManager.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of > type AttributeValue (DefaultDeletionManager.java [taskScheduler-1]) > * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of > type AttributeValue (DefaultDeletionManager.java [taskScheduler-1]) > * ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled > task. (TaskUtils.java [taskScheduler-1]) > java.lang.StackOverflowError > at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source) > at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source) > at java.util.regex.Pattern$BitClass.<init>(Unknown Source) > at java.util.regex.Pattern.clazz(Unknown Source) > at java.util.regex.Pattern.sequence(Unknown Source) > at java.util.regex.Pattern.expr(Unknown Source) > at java.util.regex.Pattern.compile(Unknown Source) > at java.util.regex.Pattern.<init>(Unknown Source) > at java.util.regex.Pattern.compile(Unknown Source) > at java.lang.String.replaceAll(Unknown Source) > at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134) > at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95) > at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396) > at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196) > at > org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java) > at java.lang.String.valueOf(Unknown Source) > at java.lang.StringBuilder.append(Unknown Source) > at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122) > at java.lang.String.valueOf(Unknown Source) > at java.lang.StringBuilder.append(Unknown Source) > at java.util.AbstractCollection.toString(Unknown Source) > at > org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327) > at java.lang.String.valueOf(Unknown Source) > at java.lang.StringBuilder.append(Unknown Source) > at > com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359) > > Any ideas? > > Regards > Calle > > On 26 June 2015 at 10:12, Calle Hedberg <calle.hedb...@gmail.com> wrote: > >> Hi, >> >> Firstly, yes we are using a custom revision - although it is SUPPOSED to >> be based on a 2.19 version from a few days ago. I will check that. >> >> BUT I have tried the import using revision 19247 too, and the >> problem/behaviour is exactly the same - it hangs. Whereas if I import into >> an older version of the same provincial instance that still runs on 2.18, >> it imports fine. >> >> The fact that the same GML file imports in the Sierra Leone demo - I >> cannot explain that, and the tomcat log only shows that >> Attribute.toString() looping... >> >> Can I create a stripped down version of the instance and upload it for >> you to test? >> >> Regards >> Calle >> >> >> >> On 26 June 2015 at 04:09, Morten Olav Hansen <morte...@gmail.com> wrote: >> >>> It seems to be looping in Attribute.toString(), I remember this was an >>> issue at some point.. but that was a long long time ago.. so maybe you need >>> a revision bump >>> >>> -- >>> Morten >>> >>> On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland <halvda...@gmail.com> >>> wrote: >>> >>>> I tried your example GML on both our demo servers (dev on 2.20 and demo >>>> on 2.19, same rev as your server) and it works just fine. >>>> >>>> If you're experiencing it hanging and not finishing you might be >>>> experiencing some kind of performance and/or memory issue. The coordinate >>>> set is fairly large (though it works well on our demos, but does take some >>>> time to finish). You can verify by trying yourself to create the orgunit >>>> (with the exact name) and import the GML on the Sierra Leone demo. >>>> >>>> The fact that the dry run seems to "work" for you (as well as the stack >>>> trace you provided) also points at the issue not residing in the GML >>>> processing but rather in the actual import stage (the GML importer uses the >>>> metadata importer as it's backend). >>>> >>>> Another thing: The instance you linked to is using a revision of 16783 >>>> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are >>>> you using a custom build? Does it contain customizations to the core in any >>>> way? That would also be a possible source of issues, obviously. >>>> >>>> You could try deploying this war >>>> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war >>>> and replicate the issue using the za South Africa orgunit. >>>> >>>> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedb...@gmail.com>: >>>> >>>>> Halvdan, >>>>> >>>>> The GML files I'm trying to import using 2.19 is the same GML files >>>>> I've been using successfully for 2.17, 2.18 (so it's using NAME). >>>>> Something >>>>> in the import has changed: >>>>> >>>>> I have attached the small GML file I'm trying to import - it has the >>>>> international boundary of SA, so there is only ONE GIS record (top node in >>>>> all our OrgHierarchies). It imports with no problems in 2.18. >>>>> >>>>> In 2.19, if I choose YES for dry run, it will complete with usual >>>>> import summary etc. >>>>> >>>>> If I choose NO for dry run, I get >>>>> 2015-06-25 22:18:15Importing 1 OrganisationUnits 2015-06-25 >>>>> 22:18:11Importing >>>>> meta-data >>>>> and there it hangs. Nothing gets imported. >>>>> >>>>> The online instance you can try to import into is >>>>> http://fs.dhis.dhmis.org/staging/ >>>>> >>>>> When I try to import into a local instance, it seems to enter some >>>>> kind of endless loop based on the tomcat log: >>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397) >>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source) >>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>> at >>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192) >>>>> at >>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at >>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at java.util.AbstractCollection.toString(Unknown Source) >>>>> at >>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at >>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359) >>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397) >>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source) >>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>> at >>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192) >>>>> at >>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at >>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at java.util.AbstractCollection.toString(Unknown Source) >>>>> at >>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327) >>>>> at java.lang.String.valueOf(Unknown Source) >>>>> at java.lang.StringBuilder.append(Unknown Source) >>>>> at >>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359) >>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397) >>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source) >>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>>> >>>>> On 25 June 2015 at 18:16, Halvdan Grelland <halvda...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Calle, >>>>>> >>>>>> The commit you've replied to implements (but not fully, therefore >>>>>> 'WIP', there's another commit following it pretty soon after) better >>>>>> error >>>>>> reporting feedback to the user through the interface. I.e. not having to >>>>>> see the stacktrace on the server to determine if and why something went >>>>>> wrong. I believe you have been among the people requesting this too :) >>>>>> >>>>>> Also, as you're referencing, we've made some other changes to GML >>>>>> import. In particular the importer will now match on uid, code or name, >>>>>> as >>>>>> oppsed to the former mode of matching only on name. This is not reflected >>>>>> in the UI as it is a backend enhancement only, and does not change the >>>>>> import options at all. It IS, however, documented and ready for you to >>>>>> read >>>>>> about. Short version: it matches on a prioritized list of uid -> code -> >>>>>> name. Long version: >>>>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html >>>>>> >>>>>> Be adviced that the commit you've referenced is towards development >>>>>> trunk, and is not backported to 2.19, the match on multiple identifiers, >>>>>> however, was introduced in 2.19 and should work in current 2.19 stable. >>>>>> If >>>>>> you're having problems with it you can send me a full stack trace and >>>>>> preferably the GML input data. >>>>>> >>>>>> >>>>>> >>>>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <calle.hedb...@gmail.com>: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm unable to import GML files using 2.19 (the same files import >>>>>>> fine in 2.18) - the import simply hangs (endless rolling progress bar >>>>>>> icon). >>>>>>> >>>>>>> Will the above bug-fix sort out that problem - I do not understand >>>>>>> what the above actually means, or if the GML format has changed in some >>>>>>> way. >>>>>>> >>>>>>> Note also that the final 2.19 release was supposed to include a more >>>>>>> diversified GML import - but I cannot see WHERE to do that (I still see >>>>>>> the >>>>>>> same two buttons: choose file and import, either dry run or not) >>>>>>> >>>>>>> Regards >>>>>>> Calle >>>>>>> >>>>>>> On 5 June 2015 at 14:57, <nore...@launchpad.net> wrote: >>>>>>> >>>>>>>> ------------------------------------------------------------ >>>>>>>> revno: 19296 >>>>>>>> committer: Halvdan Hoem Grelland <halvda...@gmail.com> >>>>>>>> branch nick: dhis2 >>>>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200 >>>>>>>> message: >>>>>>>> WIP proper error reporting on failed GML parsing during import >>>>>>>> added: >>>>>>>> >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java >>>>>>>> modified: >>>>>>>> >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java >>>>>>>> >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java >>>>>>>> >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java >>>>>>>> >>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java >>>>>>>> >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java >>>>>>>> >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java >>>>>>>> >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties >>>>>>>> >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> lp:dhis2 >>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk >>>>>>>> >>>>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2. >>>>>>>> To unsubscribe from this branch go to >>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java >>>>>>>> 2015-06-03 15:00:19 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -32,6 +32,8 @@ >>>>>>>> import com.google.common.base.Strings; >>>>>>>> import com.google.common.collect.Iterators; >>>>>>>> import com.google.common.collect.Maps; >>>>>>>> +import org.apache.commons.io.IOUtils; >>>>>>>> +import org.hibernate.Hibernate; >>>>>>>> import org.hisp.dhis.common.IdentifiableObjectManager; >>>>>>>> import org.hisp.dhis.common.IdentifiableProperty; >>>>>>>> import org.hisp.dhis.common.MergeStrategy; >>>>>>>> @@ -88,13 +90,91 @@ >>>>>>>> // GmlImportService implementation >>>>>>>> // >>>>>>>> ------------------------------------------------------------------------- >>>>>>>> >>>>>>>> + @Transactional >>>>>>>> + @Override >>>>>>>> + public GmlPreProcessingResult preProcessGml( InputStream >>>>>>>> inputStream ) >>>>>>>> + { >>>>>>>> + InputStream dxfStream = null; >>>>>>>> + MetaData metaData = null; >>>>>>>> + >>>>>>>> + try >>>>>>>> + { >>>>>>>> + dxfStream = transformGml( inputStream ); >>>>>>>> + metaData = renderService.fromXml( dxfStream, >>>>>>>> MetaData.class ); >>>>>>>> + } >>>>>>>> + catch ( IOException | TransformerException e ) >>>>>>>> + { >>>>>>>> + return GmlPreProcessingResult.failure( e ); >>>>>>>> + } >>>>>>>> + finally >>>>>>>> + { >>>>>>>> + IOUtils.closeQuietly( dxfStream ); >>>>>>>> + } >>>>>>>> + >>>>>>>> + Map<String, OrganisationUnit> uidMap = Maps.newHashMap(), >>>>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap(); >>>>>>>> + >>>>>>>> + matchAndFilterOnIdentifiers( >>>>>>>> metaData.getOrganisationUnits(), uidMap, codeMap, nameMap ); >>>>>>>> + >>>>>>>> + Map<String, OrganisationUnit> persistedUidMap = >>>>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(), >>>>>>>> IdentifiableProperty.UID ); >>>>>>>> + Map<String, OrganisationUnit> persistedCodeMap = >>>>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), >>>>>>>> IdentifiableProperty.CODE ); >>>>>>>> + Map<String, OrganisationUnit> persistedNameMap = >>>>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), >>>>>>>> IdentifiableProperty.NAME ); >>>>>>>> + >>>>>>>> + Iterator<OrganisationUnit> persistedIterator = >>>>>>>> Iterators.concat( persistedUidMap.values().iterator(), >>>>>>>> + persistedCodeMap.values().iterator(), >>>>>>>> persistedNameMap.values().iterator() ); >>>>>>>> + >>>>>>>> + while ( persistedIterator.hasNext() ) >>>>>>>> + { >>>>>>>> + OrganisationUnit persisted = persistedIterator.next(), >>>>>>>> imported = null; >>>>>>>> + >>>>>>>> + if ( !Strings.isNullOrEmpty( persisted.getUid() ) && >>>>>>>> uidMap.containsKey( persisted.getUid() ) ) >>>>>>>> + { >>>>>>>> + imported = uidMap.get( persisted.getUid() ); >>>>>>>> + } >>>>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getCode() >>>>>>>> ) && codeMap.containsKey( persisted.getCode() ) ) >>>>>>>> + { >>>>>>>> + imported = codeMap.get( persisted.getCode() ); >>>>>>>> + } >>>>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getName() >>>>>>>> ) && nameMap.containsKey( persisted.getName() ) ) >>>>>>>> + { >>>>>>>> + imported = nameMap.get( persisted.getName() ); >>>>>>>> + } >>>>>>>> + >>>>>>>> + if ( imported == null || imported.getCoordinates() == >>>>>>>> null || imported.getFeatureType() == null ) >>>>>>>> + { >>>>>>>> + continue; // Failed to dereference a persisted >>>>>>>> entity for this org unit or geo data incomplete/missing, therefore >>>>>>>> ignore >>>>>>>> + } >>>>>>>> + >>>>>>>> + mergeNonGeoData( persisted, imported ); >>>>>>>> + } >>>>>>>> + >>>>>>>> + return GmlPreProcessingResult.success( metaData ); >>>>>>>> + } >>>>>>>> + >>>>>>>> @Override >>>>>>>> public MetaData fromGml( InputStream inputStream ) >>>>>>>> throws IOException, TransformerException >>>>>>>> { >>>>>>>> - InputStream dxfStream = transformGml( inputStream ); >>>>>>>> - MetaData metaData = renderService.fromXml( dxfStream, >>>>>>>> MetaData.class ); >>>>>>>> - dxfStream.close(); >>>>>>>> + InputStream dxfStream; >>>>>>>> + MetaData metaData; >>>>>>>> + >>>>>>>> + try >>>>>>>> + { >>>>>>>> + dxfStream = transformGml( inputStream ); >>>>>>>> + } >>>>>>>> + catch (Exception e) >>>>>>>> + { >>>>>>>> + dxfStream = null; >>>>>>>> + } >>>>>>>> + >>>>>>>> + if(dxfStream != null) >>>>>>>> + { >>>>>>>> + metaData = renderService.fromXml( dxfStream, >>>>>>>> MetaData.class ); >>>>>>>> + dxfStream.close(); >>>>>>>> + } >>>>>>>> + else >>>>>>>> + { >>>>>>>> + return null; >>>>>>>> + } >>>>>>>> >>>>>>>> Map<String, OrganisationUnit> uidMap = Maps.newHashMap(), >>>>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap(); >>>>>>>> >>>>>>>> @@ -143,6 +223,7 @@ >>>>>>>> importService.importMetaData( userUid, fromGml( >>>>>>>> inputStream ), importOptions, taskId ); >>>>>>>> } >>>>>>>> >>>>>>>> + @Transactional >>>>>>>> @Override >>>>>>>> public void importGml( MetaData metaData, String userUid, >>>>>>>> ImportOptions importOptions, TaskId taskId ) >>>>>>>> { >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java >>>>>>>> 2015-06-03 15:00:19 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -62,6 +62,24 @@ >>>>>>>> throws IOException, TransformerException; >>>>>>>> >>>>>>>> /** >>>>>>>> + * Pre-process a GML document. The process, in short, entails >>>>>>>> the following: >>>>>>>> + * <ol> >>>>>>>> + * <li>Parse the GML payload and transform it into DXF2 >>>>>>>> format</li> >>>>>>>> + * <li>Get the given identifiers (uid, code or name) from >>>>>>>> the parsed payload and fetch >>>>>>>> + * the corresponding entities from the DB</li> >>>>>>>> + * <li>Merge the geospatial data given in the input GML >>>>>>>> into DB entities and return</li> >>>>>>>> + * </ol> >>>>>>>> + * >>>>>>>> + * The result of this process in returned in a {@link >>>>>>>> GmlPreProcessingResult} which >>>>>>>> + * encapsulates the returned {@link MetaData} object or the >>>>>>>> exception in cause of parse >>>>>>>> + * failure due to IO errors or malformed input. >>>>>>>> + * >>>>>>>> + * @param gmlInputStream the InputStream providing the GML >>>>>>>> input. >>>>>>>> + * @return a GmlPreProcessingResult representing the end >>>>>>>> result of the process. >>>>>>>> + */ >>>>>>>> + GmlPreProcessingResult preProcessGml( InputStream >>>>>>>> gmlInputStream ); >>>>>>>> + >>>>>>>> + /** >>>>>>>> * Imports GML data and merges the geospatial data updates >>>>>>>> into the database. >>>>>>>> * See {@link #fromGml(InputStream)} for details on the >>>>>>>> underlying process. >>>>>>>> * >>>>>>>> >>>>>>>> === added file >>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java >>>>>>>> 1970-01-01 00:00:00 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -0,0 +1,70 @@ >>>>>>>> +package org.hisp.dhis.dxf2.gml; >>>>>>>> + >>>>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData; >>>>>>>> + >>>>>>>> +import java.io.InputStream; >>>>>>>> + >>>>>>>> +/** >>>>>>>> + * Wraps the result of {@link >>>>>>>> GmlImportService#preProcessGml(InputStream)}. >>>>>>>> + * This is necessary when performing GML import on a context where >>>>>>>> exceptions >>>>>>>> + * due to malformed input cannot be caught, thus leaving the user >>>>>>>> uninformed of >>>>>>>> + * the error. This class will wrap the failure and provide the >>>>>>>> Throwable to the >>>>>>>> + * consuming class on error or the resulting MetaData object on >>>>>>>> success. >>>>>>>> + * >>>>>>>> + * @author Halvdan Hoem grelland <halvda...@gmail.com> >>>>>>>> + */ >>>>>>>> +public final class GmlPreProcessingResult >>>>>>>> +{ >>>>>>>> + private boolean isSuccess; >>>>>>>> + private MetaData resultMetaData; >>>>>>>> + private Throwable throwable; >>>>>>>> + >>>>>>>> + private GmlPreProcessingResult(){} >>>>>>>> + >>>>>>>> + public static GmlPreProcessingResult success( MetaData >>>>>>>> resultMetaData ) { >>>>>>>> + GmlPreProcessingResult result = new >>>>>>>> GmlPreProcessingResult(); >>>>>>>> + result.setResultMetaData( resultMetaData ); >>>>>>>> + result.setSuccess( true ); >>>>>>>> + >>>>>>>> + return result; >>>>>>>> + } >>>>>>>> + >>>>>>>> + public static GmlPreProcessingResult failure( Throwable >>>>>>>> throwable ) >>>>>>>> + { >>>>>>>> + GmlPreProcessingResult result = new >>>>>>>> GmlPreProcessingResult(); >>>>>>>> + result.setThrowable( throwable ); >>>>>>>> + result.setSuccess( false ); >>>>>>>> + >>>>>>>> + return result; >>>>>>>> + } >>>>>>>> + >>>>>>>> + private void setSuccess( boolean isSuccess ) >>>>>>>> + { >>>>>>>> + this.isSuccess = isSuccess; >>>>>>>> + } >>>>>>>> + >>>>>>>> + public boolean isSuccess() >>>>>>>> + { >>>>>>>> + return isSuccess; >>>>>>>> + } >>>>>>>> + >>>>>>>> + private void setResultMetaData( MetaData resultMetaData ) >>>>>>>> + { >>>>>>>> + this.resultMetaData = resultMetaData; >>>>>>>> + } >>>>>>>> + >>>>>>>> + public MetaData getResultMetaData() >>>>>>>> + { >>>>>>>> + return resultMetaData; >>>>>>>> + } >>>>>>>> + >>>>>>>> + private void setThrowable( Throwable throwable ) >>>>>>>> + { >>>>>>>> + this.throwable = throwable; >>>>>>>> + } >>>>>>>> + >>>>>>>> + public Throwable getThrowable() >>>>>>>> + { >>>>>>>> + return throwable; >>>>>>>> + } >>>>>>>> +} >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java >>>>>>>> 2015-01-17 07:41:26 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -117,4 +117,6 @@ >>>>>>>> assertEquals( 1, units.get( >>>>>>>> "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() ); >>>>>>>> assertEquals( 76, units.get( "Forskningsparken" >>>>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() ); >>>>>>>> } >>>>>>>> + >>>>>>>> + // TODO Add test for >>>>>>>> GmlImportService#preProcessGml(InputStream) >>>>>>>> } >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java >>>>>>>> 2015-05-30 13:36:07 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -77,7 +77,7 @@ >>>>>>>> { >>>>>>>> taskExecutor.execute( task ); >>>>>>>> } >>>>>>>> - >>>>>>>> + >>>>>>>> @Override >>>>>>>> public boolean scheduleTask( String key, Runnable task, String >>>>>>>> cronExpr ) >>>>>>>> { >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java >>>>>>>> 2015-05-28 16:10:07 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -187,7 +187,7 @@ >>>>>>>> } >>>>>>>> else if ( "gml".equals( importFormat ) ) >>>>>>>> { >>>>>>>> - scheduler.executeTask( new ImportMetaDataGmlTask( >>>>>>>> userId, gmlImportService, importOptions, in, taskId ) ); >>>>>>>> + scheduler.executeTask( new ImportMetaDataGmlTask( >>>>>>>> userId, gmlImportService, notifier, importOptions, in, taskId ) ); >>>>>>>> } >>>>>>>> else if ( "json".equals( importFormat ) || "xml".equals( >>>>>>>> importFormat ) ) >>>>>>>> { >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java' >>>>>>>> --- >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java >>>>>>>> 2015-04-11 14:06:51 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -28,14 +28,18 @@ >>>>>>>> * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >>>>>>>> */ >>>>>>>> >>>>>>>> +import org.apache.commons.lang.exception.ExceptionUtils; >>>>>>>> import org.apache.commons.logging.Log; >>>>>>>> import org.apache.commons.logging.LogFactory; >>>>>>>> import org.hisp.dhis.dxf2.common.ImportOptions; >>>>>>>> import org.hisp.dhis.dxf2.gml.GmlImportService; >>>>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult; >>>>>>>> import org.hisp.dhis.scheduling.TaskId; >>>>>>>> +import org.hisp.dhis.system.notification.NotificationLevel; >>>>>>>> +import org.hisp.dhis.system.notification.Notifier; >>>>>>>> +import org.springframework.web.util.HtmlUtils; >>>>>>>> +import org.xml.sax.SAXParseException; >>>>>>>> >>>>>>>> -import javax.xml.transform.TransformerException; >>>>>>>> -import java.io.IOException; >>>>>>>> import java.io.InputStream; >>>>>>>> >>>>>>>> /** >>>>>>>> @@ -50,12 +54,15 @@ >>>>>>>> >>>>>>>> private String userUid; >>>>>>>> >>>>>>>> + >>>>>>>> // >>>>>>>> ------------------------------------------------------------------------- >>>>>>>> // Dependencies >>>>>>>> // >>>>>>>> ------------------------------------------------------------------------- >>>>>>>> >>>>>>>> private GmlImportService gmlImportService; >>>>>>>> >>>>>>>> + private Notifier notifier; >>>>>>>> + >>>>>>>> private ImportOptions importOptions; >>>>>>>> >>>>>>>> private InputStream inputStream; >>>>>>>> @@ -64,11 +71,12 @@ >>>>>>>> // Constructors >>>>>>>> // >>>>>>>> ------------------------------------------------------------------------- >>>>>>>> >>>>>>>> - public ImportMetaDataGmlTask( String userUid, GmlImportService >>>>>>>> gmlImportService, >>>>>>>> + public ImportMetaDataGmlTask( String userUid, GmlImportService >>>>>>>> gmlImportService, Notifier notifier, >>>>>>>> ImportOptions importOptions, InputStream inputStream, >>>>>>>> TaskId taskId ) >>>>>>>> { >>>>>>>> this.userUid = userUid; >>>>>>>> this.gmlImportService = gmlImportService; >>>>>>>> + this.notifier = notifier; >>>>>>>> this.importOptions = importOptions; >>>>>>>> this.inputStream = inputStream; >>>>>>>> this.taskId = taskId; >>>>>>>> @@ -83,15 +91,37 @@ >>>>>>>> { >>>>>>>> importOptions.setImportStrategy( "update" ); // Force >>>>>>>> update only for GML import >>>>>>>> >>>>>>>> - try >>>>>>>> - { >>>>>>>> - gmlImportService.importGml( inputStream, userUid, >>>>>>>> importOptions, taskId ); >>>>>>>> - } >>>>>>>> - catch ( IOException | TransformerException e ) >>>>>>>> - { >>>>>>>> - log.error( "Unable to read GML data from input >>>>>>>> stream", e ); >>>>>>>> - >>>>>>>> - throw new RuntimeException( "Failed to parse GML input >>>>>>>> stream", e ); >>>>>>>> - } >>>>>>>> + GmlPreProcessingResult gmlPreProcessingResult = >>>>>>>> gmlImportService.preProcessGml( inputStream ); >>>>>>>> + >>>>>>>> + if ( !gmlPreProcessingResult.isSuccess() ) >>>>>>>> + { >>>>>>>> + Throwable throwable = >>>>>>>> gmlPreProcessingResult.getThrowable(); >>>>>>>> + String message = createErrorMessage( throwable ); >>>>>>>> + >>>>>>>> + notifier.notify( taskId, NotificationLevel.ERROR, >>>>>>>> message, false ); >>>>>>>> + log.error( "GML import failed: " + message, throwable >>>>>>>> ); >>>>>>>> + >>>>>>>> + return; >>>>>>>> + } >>>>>>>> + >>>>>>>> + gmlImportService.importGml( >>>>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, >>>>>>>> taskId >>>>>>>> ); >>>>>>>> + } >>>>>>>> + >>>>>>>> + private String createErrorMessage( Throwable throwable ) >>>>>>>> + { >>>>>>>> + String message = ""; >>>>>>>> + Throwable rootThrowable = ExceptionUtils.getRootCause( >>>>>>>> throwable ); >>>>>>>> + >>>>>>>> + if ( rootThrowable instanceof SAXParseException ) >>>>>>>> + { >>>>>>>> + SAXParseException e = (SAXParseException) >>>>>>>> rootThrowable; >>>>>>>> + message += "Syntax error on line " + e.getLineNumber() >>>>>>>> + ". " + e.getMessage(); >>>>>>>> + } >>>>>>>> + else >>>>>>>> + { >>>>>>>> + message += rootThrowable.getMessage(); >>>>>>>> + } >>>>>>>> + >>>>>>>> + return HtmlUtils.htmlEscape( message ); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties' >>>>>>>> --- >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties >>>>>>>> 2015-02-20 11:17:31 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -300,6 +300,7 @@ >>>>>>>> ignored=Ignored >>>>>>>> conflicts=Conflicts >>>>>>>> no_conflicts_found=No conflicts found >>>>>>>> +no_import_summary_available=No summary available >>>>>>>> type=Type >>>>>>>> count=Count >>>>>>>> export_as_xml=Export as XML >>>>>>>> >>>>>>>> === modified file >>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm' >>>>>>>> --- >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm >>>>>>>> 2013-03-21 09:15:24 +0000 >>>>>>>> +++ >>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm >>>>>>>> 2015-06-05 12:56:34 +0000 >>>>>>>> @@ -1,4 +1,6 @@ >>>>>>>> <h3>$i18n.getString( "import_summary" )</h3> >>>>>>>> + >>>>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML import >>>>>>>> (pre-processing fails) >>>>>>>> <h4>$i18n.getString( "import_count" )</h4> >>>>>>>> >>>>>>>> $summary.importCount.imported Imported<br/> >>>>>>>> @@ -54,3 +56,6 @@ >>>>>>>> #else >>>>>>>> <p>$i18n.getString( "no_conflicts_found" )</p> >>>>>>>> #end >>>>>>>> +#else >>>>>>>> + $i18n.getString( "no_summary_available" ) >>>>>>>> +#end >>>>>>>> \ No newline at end of file >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> ******************************************* >>>>>>> >>>>>>> Calle Hedberg >>>>>>> >>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>>>>> >>>>>>> Tel/fax (home): +27-21-685-6472 >>>>>>> >>>>>>> Cell: +27-82-853-5352 >>>>>>> >>>>>>> Iridium SatPhone: +8816-315-19274 >>>>>>> >>>>>>> Email: calle.hedb...@gmail.com >>>>>>> >>>>>>> Skype: calle_hedberg >>>>>>> >>>>>>> ******************************************* >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> ******************************************* >>>>> >>>>> Calle Hedberg >>>>> >>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>>> >>>>> Tel/fax (home): +27-21-685-6472 >>>>> >>>>> Cell: +27-82-853-5352 >>>>> >>>>> Iridium SatPhone: +8816-315-19274 >>>>> >>>>> Email: calle.hedb...@gmail.com >>>>> >>>>> Skype: calle_hedberg >>>>> >>>>> ******************************************* >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~dhis2-devs >>>> Post to : dhis2-devs@lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>> >> >> >> -- >> >> ******************************************* >> >> Calle Hedberg >> >> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >> >> Tel/fax (home): +27-21-685-6472 >> >> Cell: +27-82-853-5352 >> >> Iridium SatPhone: +8816-315-19274 >> >> Email: calle.hedb...@gmail.com >> >> Skype: calle_hedberg >> >> ******************************************* >> >> > > > -- > > ******************************************* > > Calle Hedberg > > 46D Alma Road, 7700 Rosebank, SOUTH AFRICA > > Tel/fax (home): +27-21-685-6472 > > Cell: +27-82-853-5352 > > Iridium SatPhone: +8816-315-19274 > > Email: calle.hedb...@gmail.com > > Skype: calle_hedberg > > ******************************************* > > _______________________________________________ > Mailing list: https://launchpad.net/~dhis2-devs > Post to : dhis2-devs@lists.launchpad.net > Unsubscribe : https://launchpad.net/~dhis2-devs > More help : https://help.launchpad.net/ListHelp > > > > _______________________________________________ > Mailing list: https://launchpad.net/~dhis2-devs > Post to : dhis2-devs@lists.launchpad.net > Unsubscribe : https://launchpad.net/~dhis2-devs > More help : https://help.launchpad.net/ListHelp > > -- Knut Staring Dept. of Informatics, University of Oslo Norway: +4791880522 Skype: knutstar http://dhis2.org
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
