Hi Morten, Sultan, I think we know the reason of duplicates: categoryOptionGroups are shared between categoryOptionGroupSets, for example, ">= 15" is used in three categoryOptionGroupSets (Paediatrics, HIV and general). In the API, categoryOptionGroups appear duplicated as many times as they are re-used.
We don't know if sharing categoryOptionGroups is something to avoid, but it seems to be the cause of this issue. Regards, Víctor On 12 September 2016 at 09:41, Morten Olav Hansen <mor...@dhis2.org> wrote: > Hi again > > Would it be possible to share your database with me? I haven't seen this > issues anywhere else. > > -- > Morten Olav Hansen > Senior Engineer, DHIS 2 > University of Oslo > http://www.dhis2.org > > On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad < > sult...@thoughtworks.com> wrote: > > > Hi Morten, > > > > This is what our table is showing up when we say \d dataelement. I think > > there is nothing suspicious out here. > > > > > > > > Regards, > > Sultan Ahamar. > > > > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen <mor...@dhis2.org> > > wrote: > > > >> I'm not sure what is causing it, but it shouldn't be allowed to begin > >> with. Is there any constraint on the UID column in your database? > >> > >> [image: Inline image 1] > >> > >> -- > >> Morten Olav Hansen > >> Senior Engineer, DHIS 2 > >> University of Oslo > >> http://www.dhis2.org > >> > >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad < > >> sult...@thoughtworks.com> wrote: > >> > >>> Hi Morten, > >>> > >>> Hope you are doing good. Let us know your thoughts on the above issue > >>> and please let us know if you need any information that you might > require. > >>> > >>> Regards, > >>> Sultan Ahamar. > >>> > >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad < > >>> sult...@thoughtworks.com> wrote: > >>> > >>>> Hi Morten, > >>>> > >>>> We have pulled latest code from 2.24 and tried import / export again. > >>>> We have some interesting observations > >>>> > >>>> *DB snapshot:* > >>>> > >>>> > >>>> > >>>> There are no duplicate items in DB. We double checked. > >>>> > >>>> > >>>> > >>>> > >>>> *Entity endpoint snapshot:* > >>>> > >>>> Interestingly we found few duplicate entities when we try to fetch > >>>> this data through the API. In Maintenance app, we are not able to see > these > >>>> duplicate entities. Does it also explain why we get repeating entries > (with > >>>> same UID's) in the full export JSON as well? Let us know on how to > proceed > >>>> and if you need more input. > >>>> > >>>> Thanks for all the help in advance. > >>>> > >>>> Regards, > >>>> Sultan Ahamar. > >>>> > >>>> > >>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen <mor...@dhis2.org> > >>>> wrote: > >>>> > >>>>> Hi Vanya > >>>>> > >>>>> This should now have been fixed in master and 224. It will now do a > >>>>> global UID check first, and report back any duplicates, if it finds a > >>>>> duplicate it will remove them from the import and report back an > >>>>> `ErrorReport` (as it would do with other validation issues). So if > it finds > >>>>> a duplicate, you will need to set atomic mode to NONE to have it > import and > >>>>> ignore the duplicates. > >>>>> > >>>>> (btw, there was several duplicates in your file, not sure how that > >>>>> happened.. maybe something needs to be cleaned in your source > instance) > >>>>> > >>>>> -- > >>>>> Morten Olav Hansen > >>>>> Senior Engineer, DHIS 2 > >>>>> University of Oslo > >>>>> http://www.dhis2.org > >>>>> > >>>>> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <mor...@dhis2.org > > > >>>>> wrote: > >>>>> > >>>>>> Hi Vanya > >>>>>> > >>>>>> This should have been caught by the importer... but there are > several > >>>>>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 > category > >>>>>> option groups, same with oqeVQ71LCgY.. > >>>>>> > >>>>>> I will look into making the validation process more robust.. but at > >>>>>> least you should know that this file have issues > >>>>>> > >>>>>> -- > >>>>>> Morten Olav Hansen > >>>>>> Senior Engineer, DHIS 2 > >>>>>> University of Oslo > >>>>>> http://www.dhis2.org > >>>>>> > >>>>>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen < > mor...@dhis2.org> > >>>>>> wrote: > >>>>>> > >>>>>>> Ok, thanks Vanya, I'm looking into it now > >>>>>>> > >>>>>>> -- > >>>>>>> Morten Olav Hansen > >>>>>>> Senior Engineer, DHIS 2 > >>>>>>> University of Oslo > >>>>>>> http://www.dhis2.org > >>>>>>> > >>>>>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth < > van...@thoughtworks.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi Morten > >>>>>>>> > >>>>>>>> Thanks for the response. > >>>>>>>> > >>>>>>>> We ran into another issue while trying to do a full export and > >>>>>>>> import. > >>>>>>>> > >>>>>>>> As per the previous conversations we took care of all data > >>>>>>>> migrations (as per the new constraints on the DB- pretty much > caught by the > >>>>>>>> Importer during the validation phase itself). > >>>>>>>> > >>>>>>>> Having done that we run into this issue: > >>>>>>>> The 'categoryoptiongroup' import is failing with constraint > >>>>>>>> violation for the UID. We checked the data and there is no > repeating UID in > >>>>>>>> the database. > >>>>>>>> > >>>>>>>> The error text is attached for your reference, as well the payload > >>>>>>>> used for the import. > >>>>>>>> > >>>>>>>> Thanks for all the help in advance. > >>>>>>>> > >>>>>>>> Regards > >>>>>>>> Vanya > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen < > >>>>>>>> mor...@dhis2.org> wrote: > >>>>>>>> > >>>>>>>>> Thanks Aamer, I'm looking at a similar bug right now.. seems some > >>>>>>>>> objects don't get their deps properly attached.. > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Morten Olav Hansen > >>>>>>>>> Senior Engineer, DHIS 2 > >>>>>>>>> University of Oslo > >>>>>>>>> http://www.dhis2.org > >>>>>>>>> > >>>>>>>>> On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed < > >>>>>>>>> aam...@thoughtworks.com> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Morten, > >>>>>>>>>> > >>>>>>>>>> Please find the payload for below request > >>>>>>>>>> *curl -H "Content-Type: application/json" -X GET -u > >>>>>>>>>> username:password > >>>>>>>>>> "http://localhost:8888/api/24/metadata?filter=created:gt: > 2016-07 > >>>>>>>>>> <http://localhost:8888/api/24/metadata?filter=created:gt: > 2016-07>" > > >>>>>>>>>> metadata.224.newapi.created.json* > >>>>>>>>>> > >>>>>>>>>> Import the payload in a fresh instance for import using > >>>>>>>>>> > >>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data > >>>>>>>>>> @metadata.224.newapi.created.json -u admin:district > >>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE > >>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” > > >>>>>>>>>> output_created_besteffort.txt* > >>>>>>>>>> > >>>>>>>>>> If the same payload is run with atomicMode=ALL, I am getting > >>>>>>>>>> 'Invalid references' errors which is acceptable. But if it is > run with > >>>>>>>>>> atomicMode=NONE, it is throwing the error for which the > complete stack > >>>>>>>>>> trace is attached earlier. filename: ' > >>>>>>>>>> *output_created_besteffort_trace.txt'* > >>>>>>>>>> > >>>>>>>>>> Let me know if the complete payload or any other details are > >>>>>>>>>> required from my end. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> Aamer. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Sep 1, 2016 at 8:52 AM, Morten Olav Hansen < > >>>>>>>>>> mor...@dhis2.org> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Aamer > >>>>>>>>>>> > >>>>>>>>>>> Could you please share the payload of the object where this > >>>>>>>>>>> happen? > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Morten Olav Hansen > >>>>>>>>>>> Senior Engineer, DHIS 2 > >>>>>>>>>>> University of Oslo > >>>>>>>>>>> http://www.dhis2.org > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Aug 26, 2016 at 9:13 PM, Aamer Mohammed < > >>>>>>>>>>> aam...@thoughtworks.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Morten, > >>>>>>>>>>>> > >>>>>>>>>>>> The Importer was run with atomicMode=ALL. Though the payload > is > >>>>>>>>>>>> huge, only few errors were shown in ImportSummary. As per > errors in > >>>>>>>>>>>> previous ImportSummary, I have done below. > >>>>>>>>>>>> 1) ensured the 'short name' field is unique for each of the > >>>>>>>>>>>> 'data elements' and 'data element group' as well > >>>>>>>>>>>> 2) UID of admin, Super User, tracked entity was changed in > >>>>>>>>>>>> payload to be same to where it is getting imported. > >>>>>>>>>>>> I have manually resolved them and ran the importer again with > >>>>>>>>>>>> atomicMode=ALL. > >>>>>>>>>>>> > >>>>>>>>>>>> I got exception as attached in trace file > >>>>>>>>>>>> 'output_besteffort_trace.txt' > >>>>>>>>>>>> Would these kind of issues be known only after the import has > >>>>>>>>>>>> failed? and we need to resolve them one-by-one and > as-and-when the importer > >>>>>>>>>>>> throws them? Is there any note which has the details about > the constraints > >>>>>>>>>>>> being introduced in a new DHIS version? > >>>>>>>>>>>> > >>>>>>>>>>>> Alternatively, If we export a filtered set of metadata using ' > >>>>>>>>>>>> /api/24/metadata?filter=lastUpdated:gt:2016-05' and run the > >>>>>>>>>>>> importer with atomicMode=NONE > >>>>>>>>>>>> > >>>>>>>>>>>> *curl -H "Content-Type: application/json" -X POST --data > >>>>>>>>>>>> @metadata.224.newapi.json -u admin:district > >>>>>>>>>>>> “http://localhost:8080/api/24/metadata?atomicMode=NONE > >>>>>>>>>>>> <http://localhost:8080/api/24/metadata?atomicMode=NONE>” > > >>>>>>>>>>>> output_created_besteffort_trace.txt* > >>>>>>>>>>>> > >>>>>>>>>>>> Getting below exception in trace. Complete trace also > attached. > >>>>>>>>>>>> ----- > >>>>>>>>>>>> * INFO 2016-08-26 15:58:38,217 (admin) Creating 57 object(s) > of > >>>>>>>>>>>> type ReportTable (DefaultObjectBundleService.java > >>>>>>>>>>>> [qtp289378424-12]) org.hibernate.TransientObjectException: > >>>>>>>>>>>> object references an unsaved transient instance - save the > transient > >>>>>>>>>>>> instance before flushing: org.hisp.dhis.indicator.Indicator > >>>>>>>>>>>> ------ > >>>>>>>>>>>> Any suggestions please. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> Aamer. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Aug 23, 2016 at 11:19 AM, Morten Olav Hansen < > >>>>>>>>>>>> mor...@dhis2.org> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Aamer > >>>>>>>>>>>>> > >>>>>>>>>>>>> Probably what is happening is that the new database has some > >>>>>>>>>>>>> constraints that the old one doesn't. We have had some > issues with > >>>>>>>>>>>>> hibernate in the past, and for certain databases that means > that many of > >>>>>>>>>>>>> the constraints was not applied.. > >>>>>>>>>>>>> > >>>>>>>>>>>>> If you look at the messages, you will see that e.g > `shortName` > >>>>>>>>>>>>> must be unique for data elements.. the list of errors is not > big though, > >>>>>>>>>>>>> probably you can go through > [truncated for moderation] > _______________________________________________ > Mailing list: https://launchpad.net/~dhis2-devs > Post to : dhis2-devs@lists.launchpad.net > Unsubscribe : https://launchpad.net/~dhis2-devs > More help : https://help.launchpad.net/ListHelp > >
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
