Hello Morten, Thanks for the advice. For what I know the server looks big enough (4gb mem). I might try to scale it to 8gb change some vm parameters and see how it works. But knowing that I have the backup solution to split it in level is always good.
Regards, Nicolas On Fri, Oct 23, 2015 at 3:01 AM, Morten Olav Hansen <morte...@gmail.com> wrote: > It's also fine splitting it up in many small pieces. As you say, you must > ensure that the parents exists, so doing it one level at the time is the > approach. If not, you must do import multiple times, so it can update the > parents (not really a proper solution). > > Does the server have sufficient memory? because 37k OUs (8mb) should not > really be a problem.. I have imported 50-60mb on my local machine before > without any issues (although that was more than just OUs) > > -- > Morten > > On Fri, Oct 23, 2015 at 12:21 AM, Nicolas Germeau <ngerm...@bluesquare.org > > wrote: > >> Jason, >> >> I guess I'm going on the XSLT solution and split it per level, this is >> the fastest line for me. >> >> Thanks a lot for the advices. Enjoy you mushrooms wanderings. >> >> See you >> Nicolas >> >> >> >> On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering < >> jason.p.picker...@gmail.com> wrote: >> >>> Hi Nicolas >>> >>> Not sure there is a tool per se, but there are approaches. Not sure >>> where your data is coming from, but if you are transforming to JSON/XML, it >>> may be just as easy to transform to SQL insert statements. >>> >>> The trick is to do it level by level and be sure you get the parent ID >>> correct. Otherwise, set the parent level and be sure you reset the >>> hibernate sequence of the database to avoid collisions with future inserts. >>> >>> Otherwise, maybe try what you have but in batches of a few thousand at a >>> time since you seem to have it already in an importable format. >>> >>> Thanks. I need to do something other than dhis2 and looking for >>> mushrooms helps to clear the mind. >>> >>> Regards, >>> Jason >>> >>> On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngerm...@bluesquare.org> >>> wrote: >>> >>>> Hello Jason, >>>> >>>> Nope, no stacktrace. I suppose the transaction is too big and blow the >>>> whole thing up. Yep sure, I don't really care about the method as long as >>>> it doesn't take to much time to setup. I'm thinking about creating a quick >>>> XSLT to extract org unit per level but if you have any tool/idea which >>>> allow direct SQL I'm in. >>>> >>>> Nice mushroom picture by the way. >>>> Regards. >>>> Nicolas >>>> >>>> >>>> >>>> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering < >>>> jason.p.picker...@gmail.com> wrote: >>>> >>>>> Hi Nicolas, >>>>> >>>>> Hard to say without a stack trace. It just bombs with no explanation? >>>>> >>>>> Could you consider using direct SQL injection for such a large number >>>>> of orgunits? >>>>> >>>>> Regards, >>>>> Jason >>>>> >>>>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngerm...@bluesquare.org> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I'm trying to make a rather large import of orgunit. The size of the >>>>>> file is around 8megs zipped and it contains more than 37000 orgunits. >>>>>> >>>>>> When I'm launching the import it takes a little bit more than an >>>>>> hour. I can see in the log file (dhis.log) outputs telling me that org >>>>>> unit >>>>>> are being imported. At the end I'm receiving the following message in >>>>>> catalina.out >>>>>> >>>>>> * INFO 2015-10-22 15:26:50,878 [Level: INFO, category: >>>>>> METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import >>>>>> done. >>>>>> Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1]) >>>>>> >>>>>> and then the server crash without any message on what when wrong. >>>>>> Nothing is imported. >>>>>> >>>>>> I'm wondering how the import works, can I split the file arbitrary in >>>>>> multiple smaller parts (will it handle the fk parent id correctly) ? Or >>>>>> do >>>>>> I have to split the file per level to ensure parent finding ? >>>>>> >>>>>> Thanks >>>>>> Regards >>>>>> Nicolas >>>>>> >>>>>> >>>>>> -- >>>>>> *Nicolas Germeau* >>>>>> *Software Developer* >>>>>> >>>>>> GSM: +32 487 129820 >>>>>> >>>>>> Tel: +32 2 737 65 16 >>>>>> Skype: nicolas.germeau >>>>>> >>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org> >>>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts> >>>>>> >>>>>> Visit our new website <http://www.bluesquare.org/> >>>>>> _______________________________________________ >>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>> More help : https://help.launchpad.net/ListHelp >>>>>> >>>>> >>>> >>>> >>>> -- >>>> *Nicolas Germeau* >>>> *Software Developer* >>>> >>>> GSM: +32 487 129820 >>>> >>>> Tel: +32 2 737 65 16 >>>> Skype: nicolas.germeau >>>> >>>> Follow us on Twitter <https://twitter.com/bluesquare_org> >>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts> >>>> Visit our new website <http://www.bluesquare.org/> >>>> >>> >> >> >> -- >> *Nicolas Germeau* >> *Software Developer* >> >> GSM: +32 487 129820 >> >> Tel: +32 2 737 65 16 >> Skype: nicolas.germeau >> >> Follow us on Twitter <https://twitter.com/bluesquare_org> >> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts> >> Visit our new website <http://www.bluesquare.org/> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-devs >> Post to : dhis2-devs@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~dhis2-devs >> More help : https://help.launchpad.net/ListHelp >> >> > -- *Nicolas Germeau* *Software Developer* GSM: +32 487 129820 Tel: +32 2 737 65 16 Skype: nicolas.germeau Follow us on Twitter <https://twitter.com/bluesquare_org> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts> Visit our new website <http://www.bluesquare.org/>
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
