Re: [Dhis2-devs] Upgrading DHIS2 from 2.21 to 2.24 and using new exporter/importer

[Sultanahamar Mohammad]

Hi Morten,

This is what our table is showing up when we say \d dataelement. I think
there is nothing suspicious out here.


​
Regards,
Sultan Ahamar.

On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen  wrote:

> I'm not sure what is causing it, but it shouldn't be allowed to begin
> with. Is there any constraint on the UID column in your database?
>
> [image: Inline image 1]
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
> sult...@thoughtworks.com> wrote:
>
>> Hi Morten,
>>
>> Hope you are doing good. Let us know your thoughts on the above issue and
>> please let us know if you need any information that you might require.
>>
>> Regards,
>> Sultan Ahamar.
>>
>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>> sult...@thoughtworks.com> wrote:
>>
>>> Hi Morten,
>>>
>>> We have pulled latest code from 2.24 and tried import /  export again.
>>> We have some interesting observations
>>>
>>> *DB snapshot:*
>>>
>>>
>>> ​
>>> There are no duplicate items in DB. We double checked.
>>>
>>>
>>>
>>>
>>> *Entity endpoint snapshot:*
>>>
>>> ​Interestingly we found few duplicate entities when we try to fetch this
>>> data through the API. In Maintenance app, we are not able to see these
>>> duplicate entities. Does it also explain why we get repeating entries (with
>>> same UID's) in the full export JSON as well?  Let us know on how to proceed
>>> and if you need more input.
>>>
>>> Thanks for all the help in advance.
>>>
>>> Regards,
>>> Sultan Ahamar.
>>>
>>>
>>> On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen 
>>> wrote:
>>>
 Hi Vanya

 This should now have been fixed in master and 224. It will now do a
 global UID check first, and report back any duplicates, if it finds a
 duplicate it will remove them from the import and report back an
 `ErrorReport` (as it would do with other validation issues). So if it finds
 a duplicate, you will need to set atomic mode to NONE to have it import and
 ignore the duplicates.

 (btw, there was several duplicates in your file, not sure how that
 happened.. maybe something needs to be cleaned in your source instance)

 --
 Morten Olav Hansen
 Senior Engineer, DHIS 2
 University of Oslo
 http://www.dhis2.org

 On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen 
 wrote:

> Hi Vanya
>
> This should have been caught by the importer... but there are several
> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 category
> option groups, same with oqeVQ71LCgY..
>
> I will look into making the validation process more robust.. but at
> least you should know that this file have issues
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen 
> wrote:
>
>> Ok, thanks Vanya, I'm looking into it now
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth 
>> wrote:
>>
>>> Hi Morten
>>>
>>> Thanks for the response.
>>>
>>> We ran into another issue while trying to do a full export and
>>> import.
>>>
>>> As per the previous conversations we took care of all data
>>> migrations (as per the new constraints on the DB- pretty much caught by 
>>> the
>>> Importer during the validation phase itself).
>>>
>>> Having done that we run into this issue:
>>> The 'categoryoptiongroup' import is failing with constraint
>>> violation for the UID. We checked the data and there is no repeating 
>>> UID in
>>> the  database.
>>>
>>> The error text is attached for your reference, as well the payload
>>> used for the import.
>>>
>>> Thanks for all the help in advance.
>>>
>>> Regards
>>> Vanya
>>>
>>>
>>>
>>> On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
>>> mor...@dhis2.org> wrote:
>>>
 Thanks Aamer, I'm looking at a similar bug right now.. seems some
 objects don't get their deps properly attached..

 --
 Morten Olav Hansen
 Senior Engineer, DHIS 2
 University of Oslo
 http://www.dhis2.org

 On Thu, Sep 1, 2016 at 12:36 PM, Aamer Mohammed <
 aam...@thoughtworks.com> wrote:

> Hi Morten,
>
> Please find the payload for below request
> *curl -H "Content-Type: application/json" -X GET -u
> username:password
> "http://localhost:/api/24/metadata?filter=created:gt:2016-07
> " >
> metadata.224.newapi.created.json*
>
> Import the 

Re: [Dhis2-devs] Upgrading DHIS2 from 2.21 to 2.24 and using new exporter/importer

[Morten Olav Hansen]

Hi again

Would it be possible to share your database with me? I haven't seen this
issues anywhere else.

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
sult...@thoughtworks.com> wrote:

> Hi Morten,
>
> This is what our table is showing up when we say \d dataelement. I think
> there is nothing suspicious out here.
>
>
> ​
> Regards,
> Sultan Ahamar.
>
> On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen 
> wrote:
>
>> I'm not sure what is causing it, but it shouldn't be allowed to begin
>> with. Is there any constraint on the UID column in your database?
>>
>> [image: Inline image 1]
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>> sult...@thoughtworks.com> wrote:
>>
>>> Hi Morten,
>>>
>>> Hope you are doing good. Let us know your thoughts on the above issue
>>> and please let us know if you need any information that you might require.
>>>
>>> Regards,
>>> Sultan Ahamar.
>>>
>>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>>> sult...@thoughtworks.com> wrote:
>>>
 Hi Morten,

 We have pulled latest code from 2.24 and tried import /  export again.
 We have some interesting observations

 *DB snapshot:*


 ​
 There are no duplicate items in DB. We double checked.




 *Entity endpoint snapshot:*

 ​Interestingly we found few duplicate entities when we try to fetch
 this data through the API. In Maintenance app, we are not able to see these
 duplicate entities. Does it also explain why we get repeating entries (with
 same UID's) in the full export JSON as well?  Let us know on how to proceed
 and if you need more input.

 Thanks for all the help in advance.

 Regards,
 Sultan Ahamar.


 On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen 
 wrote:

> Hi Vanya
>
> This should now have been fixed in master and 224. It will now do a
> global UID check first, and report back any duplicates, if it finds a
> duplicate it will remove them from the import and report back an
> `ErrorReport` (as it would do with other validation issues). So if it 
> finds
> a duplicate, you will need to set atomic mode to NONE to have it import 
> and
> ignore the duplicates.
>
> (btw, there was several duplicates in your file, not sure how that
> happened.. maybe something needs to be cleaned in your source instance)
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen 
> wrote:
>
>> Hi Vanya
>>
>> This should have been caught by the importer... but there are several
>> duplicates without the file you sent me, MaO4Ik8f34O is used in 3 
>> category
>> option groups, same with oqeVQ71LCgY..
>>
>> I will look into making the validation process more robust.. but at
>> least you should know that this file have issues
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen 
>> wrote:
>>
>>> Ok, thanks Vanya, I'm looking into it now
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth 
>>> wrote:
>>>
 Hi Morten

 Thanks for the response.

 We ran into another issue while trying to do a full export and
 import.

 As per the previous conversations we took care of all data
 migrations (as per the new constraints on the DB- pretty much caught 
 by the
 Importer during the validation phase itself).

 Having done that we run into this issue:
 The 'categoryoptiongroup' import is failing with constraint
 violation for the UID. We checked the data and there is no repeating 
 UID in
 the  database.

 The error text is attached for your reference, as well the payload
 used for the import.

 Thanks for all the help in advance.

 Regards
 Vanya



 On Thu, Sep 1, 2016 at 11:10 AM, Morten Olav Hansen <
 mor...@dhis2.org> wrote:

> Thanks Aamer, I'm looking at a similar bug right now.. seems some
> objects don't get their deps properly attached..
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Thu, Sep 1, 2016 at 12:3

[Dhis2-devs] OU Group does not filter in Analytics (pivot table, Data viz, etc.)

[Derek Kahongo Mukenge]

Hi Everybody,

I have a OU group set named Coordination team that have 8 groups that group
Health district. Some of those group have the same health district as
Province level. When I generate a pivot table filtered by OU (province
level), I get a result different from when I filtered based on Coordination
Group. The bad side of this is that I got no value in a customized Html
standard report that work with customized OU group.

I think it's a bug, but if someone have faced this before can tell me how
to tacle its

Here is my config:

Version:
2.23
Build revision:
22980
Build date:
2016-05-31 05:47
Jasper reports version:
6.1.0
User agent:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/52.0.2743.116 Safari/537.36

Java version:
1.8.0_91
Java vendor:
Oracle Corporation
OS name:
Linux
OS architecture:
amd64
OS version:
2.6.32-504.12.2.el6.x86_64
Server memory:
Mem Total in JVM: 1166 Free in JVM: 744 Max Limit: 1166
CPU cores:
2
Cordially,
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Groups - Tables

[Raminosoa Rabemanantsoa, Tantely]

Hi Jason,

Thank your advice.

Regards,

Tantely.

On Mon, Sep 12, 2016 at 9:31 AM, Jason Pickering <
jason.p.picker...@gmail.com> wrote:

> Hi Tantely,
>
> 1- the table where are stored all organization units groups with the list
> of their members.
>
> \d orgunitgroupmembers
>
> 2- the table where are stored the datasets assigned to the organization
> units
>
> \d datasetsource
>
> 3- the table where are stored the organization units assigned to all users
>
> \d usermembership
>
> You should not really try and update these tables manually. Its much
> better to use the API.
>
>
> Regards,
> Jason
>
>
>
> On Mon, Sep 12, 2016 at 7:52 AM, Raminosoa Rabemanantsoa, Tantely <
> tramino...@mikolo.org> wrote:
>
>> Dear Members,
>>
>> I am looking for 3 database tables:
>>
>> 1- the table where are stored all organization units groups with the list
>> of their members.
>> 2- the table where are stored the datasets assigned to the organization
>> units
>> 3- the table where are stored the organization units assigned to all
>> users
>>
>> Does anyone know in which tables should I query to select and update
>> their records? I am using DHIS2 2.19 and I connect to the DHIS2 database
>> directly with psql.
>>
>> Thank you for your advice.
>>
>> Regards,
>>
>> Tantely.
>>
>> *This message and its attachments are confidential and solely for the
>> intended recipients. If received in error, please delete them and notify
>> the sender via reply e-mail immediately.*
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
> --
> Jason P. Pickering
> email: jason.p.picker...@gmail.com
> tel:+46764147049
>

-- 
*This message and its attachments are confidential and solely for the 
intended recipients. If received in error, please delete them and notify 
the sender via reply e-mail immediately.*
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Render Sections as Tab option in program/stage

[Alex Tumwesigye]

Dear Nayeem,

Try the Tabular Widget. It has similar functionality. You can turn it on
from the Tracker Dashboard (Default is DataEntry Widget)

Alex

On Sun, Sep 11, 2016 at 8:54 AM, Nayeem Al Mifthah 
wrote:

> Dear Developers
>
> It would be very helpful if you could add "Render sections as tab"
> function in program stage management. It will enable to view sections in
> tab on the time of tracker capture and event capture data entry.
>
> It is available for aggregated dataset management.
>
>
> *Nayeem Al Mifthah*
> *Consultant for HSS, Health MIS, M&E*
> Email: n.mift...@gmail.com
> Cell:+8801914030574
> Skype: nayeem.a.m
>
> ___
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET  | HISP Uganda
Kampala
Uganda
+256 774149 775, + 256 759 800161
Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems -
DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant


"I don't want to be anything other than what I have been - one tree hill "
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Analytics and disk space

[Jason Pickering]

Hi Lars,

You are right in that most of this is available via the API, but making a
little bit easier for people through the UI would seem to be a quick win
really. Previously when the datamart was in use, one could choose specific
time periods to regenerate data for, but no longer.Currently, users who use
the UI can only execute a full-analytics run, which is really the basis of
this request. In this particular situation, a full analytics run caused the
server to choke up, due to lack of disk space and the manner in which the
analytics tables are (re)generated.

 Hopefully the new scheduling functionality will allow for better control
over analytics, without users always having to resort to a curl call to the
API.

Regards,
Jason


On Mon, Sep 12, 2016 at 8:42 AM, Lars Helge Øverland  wrote:

> Hi there,
>
> thanks for the feedback. Most of what's requested is available in the API.
> It's on our list to rewrite the import-export app and write a better
> scheduling manager for background tasks such analytics generation.
>
> In the meantime:
>
> - Analytics tables generation
> 
>  for
> last x years
> - Data value export
> 
> (lastUpdated, lastUpdatedDuration, orgUnit params)
>
>
> regards,
>
> Lars
>
>
>
> On Sun, Sep 11, 2016 at 5:20 PM, David Siang Fong Oh  > wrote:
>
>> I think Jason also pointed out that this could be achieved from the API,
>> but the question is whether it needs to be more user-friendly, i.e.
>> customisable using the web application as opposed to requiring a custom
>> script triggered by a cron job.
>>
>> Cheers,
>>
>> -doh
>>
>> On Sun, Sep 11, 2016 at 8:36 PM, Dan Cocos  wrote:
>>
>>> Hi All,
>>>
>>> You could run this
>>> /api/24/maintenance/analyticsTablesClear
>>> and this possibly this
>>> /api/24/maintenance/periodPruning
>>>
>>> I don't see it in the documentation but we use call this
>>>  /api/resourceTables/analytics?lastYears=2 quite often for clients with
>>> a lot of historical data.
>>>
>>> Good luck,
>>> Dan
>>>
>>> *Dan Cocos*
>>> Principal, BAO Systems
>>> dco...@baosystems.com  | http:/
>>> /www.baosystems.com |  2900 K Street, Suite 404, Washington D.C. 20007
>>>
>>>
>>>
>>>
>>>
>>> On Sep 11, 2016, at 10:05 AM, Calle Hedberg 
>>> wrote:
>>>
>>> Hi,
>>>
>>> It's not only analytics that would benefit from segmented/staggered
>>> processing: I exported around 100 mill data values yesterday from a number
>>> of instance, and found that the export process was (seemingly)
>>> exponentially slower with increasing number of records exported. Most of
>>> the export files contained well under 10 mill records, which was pretty
>>> fast. In comparison, the largest export file with around 30 mill data
>>> values probably took 20 times as much time as an 8 mill value export. Based
>>> on just keeping an eye on the "progress bar", it seemed like some kind of
>>> cache staggering was taking place - the amount exported would increase
>>> quickly by 2-3mb, then "hang" for a good while, then increase quickly by
>>> 2-3mb again.
>>>
>>> Note also that there are several fundamental strategies one could use to
>>> reducing heavy work processes like analytics, exports (and thus imports),
>>> etc:
>>> - to be able to specify a sub-period as Jason's suggest
>>> - to be able to specify the "dirty" part of the instance by using e.g.
>>> LastUpdated >= x
>>> - to be able to specify a sub-OrgUnit-area
>>>
>>> These partial strategies are of course mostly relevant for very large
>>> instances, but such large instances are also the ones where you typically
>>> only have changes made to a small segment of the total - like if you have
>>> data for 30 years, 27 of those might be locked down and no longer available
>>> for updates.
>>>
>>> Regards
>>> Calle
>>>
>>> On 11 September 2016 at 15:47, David Siang Fong Oh >> > wrote:
>>>
 +1 to Calle's idea of staggering analytics year by year

 I also like Jason's suggestion of being able to configure the time
 period for which analytics is regenerated. If the general use-case has data
 being entered only for the current year, then is it perhaps unnecessary to
 regenerate data for previous years?

 Cheers,

 -doh

 On Tue, Jul 26, 2016 at 2:36 PM, Calle Hedberg >>> > wrote:

> Hi,
>
> One (presumably) simple solution is to stagger analytics on a year by
> year basis - i.e. run and complete 2009 before processing 2010. That would
> reduce temp disk space requirements significantly while (presumably) not
> changing the general design.
>
> Regards
> Calle
>
> On 26 July 2016 at 10:24, Jason Pickering  > wrote:
>
>> Hi Devs,
>> I am seeking some advice on how to try and decrease the amount of
>> disk usage with DHIS2.
>>
>> Here is a l

Re: [Dhis2-devs] Upgrading DHIS2 from 2.21 to 2.24 and using new exporter/importer

[Victor Garcia]

Hi Morten, Sultan,

I think we know the reason of duplicates: categoryOptionGroups are shared
between categoryOptionGroupSets, for example, ">= 15" is used in three
categoryOptionGroupSets (Paediatrics, HIV and general). In the API,
categoryOptionGroups appear duplicated as many times as they are re-used.

We don't know if sharing categoryOptionGroups is something to avoid, but it
seems to be the cause of this issue.

Regards,

Víctor

On 12 September 2016 at 09:41, Morten Olav Hansen  wrote:

> Hi again
>
> Would it be possible to share your database with me? I haven't seen this
> issues anywhere else.
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
> sult...@thoughtworks.com> wrote:
>
> > Hi Morten,
> >
> > This is what our table is showing up when we say \d dataelement. I think
> > there is nothing suspicious out here.
> >
> >
> > ​
> > Regards,
> > Sultan Ahamar.
> >
> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen 
> > wrote:
> >
> >> I'm not sure what is causing it, but it shouldn't be allowed to begin
> >> with. Is there any constraint on the UID column in your database?
> >>
> >> [image: Inline image 1]
> >>
> >> --
> >> Morten Olav Hansen
> >> Senior Engineer, DHIS 2
> >> University of Oslo
> >> http://www.dhis2.org
> >>
> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
> >> sult...@thoughtworks.com> wrote:
> >>
> >>> Hi Morten,
> >>>
> >>> Hope you are doing good. Let us know your thoughts on the above issue
> >>> and please let us know if you need any information that you might
> require.
> >>>
> >>> Regards,
> >>> Sultan Ahamar.
> >>>
> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
> >>> sult...@thoughtworks.com> wrote:
> >>>
>  Hi Morten,
> 
>  We have pulled latest code from 2.24 and tried import /  export again.
>  We have some interesting observations
> 
>  *DB snapshot:*
> 
> 
>  ​
>  There are no duplicate items in DB. We double checked.
> 
> 
> 
> 
>  *Entity endpoint snapshot:*
> 
>  ​Interestingly we found few duplicate entities when we try to fetch
>  this data through the API. In Maintenance app, we are not able to see
> these
>  duplicate entities. Does it also explain why we get repeating entries
> (with
>  same UID's) in the full export JSON as well?  Let us know on how to
> proceed
>  and if you need more input.
> 
>  Thanks for all the help in advance.
> 
>  Regards,
>  Sultan Ahamar.
> 
> 
>  On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen 
>  wrote:
> 
> > Hi Vanya
> >
> > This should now have been fixed in master and 224. It will now do a
> > global UID check first, and report back any duplicates, if it finds a
> > duplicate it will remove them from the import and report back an
> > `ErrorReport` (as it would do with other validation issues). So if
> it finds
> > a duplicate, you will need to set atomic mode to NONE to have it
> import and
> > ignore the duplicates.
> >
> > (btw, there was several duplicates in your file, not sure how that
> > happened.. maybe something needs to be cleaned in your source
> instance)
> >
> > --
> > Morten Olav Hansen
> > Senior Engineer, DHIS 2
> > University of Oslo
> > http://www.dhis2.org
> >
> > On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen  >
> > wrote:
> >
> >> Hi Vanya
> >>
> >> This should have been caught by the importer... but there are
> several
> >> duplicates without the file you sent me, MaO4Ik8f34O is used in 3
> category
> >> option groups, same with oqeVQ71LCgY..
> >>
> >> I will look into making the validation process more robust.. but at
> >> least you should know that this file have issues
> >>
> >> --
> >> Morten Olav Hansen
> >> Senior Engineer, DHIS 2
> >> University of Oslo
> >> http://www.dhis2.org
> >>
> >> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
> mor...@dhis2.org>
> >> wrote:
> >>
> >>> Ok, thanks Vanya, I'm looking into it now
> >>>
> >>> --
> >>> Morten Olav Hansen
> >>> Senior Engineer, DHIS 2
> >>> University of Oslo
> >>> http://www.dhis2.org
> >>>
> >>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
> van...@thoughtworks.com>
> >>> wrote:
> >>>
>  Hi Morten
> 
>  Thanks for the response.
> 
>  We ran into another issue while trying to do a full export and
>  import.
> 
>  As per the previous conversations we took care of all data
>  migrations (as per the new constraints on the DB- pretty much
> caught by the
>  Importer during the validation phase itself).
> 
>  Having done that we run into this issue:
>  The 'categoryoptiongroup'

Re: [Dhis2-devs] Upgrading DHIS2 from 2.21 to 2.24 and using new exporter/importer

[Morten Olav Hansen]

That's interesting.

@Lars any suggestions why this happens? I guess there is something wrong
with our mapping. Will have a look tomorrow.

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Sep 12, 2016 at 5:42 PM, Victor Garcia  wrote:

> Hi Morten, Sultan,
>
> I think we know the reason of duplicates: categoryOptionGroups are shared
> between categoryOptionGroupSets, for example, ">= 15" is used in three
> categoryOptionGroupSets (Paediatrics, HIV and general). In the API,
> categoryOptionGroups appear duplicated as many times as they are re-used.
>
> We don't know if sharing categoryOptionGroups is something to avoid, but
> it seems to be the cause of this issue.
>
> Regards,
>
> Víctor
>
> On 12 September 2016 at 09:41, Morten Olav Hansen 
> wrote:
>
>> Hi again
>>
>> Would it be possible to share your database with me? I haven't seen this
>> issues anywhere else.
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Mon, Sep 12, 2016 at 2:28 PM, Sultanahamar Mohammad <
>> sult...@thoughtworks.com> wrote:
>>
>> > Hi Morten,
>> >
>> > This is what our table is showing up when we say \d dataelement. I think
>> > there is nothing suspicious out here.
>> >
>> >
>> > ​
>> > Regards,
>> > Sultan Ahamar.
>> >
>> > On Fri, Sep 9, 2016 at 2:13 PM, Morten Olav Hansen 
>> > wrote:
>> >
>> >> I'm not sure what is causing it, but it shouldn't be allowed to begin
>> >> with. Is there any constraint on the UID column in your database?
>> >>
>> >> [image: Inline image 1]
>> >>
>> >> --
>> >> Morten Olav Hansen
>> >> Senior Engineer, DHIS 2
>> >> University of Oslo
>> >> http://www.dhis2.org
>> >>
>> >> On Fri, Sep 9, 2016 at 12:31 PM, Sultanahamar Mohammad <
>> >> sult...@thoughtworks.com> wrote:
>> >>
>> >>> Hi Morten,
>> >>>
>> >>> Hope you are doing good. Let us know your thoughts on the above issue
>> >>> and please let us know if you need any information that you might
>> require.
>> >>>
>> >>> Regards,
>> >>> Sultan Ahamar.
>> >>>
>> >>> On Tue, Sep 6, 2016 at 6:14 PM, Sultanahamar Mohammad <
>> >>> sult...@thoughtworks.com> wrote:
>> >>>
>>  Hi Morten,
>> 
>>  We have pulled latest code from 2.24 and tried import /  export
>> again.
>>  We have some interesting observations
>> 
>>  *DB snapshot:*
>> 
>> 
>>  ​
>>  There are no duplicate items in DB. We double checked.
>> 
>> 
>> 
>> 
>>  *Entity endpoint snapshot:*
>>
>> 
>>  ​Interestingly we found few duplicate entities when we try to fetch
>>  this data through the API. In Maintenance app, we are not able to
>> see these
>>  duplicate entities. Does it also explain why we get repeating
>> entries (with
>>  same UID's) in the full export JSON as well?  Let us know on how to
>> proceed
>>  and if you need more input.
>> 
>>  Thanks for all the help in advance.
>> 
>>  Regards,
>>  Sultan Ahamar.
>> 
>> 
>>  On Tue, Sep 6, 2016 at 9:34 AM, Morten Olav Hansen > >
>>  wrote:
>> 
>> > Hi Vanya
>> >
>> > This should now have been fixed in master and 224. It will now do a
>> > global UID check first, and report back any duplicates, if it finds
>> a
>> > duplicate it will remove them from the import and report back an
>> > `ErrorReport` (as it would do with other validation issues). So if
>> it finds
>> > a duplicate, you will need to set atomic mode to NONE to have it
>> import and
>> > ignore the duplicates.
>> >
>> > (btw, there was several duplicates in your file, not sure how that
>> > happened.. maybe something needs to be cleaned in your source
>> instance)
>> >
>> > --
>> > Morten Olav Hansen
>> > Senior Engineer, DHIS 2
>> > University of Oslo
>> > http://www.dhis2.org
>> >
>> > On Mon, Sep 5, 2016 at 2:37 PM, Morten Olav Hansen <
>> mor...@dhis2.org>
>> > wrote:
>> >
>> >> Hi Vanya
>> >>
>> >> This should have been caught by the importer... but there are
>> several
>> >> duplicates without the file you sent me, MaO4Ik8f34O is used in 3
>> category
>> >> option groups, same with oqeVQ71LCgY..
>> >>
>> >> I will look into making the validation process more robust.. but at
>> >> least you should know that this file have issues
>> >>
>> >> --
>> >> Morten Olav Hansen
>> >> Senior Engineer, DHIS 2
>> >> University of Oslo
>> >> http://www.dhis2.org
>> >>
>> >> On Mon, Sep 5, 2016 at 1:20 PM, Morten Olav Hansen <
>> mor...@dhis2.org>
>> >> wrote:
>> >>
>> >>> Ok, thanks Vanya, I'm looking into it now
>> >>>
>> >>> --
>> >>> Morten Olav Hansen
>> >>> Senior Engineer, DHIS 2
>> >>> University of Oslo
>> >>> http://www.dhis2.org
>> >>>
>> >>> On Thu, Sep 1, 2016 at 7:28 PM, Vanya Seth <
>> van...@thoughtworks.com>
>> >>> wrote:
>> >>>
>> >>>

Re: [Dhis2-devs] [Dhis2-users] DHIS2 Network Out

[Halvdan Hoem Grelland]

You should probably look into what specific processes on the server are
hogging the CPU. Try using 'htop' from the console (or similar). The logs
you've provided unfortunately do not provide any useful information.

Also, does a simple server reboot solve the issue? If so, does it re-occur?

On Mon, Sep 12, 2016 at 4:37 PM, Raminosoa Rabemanantsoa, Tantely <
tramino...@mikolo.org> wrote:

> Hi All,
>
> We are using DHIS2 2.19 and it was working really fine for several months.
> We have followed the HISP recommendations for the JVM parameters and the
> PostgreSQL memory tunning. We noticed few days ago a slow access to our
> DHIS2 server on Amazon from web version as well as from data capture. When
> checking the status, the server CPU is constantly running at 100%. We would
> like to have your advice on the possible sources of the errors. I note that
> Amazon has upgraded the Linux system to a newer version (2016.03). Does
> this change have impact on the installation of Tomact and postgreSQL? I am
> attaching the DHIS2 log file which is containing several errors.
>
> We are stuck, your help are welcome!
>
> Regards,
>
> Tantely.
>
> --
> *This message and its attachments are confidential and solely for the
> intended recipients. If received in error, please delete them and notify
> the sender via reply e-mail immediately.*
>
> ___
> Mailing list: https://launchpad.net/~dhis2-users
> Post to : dhis2-us...@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Halvdan Hoem Grelland
Software developer, DHIS 2
University of Oslo
http://www.dhis2.org 
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] DHIS2 Network Out

[Raminosoa Rabemanantsoa, Tantely]

Hi All,

We are using DHIS2 2.19 and it was working really fine for several months.
We have followed the HISP recommendations for the JVM parameters and the
PostgreSQL memory tunning. We noticed few days ago a slow access to our
DHIS2 server on Amazon from web version as well as from data capture. When
checking the status, the server CPU is constantly running at 100%. We would
like to have your advice on the possible sources of the errors. I note that
Amazon has upgraded the Linux system to a newer version (2016.03). Does
this change have impact on the installation of Tomact and postgreSQL? I am
attaching the DHIS2 log file which is containing several errors.

We are stuck, your help are welcome!

Regards,

Tantely.

-- 
*This message and its attachments are confidential and solely for the 
intended recipients. If received in error, please delete them and notify 
the sender via reply e-mail immediately.*
* ERROR 2016-09-12 00:45:52,280 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6547])
* INFO  2016-09-12 00:45:52,562 Importing data values 
(DefaultDataValueSetService.java [http-bio-8080-exec-6540])
* ERROR 2016-09-12 00:45:56,789 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6547])
* INFO  2016-09-12 00:46:09,773 Data value import done, total: 558, import: 0, 
update: 69 (DefaultDataValueSetService.java [http-bio-8080-exec-6535])
* ERROR 2016-09-12 00:46:10,458 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6551])
* ERROR 2016-09-12 00:46:15,648 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6551])
* ERROR 2016-09-12 00:46:42,437 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6546])
* ERROR 2016-09-12 00:46:47,430 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6546])
* INFO  2016-09-12 00:46:49,986 Login success for user: 'dhis120asos10', ip: 
'172.31.14.22' (DefaultUserAuditService.java [http-bio-8080-exec-6535])
* INFO  2016-09-12 00:46:56,088 Import options: [Id scheme: null, data element 
id scheme: null, org unit id scheme: null, dry run: false, preheat cache: true, 
async: false, strategy: null, skip check: false] 
(DefaultDataValueSetService.java [http-bio-8080-exec-6535])
* INFO  2016-09-12 00:46:56,088 Data value set scheme: null, data element 
scheme: null, org unit scheme: null (DefaultDataValueSetService.java 
[http-bio-8080-exec-6535])
* INFO  2016-09-12 00:46:56,088 Scheme: UID, data element scheme: UID, org unit 
scheme: UID (DefaultDataValueSetService.java [http-bio-8080-exec-6535])
* INFO  2016-09-12 00:46:59,184 Importing data values 
(DefaultDataValueSetService.java [http-bio-8080-exec-6535])
* INFO  2016-09-12 00:47:00,903 Data value import done, total: 558, import: 0, 
update: 97 (DefaultDataValueSetService.java [http-bio-8080-exec-6540])
* ERROR 2016-09-12 00:47:34,676 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6525])
* ERROR 2016-09-12 00:47:37,250 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6525])
* ERROR 2016-09-12 00:47:49,313 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6525])
* ERROR 2016-09-12 00:47:52,326 Initialised statement manager 
(StatementInterceptor.java [http-bio-8080-exec-6525])
* INFO  2016-09-12 00:47:57,969 Login success for user: 'dhis120asos10', ip: 
'172.31.14.22' (DefaultUserAuditService.java [http-bio-8080-exec-6547])
* INFO  2016-09-12 00:48:06,793 Data value import done, total: 558, import: 0, 
update: 207 (DefaultDataValueSetService.java [http-bio-8080-exec-6535])
* INFO  2016-09-12 00:48:09,247 Import options: [Id scheme: null, data element 
id scheme: null, org unit id scheme: null, dry run: false, preheat cache: true, 
async: false, strategy: null, skip check: false] 
(DefaultDataValueSetService.java [http-bio-8080-exec-6547])
* INFO  2016-09-12 00:48:09,247 Data value set scheme: null, data element 
scheme: null, org unit scheme: null (DefaultDataValueSetService.java 
[http-bio-8080-exec-6547])
* INFO  2016-09-12 00:48:09,248 Scheme: UID, data element scheme: UID, org unit 
scheme: UID (DefaultDataValueSetService.java [http-bio-8080-exec-6547])
* INFO  2016-09-12 00:48:11,954 Importing data values 
(DefaultDataValueSetService.java [http-bio-8080-exec-6547])
* INFO  2016-09-12 00:49:11,426 Login success for user: 'dhis120asos10', ip: 
'172.31.14.22' (DefaultUserAuditService.java [http-bio-8080-exec-6544])
* INFO  2016-09-12 00:49:15,672 Import options: [Id scheme: null, data element 
id scheme: null, org unit id scheme: null, dry run: false, preheat cache: true, 
async: false, strategy: null, skip check: false] 
(DefaultDataValueSetService.java [http-bio-8080-exec-6544])
* INFO  2016-09-12 00:49:15,673 Data value set scheme: null, data element 
scheme: null, org unit scheme: null (DefaultDataValueSetService.java 
[http-bio-8080-exec-6544])
* INFO  2016-09-12 00:49:15

Re: [Dhis2-devs] [Dhis2-users] DHIS2 Network Out

[Raminosoa Rabemanantsoa, Tantely]

Hi Halvdan,

We have already restarted the server and it has solved temporarily the
problem. We are going to try the command you mentioned and see what is
happening. Thank you.

Regards,

Tantely.

Le 12 sept. 2016 5:55 PM, "Halvdan Hoem Grelland"  a
écrit :

> You should probably look into what specific processes on the server are
> hogging the CPU. Try using 'htop' from the console (or similar). The logs
> you've provided unfortunately do not provide any useful information.
>
> Also, does a simple server reboot solve the issue? If so, does it re-occur?
>
> On Mon, Sep 12, 2016 at 4:37 PM, Raminosoa Rabemanantsoa, Tantely <
> tramino...@mikolo.org> wrote:
>
>> Hi All,
>>
>> We are using DHIS2 2.19 and it was working really fine for several months.
>> We have followed the HISP recommendations for the JVM parameters and the
>> PostgreSQL memory tunning. We noticed few days ago a slow access to our
>> DHIS2 server on Amazon from web version as well as from data capture. When
>> checking the status, the server CPU is constantly running at 100%. We
>> would
>> like to have your advice on the possible sources of the errors. I note
>> that
>> Amazon has upgraded the Linux system to a newer version (2016.03). Does
>> this change have impact on the installation of Tomact and postgreSQL? I am
>> attaching the DHIS2 log file which is containing several errors.
>>
>> We are stuck, your help are welcome!
>>
>> Regards,
>>
>> Tantely.
>>
>> --
>> *This message and its attachments are confidential and solely for the
>> intended recipients. If received in error, please delete them and notify
>> the sender via reply e-mail immediately.*
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to : dhis2-us...@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Halvdan Hoem Grelland
> Software developer, DHIS 2
> University of Oslo
> http://www.dhis2.org 
>
>

-- 
*This message and its attachments are confidential and solely for the 
intended recipients. If received in error, please delete them and notify 
the sender via reply e-mail immediately.*
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

[Dhis2-devs] Issue with Combination of Categories in Dataset/Data Entry Form

[Ifeanyi Okoye]

Dear Devs/Users,Good evening.
DHIS2 ver 2.22 build 22086
I am having a bit of a challenge with our data entry forms.
When I add a combination of categories to the dataset, once data is entered on 
the data entry form, the data field turns from yellow to red instead of green 
(see attachment). The combination of categories is important in this dataset so 
we can use it as a data dimension in the analytics. Once I set the combination 
of categories to default, then the issue goes away.
Since this issue started, I can no longer open, edit, or delete the first 
category combination I used with the data set. Running maintenance just runs in 
an endless loop. Generating resource tables or analytics do not complete. 
Not sure what is going on. 
Has anyone encountered this issue and what can we do about it?
Thanks

 
Warm Regards, 

Ifeanyi

Read DrMaestro's Blog
www.ifeanyiokoye.com
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

Re: [Dhis2-devs] Groups - Tables

[Raminosoa Rabemanantsoa, Tantely]

Hi there,
.
I am trying to connect to those resources using the web API and it seems
that the requested resources are not available. Could you please mention
the way to connect to them by using DHIS2 2.19?

Regards,

Tantely

On Mon, Sep 12, 2016 at 9:31 AM, Jason Pickering <
jason.p.picker...@gmail.com> wrote:

> Hi Tantely,
>
> 1- the table where are stored all organization units groups with the list
> of their members.
>
> \d orgunitgroupmembers
>
> 2- the table where are stored the datasets assigned to the organization
> units
>
> \d datasetsource
>
> 3- the table where are stored the organization units assigned to all users
>
> \d usermembership
>
> You should not really try and update these tables manually. Its much
> better to use the API.
>
>
> Regards,
> Jason
>
>
>
> On Mon, Sep 12, 2016 at 7:52 AM, Raminosoa Rabemanantsoa, Tantely <
> tramino...@mikolo.org> wrote:
>
>> Dear Members,
>>
>> I am looking for 3 database tables:
>>
>> 1- the table where are stored all organization units groups with the list
>> of their members.
>> 2- the table where are stored the datasets assigned to the organization
>> units
>> 3- the table where are stored the organization units assigned to all
>> users
>>
>> Does anyone know in which tables should I query to select and update
>> their records? I am using DHIS2 2.19 and I connect to the DHIS2 database
>> directly with psql.
>>
>> Thank you for your advice.
>>
>> Regards,
>>
>> Tantely.
>>
>> *This message and its attachments are confidential and solely for the
>> intended recipients. If received in error, please delete them and notify
>> the sender via reply e-mail immediately.*
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Jason P. Pickering
> email: jason.p.picker...@gmail.com
> tel:+46764147049
>

-- 
*This message and its attachments are confidential and solely for the 
intended recipients. If received in error, please delete them and notify 
the sender via reply e-mail immediately.*
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp