Here is 4.12.0.0 debug and trace log files, as requested Not so much info here
https://i.pozitis.ru/a2013a0ad9f2a26dd40051f9ab189faf.txt and trace file https://i.pozitis.ru/d24ce987120417714aa775764b682d14.txt Is you can see, I updated usage.stats.job.exec.time every time before usage server start to force job start There is nothing after line (main:null) (logid:) UsageServer ready... in both cases. in the same time the version 4.11.2 with same approach, when I changed usage.stats.job.exec.time to the next minute at the start moment gave me this result: https://i.pozitis.ru/21eba0073b51cdd102180d1c605eda9c.txt As you can see, the job started and proceed as it should be I made the test on same VM, by replacing the repos from 11.2.0 to 12.0.0 and back and reinstalling usage server without any changes to other software bug is in cloudstack-usage-4.12.0.0-1.el7.centos.x86_64 this version works fine cloudstack-usage-4.11.2.0-1.el7.centos.x86_64 On Fri, Mar 8, 2019 at 12:37 PM Rohit Yadav <[email protected]> wrote: > Can you increase the logging to DEBUG (or TRACE if you don't see anything > in logs) for the usage server via the log4j xml at /etc/cloudstack/usage ? > > > > Regards, > > Rohit Yadav > > Software Architect, ShapeBlue > > https://www.shapeblue.com > > ________________________________ > From: Konstantin <[email protected]> > Sent: Friday, March 8, 2019 2:53:06 PM > To: [email protected] > Subject: Re: cloudstack usage server issue > > here is a link to the issue on github > > https://github.com/apache/cloudstack/issues/3203 > > On Fri, Mar 8, 2019 at 12:11 PM Konstantin <[email protected]> > wrote: > > > Yes, Andrij, > > > > This is exactly my story as well. The Usage Server completely ignore " > > db.usage " settings from /etc/cloudstack/usage/db.properties > > > > I will try to manage with github, will let you know. > > > > Thanks again for the valuable support > > > > > > On Fri, Mar 8, 2019 at 1:59 AM Andrija Panic < > [email protected]> > > wrote: > > > >> Hi Konstantin, > >> > >> Thanks for raising the issue - seems it's really broken in 4.12 (usage > >> server) > >> > >> [root@ref-trl-1013-k-M7-apanic-mgmt1 usage]# grep -i usage > >> /etc/cloudstack/usage/db.properties | grep invalid > >> db.usage.name=invalid_database_name > >> db.usage.port=invalid_port > >> db.usage.username=invalid_username > >> db.usage.password=invalid_password! > >> [root@ref-trl-1013-k-M7-apanic-mgmt1 usage]# tail -n 5 > >> /var/log/cloudstack/usage/usage.log > >> 2019-03-07 22:40:36,738 INFO [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Implementation Version is 4.12.0.0 > >> 2019-03-07 22:40:37,569 DEBUG [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Usage stats aggregation time zone: GMT > >> 2019-03-07 22:40:37,570 DEBUG [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Execution Time: Thu Mar 07 22:28:00 UTC 2019 > >> 2019-03-07 22:40:37,570 DEBUG [cloud.usage.UsageManagerImpl] (main:null) > >> (logid:) Current Time: Thu Mar 07 22:40:37 UTC 2019 > >> 2019-03-07 22:40:37,575 INFO [cloud.usage.UsageServer] (main:null) > >> (logid:) UsageServer ready.. > >> > >> So even with all usage server settings in db.properties, it's still > >> running happily > >> > >> And yes, the usage job doesn't kick in at all - but let me let it wait > >> until morning and if no job run (I configured hourly run), I will raise > the > >> blocker issue - or eve better - you can do it for that matter - as a > >> valuable community member 😊 ) > >> > >> Feel free to create an GitHub issue (on link I shared before) and I will > >> comment and please feel free to vote -1 on the mailing list for 4.12 > RC4 - > >> and please reference the issue you raised previously on GitHub. > >> > >> Kind regards, > >> Andrija > >> > >> [email protected] > >> www.shapeblue.com<http://www.shapeblue.com> > >> Amadeus House, Floral Street, London WC2E 9DPUK > >> @shapeblue > >> > >> > >> > >> > >> -----Original Message----- > >> From: Andrija Panic <[email protected]> > >> Sent: 07 March 2019 20:53 > >> To: [email protected] > >> Subject: RE: cloudstack usage server issue > >> > >> Hi Konstantin, > >> > >> Issues/bugs are raised here https://github.com/apache/cloudstack/issues > >> - so please feel free to raise it ASAP - since we have a RC4 voting > process > >> in place ! (it's very simple) - also feel free to jump on the voting > email > >> thread (before giving -1, let's first try to reproduce the issue once > >> more? ) > >> > >> I will try to look into this problem, but can't promise anything (into > >> 4.12 issue). > >> > >> As for the very first link you shared - the jump in time visible in logs > >> - is this because of restart, right ? > >> > >> Also as for aggregation period, 1440 is default (and in production, > >> believe me, should be kept this way if possible). > >> Hourly jobs are ok for testing (value of 60 minutes), not sure setting 5 > >> (minutes) makes sense (perhaps for testing, but I would better stick to > 60 > >> min period, and then change execution time to 5min from now() to be > able to > >> test (and restart usage service) - please give it some time to actually > >> start processing data... > >> > >> As for the issues changing db.properties file, you should have it like > >> following: > >> > >> ls -la /etc/cloudstack/usage/ > >> total 4 > >> lrwxrwxrwx. 1 root root 40 Feb 13 20:27 db.properties -> > >> /etc/cloudstack/management/db.properties > >> lrwxrwxrwx. 1 root root 30 Feb 13 20:27 key -> > >> /etc/cloudstack/management/key > >> -rw-r--r--. 1 root root 2980 Mar 7 17:22 log4j-cloud.xml First two are > >> links as you see - assuming you tried to change correct values in > >> db.properties (to make usage fail to connect to DB), that should work, > >> otherwise, it's a bug... > >> > >> Kind regards, > >> Andrija > >> > >> [email protected] > >> www.shapeblue.com<http://www.shapeblue.com> > >> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > >> > >> > >> > >> > >> -----Original Message----- > >> From: Konstantin <[email protected]> > >> Sent: 07 March 2019 16:59 > >> To: [email protected] > >> Subject: Re: cloudstack usage server issue > >> > >> I reinstalled 4.11.2 and logs are changed. New error appear > >> > >> https://i.pozitis.ru/77a661462b182d54a6405484074dcee2.txt > >> > >> 2019-03-07 15:50:00,001 INFO [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) starting usage job... > >> 2019-03-07 15:50:00,014 DEBUG [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) Not owner of usage job, skipping... > >> 2019-03-07 15:50:00,014 INFO [cloud.usage.UsageManagerImpl] > >> (Usage-Job-1:null) (logid:) usage job complete > >> > >> I cleaned the usage_job table from the records and FINALLY its start > >> working! > >> > >> > >> How to submit a bug report for the version 4.12.0.0 ? > >> > >> Could you please do it on my behalf? > >> > >> > >> > >> > >> On Thu, Mar 7, 2019 at 6:23 PM Konstantin < > [email protected]> > >> wrote: > >> > >> > Dear Andrija, > >> > > >> > Thanks a lot for the attention to my issue. > >> > > >> > There is logs from usage server: > >> > > >> > https://i.pozitis.ru/80d76e3cc5e605be037a088fd014a986.txt > >> > > >> > There is config > >> > https://i.pozitis.ru/8419e465b3ffa63b4d677ae5be85688a.txt > >> > > >> > its looks like usage server just ignore cloud_usage DB settings, I > >> > tried to put wrong credentials to db.usage.name=cloud_usage params, > No > >> > matter, Its load the usage details from db.cloud.name=cloud, I think > >> > > >> > Here is my global settings > >> > > >> > https://i.pozitis.ru/9d3d53fd86997dd649b0e8171ca14c73.txt > >> > > >> > I did checked management server log, parsed by "usage", here is > >> > result, I see error checking health of usage server, usage server > >> > running? false, > >> > heartbeat: Wed Mar 06 20:00:06 UTC 2019 > >> > > >> > https://i.pozitis.ru/fd56fdc8a3f7d80fabdec8d4cf563325.txt > >> > > >> > I restarted both services, here is debug results > >> > > >> > https://i.pozitis.ru/d8a8c06ea31a6ec89e0ca3e5aa89b178.txt > >> > > >> > here is usage server logs after restart > >> > https://i.pozitis.ru/3bf38d723d2ccfc0e3bb6be785225c48.txt > >> > > >> > Would you recommend something to check more? > >> > > >> > > >> > Regads, > >> > Konstantin > >> > > >> > > >> > On Thu, Mar 7, 2019 at 2:23 PM Andrija Panic > >> > <[email protected]> > >> > wrote: > >> > > >> >> Hi Konstantin, > >> >> > >> >> Can you please usage upload the logs to somewhere (pastebin is OK) so > >> >> we can check ? > >> >> > >> >> Did you/can you try with 4.11.2 ? > >> >> > >> >> Kind regards, > >> >> Andrija > >> >> > >> >> [email protected] > >> >> www.shapeblue.com<http://www.shapeblue.com> > >> >> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > >> >> > >> >> > >> >> > >> >> > >> >> -----Original Message----- > >> >> From: Konstantin <[email protected]> > >> >> Sent: 06 March 2019 23:20 > >> >> To: [email protected] > >> >> Subject: cloudstack usage server issue > >> >> > >> >> I did installed the management server, connect it to ESX HV, the > >> >> control part working fine. > >> >> > >> >> I deployed the usage server with all guidelines awareness but have no > >> >> luck with processing a usage data from "cloud" DB to "cloud_usage" DB > >> >> by usage server. > >> >> > >> >> I tried to configure, reconfigure and even reinstall everything, but > >> >> the last usage server log record is always same: > >> >> > >> >> 2019-03-06 18:21:28,903 DEBUG [cloud.usage.UsageManagerImpl] > >> >> (main:null) > >> >> (logid:) Checking to see if usage.vmops.pid exists. > >> >> 2019-03-06 18:21:28,903 INFO [cloud.usage.UsageManagerImpl] > >> >> (main:null) > >> >> (logid:) Implementation Version is 4.12.0.0 > >> >> 2019-03-06 18:21:30,865 DEBUG [cloud.usage.UsageManagerImpl] > >> >> (main:null) > >> >> (logid:) Usage stats aggregation time zone: UTC > >> >> 2019-03-06 18:21:30,866 DEBUG [cloud.usage.UsageManagerImpl] > >> >> (main:null) > >> >> (logid:) Execution Time: Wed Mar 06 18:22:00 UTC 2019 > >> >> 2019-03-06 18:21:30,866 DEBUG [cloud.usage.UsageManagerImpl] > >> >> (main:null) > >> >> (logid:) Current Time: Wed Mar 06 18:21:30 UTC 2019 > >> >> 2019-03-06 18:21:30,874 INFO [cloud.usage.UsageServer] (main:null) > >> >> (logid:) UsageServer ready... > >> >> > >> >> As you may see, I played with starting time to force usage server to > >> >> start the job, but no luck... > >> >> Its never gave me any logs written and any action taken after " > >> >> UsageServer ready... " > >> >> I tried to call > >> >> > >> >> [root@cloudstack usage]# cloudmonkey generateUsageRecords > >> >> startdate=2018-09-01 enddate=2019-09-30 success = True > >> >> > >> >> but the "cloud_usage" DB is always empty, the all tables are empty, > >> >> no single line in it. > >> >> In the same time, the "cloud" DB and usage_event table is full of > >> >> data with prossessed column "0" for any record in the table. > >> >> > >> >> Do you know how to force usage server to start the job? > >> >> > >> >> here is my usage params info: > >> >> > >> >> enable.usage.server true > >> >> publish.usage.events true > >> >> quota.usage.smtp.connection.timeout 60 > >> >> quota.usage.smtp.host > >> >> quota.usage.smtp.password > >> >> quota.usage.smtp.port > >> >> quota.usage.smtp.sender > >> >> quota.usage.smtp.useAuth > >> >> quota.usage.smtp.user > >> >> usage.aggregation.timezone UTC > >> >> usage.execution.timezone UTC > >> >> usage.sanity.check.interval 1 > >> >> usage.snapshot.virtualsize.select false > >> >> usage.stats.job.aggregation.range 5 //I changed it from 5 to 60 > and > >> >> 1440 > >> >> with same negative result > >> >> usage.stats.job.exec.time 18:22 /changed many times to fire job, > no > >> >> luck > >> >> > >> >> > >> >> Regards, > >> >> Konstantin > >> >> > >> > > >> > > > > [email protected] > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > >
