The only thing I can think of is that the VM is not on a network that provides routing. for instance in a basic zone or on a shared network with a public ip range.
On Fri, Jun 25, 2021 at 12:23 PM Hean Seng <[email protected]> wrote: > HI Daan > > > My usage record have this , this is one of the example that do not have. > record of type 4 and 5, but have type 1, > > select * from usage_event where account_id=3391; > > > -------+-------------------------+------------+---------------------+---------+-------------+--------------------+-------------+-------------+-------------+----------------+-----------+--------------+ > > | id | type | account_id | created | > zone_id | resource_id | resource_name | offering_id | template_id | > size | resource_type | processed | virtual_size | > > > +-------+-------------------------+------------+---------------------+---------+-------------+--------------------+-------------+-------------+-------------+----------------+-----------+--------------+ > > | 75910 | NET.IPASSIGN | 3391 | 2021-05-06 10:16:55 | > 1 | 2396 | 11.11.1.11 | NULL | 0 | > 0 > | DirectAttached | 1 | NULL | > > | 75911 | VOLUME.CREATE | 3391 | 2021-05-06 10:16:55 | > 1 | 6472 | ROOT-3649 | NULL | 210 | > 21474836480 | NULL | 1 | NULL | > > | 75912 | VM.CREATE | 3391 | 2021-05-06 10:16:55 | > 1 | 3649 | VM-38629-vasuflor | 51 | 210 | > NULL > | KVM | 1 | NULL | > > | 75913 | NETWORK.OFFERING.ASSIGN | 3391 | 2021-05-06 10:17:06 | > 1 | 3649 | 3753 | 6 | NULL | > 1 | NULL | 1 | NULL | > > | 75914 | SG.ASSIGN | 3391 | 2021-05-06 10:17:06 | > 1 | 3649 | NULL | 3664 | NULL | > NULL > | NULL | 1 | NULL | > > | 75915 | VM.START | 3391 | 2021-05-06 10:17:06 | > 1 | 3649 | VM-38629-vasuflor | 51 | 210 | > NULL > | KVM | 1 | NULL | > > > +-------+-------------------------+------------+---------------------+---------+-------------+--------------------+-------------+-------------+-------------+----------------+-----------+--------------+ > > 6 row > > mysql> select * from usage_network where account_id=3391; > > Empty set (0.00 sec) > > > > mysql> select * from usage_network_offering where account_id=3391 > > -> ; > > > +---------+------------+-----------+----------------+---------------------+------------+---------------------+---------+--------+ > > | zone_id | account_id | domain_id | vm_instance_id | network_offering_id | > is_default | created | deleted | nic_id | > > > +---------+------------+-----------+----------------+---------------------+------------+---------------------+---------+--------+ > > | 1 | 3391 | 3383 | 3649 | 6 | > 1 | 2021-05-06 10:17:06 | NULL | 3753 | > > > +---------+------------+-----------+----------------+---------------------+------------+---------------------+---------+--------+ > > 1 row in set (0.01 sec) > > > > > > On Fri, Jun 25, 2021 at 3:40 PM Daan Hoogland <[email protected]> > wrote: > > > Hean, > > To be honest, no i don't. You can start looking at any log records > related > > to the VMs and than track to their nics. Also look at the > > usage_event-table, to see if any records are there. First in the cloud > DB, > > and if these are there in the cloud_usage DB. > > Sorry to not have an obviously useful clue for you. > > > > On Thu, Jun 24, 2021 at 7:49 PM Hean Seng <[email protected]> wrote: > > > > > Hi Daan, > > > > > > Thanks for reply. I deleted the null record, and manage to make it > work > > > again. The it will auto generate back the record with correct ID . > > > > > > > > > However, I am facing the issue is the Usage Type 4, and 5 (Network > sent > > > and received) are not populated . Other Type like Type1 is always > > working. > > > > > > Fyi, I am using share network, and set the parameter > > > vm.network.stats.interval=300 > > > so that the VM statistic is updated . > > > > > > This issue not happen on all VM, but part of it. ( about 50%) . I > > > tried look at the DB, and seem cannot find the clue what actually diff > , > > in > > > term of those updated and those is not. > > > > > > The record is appearing in usage_network_offering and seems good. Do > you > > > have any idea of this ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 21, 2021 at 3:19 PM Daan Hoogland <[email protected] > > > > > wrote: > > > > > > > Sorry Hean, wasn't on line most of the weekend, Is it running now? > > > > as you wrote this two days ago, it should already have run twice > > > (assuming > > > > you didn't change a lot) > > > > > > > > On Sat, Jun 19, 2021 at 8:41 PM Hean Seng <[email protected]> > wrote: > > > > > > > > > And it will insert another success is NULL to usage_job table , > > right ? > > > > > > > > > > i deleted the null record and restart usage servoce and see > > following > > > > in > > > > > usage log : > > > > > > > > > > 2021-06-19 18:37:52,912 INFO [cloud.usage.UsageManagerImpl] > > > (main:null) > > > > > (logid:) Implementation Version is 4.15.0.0 > > > > > > > > > > 2021-06-19 18:37:55,942 DEBUG [cloud.usage.UsageManagerImpl] > > > (main:null) > > > > > (logid:) Usage stats aggregation time zone: GMT > > > > > > > > > > 2021-06-19 18:37:55,952 DEBUG [cloud.usage.UsageManagerImpl] > > > (main:null) > > > > > (logid:) Execution Time: Sat Jun 19 00:15:00 UTC 2021 > > > > > > > > > > 2021-06-19 18:37:55,953 DEBUG [cloud.usage.UsageManagerImpl] > > > (main:null) > > > > > (logid:) Current Time: Sat Jun 19 18:37:55 UTC 2021 > > > > > > > > > > It seem execution time before current time ? Is that normal ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jun 20, 2021 at 2:39 AM Hean Seng <[email protected]> > > wrote: > > > > > > > > > > > > > > > > > So, I shall stop the Cloudstack-Usage services and delete all > > record > > > > > where > > > > > > success is NULL , and start back the usage service ? > > > > > > > > > > > > > > > > > > On Sat, Jun 19, 2021 at 11:00 PM Daan Hoogland < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> Heanm, > > > > > >> > > > > > >> On Sat, Jun 19, 2021 at 5:27 AM Hean Seng <[email protected]> > > > wrote: > > > > > >> > > > > > >> > i have no unprocess job , but still have this issue: > > > > > >> > > > > > > >> > 2021-06-19 10:09:57,657 DEBUG [cloud.usage.UsageManagerImpl] > > > > > >> > (Usage-HB-1:null) (logid:) Scheduling Usage job... > > > > > >> > > > > > > >> > 2021-06-19 10:09:57,657 INFO [cloud.usage.UsageManagerImpl] > > > > > >> > (Usage-Job-1:null) (logid:) starting usage job... > > > > > >> > > > > > > >> > 2021-06-19 10:09:57,661 DEBUG [cloud.usage.UsageManagerImpl] > > > > > >> > (Usage-Job-1:null) (logid:) Not owner of usage job, > skipping... > > > > > >> > > > > > > >> this line ^^ means there is a unprocessed job owned by another > > > > proccess > > > > > >> ... > > > > > >> > > > > > >> > > > > > >> > 2021-06-19 10:09:57,661 INFO [cloud.usage.UsageManagerImpl] > > > > > >> > (Usage-Job-1:null) (logid:) usage job complete > > > > > >> > > > > > > >> > > > > > > >> > mysql> select id, pid, start_date ,end_date , success, > > heartbeat > > > > from > > > > > >> > usage_job where success=0 order by start_date desc limit 100; > > > > > >> > > > > > > >> you should search for `where success is null` > > > > > >> > > > > > >> > > > > > >> > Empty set (0.00 sec) > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > On Fri, Jun 18, 2021 at 11:01 PM Hean Seng < > [email protected]> > > > > > wrote: > > > > > >> > > > > > > >> > > yes, I restarted the mgmt server, wait till can login web > > > > > interface > > > > > >> . > > > > > >> > > then only restart usage . > > > > > >> > > > > > > > >> > > I did follwoig also : > > > > > >> > > > > > > > >> > > * stop usage server > > > > > >> > > > > > > > >> > > * remove the record from usage_job for the unprocessed job > > > > > >> > > > > > > > >> > > * start usage server again > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > Still same thing. Not sure where the Cloudstack keep the > > Usage > > > > > server > > > > > >> > PID > > > > > >> > > id , and keep getting wrong id. > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > On Fri, Jun 18, 2021 at 4:31 PM Boris Stoyanov < > > > > > >> > > [email protected]> wrote: > > > > > >> > > > > > > > >> > >> Hi Hean Seng, > > > > > >> > >> > > > > > >> > >> Have you waited the management service to come up and then > > > > > restarted > > > > > >> > >> usage service? It might be some race condition? > > > > > >> > >> > > > > > >> > >> Bobby. > > > > > >> > >> > > > > > >> > >> From: Hean Seng <[email protected]> > > > > > >> > >> Date: Friday, 18 June 2021, 7:31 > > > > > >> > >> To: [email protected] < > [email protected] > > > > > > > > >> > >> Subject: Cloudstack Usage --- not owner > > > > > >> > >> Hi > > > > > >> > >> > > > > > >> > >> I have mysql hang, and restarted mysql, restarted > cloudstack > > > > mgmt, > > > > > >> and > > > > > >> > >> usage. > > > > > >> > >> > > > > > >> > >> After that facing issue of following: > > > > > >> > >> > > > > > >> > >> duration is 120 minutes) > > > > > >> > >> > > > > > >> > >> 2021-06-18 04:25:49,656 DEBUG > [cloud.usage.UsageManagerImpl] > > > > > >> > >> (Usage-HB-1:null) (logid:) Scheduling Usage job... > > > > > >> > >> > > > > > >> > >> 2021-06-18 04:25:49,657 INFO > [cloud.usage.UsageManagerImpl] > > > > > >> > >> (Usage-Job-1:null) (logid:) starting usage job... > > > > > >> > >> > > > > > >> > >> 2021-06-18 04:25:49,669 DEBUG > [cloud.usage.UsageManagerImpl] > > > > > >> > >> (Usage-Job-1:null) (logid:) Not owner of usage job, > > skipping... > > > > > >> > >> > > > > > >> > >> 2021-06-18 04:25:49,669 INFO > [cloud.usage.UsageManagerImpl] > > > > > >> > >> (Usage-Job-1:null) (logid:) usage job complete > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> Seems usage pid is not updated to db . > > > > > >> > >> > > > > > >> > >> Andybody know how to fix this ? > > > > > >> > >> > > > > > >> > >> -- > > > > > >> > >> Regards, > > > > > >> > >> Hean Seng > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > > > > > > > >> > > -- > > > > > >> > > Regards, > > > > > >> > > Hean Seng > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > -- > > > > > >> > Regards, > > > > > >> > Hean Seng > > > > > >> > > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Daan > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > Hean Seng > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Hean Seng > > > > > > > > > > > > > > > > > -- > > > > Daan > > > > > > > > > > > > > -- > > > Regards, > > > Hean Seng > > > > > > > > > -- > > Daan > > > > > -- > Regards, > Hean Seng > -- Daan
