Re: Controllers running out of disk space

2016-11-24 Thread roger peppe
Log size limits are great, and a necessary thing, but for me the crucial thing is to have some final fail safe if disk space does end up getting critically low for *any* reason. Almost all the really borked Juju installations I've seen have been due to running out of disk space. Shutting down the

Re: Controllers running out of disk space

2016-11-22 Thread John Meinel
Juju records a longer history in the database, but for ease of debugging we also save to a plain text file, which gets rotated. It is set to rotate at 300MB and save at most 2 backups. Which means it should go to potentially ~1GB but not grow beyond that. We could compress on rotation, which we don

Re: Controllers running out of disk space

2016-11-22 Thread Jacek Nykis
On 21/11/16 23:26, Menno Smits wrote: > On 18 November 2016 at 05:07, Nate Finch wrote: > >> Resources are also stored in mongo and can be unlimited in size (not much >> different than fat charms, except that at least they're only pulled down on >> demand). >> >> We should let admins configure th

Re: Controllers running out of disk space

2016-11-21 Thread Mark Shuttleworth
On 21/11/16 23:26, Menno Smits wrote: > On 18 November 2016 at 05:07, Nate Finch >wrote: > > Resources are also stored in mongo and can be unlimited in size > (not much different than fat charms, except that at least they're > only pulled down on demand

Re: Controllers running out of disk space

2016-11-21 Thread Menno Smits
On 18 November 2016 at 05:07, Nate Finch wrote: > Resources are also stored in mongo and can be unlimited in size (not much > different than fat charms, except that at least they're only pulled down on > demand). > > We should let admins configure their max log size... our defaults may not > be w

Re: Controllers running out of disk space

2016-11-17 Thread Nate Finch
Resources are also stored in mongo and can be unlimited in size (not much different than fat charms, except that at least they're only pulled down on demand). We should let admins configure their max log size... our defaults may not be what they like, but I bet that's not really the issue, since w

Re: Controllers running out of disk space

2016-11-17 Thread Rick Harding
I'm definitely agreeing we need to provide some better tools for the admin of the controller to track and garden things such as charms and resources which can be quite large and grows over time. My main point with Uros was that we have this way due to model migrations to stick a controller/model i

Re: Controllers running out of disk space

2016-11-17 Thread John Meinel
So logs in mongo and logs on disk should be capped, and purged when they get above a certain size. 'audit.log' should never be automatically purged. Charms in the blobstore are potentially local data that we can't reproduce, so hard to automatically purge them. I think there has been some work done

Re: Controllers running out of disk space

2016-11-17 Thread Jacek Nykis
On 17/11/16 08:20, Uros Jovanovic wrote: > Hi all, > > I'd like to start a discussion on how to handle storage issues on > controller machines, especially what we can do when storage is getting 95% > or even 98% full. There are many processes that are storing data, we have > at least: > - charms a