Hi All, I hope some of us have been able to read through the proposal. Awaiting thoughts/suggestion/feedback. I tried to upload more details on this proposal to the wiki but unable to add pages.
Can somebody please add permission so that i could add a design doc to wiki. wiki User - (saurav.lah...@sungard.com) Thanks Saurav On Wed, Dec 4, 2013 at 7:56 PM, Saurav Lahiri <saurav.lah...@sungard.com>wrote: > All, > This is to initiate discussion towards implementing an api to extract the > log messages > specific to a jobid. > > As has been described in the bug descrition as of cloudstack 4.3 there is > no api that can > aggregate log messages by the job id. An api to extract logs by the jobid > would make it > easier to identify the sequence of steps that could have caused any issue > at hand. Manually > sifting through the huge log files is always an inconvenience. > > To resolve this issue I am looking to implement the api which will extract > log messages by > job id. This will be available for use by any required consumers like UI, > cloudmonkey etc. > The api would in almost all cases be used to identify the sequence of > steps which have > occurred in relation to a job. In cases of failure this could potentially > point out the steps > which caused the issue. In other scenarios it could to an extent point out > how much a job has > progressed. Though progress indicator is not the purpose of the api. > > Being an api, it will follow the standard api invocation procedures. It > can be invoked programatically > using the desired programming languages. > > Possible inputs to the api could be filter parameters like jobid, start > timestap, end timestamp, > severity of the message. It will produce a response which will comprise of > all the lines from the > logfiles which correspond to this jobid and satisfy the specified > filtering criteria. The api would be > available for root admin, domain admin and users. The security > consideration should be that users > should only be able to query log messages for jobs that have been > triggerred by them.Since the > user is expecting an output it would be a good idea to implement it as a a > blocking/synchronous > API. > > [QUESTION] Does anybody other than root admin need this API? > > To filter and search the log messages instead of implementing the > filtering code in cloudstack the > bulk of the job can be done through a opensource tool like > logstash+elasticsearch or fluentd. From > preliminary investigation it appears that logstash+elasticsearch could be > a feasible option. > Inputs on any other tool which can better accomplish the task is also > welcome. A point of consideration > with logstash+elasticsearch is that if the log files grow very large then > to function optimally it could > require a dedicated hardware since these tools tend to cache large amount > of data in memory. > Currently I do not have any data to point to what large means but I shall > be running further tests during > evaluation of this combination. > > Above is a brief description on what is being proposed, do let me know if > above sounds workable. > Suggestions/Objections are welcome. > > > Thanks > Saurav >
