All, This is to initiate discussion towards implementing an api to extract the log messages specific to a jobid.
As has been described in the bug descrition as of cloudstack 4.3 there is no api that can aggregate log messages by the job id. An api to extract logs by the jobid would make it easier to identify the sequence of steps that could have caused any issue at hand. Manually sifting through the huge log files is always an inconvenience. To resolve this issue I am looking to implement the api which will extract log messages by job id. This will be available for use by any required consumers like UI, cloudmonkey etc. The api would in almost all cases be used to identify the sequence of steps which have occurred in relation to a job. In cases of failure this could potentially point out the steps which caused the issue. In other scenarios it could to an extent point out how much a job has progressed. Though progress indicator is not the purpose of the api. Being an api, it will follow the standard api invocation procedures. It can be invoked programatically using the desired programming languages. Possible inputs to the api could be filter parameters like jobid, start timestap, end timestamp, severity of the message. It will produce a response which will comprise of all the lines from the logfiles which correspond to this jobid and satisfy the specified filtering criteria. The api would be available for root admin, domain admin and users. The security consideration should be that users should only be able to query log messages for jobs that have been triggerred by them.Since the user is expecting an output it would be a good idea to implement it as a a blocking/synchronous API. [QUESTION] Does anybody other than root admin need this API? To filter and search the log messages instead of implementing the filtering code in cloudstack the bulk of the job can be done through a opensource tool like logstash+elasticsearch or fluentd. From preliminary investigation it appears that logstash+elasticsearch could be a feasible option. Inputs on any other tool which can better accomplish the task is also welcome. A point of consideration with logstash+elasticsearch is that if the log files grow very large then to function optimally it could require a dedicated hardware since these tools tend to cache large amount of data in memory. Currently I do not have any data to point to what large means but I shall be running further tests during evaluation of this combination. Above is a brief description on what is being proposed, do let me know if above sounds workable. Suggestions/Objections are welcome. Thanks Saurav
