I use Logstash as the sidecar container to drain action container's logs to Elastic
-----Original Message----- From: "Rodric Rabbah"<rod...@gmail.com> To: <dev@openwhisk.apache.org>; Cc: Sent: 2020/9/1周二 12:55 (GMT+08:00) Subject: Re: [Discussion]Runtime log collection issue on K8S Do you drain logs directly from memory to Elastic or do you use the FileLogStore which then drains the logs to Elastic? -r On Mon, Aug 31, 2020 at 5:43 AM 蒋鹏程 <jiang.pengch...@navercorp.com> wrote: > Hello guys: > > Recently we are trying to migrate OpenWhisk from docker to K8S, but faced > some problems, and one of it is: the performance droped a lot(the tps > changed from thousands to dozens in our environment during benchmarks) > after migrating to K8S, and the reason we founded is that all action > containers will trying to fetch action logs from K8S api server after `run` > are finished, and these log requests become very slow when there are logs > of concurrent activations comes, which in turn leads to a very low tps of > activations. > > To resolve such issue, we tried below 2 ways: > > > mount a shared volume for all action pods and invoker pods, and make > invoker read action logs from shared volume directly > > use sidecar container for each action container, read and send action logs > to an external database(ElasticSearch here), and then invokers fetch logs > from the external database > > > > But both 2 approaches has some problems, the shared volume needs some > time(several seconds) to sync file contents and ElasticSearch also need to > take some seconds to save action logs and return them to invokers. > > Then we finally come up with an another idea, which I think is preferable > than above two, that to make action container capture the output of user's > codes and then return them within the `run` request in a specified field > like `__OW_LOGS`, so invokers can get action logs directly from the > response of `run` uri, and this do improve the tps of activations. But this > method also has some flaws needs to be solved. > > Any idea about this? I'm also curious that have you faced same > performance issue when run openwhisk on K8S and how did you resolved it? > > Sincerely > Jiang PengCheng >
