2020-08-25 02:21:30 UTC - Dominic Kim: How many invokers did you use and what is the value of `userMemory` configuration? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598322090014900?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 05:36:50 UTC - Shubham Gupta: CONFIG_whisk_containerPool_userMemory=40960m https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598333810015100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 05:46:55 UTC - Shubham Gupta: currently there is only one invoker. but i tried to increase invokers by changing the value of `INVOKER_INSTANCES` in docker-whisk-controller.env file. but that did not work and then i found why invokers did not increase on this link <https://github.com/apache/openwhisk-devtools/issues/305> https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598334415015300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:26:45 UTC - Shubham Gupta: In controller logs, it is showing No invokers available https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598340405015500?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:28:28 UTC - Shubham Gupta: and when i check invokers status, it is showing invoker0 is unresponsive https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598340508016000?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:31:36 UTC - Shubham Gupta: @Rodric Rabbah @Dominic Kim if you guys can help in this issue would be really appreciated. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598340696016400?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:38:47 UTC - Dominic Kim: Hm.. it seems the invoker is not responsive, are there any logs in Invoker side? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341127016900?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:41:32 UTC - Shubham Gupta: In invoker logs i found these errors https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341292017100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:43:10 UTC - Dominic Kim: hm.. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341390017700?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:43:32 UTC - Dominic Kim: It seems there are some issues with docker/runc. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341412018000?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:44:02 UTC - Dominic Kim: Which version of docker are you using? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341442018300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:45:56 UTC - Dominic Kim: hm.. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341556018500?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:46:54 UTC - Dominic Kim: You may try to disable the runc as well if you do not need it. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341614018700?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:50:13 UTC - Shubham Gupta: Docker version 18.06.3-ce, build d7080c1 https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598341813018900?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 07:58:50 UTC - Shubham Gupta: okk, i will try again after disabling runc and see if i get success. +1 : Dominic Kim https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598342330019100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 08:07:29 UTC - Dominic Kim: ok https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598342849019300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 08:08:37 UTC - Dominic Kim: When I faced those logs, the volume configuration for runc was different with the location of container directory and it could not find the target containers while pausing/unpausing them. (Runc is used to pause/unpause containers.) https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598342917019500?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 08:17:04 UTC - Shubham Gupta: I think the volumes are correct in my case. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598343424019800?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 08:17:44 UTC - Dominic Kim: Are you able to list up all containers with docker-runc list? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598343464020200?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 08:26:51 UTC - Shubham Gupta: i can execute docker-run list in invoker container but there are no containers in the output https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598344011020400?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 08:27:47 UTC - Shubham Gupta: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598344067020600?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:47:39 UTC - Dominic Kim: I think you can list up in the host as well. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598348859021000?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:47:45 UTC - Dominic Kim: ```docker-runc --root /var/run/docker/runtime-runc/moby/ list``` Could you try this? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598348865021200?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:48:26 UTC - Dominic Kim: Anyway, it looks runc is pointing a different location. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598348906021400?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:49:00 UTC - Dominic Kim: Please check on the invoker container arguments. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598348940021600?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:49:25 UTC - Dominic Kim: ``` $ docker inspect invoker4 | grep runc "/run/docker/runtime-runc/moby:/run/runc:rw" "Runtime": "runc", "Source": "/run/docker/runtime-runc/moby", "Destination": "/run/runc", "/run/runc": {},``` This is mine. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598348965021800?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:55:52 UTC - Shubham Gupta: https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349352022000?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:56:35 UTC - Dominic Kim: When using `--root` option, you should run it in your host rather than inside of the invoker container. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349395022600?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:56:42 UTC - Dominic Kim: And it looks some configurations are missing for runc. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349402022800?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:57:01 UTC - Dominic Kim: So I think disabling the runc would work. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349421023100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:57:25 UTC - Dominic Kim: You are using devtools for openwhisk, right? I think that's a contribution point. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349445023300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 09:59:24 UTC - Shubham Gupta: I have disabled runc and now i am not getting any runc error in invoker logs, but `No invokers available` error still persist in controller logs https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349564023500?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:00:15 UTC - Dominic Kim: So `/invokers` still shows the unresponsive invoker? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349615023700?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:03:24 UTC - Shubham Gupta: yes https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349804023900?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:03:48 UTC - Dominic Kim: when disalbing the invoker, did you restart all components or just invoker? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349828024100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:04:20 UTC - Shubham Gupta: all components https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349860024300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:04:44 UTC - Dominic Kim: hm.. then have you observed any normal logs for health activations in the invoker side? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349884024700?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:05:20 UTC - Dominic Kim: If there was no error in invoker side, it is supposed to invoke some health actions and become healthy. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598349920025000?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:16:17 UTC - Shubham Gupta: and when i am executing api list command or invoking any gateway api. it is accepting every request but giving error and saying processing not completed yet. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598350577025200?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:18:01 UTC - Dominic Kim: hm.. I'd suggest using standalone openwhisk for easier use. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598350681025700?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:18:03 UTC - Dominic Kim: Have you tried it? https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598350683025900?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:24:37 UTC - Shubham Gupta: No https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598351077026100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 10:25:21 UTC - Shubham Gupta: is there any way to increase invokers in openwhisk-devtools ? i tried one but that did not work https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598351121026300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 11:37:52 UTC - Dominic Kim: It has been some time since I tried openwhisk-devtools as standalone openwhisk come to the fore. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598355472027500?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 11:38:16 UTC - Dominic Kim: I would look into it tomorrow. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598355496028100?thread_ts=1598282887.014000&cid=C3TPCAQG1 ---- 2020-08-25 11:50:50 UTC - Shubham Gupta: okk, thank you very much. https://openwhisk-team.slack.com/archives/C3TPCAQG1/p1598356250028300?thread_ts=1598282887.014000&cid=C3TPCAQG1 ----
