setrlimit guest calls that affect memory resources
(RLIMIT_{AS,DATA,STACK}) may interfere with QEMU internal memory
management. They may result in QEMU lockup because mprotect call in
page_unprotect would fail with ENOMEM error code, causing infinite loop
of SIGSEGV. E.g. it happens when running libstdc++ testsuite for xtensa
target on x86_64 host.
Don't call host setrlimit for memory-related resources.
Reviewed-by: Peter Maydell <peter.mayd...@linaro.org>
Signed-off-by: Max Filippov <jcmvb...@gmail.com>
---
Changes v2->v3:
- add comment sugessted by Peter
Changes v1->v2:
- don't limit change to 32-bit guest on 64-bit host case
linux-user/syscall.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index bb42a225eb58..f06307ebd945 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -9275,7 +9275,21 @@ abi_long do_syscall(void *cpu_env, int num, abi_long
arg1,
rlim.rlim_cur = target_to_host_rlim(target_rlim->rlim_cur);
rlim.rlim_max = target_to_host_rlim(target_rlim->rlim_max);
unlock_user_struct(target_rlim, arg2, 0);
- ret = get_errno(setrlimit(resource, &rlim));
+ /*
+ * If we just passed through resource limit settings for memory
then
+ * they would also apply to QEMU's own allocations, and QEMU will
+ * crash or hang or die if its allocations fail. Ideally we would
+ * track the guest allocations in QEMU and apply the limits
ourselves.
+ * For now, just tell the guest the call succeeded but don't
actually
+ * limit anything.
+ */
+ if (resource != RLIMIT_AS &&
+ resource != RLIMIT_DATA &&
+ resource != RLIMIT_STACK) {
+ return get_errno(setrlimit(resource, &rlim));
+ } else {
+ return 0;
+ }
}
break;
#endif
--
2.11.0
