This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8792a8bab7c306fbc75bd99125ac0bbca5b411d7 Author: Mingyu Chen (Rayner) <[email protected]> AuthorDate: Fri Sep 25 08:44:33 2026 +0800 branch-4.1: [fix](thirdparty) Upgrade libunwind to 1.8.3 to fix aarch64 BE startup crash on 64 KiB-page kernels #67877 (#68339) Cherry-picked from #67877 (master commit e797e99fcbe) to branch-4.1. Conflict resolution: only `thirdparty/CHANGELOG.md` conflicted. The `## 20260911` libunwind entry is inserted verbatim above the branch's `## 20260901` thrift entry; the master-only `## 20260909` snappy 1.2.1 entry is not carried over because branch-4.1 still pins snappy 1.1.10. The other six files applied as on master. branch-4.1 already carries the #64093 backport and the repaired 1.6.2 patch file (#67564), so the rename to the 1.8.3 patch is a clean one. ### What problem does this PR solve? Related PR: #64093 Problem Summary: **Every aarch64 BE built from master or branch-4.1 (since 4.1.4-rc01) crashes before `main()` on a 64 KiB-page kernel** (Kunpeng 920 + Kylin V10 / openEuler / CentOS aarch64, the common ARM server stack). The core from a 4.1.4 arm64 deployment shows: ``` access_mem (addr=0) src/aarch64/Ginit.c is_plt_entry src/aarch64/Gstep.c _ULaarch64_step _ULaarch64_tdep_trace unw_backtrace <bthread/mutex.cpp static initializer: backtrace(dummy_buf, 4)> ← bRPC "Warm up backtrace before main()" __libc_csu_init ``` SIGSEGV with `si_addr=0`, AUXV `PAGESZ=65536`, and inside the libunwind cursor `ip=0`, `validate=0`, `mem_validate_func=msync_validate`, `last_good_addr[]={0,0,0,0}`. How it happens: 1. #64093 links GNU libunwind statically into the aarch64 BE (it forces `USE_UNWIND=ON` on Linux, references `unw_*` from `phdr_cache.cpp` without the `__x86_64__` guard, and builds jemalloc with `--enable-prof-libunwind`). libunwind's weak `backtrace` alias then replaces glibc's `backtrace()` for every caller in the process: bRPC's pre-main warm-up, `StackTrace::tryCapture()` on aarch64, jemalloc heap profiling. The 4.1.3 arm64 package was immune only because it was built with `USE_UNWIND=OFF` (which the ARM compilation doc still prescribes and #64093 removed): it has `backtrace@GLIBC_2.17` and zero `_ULaarch64_*` symbols. 2. libunwind 1.6.2 hard-codes `PAGE_SIZE 4096` in `src/aarch64/Ginit.c`. On a 64 KiB-page kernel `mincore()`/`msync()` reject its 4 KiB-aligned probes with `EINVAL`, so `validate_mem()` fails for 15/16 of all addresses (the core's `msync_validate` + empty `last_good_addr` are exactly this). 3. `unw_step()` therefore gets `unw_is_signal_frame() < 0` for a valid IP, takes the "IP points to non-mapped memory, use LR" recovery path and loads `uc.regs[30]` — a slot `_Uaarch64_getcontext_trace` never writes (it saves only FP/SP/PC). On the reporting host that stack garbage is 0. 4. `dwarf_step(ip=0)` finds nothing, and the fallback calls `is_plt_entry()` with `c->validate` restored to 0, so `access_mem` executes `ldr x8, [x1]` with `x1=0`. The PHDR cache being empty before `main()` is not what breaks this (step 3 has already poisoned `ip` before any FDE lookup, and the same crash reproduces without the Doris hook); changing the cache initialisation order would not fix it, and every later `backtrace()` on such a host rolls the same dice. Fix: **upgrade libunwind 1.6.2 → 1.8.3**, which was never bumped since it was introduced in 2023 (#21938): - 1.7.0+: `unw_page_size = sysconf(_SC_PAGESIZE)` instead of the hard-coded 4096. - 1.8.0+: `src/mi/Gaddress_validator.c` validates through the pipe-write probe only, so `mincore`/`msync` alignment no longer matters at all; `unw_step()` sets `c->validate = 1` before the DWARF-failure fallback, so a garbage IP is rejected instead of dereferenced. Changes: - `thirdparty/vars.sh`: libunwind 1.6.2 → 1.8.3 (md5 `13bc7b41462ac6ea157d350eaf6c1503`). - `thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch`: the #64093 hook rebased onto 1.8.3's `as->iterate_phdr_function` call site; semantics unchanged (`doris_unwind_iterate_phdr` still takes precedence whenever it is linked in). 1.8 also exposes `unw_set_iterate_phdr_function()`, which could replace the source patch in a follow-up. - `thirdparty/build-thirdparty.sh`: `--disable-tests --disable-documentation`; only `libunwind.a` is consumed. - `thirdparty/test/libunwind-page-size-test.sh` (+ run at the end of the Linux thirdparty CI build): reproduces the failure on any Linux host. An `LD_PRELOAD` shim gives libunwind a 64 KiB-page kernel's behaviour (`EINVAL` for unaligned `mincore`/`msync`, `sysconf(_SC_PAGESIZE)=65536`), and a program shaped like the bRPC warm-up calls `backtrace()` from a constructor with a zeroed stack. Against 1.6.2 it segfaults with the customer's exact stack and cursor state; against 1.8.3 it passes. No BE source change: every `unw_*` API the BE, jemalloc and the x86_64 stack-trace code use is unchanged in 1.8.3, `cxx_exceptions` stays off on x86_64/aarch64, and the weak `backtrace` alias is still provided. Follow-ups outside this PR: the doris-website ARM compilation page still says `export USE_UNWIND=OFF` is required, which has been a silent no-op since #64093. ### Release note Fix the aarch64 BE crashing before `main()` (SIGSEGV in `access_mem`/`is_plt_entry` under `unw_backtrace`) on 64 KiB-page kernels such as Kunpeng 920 with Kylin, openEuler or CentOS, by upgrading the bundled libunwind from 1.6.2 to 1.8.3. ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) - Symbolised the customer core against the matching 4.1.4 arm64 `doris_be` (gdb): crash stack, `si_addr=0`, `PAGESZ=65536`, cursor `ip=0`/`validate=0`, `mem_validate_func=msync_validate`, `last_good_addr={0}`, `uc.regs[30]=0`; `backtrace` == `unw_backtrace` in the 4.1.4 binary vs `backtrace@GLIBC_2.17` in 4.1.3. - Reproduced the identical crash on a 4 KiB arm64 Linux container by running the real 4.1.4 `doris_be --version` under the 64 KiB shim and zeroing `uc.regs[30]` in gdb. - Built libunwind 1.6.2 (+Doris patch) and 1.8.3 (+rebased patch) for aarch64 with the exact recipe from `build_libunwind()`; ran `thirdparty/test/libunwind-page-size-test.sh` against both: 1.6.2 FAIL (SIGSEGV, same stack), 1.8.3 PASS with and without the Doris hook, and with `ip` forced to 0 in gdb. - Same build + test on x86_64 (emulated ubuntu-22.04): PASS. - The rebased patch applies with both GNU `patch` and Apple `patch`; `thirdparty/test/download-thirdparty-*-test.sh`, `juicefs-default-mirror-test.sh`, `azure-vcpkg-retry-test.sh` pass; shellcheck reports no new findings. - Not done here: a Linux `doris_be` rebuilt with 1.8.3 on a real 64 KiB-page host. That is the one check that still needs an aarch64 machine. - For this pick: every hunk is identical to the master commit (only hunk offsets differ), and the rebased patch file and the new test script are byte-identical to master's. libunwind-1.8.3.tar.gz downloaded from the `vars.sh` URL has md5 `13bc7b41462ac6ea157d350eaf6c1503`; `libunwind-1.8.3-doris-phdr-cache.patch` applies to it with Apple `patch` (`--dry-run`, no fuzz, no rejects). `bash -n` on the four touched scripts, `shellcheck` on the new test script and `thirdparty/test/download-thirdparty-fallback-test.sh` pass on this branch. On master the same change passed `Build Third Party Libraries (Linux)` (with the new test at the end), `(macOS)`, `(macOS-arm64)` and `Thirdparty Script Test`. - Behavior changed: - [x] Yes. Bundled libunwind 1.6.2 → 1.8.3; libunwind's own test programs and man pages are no longer built. Unwinding semantics for the BE otherwise unchanged. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label Co-authored-by: Claude Opus 5 (1M context) <[email protected]> --- .github/workflows/build-thirdparty.yml | 5 + thirdparty/CHANGELOG.md | 11 ++ thirdparty/build-thirdparty.sh | 4 +- thirdparty/download-thirdparty.sh | 4 +- ...atch => libunwind-1.8.3-doris-phdr-cache.patch} | 8 +- thirdparty/test/libunwind-page-size-test.sh | 172 +++++++++++++++++++++ thirdparty/vars.sh | 8 +- 7 files changed, 201 insertions(+), 11 deletions(-) diff --git a/.github/workflows/build-thirdparty.yml b/.github/workflows/build-thirdparty.yml index 42f7abe95ac..3d3d11ff27d 100644 --- a/.github/workflows/build-thirdparty.yml +++ b/.github/workflows/build-thirdparty.yml @@ -174,6 +174,11 @@ jobs: #./build-thirdparty.sh -j "$(nproc)" ./build-thirdparty.sh -j 2 + # CI has no 64 KiB-page aarch64 host; this drives the libunwind just built + # through an emulated one, so it also runs on this x86_64 4 KiB runner. + cd "${GITHUB_WORKSPACE}" + thirdparty/test/libunwind-page-size-test.sh + build_macos: name: Build Third Party Libraries (macOS) needs: changes diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md index 0783980741b..38975986279 100644 --- a/thirdparty/CHANGELOG.md +++ b/thirdparty/CHANGELOG.md @@ -2,6 +2,17 @@ This file contains version of the third-party dependency libraries in the build-env image. The docker build-env image is apache/doris, and the tag is `build-env-${version}` +## 20260911 + +- Modified: libunwind 1.6.2 -> 1.8.3. 1.6.2 hard-codes a 4 KiB page size in the + AArch64 memory validator (`mincore`/`msync` on 4 KiB-aligned addresses), so on a + 64 KiB-page kernel every validation fails, `unw_step` falls back to a link register + that `getcontext_trace` never saved and dereferences it unvalidated. This crashed + every aarch64 BE on such kernels before `main()`, in bRPC's `backtrace()` warm-up. + 1.8.x reads the page size at runtime, validates through the pipe-write probe only, + and validates addresses before the DWARF-failure fallback. The Doris PHDR-cache hook + patch is rebased onto 1.8.3; the test programs and man pages are no longer built. + ## 20260901 - Modified: thrift 0.16.0 -> 0.24.0 diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh index 893825d7aa7..197b882761d 100755 --- a/thirdparty/build-thirdparty.sh +++ b/thirdparty/build-thirdparty.sh @@ -1707,7 +1707,9 @@ build_libunwind() { # LIBUNWIND_IS_NATIVE_ONLY: https://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160523/159802.html # -nostdinc++ only required for gcc compilation cflags="-I${TP_INCLUDE_DIR} -std=c99 -D_LIBUNWIND_NO_HEAP=1 -D_DEBUG -D_LIBUNWIND_IS_NATIVE_ONLY -O3 -fno-exceptions -funwind-tables -fno-sanitize=all -nostdinc++ -fno-rtti -Wno-error=incompatible-pointer-types" - CFLAGS="${cflags}" LDFLAGS="-L${TP_LIB_DIR} -llzma" ../configure --prefix="${TP_INSTALL_DIR}" --disable-shared --enable-static + # Only the library is consumed; the test programs and man pages are not. + CFLAGS="${cflags}" LDFLAGS="-L${TP_LIB_DIR} -llzma" ../configure --prefix="${TP_INSTALL_DIR}" --disable-shared --enable-static \ + --disable-tests --disable-documentation make -j "${PARALLEL}" make install diff --git a/thirdparty/download-thirdparty.sh b/thirdparty/download-thirdparty.sh index 5121d565445..d0c2eb0511c 100755 --- a/thirdparty/download-thirdparty.sh +++ b/thirdparty/download-thirdparty.sh @@ -492,10 +492,10 @@ fi # patch libunwind so Doris can force GNU libunwind to use the BE PHDR cache # without changing ordinary dl_iterate_phdr callers. if [[ " ${TP_ARCHIVES[*]} " =~ " LIBUNWIND " ]]; then - if [[ "${LIBUNWIND_SOURCE}" = "libunwind-1.6.2" ]]; then + if [[ "${LIBUNWIND_SOURCE}" = "libunwind-1.8.3" ]]; then cd "${TP_SOURCE_DIR}/${LIBUNWIND_SOURCE}" if [[ ! -f "${PATCHED_MARK}" ]]; then - patch -p1 <"${TP_PATCH_DIR}/libunwind-1.6.2-doris-phdr-cache.patch" + patch -p1 <"${TP_PATCH_DIR}/libunwind-1.8.3-doris-phdr-cache.patch" touch "${PATCHED_MARK}" fi cd - diff --git a/thirdparty/patches/libunwind-1.6.2-doris-phdr-cache.patch b/thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch similarity index 83% rename from thirdparty/patches/libunwind-1.6.2-doris-phdr-cache.patch rename to thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch index e534780b725..ea561b39a05 100644 --- a/thirdparty/patches/libunwind-1.6.2-doris-phdr-cache.patch +++ b/thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch @@ -1,8 +1,8 @@ diff --git a/src/dwarf/Gfind_proc_info-lsb.c b/src/dwarf/Gfind_proc_info-lsb.c -index 1d0d6a4..8f47463 100644 +index c11345e..837fbfe 100644 --- a/src/dwarf/Gfind_proc_info-lsb.c +++ b/src/dwarf/Gfind_proc_info-lsb.c -@@ -47,6 +47,14 @@ struct table_entry +@@ -46,6 +46,14 @@ struct table_entry #ifndef UNW_REMOTE_ONLY @@ -21,11 +21,11 @@ index 1d0d6a4..8f47463 100644 cb_data.di_debug.format = -1; SIGPROCMASK (SIG_SETMASK, &unwi_full_mask, &saved_mask); -- ret = dl_iterate_phdr (dwarf_callback, &cb_data); +- ret = as->iterate_phdr_function (dwarf_callback, &cb_data); + if (doris_unwind_iterate_phdr) + ret = doris_unwind_iterate_phdr (dwarf_callback, &cb_data, ip); + else -+ ret = dl_iterate_phdr (dwarf_callback, &cb_data); ++ ret = as->iterate_phdr_function (dwarf_callback, &cb_data); SIGPROCMASK (SIG_SETMASK, &saved_mask, NULL); if (ret > 0) diff --git a/thirdparty/test/libunwind-page-size-test.sh b/thirdparty/test/libunwind-page-size-test.sh new file mode 100755 index 00000000000..451316769cd --- /dev/null +++ b/thirdparty/test/libunwind-page-size-test.sh @@ -0,0 +1,172 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# libunwind 1.6.2 hard-coded a 4 KiB page size in its AArch64 memory validator. +# On a 64 KiB-page kernel (Kunpeng/Kylin/openEuler/CentOS aarch64) mincore() and +# msync() reject its 4 KiB-aligned probes with EINVAL, every validation fails, +# unw_step() falls back to a link register that getcontext_trace never saved, +# and is_plt_entry() dereferences that garbage unvalidated. Because the static +# libunwind's weak `backtrace` alias replaces glibc's, bRPC's pre-main +# `backtrace()` warm-up crashed every aarch64 BE on such hosts before main(). +# +# CI has no 64 KiB-page host, so this drives the installed libunwind.a through a +# program shaped like that warm-up while an LD_PRELOAD shim makes the kernel +# look like a 64 KiB-page one to libunwind: the same EINVAL for unaligned +# probes and a 64 KiB sysconf(_SC_PAGESIZE). Run it after +# `build-thirdparty.sh libunwind` on Linux; it only needs a C compiler. + +set -eo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)" +TP_INSTALL_DIR="${TP_INSTALL_DIR:-${ROOT}/installed}" + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +if [[ "$(uname -s)" != 'Linux' ]]; then + echo "SKIP: GNU libunwind is only built on Linux" + exit 0 +fi + +if [[ -n "${CC:-}" ]]; then + cc="${CC}" +else + for candidate in gcc clang cc; do + if command -v "${candidate}" >/dev/null 2>&1; then + cc="${candidate}" + break + fi + done +fi +[[ -n "${cc:-}" ]] || fail "no C compiler found; set CC" +[[ -f "${TP_INSTALL_DIR}/lib/libunwind.a" ]] || fail "${TP_INSTALL_DIR}/lib/libunwind.a not found; build libunwind first" + +tmpdir="$(mktemp -d)" +trap 'rm -rf "${tmpdir}"' EXIT + +cat >"${tmpdir}/shim64k.c" <<'EOF' +#define _GNU_SOURCE +#include <dlfcn.h> +#include <errno.h> +#include <stddef.h> +#include <stdint.h> +#include <sys/mman.h> +#include <unistd.h> + +/* What a 64 KiB-page kernel does with libunwind's validation probes. */ +#define EMU_PAGE 65536UL + +int mincore(void *addr, size_t len, unsigned char *vec) { + static int (*real)(void *, size_t, unsigned char *); + if (!real) real = dlsym(RTLD_NEXT, "mincore"); + if ((uintptr_t)addr & (EMU_PAGE - 1)) { errno = EINVAL; return -1; } + size_t n = (len + EMU_PAGE - 1) / EMU_PAGE; + for (size_t i = 0; i < n; i++) { + unsigned char tmp[EMU_PAGE / 4096]; + if (real((char *)addr + i * EMU_PAGE, EMU_PAGE, tmp) != 0) return -1; + unsigned char any = 0; + for (size_t j = 0; j < sizeof tmp; j++) any |= tmp[j] & 1; + vec[i] = any; + } + return 0; +} +int msync(void *addr, size_t len, int flags) { + static int (*real)(void *, size_t, int); + if (!real) real = dlsym(RTLD_NEXT, "msync"); + if ((uintptr_t)addr & (EMU_PAGE - 1)) { errno = EINVAL; return -1; } + return real(addr, len, flags); +} +long sysconf(int name) { + static long (*real)(int); + if (!real) real = dlsym(RTLD_NEXT, "sysconf"); + if (name == _SC_PAGESIZE) return EMU_PAGE; + return real(name); +} +int getpagesize(void) { return (int)EMU_PAGE; } +EOF + +cat >"${tmpdir}/premain.c" <<'EOF' +#define _GNU_SOURCE +#include <execinfo.h> +#include <link.h> +#include <stdio.h> +#include <string.h> +#include <unistd.h> + +/* be/src/common/phdr_cache.cpp before main(): the PHDR cache is still NULL, so the + * Doris hook that the libunwind patch routes dwarf_find_proc_info() through finds + * nothing. */ +int doris_unwind_iterate_phdr(int (*callback)(struct dl_phdr_info *, size_t, void *), void *data, + unsigned long ip) { + (void)callback; (void)data; (void)ip; + return 0; +} + +static void *dummy_buf[4]; +static int dummy_bt = -1; + +/* Zero the stack below us, like the never-touched region the warm-up ran on, so the + * uc.regs[30] slot that _Uaarch64_getcontext_trace never writes reads as 0. */ +static void __attribute__((noinline)) scrub_stack(void) { + volatile char buf[256 * 1024]; + memset((char *)buf, 0, sizeof(buf)); + __asm__ volatile("" ::: "memory"); +} + +/* bRPC src/bthread/mutex.cpp: "Warm up backtrace before main()." */ +__attribute__((constructor)) static void warm_up(void) { + scrub_stack(); + dummy_bt = backtrace(dummy_buf, 4); +} + +int main(void) { + void *buf[32]; + int n = backtrace(buf, 32); + printf("premain=%d runtime=%d pagesz=%ld\n", dummy_bt, n, sysconf(_SC_PAGESIZE)); + return (dummy_bt < 0 || n < 0) ? 1 : 0; +} +EOF + +extra_libs=() +[[ -f "${TP_INSTALL_DIR}/lib/libz.a" ]] && extra_libs+=(-lz) + +"${cc}" -O2 -g -shared -fPIC -o "${tmpdir}/shim64k.so" "${tmpdir}/shim64k.c" -ldl +# -u unw_backtrace pulls libunwind's backtrace.o in, whose weak `backtrace` alias then +# takes over the call, exactly as it does in doris_be. +"${cc}" -O2 -g -fno-omit-frame-pointer -o "${tmpdir}/premain" "${tmpdir}/premain.c" \ + -I"${TP_INSTALL_DIR}/include" -L"${TP_INSTALL_DIR}/lib" \ + -Wl,-u,unw_backtrace -lunwind -llzma "${extra_libs[@]}" -lpthread + +if command -v nm >/dev/null 2>&1; then + nm "${tmpdir}/premain" | grep -Eq ' [WT] backtrace$' || + fail "backtrace() is not bound to libunwind's unw_backtrace in the test program" +fi + +echo "== native page size ==" +out="$("${tmpdir}/premain")" || fail "pre-main backtrace() crashed on the native page size: ${out}" +echo "${out}" + +echo "== emulated 64 KiB page size ==" +out="$(LD_PRELOAD="${tmpdir}/shim64k.so" "${tmpdir}/premain")" || + fail "pre-main backtrace() crashed on an emulated 64 KiB-page kernel: ${out}" +echo "${out}" +[[ "${out}" == *"pagesz=65536"* ]] || fail "the 64 KiB page emulation did not take effect: ${out}" + +echo "PASS" diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh index 08bbf06681e..f86c204d579 100644 --- a/thirdparty/vars.sh +++ b/thirdparty/vars.sh @@ -320,10 +320,10 @@ JEMALLOC_DORIS_SOURCE="jemalloc-5.3.0" JEMALLOC_DORIS_MD5SUM="09a8328574dab22a7df848eae6dbbf53" # libunwind -LIBUNWIND_DOWNLOAD="https://github.com/libunwind/libunwind/releases/download/v1.6.2/libunwind-1.6.2.tar.gz"; -LIBUNWIND_NAME="libunwind-1.6.2.tar.gz" -LIBUNWIND_SOURCE="libunwind-1.6.2" -LIBUNWIND_MD5SUM="f625b6a98ac1976116c71708a73dc44a" +LIBUNWIND_DOWNLOAD="https://github.com/libunwind/libunwind/releases/download/v1.8.3/libunwind-1.8.3.tar.gz"; +LIBUNWIND_NAME="libunwind-1.8.3.tar.gz" +LIBUNWIND_SOURCE="libunwind-1.8.3" +LIBUNWIND_MD5SUM="13bc7b41462ac6ea157d350eaf6c1503" # cctz CCTZ_DOWNLOAD="https://github.com/google/cctz/archive/refs/tags/v2.5.tar.gz"; --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
