Memory tagging is used for detecting memory safety bugs. On AArch64, the
memory tagging extension (MTE) helps in reducing the overheads of memory
tagging:
- CPU: MTE instructions for efficiently tagging and untagging memory.
- Memory: New memory type, Normal Tagged Memory, added to the Arm
Architecture.
The MEMory TAGging (MEMTAG) sanitizer uses the same infrastructure as
HWASAN. MEMTAG and HWASAN are both hardware-assisted solutions, and
rely on the same sanitizer machinery in parts. So, define new
constructs that allow MEMTAG and HWASAN to share the infrastructure:
- hwassist_sanitize_p () is true when either SANITIZE_MEMTAG or
SANITIZE_HWASAN is true.
- hwassist_sanitize_stack_p () is when hwassist_sanitize_p () and
stack variables are to be sanitized.
MEMTAG and HWASAN do have differences, however, and hence, the need to
conditionalize using memtag_sanitize_p () in the relevant places. E.g.,
- Instead of generating the libcall __hwasan_tag_memory, MEMTAG needs
to invoke the target-specific hook TARGET_MEMTAG_TAG_MEMORY to tag
memory. Similar approach can be seen for handling
handle_builtin_alloca, where instead of doing the gimple
transformations, target hooks are used.
- Add a new internal function HWASAN_ALLOCA_POISON to handle
dynamically allocated stack when MEMTAG sanitizer is enabled. At
expansion, this allows to, in turn, invoke target-hooks to increment
tag, and use the generated tag to finally tag the dynamically allocated
memory.
- HWASAN uses HWASAN_STACK_BACKGROUND as the background color. In
case of MEMTAG, we simply pick stack_pointer_rtx. See
hwasan_emit_untag_frame ().
- HWASAN relies on CFN_HWASAN_CHOOSE_TAG to extract the tag if
necessary. For MEMTAG, however, since there is no hardware
instruction to extract tag from register, we make the implementation
diverge a bit.
The usual pattern:
irg x0, x0, x0
subg x0, x0, #16, #0
creates a tag in x0 and so on. For alloca, we need to apply the
generated tag to the new sp. In absense of an extract tag insn, the
implemenation in expand_HWASAN_ALLOCA_POISON resorts to invoking irg
again.
TBD:
- Not sure if we really need param_memtag_instrument_mem_intrinsics
explicitly.
- Conditionalizing using hwassist_sanitize_p (), memtag_sanitize_p ()
etc looks unappetizing in some cases. Not sure if there is a better
way. Is this generally the right thing to do, or is there some
desirable refactorings.
- adding decl to hwasan_stack_var. double check if this is necessary.
See how we update the RTL for decl at expand_one_stack_var_at. And
then use the RTL for decl in hwasan_emit_prologue. Add testcases
around this.
- In hwasan_frame_base (), see if checking for memtag_sanitize_p () for
force_reg etc is really necessary. Revisit to see what gives, fix or
add documentation.
- Error out if user specifies stack alloc alignment not a factor of 16 ?
gcc/ChangeLog:
* asan.cc (struct hwasan_stack_var):
(handle_builtin_stack_restore): Accommodate MEMTAG sanitizer.
(handle_builtin_alloca): Expand differently if MEMTAG sanitizer.
(get_mem_refs_of_builtin_call): Include MEMTAG along with
HWASAN.
(memtag_sanitize_stack_p): New definition.
(memtag_sanitize_allocas_p): Likewise.
(memtag_memintrin): Likewise.
(hwassist_sanitize_p): Likewise.
(hwassist_sanitize_stack_p): Likewise.
(report_error_func): Include MEMTAG along with HWASAN.
(build_check_stmt): Likewise.
(instrument_derefs): MEMTAG too does not deal with globals yet.
(instrument_builtin_call):
(maybe_instrument_call): Include MEMTAG along with HWASAN.
(asan_expand_mark_ifn): Likewise.
(asan_expand_check_ifn): Likewise.
(asan_expand_poison_ifn): Expand differently if MEMTAG sanitizer.
(asan_instrument):
(hwasan_frame_base):
(hwasan_record_stack_var):
(hwasan_emit_prologue): Expand differently if MEMTAG sanitizer.
(hwasan_emit_untag_frame): Likewise.
* asan.h (hwasan_record_stack_var):
(memtag_sanitize_stack_p): New declaration.
(memtag_sanitize_allocas_p): Likewise.
(hwassist_sanitize_p): Likewise.
(hwassist_sanitize_stack_p): Likewise.
(asan_sanitize_use_after_scope): Include MEMTAG along with
HWASAN.
* cfgexpand.cc (align_local_variable): Likewise.
(expand_one_stack_var_at): Likewise.
(expand_stack_vars): Likewise.
(expand_one_stack_var_1): Likewise.
(init_vars_expansion): Likewise.
(expand_used_vars): Likewise.
(pass_expand::execute): Likewise.
* gimplify.cc (asan_poison_variable): Likewise.
* internal-fn.cc (expand_HWASAN_ALLOCA_POISON): New definition.
(expand_HWASAN_ALLOCA_UNPOISON): Expand differently if MEMTAG
sanitizer.
(expand_HWASAN_MARK): Likewise.
* internal-fn.def (HWASAN_ALLOCA_POISON): Define new.
* params.opt: Document new param. FIXME.
* sanopt.cc (pass_sanopt::execute): Include MEMTAG along with
HWASAN.
---
[Changes from RFC V1]
- Bugfixes
[End of changes from RFC V1]
---
gcc/asan.cc | 245 +++++++++++++++++++++++++++++++++-----------
gcc/asan.h | 9 +-
gcc/cfgexpand.cc | 37 ++++---
gcc/gimplify.cc | 5 +-
gcc/internal-fn.cc | 73 +++++++++++--
gcc/internal-fn.def | 1 +
gcc/params.opt | 4 +
gcc/sanopt.cc | 2 +-
8 files changed, 286 insertions(+), 90 deletions(-)
diff --git a/gcc/asan.cc b/gcc/asan.cc
index 0123ed415a0a..d71a540edc52 100644
--- a/gcc/asan.cc
+++ b/gcc/asan.cc
@@ -298,6 +298,7 @@ static GTY(()) rtx_insn *hwasan_frame_base_init_seq = NULL;
tagged_base). */
struct hwasan_stack_var
{
+ tree decl;
rtx untagged_base;
rtx tagged_base;
poly_int64 nearest_offset;
@@ -762,14 +763,15 @@ static void
handle_builtin_stack_restore (gcall *call, gimple_stmt_iterator *iter)
{
if (!iter
- || !(asan_sanitize_allocas_p () || hwasan_sanitize_allocas_p ()))
+ || !(asan_sanitize_allocas_p () || hwasan_sanitize_allocas_p ()
+ || memtag_sanitize_allocas_p ()))
return;
tree restored_stack = gimple_call_arg (call, 0);
gimple *g;
- if (hwasan_sanitize_allocas_p ())
+ if (hwasan_sanitize_allocas_p () || memtag_sanitize_allocas_p ())
{
enum internal_fn fn = IFN_HWASAN_ALLOCA_UNPOISON;
/* There is only one piece of information `expand_HWASAN_ALLOCA_UNPOISON`
@@ -818,7 +820,8 @@ static void
handle_builtin_alloca (gcall *call, gimple_stmt_iterator *iter)
{
if (!iter
- || !(asan_sanitize_allocas_p () || hwasan_sanitize_allocas_p ()))
+ || !(asan_sanitize_allocas_p () || hwasan_sanitize_allocas_p ()
+ || memtag_sanitize_allocas_p ()))
return;
gassign *g;
@@ -842,23 +845,31 @@ handle_builtin_alloca (gcall *call, gimple_stmt_iterator
*iter)
e = find_fallthru_edge (gsi_bb (*iter)->succs);
}
- if (hwasan_sanitize_allocas_p ())
+ if (hwasan_sanitize_allocas_p () || memtag_sanitize_allocas_p ())
{
gimple_seq stmts = NULL;
location_t loc = gimple_location (gsi_stmt (*iter));
- /*
- HWASAN needs a different expansion.
+ /* HWASAN and MEMTAG need a different expansion.
addr = __builtin_alloca (size, align);
- should be replaced by
+ in case of HWASAN, should be replaced by
new_size = size rounded up to HWASAN_TAG_GRANULE_SIZE byte alignment;
untagged_addr = __builtin_alloca (new_size, align);
tag = __hwasan_choose_alloca_tag ();
addr = ifn_HWASAN_SET_TAG (untagged_addr, tag);
__hwasan_tag_memory (untagged_addr, tag, new_size);
- */
+
+ in case of MEMTAG, should be replaced by
+
+ new_size = size rounded up to HWASAN_TAG_GRANULE_SIZE byte alignment;
+ untagged_addr = __builtin_alloca (new_size, align);
+ addr = ifn_HWASAN_ALLOCA_POISON (untagged_addr, new_size);
+
+ where a new tag is chosen and set on untagged_addr when
+ HWASAN_ALLOCA_POISON is expanded. */
+
/* Ensure alignment at least HWASAN_TAG_GRANULE_SIZE bytes so we start on
a tag granule. */
align = align > HWASAN_TAG_GRANULE_SIZE ? align :
HWASAN_TAG_GRANULE_SIZE;
@@ -874,23 +885,30 @@ handle_builtin_alloca (gcall *call, gimple_stmt_iterator
*iter)
as_combined_fn (BUILT_IN_ALLOCA_WITH_ALIGN), ptr_type,
new_size, build_int_cst (size_type_node, align));
- /* Choose the tag.
- Here we use an internal function so we can choose the tag at expand
- time. We need the decision to be made after stack variables have been
- assigned their tag (i.e. once the hwasan_frame_tag_offset variable has
- been set to one after the last stack variables tag). */
- tree tag = gimple_build (&stmts, loc, CFN_HWASAN_CHOOSE_TAG,
- unsigned_char_type_node);
-
- /* Add tag to pointer. */
- tree addr
- = gimple_build (&stmts, loc, CFN_HWASAN_SET_TAG, ptr_type,
- untagged_addr, tag);
+ tree addr;
- /* Tag shadow memory.
- NOTE: require using `untagged_addr` here for libhwasan API. */
- gimple_build (&stmts, loc, as_combined_fn (BUILT_IN_HWASAN_TAG_MEM),
- void_type_node, untagged_addr, tag, new_size);
+ if (memtag_sanitize_p ())
+ addr = gimple_build (&stmts, loc, CFN_HWASAN_ALLOCA_POISON, ptr_type,
+ untagged_addr, new_size);
+ else
+ {
+ /* Choose the tag.
+ Here we use an internal function so we can choose the tag at expand
+ time. We need the decision to be made after stack variables have
been
+ assigned their tag (i.e. once the hwasan_frame_tag_offset variable
has
+ been set to one after the last stack variables tag). */
+ tree tag = gimple_build (&stmts, loc, CFN_HWASAN_CHOOSE_TAG,
+ unsigned_char_type_node);
+
+ /* Add tag to pointer. */
+ addr = gimple_build (&stmts, loc, CFN_HWASAN_SET_TAG, ptr_type,
+ untagged_addr, tag);
+
+ /* Tag shadow memory.
+ NOTE: require using `untagged_addr` here for libhwasan API. */
+ gimple_build (&stmts, loc, as_combined_fn (BUILT_IN_HWASAN_TAG_MEM),
+ void_type_node, untagged_addr, tag, new_size);
+ }
/* Insert the built up code sequence into the original instruction stream
the iterator points to. */
@@ -1104,7 +1122,7 @@ get_mem_refs_of_builtin_call (gcall *call,
for now we choose to just ignore `strlen` calls.
This decision was simply made because that means the special case is
limited to this one case of this one function. */
- if (hwasan_sanitize_p ())
+ if (hwassist_sanitize_p ())
return false;
source0 = gimple_call_arg (call, 0);
len = gimple_call_lhs (call);
@@ -1887,9 +1905,39 @@ hwasan_memintrin (void)
return (hwasan_sanitize_p () && param_hwasan_instrument_mem_intrinsics);
}
-/* MEMoryTAGging sanitizer (memtag) uses a hardware based capability known as
- memory tagging to detect memory safety vulnerabilities. Similar to hwasan,
- it is also a probabilistic method. */
+/* MEMoryTAGging sanitizer (MEMTAG) uses a hardware based capability known as
+ memory tagging to detect memory safety vulnerabilities. Similar to HWASAN,
+ it is also a probabilistic method.
+
+ MEMTAG relies on the optional extension in armv8.5a, known as MTE (Memory
+ Tagging Extension). The extension is available in AARCH64 only and
+ introduces two types of tags:
+ - Logical Address Tag - bits 56-59 (TARGET_MEMTAG_TAG_SIZE) of the virtual
+ address.
+ - Allocation Tag - 4 bits for each tag granule (TARGET_MEMTAG_GRANULE_SIZE
+ set to 16 bytes), stored separately.
+ Load / store instructions raise an exception if tags differ, thereby
+ providing a faster way (than HWASAN) to detect memory safety issues.
+ Further, new instructions are available in MTE to manipulate (generate,
+ update address with) tags. Load / store instructions with SP base register
+ and immediate offset do not check tags.
+
+ PS: Currently, MEMTAG sanitizer is capable of stack (variable / memory)
+ tagging only.
+
+ In general, detecting stack-related memory bugs requires the compiler to:
+ - ensure that each tag granule is only used by one variable at a time.
+ This includes alloca.
+ - Tag/Color: put tags into each stack variable pointer.
+ - Untag: the function epilogue will retag the memory.
+
+ MEMTAG sanitizer is based off the HWASAN sanitizer implementation
+ internally. Similar to HWASAN:
+ - Assigning an independently random tag to each variable is carried out by
+ keeping a tagged base pointer. A tagged base pointer allows addressing
+ variables with (addr offset, tag offset).
+ - TBD
+ */
/* Returns whether we are tagging pointers and checking those tags on memory
access. */
@@ -1899,6 +1947,42 @@ memtag_sanitize_p ()
return false;
}
+/* Are we tagging the stack? */
+bool
+memtag_sanitize_stack_p ()
+{
+ return (memtag_sanitize_p () && param_memtag_instrument_stack);
+}
+
+/* Are we tagging alloca objects? */
+bool
+memtag_sanitize_allocas_p (void)
+{
+ return (memtag_sanitize_stack_p () && param_memtag_instrument_allocas);
+}
+
+/* Are we taggin mem intrinsics? */
+bool
+memtag_memintrin (void)
+{
+ return (memtag_sanitize_p () && param_memtag_instrument_mem_intrinsics);
+}
+
+/* Returns whether we are tagging pointers and checking those tags on memory
+ access. */
+bool
+hwassist_sanitize_p ()
+{
+ return (hwasan_sanitize_p () || memtag_sanitize_p ());
+}
+
+/* Are we tagging stack objects for hwasan or memtag? */
+bool
+hwassist_sanitize_stack_p ()
+{
+ return (hwasan_sanitize_stack_p () || memtag_sanitize_stack_p ());
+}
+
/* Insert code to protect stack vars. The prologue sequence should be emitted
directly, epilogue sequence returned. BASE is the register holding the
stack base, against which OFFSETS array offsets are relative to, OFFSETS
@@ -2432,7 +2516,7 @@ static tree
report_error_func (bool is_store, bool recover_p, HOST_WIDE_INT size_in_bytes,
int *nargs)
{
- gcc_assert (!hwasan_sanitize_p ());
+ gcc_assert (!hwassist_sanitize_p ());
static enum built_in_function report[2][2][6]
= { { { BUILT_IN_ASAN_REPORT_LOAD1, BUILT_IN_ASAN_REPORT_LOAD2,
@@ -2771,7 +2855,7 @@ build_check_stmt (location_t loc, tree base, tree len,
if (is_scalar_access)
flags |= ASAN_CHECK_SCALAR_ACCESS;
- enum internal_fn fn = hwasan_sanitize_p ()
+ enum internal_fn fn = hwassist_sanitize_p ()
? IFN_HWASAN_CHECK
: IFN_ASAN_CHECK;
@@ -2871,7 +2955,7 @@ instrument_derefs (gimple_stmt_iterator *iter, tree t,
access is inside a global variable, then there's no point adding
instrumentation to check the access. N.b. hwasan currently never
sanitizes globals. */
- if ((hwasan_sanitize_p () || !param_asan_globals)
+ if ((hwassist_sanitize_p () || !param_asan_globals)
&& is_global_var (inner))
return;
if (!TREE_STATIC (inner))
@@ -2970,7 +3054,8 @@ instrument_mem_region_access (tree base, tree len,
static bool
instrument_builtin_call (gimple_stmt_iterator *iter)
{
- if (!(asan_memintrin () || hwasan_memintrin ()))
+ if (!(asan_memintrin () || hwasan_memintrin ()
+ || memtag_memintrin ()))
return false;
bool iter_advanced_p = false;
@@ -3124,7 +3209,7 @@ maybe_instrument_call (gimple_stmt_iterator *iter)
`longjmp`, thread exit, and exceptions in a different way. These
problems must be handled externally to the compiler, e.g. in the
language runtime. */
- if (! hwasan_sanitize_p ())
+ if (! hwassist_sanitize_p ())
{
tree decl = builtin_decl_implicit (BUILT_IN_ASAN_HANDLE_NO_RETURN);
gimple *g = gimple_build_call (decl, 0);
@@ -3877,7 +3962,7 @@ asan_expand_mark_ifn (gimple_stmt_iterator *iter)
gcc_checking_assert (TREE_CODE (decl) == VAR_DECL);
- if (hwasan_sanitize_p ())
+ if (hwassist_sanitize_p ())
{
gcc_assert (param_hwasan_instrument_stack);
gimple_seq stmts = NULL;
@@ -3975,7 +4060,7 @@ asan_expand_mark_ifn (gimple_stmt_iterator *iter)
bool
asan_expand_check_ifn (gimple_stmt_iterator *iter, bool use_calls)
{
- gcc_assert (!hwasan_sanitize_p ());
+ gcc_assert (!hwassist_sanitize_p ());
gimple *g = gsi_stmt (*iter);
location_t loc = gimple_location (g);
bool recover_p;
@@ -4250,7 +4335,7 @@ asan_expand_poison_ifn (gimple_stmt_iterator *iter,
int nargs;
bool store_p = gimple_call_internal_p (use, IFN_ASAN_POISON_USE);
gcall *call;
- if (hwasan_sanitize_p ())
+ if (hwassist_sanitize_p ())
{
tree fun = builtin_decl_implicit (BUILT_IN_HWASAN_TAG_MISMATCH4);
/* NOTE: hwasan has no __hwasan_report_* functions like asan does.
@@ -4351,7 +4436,7 @@ asan_expand_poison_ifn (gimple_stmt_iterator *iter,
static unsigned int
asan_instrument (void)
{
- if (hwasan_sanitize_p ())
+ if (hwassist_sanitize_p ())
{
initialize_sanitizer_builtins ();
transform_statements ();
@@ -4480,10 +4565,15 @@ hwasan_frame_base ()
if (! hwasan_frame_base_ptr)
{
start_sequence ();
- hwasan_frame_base_ptr
- = force_reg (Pmode,
- targetm.memtag.insert_random_tag (virtual_stack_vars_rtx,
- NULL_RTX));
+ if (memtag_sanitize_p ())
+ hwasan_frame_base_ptr
+ = targetm.memtag.insert_random_tag (virtual_stack_vars_rtx,
+ NULL_RTX);
+ else
+ hwasan_frame_base_ptr
+ = force_reg (Pmode,
+ targetm.memtag.insert_random_tag (virtual_stack_vars_rtx,
+ NULL_RTX));
hwasan_frame_base_init_seq = get_insns ();
end_sequence ();
}
@@ -4538,10 +4628,11 @@ hwasan_maybe_emit_frame_base_init ()
We record the `untagged_base` since the functions in the hwasan library we
use to tag memory take pointers without a tag. */
void
-hwasan_record_stack_var (rtx untagged_base, rtx tagged_base,
+hwasan_record_stack_var (tree decl, rtx untagged_base, rtx tagged_base,
poly_int64 nearest_offset, poly_int64 farthest_offset)
{
hwasan_stack_var cur_var;
+ cur_var.decl = decl;
cur_var.untagged_base = untagged_base;
cur_var.tagged_base = tagged_base;
cur_var.nearest_offset = nearest_offset;
@@ -4693,19 +4784,39 @@ hwasan_emit_prologue ()
gcc_assert (multiple_p (bot, HWASAN_TAG_GRANULE_SIZE));
gcc_assert (multiple_p (size, HWASAN_TAG_GRANULE_SIZE));
- rtx fn = init_one_libfunc ("__hwasan_tag_memory");
- rtx base_tag = targetm.memtag.extract_tag (cur.tagged_base, NULL_RTX);
- rtx tag = plus_constant (QImode, base_tag, cur.tag_offset);
- tag = hwasan_truncate_to_tag_size (tag, NULL_RTX);
-
- rtx bottom = convert_memory_address (ptr_mode,
- plus_constant (Pmode,
- cur.untagged_base,
- bot));
- emit_library_call (fn, LCT_NORMAL, VOIDmode,
- bottom, ptr_mode,
- tag, QImode,
- gen_int_mode (size, ptr_mode), ptr_mode);
+ if (memtag_sanitize_p ())
+ {
+ rtx x = NULL_RTX;
+ if (HAS_RTL_P (cur.decl))
+ x = XEXP (DECL_RTL (cur.decl), 0);
+ else if (SSA_NAME_VAR (cur.decl)
+ && (VAR_P (SSA_NAME_VAR (cur.decl))
+ || SSA_NAME_IS_DEFAULT_DEF (cur.decl)))
+ {
+ tree var = SSA_NAME_VAR (cur.decl);
+ x = XEXP (DECL_RTL (var), 0);
+ }
+ if (x != NULL_RTX)
+ targetm.memtag.tag_memory (x, gen_int_mode (size, ptr_mode), x);
+ }
+ else
+ {
+ rtx fn = init_one_libfunc ("__hwasan_tag_memory");
+
+ rtx bottom = convert_memory_address (ptr_mode,
+ plus_constant (Pmode,
+ cur.untagged_base,
+ bot));
+
+ rtx base_tag = targetm.memtag.extract_tag (cur.tagged_base, NULL_RTX);
+ rtx tag = plus_constant (QImode, base_tag, cur.tag_offset);
+ tag = hwasan_truncate_to_tag_size (tag, NULL_RTX);
+
+ emit_library_call (fn, LCT_NORMAL, VOIDmode,
+ bottom, ptr_mode,
+ tag, QImode,
+ gen_int_mode (size, ptr_mode), ptr_mode);
+ }
}
/* Clear the stack vars, we've emitted the prologue for them all now. */
hwasan_tagged_stack_vars.truncate (0);
@@ -4746,11 +4857,25 @@ hwasan_emit_untag_frame (rtx dynamic, rtx vars)
NULL_RTX, /* unsignedp = */0,
OPTAB_DIRECT);
- rtx fn = init_one_libfunc ("__hwasan_tag_memory");
- emit_library_call (fn, LCT_NORMAL, VOIDmode,
- bot_rtx, ptr_mode,
- HWASAN_STACK_BACKGROUND, QImode,
- size_rtx, ptr_mode);
+ if (memtag_sanitize_p ())
+ {
+ /* FIXME - not sure if this is OK to do. */
+ if (!cfun->calls_alloca)
+ {
+ HOST_WIDE_INT size = frame_offset.to_constant ();
+ size_rtx = gen_int_mode (size, ptr_mode);
+ }
+
+ targetm.memtag.tag_memory (bot_rtx, size_rtx, stack_pointer_rtx);
+ }
+ else
+ {
+ rtx fn = init_one_libfunc ("__hwasan_tag_memory");
+ emit_library_call (fn, LCT_NORMAL, VOIDmode,
+ bot_rtx, ptr_mode,
+ HWASAN_STACK_BACKGROUND, QImode,
+ size_rtx, ptr_mode);
+ }
do_pending_stack_adjust ();
rtx_insn *insns = get_insns ();
diff --git a/gcc/asan.h b/gcc/asan.h
index c3d5b311d300..832e743401db 100644
--- a/gcc/asan.h
+++ b/gcc/asan.h
@@ -39,7 +39,7 @@ extern void
asan_maybe_insert_dynamic_shadow_at_function_entry (function *);
extern void hwasan_record_frame_init ();
-extern void hwasan_record_stack_var (rtx, rtx, poly_int64, poly_int64);
+extern void hwasan_record_stack_var (tree, rtx, rtx, poly_int64, poly_int64);
extern void hwasan_emit_prologue ();
extern rtx_insn *hwasan_emit_untag_frame (rtx, rtx);
extern rtx hwasan_get_frame_extent ();
@@ -58,6 +58,11 @@ extern bool hwasan_expand_mark_ifn (gimple_stmt_iterator *);
extern bool gate_hwasan (void);
extern bool memtag_sanitize_p (void);
+extern bool memtag_sanitize_stack_p (void);
+extern bool memtag_sanitize_allocas_p (void);
+
+bool hwassist_sanitize_p (void);
+bool hwassist_sanitize_stack_p (void);
extern gimple_stmt_iterator create_cond_insert_point
(gimple_stmt_iterator *, bool, bool, bool, basic_block *, basic_block *);
@@ -227,7 +232,7 @@ inline bool
asan_sanitize_use_after_scope (void)
{
return (flag_sanitize_address_use_after_scope
- && (asan_sanitize_stack_p () || hwasan_sanitize_stack_p ()));
+ && (asan_sanitize_stack_p () || hwassist_sanitize_stack_p ()));
}
/* Return true if DECL should be guarded on the stack. */
diff --git a/gcc/cfgexpand.cc b/gcc/cfgexpand.cc
index 2b27076658fd..8b27c2c71b86 100644
--- a/gcc/cfgexpand.cc
+++ b/gcc/cfgexpand.cc
@@ -381,7 +381,7 @@ align_local_variable (tree decl, bool really_expand)
else
align = LOCAL_DECL_ALIGNMENT (decl);
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
align = MAX (align, (unsigned) HWASAN_TAG_GRANULE_SIZE * BITS_PER_UNIT);
if (TREE_CODE (decl) != SSA_NAME && really_expand)
@@ -1328,7 +1328,7 @@ expand_one_stack_var_at (tree decl, rtx base, unsigned
base_align,
/* If this fails, we've overflowed the stack frame. Error nicely? */
gcc_assert (known_eq (offset, trunc_int_for_mode (offset, Pmode)));
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
x = targetm.memtag.add_tag (base, offset,
hwasan_current_frame_tag ());
else
@@ -1463,14 +1463,14 @@ expand_stack_vars (bool (*pred) (unsigned), class
stack_vars_data *data)
if (pred && !pred (i))
continue;
- base = (hwasan_sanitize_stack_p ()
+ base = (hwassist_sanitize_stack_p ()
? hwasan_frame_base ()
: virtual_stack_vars_rtx);
alignb = stack_vars[i].alignb;
if (alignb * BITS_PER_UNIT <= MAX_SUPPORTED_STACK_ALIGNMENT)
{
poly_int64 hwasan_orig_offset;
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
{
/* There must be no tag granule "shared" between different
objects. This means that no HWASAN_TAG_GRANULE_SIZE byte
@@ -1569,7 +1569,7 @@ expand_stack_vars (bool (*pred) (unsigned), class
stack_vars_data *data)
offset = alloc_stack_frame_space (stack_vars[i].size, alignb);
base_align = crtl->max_used_stack_slot_alignment;
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
{
/* Align again since the point of this alignment is to handle
the "end" of the object (i.e. smallest address after the
@@ -1583,7 +1583,8 @@ expand_stack_vars (bool (*pred) (unsigned), class
stack_vars_data *data)
allocated for this particular variable while `offset`
describes the address that this variable starts at. */
align_frame_offset (HWASAN_TAG_GRANULE_SIZE);
- hwasan_record_stack_var (virtual_stack_vars_rtx, base,
+ hwasan_record_stack_var (stack_vars[i].decl,
+ virtual_stack_vars_rtx, base,
hwasan_orig_offset, frame_offset);
}
}
@@ -1614,7 +1615,7 @@ expand_stack_vars (bool (*pred) (unsigned), class
stack_vars_data *data)
large_alloc = aligned_upper_bound (large_alloc, alignb);
offset = large_alloc;
large_alloc += stack_vars[i].size;
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
{
/* An object with a large alignment requirement means that the
alignment requirement is greater than the required alignment
@@ -1630,7 +1631,8 @@ expand_stack_vars (bool (*pred) (unsigned), class
stack_vars_data *data)
then use positive offsets from that. Hence the farthest
offset is `align_again` and the nearest offset from the base
is `offset`. */
- hwasan_record_stack_var (large_untagged_base, large_base,
+ hwasan_record_stack_var (stack_vars[i].decl,
+ large_untagged_base, large_base,
offset, align_again);
}
@@ -1645,7 +1647,7 @@ expand_stack_vars (bool (*pred) (unsigned), class
stack_vars_data *data)
expand_one_stack_var_at (stack_vars[j].decl,
base, base_align, offset);
}
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
hwasan_increment_frame_tag ();
}
@@ -1738,7 +1740,7 @@ expand_one_stack_var_1 (tree var)
gcc_assert (byte_align * BITS_PER_UNIT <= MAX_SUPPORTED_STACK_ALIGNMENT);
rtx base;
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
{
/* Allocate zero bytes to align the stack. */
poly_int64 hwasan_orig_offset
@@ -1754,7 +1756,7 @@ expand_one_stack_var_1 (tree var)
the "furthest" offset from the base delimiting the current stack
object. `frame_offset` will always delimit the extent that the frame.
*/
- hwasan_record_stack_var (virtual_stack_vars_rtx, base,
+ hwasan_record_stack_var (var, virtual_stack_vars_rtx, base,
hwasan_orig_offset, frame_offset);
}
else
@@ -1766,7 +1768,7 @@ expand_one_stack_var_1 (tree var)
expand_one_stack_var_at (var, base,
crtl->max_used_stack_slot_alignment, offset);
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
hwasan_increment_frame_tag ();
}
@@ -2370,7 +2372,7 @@ init_vars_expansion (void)
/* Initialize local stack smashing state. */
has_protected_decls = false;
has_short_buffer = false;
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
hwasan_record_frame_init ();
}
@@ -2696,13 +2698,14 @@ expand_used_vars (bitmap forced_stack_vars)
expand_stack_vars (NULL, &data);
}
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
hwasan_emit_prologue ();
if (asan_sanitize_allocas_p () && cfun->calls_alloca)
var_end_seq = asan_emit_allocas_unpoison (virtual_stack_dynamic_rtx,
virtual_stack_vars_rtx,
var_end_seq);
- else if (hwasan_sanitize_allocas_p () && cfun->calls_alloca)
+ else if ((hwasan_sanitize_allocas_p () || memtag_sanitize_p ())
+ && cfun->calls_alloca)
/* When using out-of-line instrumentation we only want to emit one function
call for clearing the tags in a region of shadow stack. When there are
alloca calls in this frame we want to emit a call using the
@@ -2710,7 +2713,7 @@ expand_used_vars (bitmap forced_stack_vars)
rtx we created in expand_stack_vars. */
var_end_seq = hwasan_emit_untag_frame (virtual_stack_dynamic_rtx,
virtual_stack_vars_rtx);
- else if (hwasan_sanitize_stack_p ())
+ else if (hwassist_sanitize_stack_p ())
/* If no variables were stored on the stack, `hwasan_get_frame_extent`
will return NULL_RTX and hence `hwasan_emit_untag_frame` will return
NULL (i.e. an empty sequence). */
@@ -7215,7 +7218,7 @@ pass_expand::execute (function *fun)
emit_insn_after (var_ret_seq, after);
}
- if (hwasan_sanitize_stack_p ())
+ if (hwassist_sanitize_stack_p ())
hwasan_maybe_emit_frame_base_init ();
/* Zap the tree EH table. */
diff --git a/gcc/gimplify.cc b/gcc/gimplify.cc
index 4f385b1b779b..1e6e3db5a570 100644
--- a/gcc/gimplify.cc
+++ b/gcc/gimplify.cc
@@ -1310,9 +1310,10 @@ asan_poison_variable (tree decl, bool poison,
gimple_stmt_iterator *it,
/* It's necessary to have all stack variables aligned to ASAN granularity
bytes. */
- gcc_assert (!hwasan_sanitize_p () || hwasan_sanitize_stack_p ());
+ gcc_assert (!hwassist_sanitize_p () || hwassist_sanitize_stack_p ());
unsigned shadow_granularity
- = hwasan_sanitize_p () ? HWASAN_TAG_GRANULE_SIZE : ASAN_SHADOW_GRANULARITY;
+ = (hwassist_sanitize_p ()
+ ? HWASAN_TAG_GRANULE_SIZE : ASAN_SHADOW_GRANULARITY);
if (DECL_ALIGN_UNIT (decl) <= shadow_granularity)
SET_DECL_ALIGN (decl, BITS_PER_UNIT * shadow_granularity);
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 21eac80819a5..c5cbd53372cc 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -748,6 +748,43 @@ expand_HWASAN_CHECK (internal_fn, gcall *)
gcc_unreachable ();
}
+/* For hwasan stack tagging:
+ Tag memory which is dynamically allocated. */
+static void
+expand_HWASAN_ALLOCA_POISON (internal_fn, gcall *gc)
+{
+ gcc_assert (ptr_mode == Pmode);
+ tree g_target = gimple_call_lhs (gc);
+ tree g_ptr = gimple_call_arg (gc, 0);
+ tree g_size = gimple_call_arg (gc, 1);
+
+ rtx ptr = expand_normal (g_ptr);
+ /* Get new tag for the alloca'd memory.
+ Doing a regular add_tag () like so:
+ rtx tag = targetm.memtag.add_tag (hwasan_frame_base (), 0,
+ hwasan_current_frame_tag ());
+ gets a new tag, which can be used for tagging memory. But for alloca, we
+ need both tagged memory and a tagged pointer to pass to consumers. Invoke
+ insert_random_tag () instead to add a random tag to ptr to get a tagged
+ pointer that will work for both purposes. */
+ rtx tagged_ptr = targetm.memtag.insert_random_tag (ptr, NULL_RTX);
+ rtx size = expand_normal (g_size);
+ rtx target = expand_normal (g_target);
+
+ if (memtag_sanitize_p ())
+ {
+ /* Need to put the tagged ptr into the `target` RTX for consumers
+ of alloca'd memory. */
+ if (tagged_ptr != target)
+ emit_move_insn (target, tagged_ptr);
+ /* Tag the memory. */
+ targetm.memtag.tag_memory (ptr, size, tagged_ptr);
+ hwasan_increment_frame_tag ();
+ }
+ else
+ gcc_unreachable ();
+}
+
/* For hwasan stack tagging:
Clear tags on the dynamically allocated space.
For use after an object dynamically allocated on the stack goes out of
@@ -759,14 +796,27 @@ expand_HWASAN_ALLOCA_UNPOISON (internal_fn, gcall *gc)
tree restored_position = gimple_call_arg (gc, 0);
rtx restored_rtx = expand_expr (restored_position, NULL_RTX, VOIDmode,
EXPAND_NORMAL);
- rtx func = init_one_libfunc ("__hwasan_tag_memory");
rtx off = expand_simple_binop (Pmode, MINUS, restored_rtx,
stack_pointer_rtx, NULL_RTX, 0,
OPTAB_WIDEN);
- emit_library_call_value (func, NULL_RTX, LCT_NORMAL, VOIDmode,
- virtual_stack_dynamic_rtx, Pmode,
- HWASAN_STACK_BACKGROUND, QImode,
- off, Pmode);
+
+ if (memtag_sanitize_p ())
+ {
+ emit_insn (targetm.memtag.tag_memory (virtual_stack_dynamic_rtx,
+ off,
+ virtual_stack_dynamic_rtx));
+ }
+ else
+ {
+ rtx func = init_one_libfunc ("__hwasan_tag_memory");
+ rtx off = expand_simple_binop (Pmode, MINUS, restored_rtx,
+ stack_pointer_rtx, NULL_RTX, 0,
+ OPTAB_WIDEN);
+ emit_library_call_value (func, NULL_RTX, LCT_NORMAL, VOIDmode,
+ virtual_stack_dynamic_rtx, Pmode,
+ HWASAN_STACK_BACKGROUND, QImode,
+ off, Pmode);
+ }
}
/* For hwasan stack tagging:
@@ -820,9 +870,16 @@ expand_HWASAN_MARK (internal_fn, gcall *gc)
tree len = gimple_call_arg (gc, 2);
rtx r_len = expand_normal (len);
- rtx func = init_one_libfunc ("__hwasan_tag_memory");
- emit_library_call (func, LCT_NORMAL, VOIDmode, address, Pmode,
- tag, QImode, r_len, Pmode);
+ if (memtag_sanitize_p ())
+ {
+ emit_insn (targetm.memtag.tag_memory (address, r_len, tag));
+ }
+ else
+ {
+ rtx func = init_one_libfunc ("__hwasan_tag_memory");
+ emit_library_call (func, LCT_NORMAL, VOIDmode, address, Pmode,
+ tag, QImode, r_len, Pmode);
+ }
}
/* For hwasan stack tagging:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 8edfa3540f8c..d096856d2e98 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -489,6 +489,7 @@ DEF_INTERNAL_FN (UBSAN_PTR, ECF_LEAF | ECF_NOTHROW, ". R .
")
DEF_INTERNAL_FN (UBSAN_OBJECT_SIZE, ECF_LEAF | ECF_NOTHROW, NULL)
DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN, NULL)
DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (HWASAN_ALLOCA_POISON, ECF_LEAF | ECF_NOTHROW, NULL)
DEF_INTERNAL_FN (HWASAN_ALLOCA_UNPOISON, ECF_LEAF | ECF_NOTHROW, ". R ")
DEF_INTERNAL_FN (HWASAN_CHOOSE_TAG, ECF_LEAF | ECF_NOTHROW, ". ")
DEF_INTERNAL_FN (HWASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW,
diff --git a/gcc/params.opt b/gcc/params.opt
index 77ece0c50512..3d624e38bf0d 100644
--- a/gcc/params.opt
+++ b/gcc/params.opt
@@ -102,6 +102,10 @@ When sanitizing using MTE instructions, add checks for all
stack automatics.
Target Joined UInteger Var(param_memtag_instrument_allocas) Init(1)
IntegerRange(0, 1) Param
When sanitizing using MTE instructions, add checks for all stack allocas.
+-param=memtag-instrument-mem-intrinsics=
+Common Joined UInteger Var(param_memtag_instrument_mem_intrinsics) Init(1)
IntegerRange(0, 1) Param Optimization
+When sanitizing using MTE instructions, include builtin functions.
+
-param=avg-loop-niter=
Common Joined UInteger Var(param_avg_loop_niter) Init(10) IntegerRange(1,
65536) Param Optimization
Average number of iterations of a loop.
diff --git a/gcc/sanopt.cc b/gcc/sanopt.cc
index ff5a5ff2231d..33f62540747d 100644
--- a/gcc/sanopt.cc
+++ b/gcc/sanopt.cc
@@ -1330,7 +1330,7 @@ pass_sanopt::execute (function *fun)
sanitize_asan_mark_poison ();
}
- if (asan_sanitize_stack_p () || hwasan_sanitize_stack_p ())
+ if (asan_sanitize_stack_p () || hwassist_sanitize_stack_p ())
sanitize_rewrite_addressable_params (fun);
bool use_calls = param_asan_instrumentation_with_call_threshold < INT_MAX
--
2.43.0
