jansvoboda11 updated this revision to Diff 323766.
jansvoboda11 added a comment.
Fix typos, add section heading above the list of marshalling classes, add new
section (high-level overview of adding new command line option)
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D95790/new/
https://reviews.llvm.org/D95790
Files:
clang/docs/InternalsManual.rst
Index: clang/docs/InternalsManual.rst
===================================================================
--- clang/docs/InternalsManual.rst
+++ clang/docs/InternalsManual.rst
@@ -572,6 +572,349 @@
The Frontend library contains functionality useful for building tools on top of
the Clang libraries, for example several methods for outputting diagnostics.
+Compiler Invocation
+-------------------
+
+One of the classes provided by the Frontend library is ``CompilerInvocation``,
+which holds information that describe current invocation of the Clang frontend.
+The information typically comes from the command line constructed by the Clang
+driver, or directly from clients performing custom initialization. The data
+structure is split into logical units used by different parts of the compiler,
+for example ``PreprocessorOptions``, ``LanguageOptions`` or ``CodeGenOptions``.
+
+Command Line Interface
+----------------------
+
+The command line interface of the Clang ``-cc1`` frontend is defined alongside
+the driver options in ``clang/Driver/Options.td``. The information making up an
+option definition includes the name and prefix (for example ``-std=``), form and
+position of the option value, help text, aliases and more. Each option may
+belong to a certain group and can be marked with zero or more flags. Options
+accepted by the ``-cc1`` frontend are marked with the ``CC1Option`` flag.
+
+Command Line Parsing
+--------------------
+
+Option definitions are processed by the ``-gen-opt-parser-defs`` tablegen
+backend, transformed into a list of invocations of the ``OPTION`` macro and
+stored in ``clang/Driver/Driver.inc``. This file is then used to create instance
+of ``llvm::opt::OptTable``, which acts as a command line preprocessor. The
+preprocessed command line is stored in ``llvm::opt::ArgList``, an object that
+provides an API for performing simple queries on the contents of the command
+line.
+
+Finally, the ``CompilerInvocation::CreateFromArgs`` function is responsible for
+the actual parsing of command line arguments. It maps the contents of the
+``ArgList`` onto fields of ``CompilerInvocation``, normalizing the values in the
+process.
+
+Command Line Generation
+-----------------------
+
+Any valid ``CompilerInvocation`` created from a ``-cc1`` command line can be
+also serialized back into semantically equivalent command line in a
+deterministic manner. This enables features such as implicitly discovered,
+explicitly built modules.
+
+..
+ TODO: Create and link corresponding section in Modules.rst.
+
+.. _OptionMarshalling:
+
+Option Marshalling Infrastructure
+---------------------------------
+
+The obvious way to parse command line arguments is to write the code that
+queries ``ArgList`` and constructs ``CompilerInvocation`` manually. However,
+given that behavior of most command line options is the same, this approach
+would result in lots of repetitive code. This is the reason why most of the
+code for parsing and generating arguments is automatically generated from the
+``Marshalling`` annotations that are part of the option definition in the
+tablegen file. This section goes through the details of the automatic
+marshalling system.
+
+To read and modify contents of ``CompilerInvocation``, the marshalling system
+uses key paths, which are declared in two steps. First, a tablegen definition
+for the ``CompilerInvocation`` member is created by inheriting from
+``KeyPathAndMacro``:
+
+.. code-block::
+
+ // Options.td
+
+ class LangOpts<string field> : KeyPathAndMacro<"LangOpts->", field, "LANG_"> {}
+ // CompilerInvocation member ^^^^^^^^^^
+ // OPTION_WITH_MARSHALLING prefix ^^^^^
+
+The first argument to the parent class is the beginning of the key path that
+references the ``CompilerInvocation`` member. This argument ends with ``->`` if
+the member is a pointer type or with ``.`` if it's a value type. The child class
+takes a single parameter ``field`` that is forwarded as the second argument to
+the base class. The child class can then be used like so:
+``LangOpts<"IgnoreExceptions">``, constructing a key path to the field
+``LangOpts->IgnoreExceptions``. The third argument passed to the parent class is
+a string that the tablegen backend uses as a prefix to the
+``OPTION_WITH_MARSHALLING`` macro. Using the key path then instructs the backend
+to generate the following code:
+
+.. code-block:: c++
+
+ // Options.inc
+
+ #ifdef LANG_OPTION_WITH_MARSHALLING
+ LANG_OPTION_WITH_MARSHALLING([...], LangOpts->IgnoreExceptions, [...])
+ #endif // LANG_OPTION_WITH_MARSHALLING
+
+Such definition can be used used in the function for parsing and generating
+command line:
+
+.. code-block:: c++
+
+ // CompilerInvoation.cpp
+
+ bool CompilerInvocation::ParseLangArgs(LangOptions *LangOpts, ArgList &Args,
+ DiagnosticsEngine &Diags) {
+ bool Success = true;
+
+ #define LANG_OPTION_WITH_MARSHALLING( \
+ PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM, \
+ HELPTEXT, METAVAR, VALUES, SPELLING, SHOULD_PARSE, ALWAYS_EMIT, KEYPATH, \
+ DEFAULT_VALUE, IMPLIED_CHECK, IMPLIED_VALUE, NORMALIZER, DENORMALIZER, \
+ MERGER, EXTRACTOR, TABLE_INDEX) \
+ PARSE_OPTION_WITH_MARSHALLING(Args, Diags, Success, ID, FLAGS, PARAM, \
+ SHOULD_PARSE, KEYPATH, DEFAULT_VALUE, \
+ IMPLIED_CHECK, IMPLIED_VALUE, NORMALIZER, \
+ MERGER, TABLE_INDEX)
+ #include "clang/Driver/Options.inc"
+ #undef LANG_OPTION_WITH_MARSHALLING
+
+ // ...
+
+ return Success;
+ }
+
+ void CompilerInvocation::GenerateLangArgs(LangOptions *LangOpts,
+ SmallVectorImpl<const char *> &Args,
+ StringAllocator SA) {
+ #define LANG_OPTION_WITH_MARSHALLING( \
+ PREFIX_TYPE, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM, \
+ HELPTEXT, METAVAR, VALUES, SPELLING, SHOULD_PARSE, ALWAYS_EMIT, KEYPATH, \
+ DEFAULT_VALUE, IMPLIED_CHECK, IMPLIED_VALUE, NORMALIZER, DENORMALIZER, \
+ MERGER, EXTRACTOR, TABLE_INDEX) \
+ GENERATE_OPTION_WITH_MARSHALLING( \
+ Args, SA, KIND, FLAGS, SPELLING, ALWAYS_EMIT, KEYPATH, DEFAULT_VALUE, \
+ IMPLIED_CHECK, IMPLIED_VALUE, DENORMALIZER, EXTRACTOR, TABLE_INDEX)
+ #include "clang/Driver/Options.inc"
+ #undef LANG_OPTION_WITH_MARSHALLING
+
+ // ...
+ }
+
+Option Marshalling Annotations
+------------------------------
+
+How does the tablegen backend know what to put in place of ``[...]`` in the
+generated ``Options.inc``? This is specified by the ``Marshalling`` utilities
+described below. All of them take a key path argument and possibly other
+information required for parsing or generating the command line argument.
+
+**Positive Flag**
+
+The key path defaults to ``false`` and is set to ``true`` when the flag is
+present on command line.
+
+.. code-block::
+
+ def fignore_exceptions : Flag<["-"], "fignore-exceptions">, Flags<[CC1Option]>,
+ MarshallingInfoFlag<LangOpts<"IgnoreExceptions">>;
+
+**Negative Flag**
+
+The key path defaults to ``true`` and is set to ``false`` when the flag is
+present on command line.
+
+.. code-block::
+
+ def fno_verbose_asm : Flag<["-"], "fno-verbose-asm">, Flags<[CC1Option]>,
+ MarshallingInfoNegativeFlag<CodeGenOpts<"AsmVerbose">>;
+
+**Negative and Positive Flag**
+
+The key path defaults to the specified value (``false``, ``true`` or some
+boolean value that's statically unknown in the tablegen file). Then, the key
+path is set to the value associated with the flag that appears last on command
+line.
+
+.. code-block::
+
+ defm legacy_pass_manager : BoolOption<"f", "legacy-pass-manager",
+ CodeGenOpts<"LegacyPassManager">, DefaultFalse,
+ PosFlag<SetTrue, [], "Use the legacy pass manager in LLVM">,
+ NegFlag<SetFalse, [], "Use the new pass manager in LLVM">,
+ BothFlags<[CC1Option]>>;
+
+With most such pair of flags, the ``-cc1`` frontend accepts only the flag that
+changes the default key path value. The Clang driver is responsible for
+accepting both and either forwarding the changing flag or discarding the flag
+that would just set the key path to its default.
+
+The first argument to ``BoolOption`` is a prefix that is used to construct the
+full names of both flags. The positive flag would then be named
+``flegacy-pass-manager`` and the negative ``fno-legacy-pass-manager``.
+``BoolOption`` also implies the ``-`` prefix for both flags. It's also possible
+to use ``BoolFOption`` that implies the ``"f"`` prefix and ``Group<f_Group>``.
+The ``PosFlag`` and ``NegFlag`` classes hold the associated boolean value, an
+array of elements passed to the ``Flag`` class and the help text. The optional
+``BothFlags`` class holds an array of ``Flag`` elements that are common for both
+the positive and negative flag and their common help text suffix.
+
+**String**
+
+The key path defaults to the specified string, or an empty one, if omitted. When
+the option appears on the command line, the argument value is simply copied.
+
+.. code-block::
+
+ def isysroot : JoinedOrSeparate<["-"], "isysroot">, Flags<[CC1Option]>,
+ MarshallingInfoString<HeaderSearchOpts<"Sysroot">, [{"/"}]>;
+
+**List of Strings**
+
+The key path defaults to an empty ``std::vector<std::string>``. Values specified
+with each appearance of the option on the command line are appended to the
+vector.
+
+.. code-block::
+
+ def frewrite_map_file : Separate<["-"], "frewrite-map-file">, Flags<[CC1Option]>,
+ MarshallingInfoStringVector<CodeGenOpts<"RewriteMapFiles">>;
+
+**Integer**
+
+The key path defaults to the specified integer value, or ``0`` if omitted. When
+the option appears on the command line, its value gets parsed by ``llvm::APInt``
+and the result is assigned to the key path on success.
+
+.. code-block::
+
+ def mstack_probe_size : Joined<["-"], "mstack-probe-size=">, Flags<[CC1Option]>,
+ MarshallingInfoStringInt<CodeGenOpts<"StackProbeSize">, "4096">;
+
+**Enumeration**
+
+The key path defaults to the value specified in ``MarshallingInfoEnum`` prefixed
+by the contents of ``NormalizedValuesScope`` and ``::``. This ensures correct
+reference to an enum case is formed even if the enum resides in different
+namespace or is an enum class. If the value present on command line does not
+match any of the comma-separated values from ``Values``, an error diagnostics is
+issued. Otherwise, the corresponding element from ``NormalizedValues`` at the
+same index is assigned to the key path (also correctly scoped). The number of
+comma-separated string values and elements of the array within
+``NormalizedValues`` must match.
+
+.. code-block::
+
+ def mthread_model : Separate<["-"], "mthread-model">, Flags<[CC1Option]>,
+ NormalizedValuesScope<"LangOptions::ThreadModelKind">,
+ MarshallingInfoEnum<LangOpts<"ThreadModel">, "POSIX">,
+ Values<"posix,single">,
+ NormalizedValues<["POSIX", "Single"]>;
+
+..
+ Intentionally omitting MarshallingInfoBitfieldFlag. It's adding some
+ complexity to the marshalling infrastructure and might be removed.
+
+It is also possible to define relationships between options.
+
+**Implication**
+
+The key path defaults to the default value from the primary ``Marshalling``
+annotation. Then, if any of the elements of ``ImpliedByAnyOf`` evaluate to true,
+the key path value is changed to the specified value or ``true`` if missing.
+Finally, the command line is parsed according to the primary annotation.
+
+.. code-block::
+
+ def fms_extensions : Flag<["-"], "fms-extensions">, Flags<[CC1Option]>,
+ MarshallingInfoFlag<LangOpts<"MicrosoftExt">>,
+ ImpliedByAnyOf<[fms_compatibility.KeyPath], "true">;
+
+**Condition**
+
+The option is parsed only if the expression in ``ShouldParseIf`` evaluates to
+true.
+
+.. code-block::
+
+ def fopenmp_enable_irbuilder : Flag<["-"], "fopenmp-enable-irbuilder">, Flags<[CC1Option]>,
+ MarshallingInfoFlag<LangOpts<"OpenMPIRBuilder">>,
+ ShouldParseIf<fopenmp.KeyPath>;
+
+Creating new Command Line Option
+--------------------------------
+
+When creating a new command line option, the first place of interest is
+``clang/include/clang/Driver/Options.td``, which contains the declarations of
+options recognized by Clang. New option always needs to be registered in this
+file.
+
+Internally, command line options are represented as instances of the ``Option``
+class which is defined in ``llvm/include/llvm/Option/OptParser.td``. An
+``Option`` instance can be created through one of the helper classes, which
+express the acceptable ways to pass the option value on the command line:
+
+* ``Flag`` - the option does not accept any value,
+* ``Joined`` - the value must immediately follow the option name within the same
+ argument,
+* ``Separate`` - the value must follow the option name in the next command line
+ argument,
+* ``JoinedOrSeparate`` - the value can be specified either as ``Joined`` or
+ ``Separate``,
+* ``CommaJoined`` - the values are comma-separated and must immediately follow
+ the option name within the same argument (see ``Wl,`` for an example).
+
+Note that besides the option name itself, the helper classes also take a list of
+prefixes (typically ``"-"``, ``"--"`` or ``"/"``). One of the prefixes must be
+present immediately before the option name on the command line.
+
+Then, additional attributes of the option may be specified via mix-in classes:
+
+* ``Alias`` denotes that the option is an alias of another option. This may be
+ combined with ``AliasArgs``, which denotes the value of the target option
+ implied by the alias option.
+* ``HelpText`` holds the text that will be printed when the user requests help
+ (e.g. via ``clang --help``).
+* ``Group`` specifies the collection of options this option belongs to, which is
+ used by various consumers of the ``Options.td`` file to filter options.
+* ``Flags`` may contain a number of flags associated with the option. This
+ enables more granular filtering than the ``Group`` attribute.
+
+After the option is specified in the tablegen file, it can be used and queried
+throught an instance of ``llvm::opt::ArgList``. This is typically done in two
+places: in the Clang driver when parsing the command line and constructing jobs,
+or in the frontend (e.g. in ``-cc1`` when parsing the command line and creating
+``CompilerInvocation`` instance).
+
+When it comes to the ``-cc1`` command line, it has one interesting feature: any
+``CompilerInvocation`` constructed from the command line can be serialized back
+into a list of semantically equivalent command line arguments. This means that
+any option that's parsed by ``CompilerInvocation::CreateFromArgs`` must be also
+generated by ``CompilerInvocation::generateCC1CommandLine``. This can be
+achieved by manually implementing both, or by using the declarative
+:ref:`marshalling infrastructure <OptionMarshalling>`.
+
+Using the marshalling infrastructure is preferred, as it reduces the amount of
+boilerplate code one needs to write. However, the number of supported command
+line constructs is intentionally limited for the sake of simplicity (both of the
+implementation and of the command line interface). This means that developers
+will find themselves in situations where the marshalling infrastructure is
+unable to encode the desired behavior. This can be solved by resorting to manual
+implementation of the parsing and generating code, or by simplifying the command
+line behavior. Note that simplification of the ``-cc1`` command line interface
+does not necessitate changes in the interface of the driver, which means making
+retroactive changes or introducing options incompatible with the GCC interface
+is entirely possible.
+
The Lexer and Preprocessor Library
==================================
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
