Acked-by: Michael D Kinney <michael.d.kin...@intel.com> > -----Original Message----- > From: Michael Kubacki <mikub...@linux.microsoft.com> > Sent: Monday, October 23, 2023 11:13 AM > To: devel@edk2.groups.io; Feng, Bob C <bob.c.f...@intel.com>; Gao, > Liming <gaolim...@byosoft.com.cn>; Kinney, Michael D > <michael.d.kin...@intel.com>; Rebecca Cran <rebe...@bsdio.com>; Sean > Brogan <sean.bro...@microsoft.com>; Chen, Christine > <yuwei.c...@intel.com> > Subject: Re: [edk2-devel] [PATCH v3 0/7] Use CodeQL CLI > > Another reminder. It would be nice to get this merged soon so actual > code fixes can follow. > > Thanks, > Michael > > On 10/19/2023 9:07 PM, Michael Kubacki wrote: > > A reminder to review this series. It's been on the mailing list for > a > > few weeks now. > > > > Thanks, > > Michael > > > > On 10/17/2023 9:04 PM, Michael Kubacki wrote: > >> From: Michael Kubacki <michael.kuba...@microsoft.com> > >> > >> CodeQL currently runs via the codeql-analysis.yml GitHub workflow > >> which uses the github/codeql-action/init@v2 action (pre-build) > >> and the github/codeql-action/analyze@v2 action (post-build) to > >> setup the CodeQL environment and extract results. > >> > >> This infrastructure is removed in preparation for a new design that > >> will directly run the CodeQL CLI as part of the build. This will > >> allow CodeQL to be run locally as part of the normal build process > >> with results that match 1:1 with CI builds. > >> > >> The CodeQL CLI design is automatically driven by a set of CodeQL > >> plugins: > >> > >> 1. `CodeQlBuildPlugin` - Used to produce a CodeQL database from > a > >> build. > >> 2. `CodeQlAnalyzePlugin` - Used to analyze a CodeQL database. > >> > >> This approach offers the following advantages: > >> > >> 1. Provides exactly the same results locally as on a CI server. > >> 2. Integrates very well into IDEs such as VS Code. > >> 3. Very simple to use - just use normal Stuart update and build > >> commands. > >> 4. Very simple to understand - minimally wraps the official > CodeQL > >> CLI. > >> 5. Very simple to integrate - works like any other Stuart build > >> plugin. > >> 6. Portable - not tied to Azure DevOps specific, GitHub > specific, > >> or other host infrastructure. > >> 7. Versioned - the query and filters are versioned in source > >> control so easy to find and track. > >> > >> The appropriate CodeQL CLI is downloaded for the host OS by passing > >> the `--codeql` argument to the update command. > >> > >> `stuart_update -c .pytool/CISettings.py --codeql` > >> > >> After that, CodeQL can be run in a build by similarly passing the > >> `--codeql` argument to the build command. For example: > >> > >> `stuart_ci_build -c .pytool/CISettings.py --codeql` > >> > >> Going forward, CI will simply use those commands in CodeQL builds > >> to get results instead of the CodeQL GitHub actions. > >> > >> When `--codeql` is specified in the build command, each package > will > >> contain two main artifacts in the Build directory. > >> > >> 1. The CodeQL database for the package > >> 2. The CodeQL SARIF (result) file for the package > >> > >> The CodeQL database (1) can be used to run queries against without > >> rebuilding any code. The SARIF result file (2) is the result of > >> running enabled queries against the database. > >> > >> SARIF stands for Static Analysis Results Interchange Format and it > >> is an industry standard format for output from static analysis > tools. > >> > >> https://sarifweb.azurewebsites.net/ > >> > >> The SARIF file can be opened with any standard SARIF file viewer > >> such as this one for VS Code: > >> > >> https://marketplace.visualstudio.com/items?itemName=MS- > SarifVSCode.sarif-viewer > >> > >> That includes the ability to jump directly to issues in the source > >> code file with relevant code highlighted and suggestions included. > >> > >> This means that after simply adding `--codeql` to the normal build > >> commands, a database will be present for future querying and a > SARIF > >> result file will be present to allow the developer to immediately > >> start fixing issues. > >> > >> More details about the location of these and usage is in the > >> BaseTools/Plugin/CodeQL/Readme.md included in this patch series. > >> > >> The CI process pushes the SARIF file to GitHub Code Scanning so the > >> results are generated exactly the same way they are locally. > >> > >> All build logs and the SARIF file for each package are uploaded to > >> the GitHub action run as artifacts. If a CodeQL issue is found, a > >> developer can download the SARIF file directly from the GitHub > action > >> run to fix the problem without needing to rebuild locally. > >> > >> An example run of these changes showing the packages built and > output > >> logs and SARIF files is available here: > >> > >> https://github.com/tianocore/edk2/actions/runs/6317077528 > >> > >> The series enables a new set of CodeQL queries that helps find > useful > >> issues in the codebase. So, new CodeQL results will appear in the > edk2 > >> GitHub Code Scanning area after the change. It is expected that the > >> community will work together to prioritize and resolve issues to > improve > >> the quality of the codebase. > >> > >> V3 Changes: > >> > >> 1. Add a "Resolution Guidelines" section to the CodeQL plugin > readme > >> file based on feedback in the October 16, 2023 Tianocore Tools > & > >> CI meeting to capture some notes useful in solving issues in > the > >> file. > >> > >> V2 Changes: > >> > >> 1. Enable CodeQL audit mode. This is because a new patch also > enables > >> queries that will result in unresolved issues so audit mode is > needed > >> for the build to succeed. > >> 2. Enable new CodeQL queries. This will enable new CodeQL queries > so the > >> issues are easier to find and track. > >> > >> Links and refernces: > >> > >> - CodeQL Overview: > >> https://codeql.github.com/docs/codeql-overview/ > >> - CodeQL open-source queries: > >> https://github.com/github/codeql > >> - CodeQL CLI: > >> https://docs.github.com/en/code-security/codeql-cli#codeql-cli > >> - SARIF Specification and Information: > >> https://sarifweb.azurewebsites.net/ > >> > >> Cc: Bob Feng <bob.c.f...@intel.com> > >> Cc: Liming Gao <gaolim...@byosoft.com.cn> > >> Cc: Michael D Kinney <michael.d.kin...@intel.com> > >> Cc: Rebecca Cran <rebe...@bsdio.com> > >> Cc: Sean Brogan <sean.bro...@microsoft.com> > >> Cc: Yuwei Chen <yuwei.c...@intel.com> > >> > >> Michael Kubacki (7): > >> Remove existing CodeQL infrastructure > >> BaseTools/Plugin/CodeQL: Add CodeQL build plugin > >> BaseTools/Plugin/CodeQL: Add integration helpers > >> .pytool/CISettings.py: Integrate CodeQL > >> .github/workflows/codeql.yml: Add CodeQL workflow > >> .pytool/CISettings: Enable CodeQL audit mode > >> BaseTools/Plugin/CodeQL: Enable 30 queries > >> > >> .github/codeql/codeql-config.yml | 29 -- > >> .github/codeql/edk2.qls | 24 -- > >> .github/workflows/codeql-analysis.yml | 118 ---- > -- > >> .github/workflows/codeql.yml | 338 > >> +++++++++++++++++ > >> .pytool/CISettings.py | 36 ++ > >> BaseTools/Plugin/CodeQL/CodeQlAnalyzePlugin.py | 222 > >> +++++++++++ > >> BaseTools/Plugin/CodeQL/CodeQlAnalyze_plug_in.yaml | 13 + > >> BaseTools/Plugin/CodeQL/CodeQlBuildPlugin.py | 172 > +++++++++ > >> BaseTools/Plugin/CodeQL/CodeQlBuild_plug_in.yaml | 13 + > >> BaseTools/Plugin/CodeQL/CodeQlQueries.qls | 118 > ++++++ > >> BaseTools/Plugin/CodeQL/Readme.md | 388 > >> ++++++++++++++++++++ > >> BaseTools/Plugin/CodeQL/analyze/__init__.py | 0 > >> BaseTools/Plugin/CodeQL/analyze/analyze_filter.py | 176 > +++++++++ > >> BaseTools/Plugin/CodeQL/analyze/globber.py | 132 > +++++++ > >> BaseTools/Plugin/CodeQL/codeqlcli_ext_dep.yaml | 26 ++ > >> BaseTools/Plugin/CodeQL/codeqlcli_linux_ext_dep.yaml | 24 ++ > >> BaseTools/Plugin/CodeQL/codeqlcli_windows_ext_dep.yaml | 24 ++ > >> BaseTools/Plugin/CodeQL/common/__init__.py | 0 > >> BaseTools/Plugin/CodeQL/common/codeql_plugin.py | 74 ++++ > >> BaseTools/Plugin/CodeQL/integration/__init__.py | 0 > >> BaseTools/Plugin/CodeQL/integration/stuart_codeql.py | 79 ++++ > >> 21 files changed, 1835 insertions(+), 171 deletions(-) > >> delete mode 100644 .github/codeql/codeql-config.yml > >> delete mode 100644 .github/codeql/edk2.qls > >> delete mode 100644 .github/workflows/codeql-analysis.yml > >> create mode 100644 .github/workflows/codeql.yml > >> create mode 100644 BaseTools/Plugin/CodeQL/CodeQlAnalyzePlugin.py > >> create mode 100644 > BaseTools/Plugin/CodeQL/CodeQlAnalyze_plug_in.yaml > >> create mode 100644 BaseTools/Plugin/CodeQL/CodeQlBuildPlugin.py > >> create mode 100644 > BaseTools/Plugin/CodeQL/CodeQlBuild_plug_in.yaml > >> create mode 100644 BaseTools/Plugin/CodeQL/CodeQlQueries.qls > >> create mode 100644 BaseTools/Plugin/CodeQL/Readme.md > >> create mode 100644 BaseTools/Plugin/CodeQL/analyze/__init__.py > >> create mode 100644 > BaseTools/Plugin/CodeQL/analyze/analyze_filter.py > >> create mode 100644 BaseTools/Plugin/CodeQL/analyze/globber.py > >> create mode 100644 BaseTools/Plugin/CodeQL/codeqlcli_ext_dep.yaml > >> create mode 100644 > BaseTools/Plugin/CodeQL/codeqlcli_linux_ext_dep.yaml > >> create mode 100644 > >> BaseTools/Plugin/CodeQL/codeqlcli_windows_ext_dep.yaml > >> create mode 100644 BaseTools/Plugin/CodeQL/common/__init__.py > >> create mode 100644 > BaseTools/Plugin/CodeQL/common/codeql_plugin.py > >> create mode 100644 > BaseTools/Plugin/CodeQL/integration/__init__.py > >> create mode 100644 > BaseTools/Plugin/CodeQL/integration/stuart_codeql.py > >>
-=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109955): https://edk2.groups.io/g/devel/message/109955 Mute This Topic: https://groups.io/mt/102031054/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/leave/9847357/21656/1706620634/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
