Directory structure
-------------------
-To keep things clean, I use the following directory structure
+To keep things clean, use the following directory structure
your root directory
- xen-release-logs [output/input directory]
- xen-release-scripts [git repo]
- - xsa-lists [output/input directory, see steps 1-5 in match-xsa]
+ - xsa-lists [output/input directory,
+ see steps 1-5 in match-xsa]
- xsa [git repo, optional]
+
Prerequisites
-------------
match-xsa and xen-release-logs, require perl and the following perl libraries
libfile-slurp-perl
- libfile-spec-perl
+ libfile-spec-perl
-See https://wiki.koha-community.org/wiki/How_to_install_missing_Perl_modules_on_Debian on how
-to install these
Tested on
---------
* MacOS Sierra, but do require an installation of pypy and wget
* Debian GNU/Linux 8 (jessie)
+
Tools overview
--------------
* xen-release-logs: produces logs for various xen-releases
-* match-xsa: checks whether XSAs have been applied to a specific Xen,
-QEMU traditional and QEMU upstream tree
-* make-webpage: creates webpages for https://xenproject.org/downloads/xen-archives.html
+* match-xsa: checks whether XSAs have been applied to a specific Xen,
+ QEMU traditional and QEMU upstream tree
+* make-webpage: creates webpages for
+ https://xenproject.org/downloads/xen-archives.html
Helper tools: can be executed by security team members only
-xsa-list-send: sends an XSA file to an e-mail address (not properly tested, needs editing)
+xsa-list-send: sends an XSA file to an e-mail address (not properly tested,
+needs editing)
+
Users
=====
* Security Team Members
* Release Managers and Release Maintainers
+
Logfiles, --version, --major, --since, --until and --logroot
------------------------------------------------------------
-The tools xen-release-logs, match-xsa and make-webpage use the following common options:
---version, --major, --since, --until and --logroot
+The tools xen-release-logs, match-xsa and make-webpage use the following
+common options:
+
+ --version, --major, --since, --until and --logroot
-Depending on --version, --major, --since and --until a log directory, which contains checked
-out git trees, will be created in the current directory or a directory specified with
---logroot
+Depending on --version, --major, --since and --until a log directory, which
+contains checked out git trees, will be created in the current directory or
+a directory specified with --logroot
These options are used in the following way
<tool> --version 4 --major 7
-Gets the logs for the entire 4.7 release up to the specific stable heads on the correct branch.
-In this example stable-4.7. The tools use the 4.6.0 release tags as common root. The log
-directory is called logs-47none-stable. Use for new major releases, when not yet tagged.
+Gets the logs for the entire 4.7 release up to the specific stable heads on
+the correct branch. In this example stable-4.7. The tools use the 4.6.0
+release tags as common root. The log directory is called logs-47none-stable.
+Use for new major releases, when not yet tagged.
<tool> --version 4 --major 7 --until 0
-This is a variant of the previous command, which is used when the tree has already been tagged
-(in this case with 4.7.0 tags). The log directory is called logs-47none-0. Use for new major
-releases, when the tree is tagged.
+This is a variant of the previous command, which is used when the tree has
+already been tagged (in this case with 4.7.0 tags). The log directory is
+called logs-47none-0. Use for new major releases, when the tree is tagged.
<tool> --version 4 --major 7 --since 0
-Gets the logs from 4.7.0 to 4.7.1 (with 4.7.1 not yet tagged). The log directory is called
-logs-470-stable.
+Gets the logs from 4.7.0 to 4.7.1 (with 4.7.1 not yet tagged). The log
+directory is called logs-470-stable.
<tool> --version 4 --major 7 --since 0 --until 1
-Gets the logs from 4.7.0 to 4.7.1 (with 4.7.1 already tagged). The log directory is called
-logs-470-1.
+Gets the logs from 4.7.0 to 4.7.1 (with 4.7.1 already tagged). The log
+directory is called logs-470-1.
Note: you can also use the options for several releases, e.g.
The log directory is called logs-470-4.
+
Developers and Users
--------------------
-For developers and users only the match-xsa tool is useful. However, the tool requires some
-input files, which currently can only be generated by security team members.
+For developers and users only the match-xsa tool is useful. However, the tool
+requires some input files, which currently can only be generated by security
+team members.
Create the following directories
- xen-release-logs [output/input directory]
- xsa-lists [save XSA list files in this directory]
*STEP 1:* go to download page of previous release (in this case:
-https://xenproject.org/downloads/xen-archives/xen-project-47-series/xen-472.html)
+https://xenproject.org/downloads/xen-archives/xen-project-47-series/
+xen-472.html)
*STEP 2:* write down last applied and verified XSA+1 (in this case XSA 210)
-*STEP 3:* ask a security team member to create an XSA file using ./Infra/xsa-list-patches
-and copy to file into xsa-lists with an appropriate name (e.g. xsa-210-222 or
-xsa-213-225)
+*STEP 3:* ask a security team member to create an XSA file using ./Infra/xsa-
+list-patches and copy to file into xsa-lists with an appropriate name (e.g.
+xsa-210-222 or xsa-213-225)
*STEP 4:* run the tool in dumb mode, which checks name signatures only.
- ./match-xsa --version 4 --major 8 --since 1 --xsa ../xsa-lists/xsa-213-225 --getlogs
- --html > ../xen-release-logs/481-stable-xsamatch.html
+ ./match-xsa --version 4 --major 8 --since 1 --getlogs --html
+ --xsa ../xsa-lists/xsa-213-225
+ > ../xen-release-logs/481-stable-xsamatch.html
-The above command line will create a report that shows which XSA patches have matching
-commit messages to those in the XSA list file. You will have to manually check whether all
-relevant patches have been applied. Typically where there are no matches, the XSA does not
-apply (e.g. the XSA may apply to Linux or older Xen releases only): you will have to go and
-double check the XSA, which can be easily accessed from within the
+The above command line will create a report that shows which XSA patches have
+matching commit messages to those in the XSA list file. You will have to
+manually check whether all relevant patches have been applied. Typically
+where there are no matches, the XSA does not apply (e.g. the XSA may apply to
+Linux or older Xen releases only): you will have to go and double check the
+XSA, which can be easily accessed from within the
-*STEP 5:* run the tool in smart mode, which checks against real patches rather then commit
-message titles. Note that this check is more fragile and can throw up false positives in the
-following situations:
+*STEP 5:* run the tool in smart mode, which checks against real patches
+rather then commit message titles. Note that this check is more fragile and
+can throw up errors in the following situations:
* A wrong patch has been applied
-* The committer has made changes to the patch when committing (this may be a real issue)
-* The *.patch file in the XSA has been created by a different diff version and file thunks
-are ordered differently (this is a tool issue that needs to be fixed).
+* The committer has made changes to the patch when committing (this may be a
+ real issue)
+* A bug in the match-xsa tool (none currently known)
- ./match-xsa --version 4 --major 8 --since 1 --xsa ../xsa-lists/xsa-213-225
- --html --smart > ../xen-release-logs/481-stable-xsamatch-smart.html
+ ./match-xsa --version 4 --major 8 --since 1 --html --smart
+ --xsa ../xsa-lists/xsa-213-225
+ > ../xen-release-logs/481-stable-xsamatch-smart.html
-Note that you can omit the --getlogs option, as you have the logs already. If there are
-unexplained. If there are unexplained discrepancies between
+Note that you can omit the --getlogs option, as you have the logs already.
-Also note that the tool will fetch XSA patches from xenbits.xenproject.org/xsa and store
-these in a ../xsaweb directory. The tool uses git show to fetch commits from tree and will
-compare these against patches.
+Also note that the tool will fetch XSA patches from xenbits.xenproject.org/
+xsa and store these in a ../xsaweb directory. The tool uses git show to fetch
+commits from tree and will compare these against patches.
-If at this stage, there no unexplained discrepancies between XSAs as published on
-xenbits.xenproject.org/xsa, you are done. If there are you can do the following...
+If at this stage, there no unexplained discrepancies between XSAs as
+published on xenbits.xenproject.org/xsa, you are done. If there are you can
+do the following...
*STEP 6:* run the tool in smart and debug mode
- ./match-xsa --version 4 --major 8 --since 1 --xsa ../xsa-lists/xsa-213-225
- --html --smart --debug > ../xen-release-logs/481-stable-xsamatch-smartd.html
+ ./match-xsa --version 4 --major 8 --since 1 --html --smart --debug
+ --xsa ../xsa-lists/xsa-213-225
+ > ../xen-release-logs/481-stable-xsamatch-smartd.html
-In this mode, git diffs, patches and logs are saved in a debug directory in the log directory
-and are accessible from the generated html (via a DEBUG link). You can look at these to
-investigate issues. Let's say a xen patch has been matched by commit message, the following
-files will be generated:
+In this mode, git diffs, patches and logs are saved in a debug directory in
+the log directory and are accessible from the generated html (via a DEBUG
+link). You can look at these to investigate issues. Let's say a xen patch has
+been matched by commit message, the following files will be generated:
* xen-git.txt : this is the original patch from git show
* xen-patch.txt : this is the matching patch from xenbits.xenproject.org/xsa
* xen-diff-n.txt: a diff between xen-git-n.txt and xen-patch-n.txt
* xen---log.txt : a log on the tool run for xen.git
-Similar files will be available for qemu traditional (qemut*) and qemu upstream (qemuu*).
+Similar files will be available for qemu traditional (qemut*) and qemu
+upstream (qemuu*).
+
Security Team Members
---------------------
-In addition to the regular workflow as outlined in the previous section, security team members
-have the capability to use match-xsa on xsa.git (only accessible for security team members).
+In addition to the regular workflow as outlined in the previous section,
+security team members have the capability to use match-xsa on xsa.git (only
+accessible for security team members).
To do this use
./match-xsa ... --xsadir <location of where you checked out xsa.git>
In addition, security team members can also create XSA list files using
-./Infra/xsa-list-patches or xsa-list-send (needs to be modified), which will call
-xsa-list-patches and send an email to either yourself or an another person.
+./Infra/xsa-list-patches or xsa-list-send (needs to be modified), which will
+call xsa-list-patches and send an email to either yourself or an another
+person.
Release Managers and Release Maintainers
Release Managers can use the following two tools:
-* xen-release-logs: creates change logs for xen, qemuu and qemut in various formats that can be copied into wikis and webpages
+* xen-release-logs: creates change logs for xen, qemuu and qemut in various
+ formats that can be copied into wikis and webpages
-* make-webpage (using the same options as the other tools with the addition of "--xsastart 210 --xsaend 225"). This tool generates text for the download page, with a table that needs to be manually populated.
+* make-webpage (using the same options as the other tools with the addition
+ of "--xsastart 210 --xsaend 225"). This tool generates text for the
+ download page, with a table that needs to be manually populated.
-TODOS AND OPEN ISSUES
-=====================
+To Dos and Open Issues
+======================
-*match-xsa*
-* The reporting can probably be improved: aka, highlight blocks with potential issues better. For example, embed relevant portions of an XSA into the output in cases where there is no match
-* --xsadir currently only works when providing an absolute path (the issue was introduced in commit 6d371d6adccc36eeeb5d2acd412ba0f3424a4a15)
-
-*xsa.git:Infra/xsa-list-patches*
+match-xsa
+---------
+* The reporting can probably be improved: aka, highlight blocks with
+ potential issues better. For example, embed relevant portions of an XSA
+ into the output in cases where there is no match
+* --xsadir currently only works when providing an absolute path (the issue
+ was introduced in commit 6d371d6adccc36eeeb5d2acd412ba0f3424a4a15)
+
+xsa.git:Infra/xsa-list-patches
+------------------------------
* Provide an option that allows to
-*make-xsa-list (new tool)*
-* Create a new tool that fetches the XSA webpage and produces output compatible to Infra/xsa-list-patches
+make-xsa-list (new tool)
+------------------------
+* Create a new tool that fetches the XSA webpage and produces output
+ compatible to Infra/xsa-list-patches
The file format is
<XSA-number><tab><patch file><tab><tab><commit message><newline>
-*make-webpage*
+make-webpage
+------------
* Make generation of XSA table fully automatic
projects / people / larsk / xen-release-scripts.git / commitdiff