From c07d6d67c735511ef39dfd2a1fb2a623c0c78957 Mon Sep 17 00:00:00 2001 From: George Dunlap Date: Fri, 16 Jun 2017 11:33:03 +0100 Subject: [PATCH] Include README.md This is based on the design document posted before, but is updated with more workflow description, as well as future development plans. Signed-off-by: George Dunlap --- README.md | 320 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 320 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..7b9da84 --- /dev/null +++ b/README.md @@ -0,0 +1,320 @@ +# Purpose + +The primary goal of xsatool is to help verify that the patches we're +going to send out apply and build properly. + +A secondary goal is to help in the generation of those patches. + +In all cases it is meant to do a lot of the "busy-work" which is +boring, repetitive, and error-prone when humans do it, but quick, +efficient, and predictable when computers do it. + +See the **Workflow** section for a quick usage guide. + +# Design + +Our current standard practice is to provide a patch or series (or +multiple patches or series), such `master`, as well as all currently +supported versions under security support, will have fixes which apply +via `git am`. + +However, these patches: + +1. Apply to a specific tree (`xen.git`, `qemu-xen.git`, + `qemu-xen-traditional.git`) + +2. Apply to a specific point on some branch (`master` for + `xen-unstable`, `stable-${v}` for released versions). + +3. May rely on previous unreleased XSAs. + +This tool introduces the concept of a *recipe*. Each supported +release will have its own recipe. The recipe will include the +specific tree it's targetet at, the *baseline* in `stable-${v}` on +which it's applied, as well as any XSA prerequisites. The recipe +would ideally be both machine- and human-readiable, and live in +`xsa.git`. + +Which in turns leads us to the next thing: rather than having to +specify each time what XSAs need to be applied, this should happen +automatically. There should be global metadata that contains: + +- All the currently supported Xen versions + +- All XSAs not yet in the various stable branches + +When a new XSA metadata is created with xsatool, "empty" recipes +should be created based for each currently-supported version, +including all the previous XSAs on that tree for that version. The +new XSA should also be added to the list of "Current XSAs". (This in +particular will need humans to do adjustment when circumstances don't +allow the release in a nice clean order.) + +It should probably be obvious that creating this XSA 'recipe' (as +opposed to allocating the XSA number) shouldn't be done until a patch +is reasonably close to being ready; XSA-115 provides a good reason +why. + +The actual creation and manipulation of the patches will be highly +context-dependent and developer-driven; to that end the standard will +be to create branches with a specific naming scheme in the various +trees, which developers can then manipuate. xsatool will have +commands to 'sync' between the trees. + +# Data files + +## .xsatool + +xsatool expects a top-level directory that contains a file named +`.xsatool`. This configuration file contains paths for xen, +qemu-upstream, qemu-traditional, and xsa.git. `xsatool` will +manipulate these working trees freely, so it is highly advised that +these trees be separate from normal development trees. + +## xsa.git/global.meta + +Lives in `xsa.git`. Contains metadata about the XSA tree as a whole. +At the moment this includes what versions of Xen are currently +supported, as well as any outstanding (uncommitted) XSAs. + +## xsa.git/xsa${N}.meta + +Lives in `xsa.git`. Contains metadata about XSA ${N}. Includes which +versions of Xen and which trees this particular XSA applies to, as +well as recipes for applying the patch. The recipe includes a +reference to the point in the stable tree to which the patch will +apply, other prerequisite XSAs which must be applied first, and a +"glob" of files to apply for this patch. + +## Data format + +The current version of the tool uses JSON to store data. It is very +easy in Go to marshall and unmarshall structures to and from JSON, but +it is still somewhat human-readable and editable (although it's not +particularly great). + +How human-readable it is depends somewhat on the way the Go structure +is formatted; I've tried to format the structures such that they have +reasonable-looking JSON output which can be edited by hand without too +much effort if need be. + +# Trees + +## Branch names + +When an "empty" XSA is created, two three will be created in each tree +for which patches are required: + + xsa/${n}/${v}-stable + xsa/${n}/${v}-baseline + xsa/${n}/${v} + +`${v}` is the major release version. The `-stable` branch points to +the commit in the `stable-${v}` tree at the point at which the xsa was +created. `baseline` is `stable` with all of the prerequisite XSAs +applied. The last branch is the branch that contains the actual fix. + +## Generating patches from a branch + +To create a set of patches for a particular xen version for a +particular xsa, the tool will run `git format-patch +xsa/${n}/${v}-baseline..xsa/${n}/${v}` into a temporary directory. + +If the result is only a single patch, then the patch will be copied to +xsa.git/xsa${n}-${VERSION}.patch. If the result is more than one patch, +the patches will be copied into xsa.git/xsa${n}-${version}/. + +# Initial setup + +## Creating initial setup + +`xsatool repo init` + +This command will, in the current directory: + +- Create a new `.xsatool` if one doesn't exist. If it does, it will +be consulted. + +- For each of `xen.git`, `qemu-xen.git`, `qemu-xen-traditional.git`, +and `xsa.git`, it will clone a new copy if one doesn't exist already. +Existing repositories will be left alone. + +- It will also disable the default push on all of those repositories, +even if they exist already. (Command `git config +config.origin.pushurl http://localhost/fail`.) + +**TODO**: Enable `git rerere` when creating new repositories + +## Creating or updating the initial global metadata + +`xsatool global update` + +If this is run and `xsa.git/global.meta` doesn't exist, it will create +the file. It firsts asks what the oldest supported Xen version is. +It then goes through `xen.git` and looks for `RELEASE-M.m.p` tags, and +marks each version greater than or equal to the specified version as +supported. + +If `global.meta` does exist, it simply does a refresh; this will +automatically add support for new versions. + +# Workflow + +## New Xen XSA + +Using XSA 213 as an example. + +`xsatool 213 init xen` + + - Create `xsa213.meta` with "empty" recipes for master and all + currently-supported versions; in this case these versions will + probably be `master`, `4.8`, `4.7`, `4.6`, and `4.5`. Tell + `xsatool` that `xen` is the only tree that needs patches. + + - Take the current value of `xen.git` `stable-${v}` for the "stable + baseline" for each version + + - In `xen.git`: + + - create branches `xsa/213/xsa/${v}-{stable,baseline}` + + - create branch `xsa/213/master` + +The developer now generates patch for XSA 213 on `xsa/213/master`. + +`xsatool 213 backport` + + - Starting with the latest release and going backwards, looks for + versions such that `xsa/213/$v` doesn't exist. + + - It will then create a branch which is a clone of `xsa/213/${v+1}`, + and then rebase onto `xsa/213/${v}-baseline`. + +If the rebase fails, the developer fixes it up just like she would +normally. + +If the rebase succeeds, the tool still stops, giving the developer a +chance to inspect and test the backport. + +The developer repeats `xsatool 213 backport` until all backports have +been made. + +`xsatool 213 sync-patches` + +For each version fro which `xsa/213/${v}` exists, this will: + + - Delete any patches in xsa.git which match its file generation pattern. + + - Generate patches in xsa.git + + - Update the recipe for that version in `xsa213.meta` + +## Checking patches before sending them out + +`xsatool 213 test-apply` + +For each version for which patches exist in `xsa213.meta`, it creates +`test/213/$v-stable`, `test/213/$v-baseline`, and `test/213/$v` + +Individual versions can be specified as well: `xsatool 213 test-apply 4.6 4.5` + +`xsatool 213 test-build` + +For each version for which patches exist in `xsa213.meta`, it will +attempt to check out a tree based on `test/213/$v` and do a build. If +the branch does not exist it will throw an error. + +As with `test-apply`, you can also specify individual versions: +`xsatool 213 test-build 4.5` + +`xsatool 213 test` + +Do both `test-apply` and `test-build` (i.e., a full test) for each +version. + +## Viewing patches that another developer has created + +Suppose instead you want to review another developer's series; or +suppose another developer has made update to your own series, and you +want to update your copy. + +Assuming the developer has checked her changes into `xsa.git`, update +your copy of `xsa.git`, and then + +`xsatool 213 sync-branch` + +For each version specified, it will reset all branches (`-stable`, +`-baseline`, and patch branch) according to the updated recipe in +`xsa213.meta`. + +As with `sync-patches`, you can either specify a specific version, or +you can leave the version blank and it will make branches for versions +*for which recipes exist*. + +This could also be useful, for example, if prerequisites are manually +added or removed. + +# Potential future functionality + +## Automatically de-duplicating patches + +At the moment the tool will always create a patch or a patch series +for each supported version, even if a single patch can apply to +multiple versions. Detecting this automatically would be one of the +obvious next steps. + +## Removing xsas from the 'outstanding' list + +Either something like: + +`xsatool global end-support nnn` + +Or having `global update` automatically detect that patches have been +applied to stable branches. + +## Generating osstest test runes automatically + +## Rebasing + +- Rebasing to new versions of `stable` or `master` + +- Automatically rebasing when prerequisites are update. + +## Resetting backports + +Have a command to automatically blow away all patch branches except +for `master`, so that `backport` has a clean slate to work from. + +## Using `fixup` functionality for backports + +Developer checks out xsa/213/master, makes a fix, and commits it using git commit --fixup=$ref + +`xsatool 213 backport-fixup [version]` + + - Starting with either xsa/213/master or xsa/213/${version}, looks + for patches with 'fixup' in the header, and cherry-picks them (or + rebases them?) onto previous versions + +`xsatool 213 backport-squash [version]` + + - Starting with xsa/213/master (or the version specified), run `git + rebase --autosquash` + +## Getting information in a more human-friendly way + +`xsatool NNN info` + +`xsatool repo info` + +`xsatool global info` + +# Open issues + +# Inter-tree dependcies + +Do we need to worry about inter-tree dependencies? For example, could +it ever be the case that xsa 251, for xen.git, depends on xsa250, +which contains both xen.git and qemu-upstream.git components? If so +then when testing xsa251 we'd have to make sure that *all* the trees +were in the appropriate state. + -- 2.39.5