Handle osstest's own local push gate in non-master production instances

We want to arrange that the master XenProject instance continues to
test its own pretest branch while any downstream instances will pickup
changes from the master instance's production (i.e. tested) branch,
which is published at git://xenbits.xen.org/osstest.git#master. We
want to also be able to use local pretest for local changes (which may
or may not get merged back upstream).

Add a new configuration option OsstestUpstream which by default is
"git://xenbits.xen.org/osstest.git master" and which is cleared to
nothing on the master instance via production-config.

If the option is not set then the existing behaviour is unchanged.

If the option is set then osstest branch flights will still prefer to
test the local pretest branch, but if nothing is pending there then it
will proceed by merging the upstream branch into the local production
branch and testing the result.

This merge must be done:
 - in a clone not in the main testing.git in order to avoid inserting
   merge conflict markers into the active set of scripts.
 - in a non-bare repo because git merge requires it.

$repos/osstest is a bare repo which we want to keep that way because
using repo_tree_rev_fetch_git to fetch the remote branch is
convenient.

So we use $repos/osstest-merge as a temporary merge repo and reclone
from the active local repo each time.

All of this happens in ap-fetch-version.

As part of this arrange that the result is always left in the ap-fetch
branch of the for-osstest.git repo (even for existing cases) and the
sha1 is produced as output. Resetting to that revision is handled by
cr-daily-branch.

If the merge fails then manual intervention (i.e. a manual merge and
push to the _local_ pretest) will be required. Likewise if local
pretest and local production have diverged manual intervention will be
required.

In ap-push we stop pushing to xenbits#master except for the master
instance if an upstream is defined. At some point it might be useful
to add a configuration option for where to push to but I don't have
that requirement right now.

ap-fetch-version-old requires no changes.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4:
 - Use git update-ref properly, i.e. with the full ref name, otherwise
   it creates random .git/ap-fetch refs

v3:
 - Only merge from upstream if there is nothing pending locally.
 - Always update ap-fetch.

v2:
 - Arrange for $OSSTEST_USE_HEAD=y to take precendence
 - drop LOCALREV (which was wrong anyway) in favour of inline
   branchname
 - Rename OSSTEST_REVISION_MERGE as revision_merge to avoid implying
   it can be set and will be honoured.
 - Git in Debian Squeeze lacks -C and --no-edit, adjust accordingly.

full name to update-ref

diff --git a/Osstest.pm b/Osstest.pm
index 79891290f2b8dd315c2fa91f04025dd7c54d3104..7f72bc69a1ffc0e76462ceec68c9c26a5bc7e018 100644 (file)
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -102,6 +102,8 @@ sub readglobalconfig () {
     return if $readglobalconfig_done;
     $readglobalconfig_done=1;
 
+    $c{OsstestUpstream} = 'git://xenbits.xen.org/osstest.git master';
+
     $c{HostProp_DhcpWatchMethod} = 'leases dhcp3 infra:5556';
     $c{AuthorizedKeysFiles} = '';
     $c{AuthorizedKeysAppend} = '';

diff --git a/ap-common b/ap-common
index 64749e3b8e0596a65cf785042a226ffb7f882ebd..de7bb1dd6a420e7cb6174ffd7b7ddb22fe32f55b 100644 (file)
--- a/ap-common
+++ b/ap-common
@@ -21,6 +21,8 @@
 
 : ${XENBITS:=osstest@xenbits.xen.org}
 
+: ${TREEBRANCH_OSSTEST_UPSTREAM=`getconfig OsstestUpstream`}
+
 : ${TREE_XEN:=git://xenbits.xen.org/xen.git}
 : ${PUSH_TREE_XEN:=$XENBITS:/home/xen/git/xen.git}
 

diff --git a/ap-fetch-version b/ap-fetch-version
index 33aaf00b64456ae95dfc199884a16249478c66ce..ec52889a0cb644dcf269f9ac684dab8771df6358 100755 (executable)
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -90,12 +90,77 @@ ovmf)
                $TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
        ;;
 osstest)
-       if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
-           git fetch $HOME/testing.git pretest:ap-fetch >&2
-            git rev-parse ap-fetch^0
+        if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
+           git update-ref -m "Arranging to test HEAD" \
+               ap-fetch HEAD^0
+       elif [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then
+           # Deciding what to test here is somewhat complex because
+           # we would prefer to avoid octopus merges and to try and
+           # keep the change history as clean as reasonably possible.
+           #
+           # Therefore we prefer to test anything which is in local
+           # pretest before testing anything which is in
+           # upstream. Whether or not there is anything in local
+           # pretest is determined by using "git merge base"
+
+           git fetch $HOME/testing.git +production:production
+           git fetch $HOME/testing.git +pretest:pretest
+
+           local_production=$(git rev-parse production^0)
+           local_pretest=$(git rev-parse pretest^0)
+
+           merge_base=$(git merge-base $local_production $local_pretest)
+
+           echo >&2 "Considering merge from upstream"
+           echo >&2 "Local production: $local_production"
+           echo >&2 "Local pretest:    $local_pretest"
+           echo >&2 "Merge base:       $merge_base"
+
+           if [ x$merge_base = x$local_pretest ] ; then
+               # local production is a ff of local pretest
+               # (i.e. there is nothing local to test). So merge from
+               # upstream master into production and test that.
+
+               revision_merge=`repo_tree_rev_fetch_git osstest \
+                   $TREEBRANCH_OSSTEST_UPSTREAM daily-cron.$branch`
+               echo >&2 "Testing $TREEBRANCH_OSSTEST_UPSTREAM = $revision_merge"
+
+               rm -rf $repos/osstest-merge >&2
+               git clone -b production $HOME/testing.git $repos/osstest-merge >&2
+
+               (
+                   cd $repos/osstest-merge
+                   git fetch $repos/osstest daily-cron.$branch:ap-merge >&2
+                   git merge \
+                       -m "Automerge of $TREEBRANCH_OSSTEST_UPSTREAM into production" \
+                       ap-merge >&2
+               )
+
+               # Fetch the result of the above merge into local tree
+               # ready for use. cr-daily-branch handles reseting the
+               # working tree to this version
+               git fetch $repos/osstest-merge +production:ap-fetch
+           elif [ x$merge_base = x$local_production ] ; then
+               # there are commits in pretest which we should test in
+               # an attempt to progress production to this point.
+               echo >&2 "Testing local pretest"
+               git update-ref -m "Arranging to test local pretest" \
+                   refs/heads/ap-fetch $local_pretest
+           else
+               # It's too hard for to deal with automatically. This
+               # might happen if a push to pretest races with a
+               # successful flight which merges from upstream. We
+               # expect pushes to pretest to be rare in practice.
+               echo >&2 "************************************************"
+               echo >&2 "production and pretest have diverged."
+               echo >&2 "need manual fixup/rebase and push to pretest."
+               echo >&2 "************************************************"
+           fi
        else
-           git rev-parse HEAD^0
+           git fetch $HOME/testing.git pretest:ap-fetch >&2
        fi
+
+       git rev-parse ap-fetch^0
         ;;
 *)
        echo >&2 "branch $branch ?"

diff --git a/ap-push b/ap-push
index c1414640bd463543486fa31d55379a80bb814c25..5967b426a2c5cbd70e4b786e69e09da6443fb1d9 100755 (executable)
--- a/ap-push
+++ b/ap-push
@@ -99,7 +99,13 @@ ovmf)
        ;;
 osstest)
        git push $HOME/testing.git $revision:production
-       git push $XENBITS:/home/xen/git/osstest.git $revision:master
+       if [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then
+           # could push to instance specific location, but
+           # certainly not to master instance's xenbits repo!
+           :
+       else
+           git push $XENBITS:/home/xen/git/osstest.git $revision:master
+       fi
        ;;
 *)
        echo >&2 "branch $branch ?"

diff --git a/production-config b/production-config
index 8f20c27756fe23d353eeae1b95a1303a8fbc5c31..84a3bb755ea8ff1c39f541cc47051a5d808e2843 100644 (file)
--- a/production-config
+++ b/production-config
@@ -15,6 +15,8 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+# This is the master instance, therefore it has no upstream
+OsstestUpstream ''
 
 JobDB Executive
 HostDB Executive