]> xenbits.xensource.com Git - osstest.git/log
osstest.git
10 years agoselectguest: log a message
Ian Jackson [Tue, 15 Jul 2014 14:15:02 +0000 (15:15 +0100)]
selectguest: log a message

Print the guest's name and the host it's on; this can help with
interpreting the logs.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agots-rumpuserxen-build: Cope with new layout, disabling bodges flight-29255 flight-29275 flight-29276 flight-29277 flight-29278 flight-29279 flight-29280 flight-29281 flight-29282 flight-29283 flight-29284 flight-29285 flight-29286 flight-29287 flight-29288 flight-29289 flight-29290 flight-29291 flight-29292 flight-29293 flight-29294 flight-29295 flight-29296 flight-29297 flight-29298 flight-29299 flight-29300 flight-29301 flight-29302 flight-29303 flight-29304 flight-29305 flight-29306 flight-29307 flight-29308 flight-29309 flight-29310 flight-29311 flight-29312 flight-29313 flight-29314 flight-29315 flight-29316 flight-29317 flight-29319 flight-29329 flight-29330 flight-29331 flight-29332 flight-29333 flight-29334 flight-29335 flight-29336 flight-29337 flight-29338 flight-29339 flight-29340 flight-29341 flight-29343 flight-29345 flight-29346 flight-29347 flight-29350 flight-29351 flight-29352 flight-29353 flight-29354 flight-29355 flight-29356 flight-29357 flight-29358 flight-29359 flight-29360 flight-29361 flight-29362 flight-29363 flight-29364 flight-29365 flight-29366 flight-29367 flight-29368 flight-29369 flight-29371 flight-29372 flight-29374 flight-29375 flight-29376 flight-29377 flight-29378 flight-29379
Ian Jackson [Fri, 11 Jul 2014 14:27:11 +0000 (15:27 +0100)]
ts-rumpuserxen-build: Cope with new layout, disabling bodges

The rumpsrc psuedo-submodule has been integrated with the nblibs
submodule and made into a proper submodule.

Also, in the intervening time, rumpuserxen/Config.mk now uses ?= when
setting XEN_HEADERS.

So if we detect that we have a new rumpuser-xen, we can disable:
 - ad-hoc arrangements for rumpsrc
 - grievous-bodge
 - patching Config.mk

We consider a rumpuserxen old iff it has nblibs as a submodule and
lacks rumpsrc as a submodule.

We have to have a different name for the rumpsrc submodule (rather
than using rumpsrc), because in the new arrangements it needs to be
fetched from the location specified in rumpuserxen's .gitmodules
rather than our own configured location for the old rumpsrc.  (And the
local name is in the runvars etc.)

The old runvars etc. for rumpsrc need to stay around to cope with old
rumpuserxen, for now, but are going to be obsolete.  Leave some
comments about this in relevant places.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoBuildSupport: provide submodule_have
Ian Jackson [Fri, 11 Jul 2014 14:13:03 +0000 (15:13 +0100)]
BuildSupport: provide submodule_have

Convenience function.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoadhoc-revtuple-generator: Explicitly specify refs to fetch flight-29176 flight-29177 flight-29178 flight-29179 flight-29180 flight-29181 flight-29182 flight-29183 flight-29184 flight-29185 flight-29186 flight-29187 flight-29188 flight-29189 flight-29190 flight-29191 flight-29192 flight-29193 flight-29194 flight-29195 flight-29196 flight-29197 flight-29198 flight-29199 flight-29202 flight-29203 flight-29204 flight-29205 flight-29206 flight-29207 flight-29208 flight-29209 flight-29210 flight-29211 flight-29212 flight-29213 flight-29214 flight-29215 flight-29216 flight-29217 flight-29218 flight-29219 flight-29220 flight-29221 flight-29222 flight-29223 flight-29224 flight-29225 flight-29226 flight-29227 flight-29228 flight-29229 flight-29230 flight-29231 flight-29232 flight-29233 flight-29234 flight-29235 flight-29236 flight-29237 flight-29238 flight-29239 flight-29240 flight-29241 flight-29242 flight-29243 flight-29244 flight-29245 flight-29246 flight-29247 flight-29248 flight-29249 flight-29250 flight-29251 flight-29256 flight-29257 flight-29258 flight-29259 flight-29260 flight-29261 flight-29262 flight-29263 flight-29264 flight-29265 flight-29266 flight-29267 flight-29268 flight-29269 flight-29270 flight-29271 flight-29272 flight-29273 flight-29274
Ian Jackson [Tue, 8 Jul 2014 10:27:59 +0000 (11:27 +0100)]
adhoc-revtuple-generator: Explicitly specify refs to fetch

Some of the git trees in $c{Repos} may have been created (eg by
earlier versions of git or earlier versions of osstest) without the
usual
  remote.origin.fetch = +refs/heads/*:refs/remotes/origin/*
git config setting.

Pass this refspec explicitly, instead of relying on it being set as
the default.

Amongst other things, this will prevent a possible error during
bisection
  git fetch -p origin
  [stuff]
  fatal: bad object 3e3d5f6146c3f684f4b7bd71feffc31e1c668754
which is currently appearing in our cron mail.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoexecutive-postgresql-schema: Schema dump flight-27830 flight-27839 flight-27852 flight-28807 flight-28808 flight-28809 flight-28810 flight-28811 flight-28812 flight-28813 flight-28814 flight-28815 flight-28816 flight-28817 flight-28818 flight-28819 flight-28820 flight-28821 flight-28822 flight-28823 flight-28824 flight-28825 flight-28826 flight-28827 flight-28828 flight-28829 flight-28830 flight-28831 flight-28832 flight-28833 flight-28834 flight-28835 flight-28836 flight-28837 flight-28838 flight-28839 flight-28840 flight-28841 flight-28842 flight-28843 flight-28844 flight-28845 flight-28846 flight-28847 flight-28848 flight-28849 flight-28850 flight-28851 flight-28852 flight-28853 flight-28854 flight-28855 flight-28856 flight-28857 flight-28858 flight-28859 flight-28860 flight-28861 flight-28862 flight-28863 flight-28864 flight-28865 flight-28866 flight-28869 flight-28870 flight-28871 flight-28872 flight-28873 flight-28874 flight-28875 flight-28876 flight-28877 flight-28878 flight-28879 flight-28880 flight-28881 flight-28882 flight-28884 flight-28885 flight-28886 flight-28888 flight-28889 flight-28890 flight-28891 flight-28892 flight-28893 flight-28894 flight-28895 flight-28896 flight-28897 flight-28898 flight-28899 flight-28900 flight-28901 flight-28902 flight-28903 flight-28905 flight-28906 flight-28907 flight-28908 flight-28911 flight-28912 flight-28913 flight-28914 flight-28915 flight-28916 flight-28917 flight-28918 flight-28919 flight-28920 flight-28921 flight-28922 flight-28923 flight-28924 flight-28925 flight-28926 flight-28927 flight-28928 flight-28929 flight-28930 flight-28931 flight-28932 flight-28935 flight-28937 flight-28938 flight-28940 flight-28943 flight-28945 flight-28947 flight-28949 flight-28953 flight-28955 flight-28979 flight-28988 flight-29009 flight-29031 flight-29052 flight-29081 flight-29086 flight-29087 flight-29138 flight-29151
Ian Jackson [Mon, 30 Jun 2014 10:13:37 +0000 (11:13 +0100)]
executive-postgresql-schema: Schema dump

We really ought to have a proper schema script.  But for now, include
the output of pg_dump --schema-only on the osstestdb, for the benefit
of anyone trying to set up a hosted (rather than standalone) instance
(or just trying to see how the code works).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch in v2 of the series.

10 years agomake-flight: Test rump kernels only when needed
Ian Jackson [Fri, 27 Jun 2014 10:20:19 +0000 (11:20 +0100)]
make-flight: Test rump kernels only when needed

Introduce a new subroutine branch_wants_rumpkernel_tests and call it
before making the rumpuserxen build and test jobs.

The default is not to include a rumpuserxen test.  We include it for
rumpuserxen itself, xen, linux (in case there is a frontend/backend
incompatibility), and osstest.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomake-flight: Break out do_rumpkernel_tests
Ian Jackson [Fri, 27 Jun 2014 10:19:47 +0000 (11:19 +0100)]
make-flight: Break out do_rumpkernel_tests

No functional change in this patch.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agostandalone-generate-dump-flight-runvars: New utility
Ian Jackson [Fri, 27 Jun 2014 14:40:57 +0000 (15:40 +0100)]
standalone-generate-dump-flight-runvars: New utility

This is helpful for showing the effect of changes to make-flight,
cr-daily-branch, et al.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Do not set DAILY_BRANCH_POSTMAKE_HOOK, since standalone now does
    that.

10 years agomg-list-all-branches: New script.
Ian Jackson [Fri, 27 Jun 2014 11:45:11 +0000 (12:45 +0100)]
mg-list-all-branches: New script.

The approach here is a little unpleasant but it has the virtue of
getting the ones out of crontab too, and of not depending on the
production database (which will in any case contain obsolete flights).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agostandalone-config-example: Provide a setting of Repos
Ian Jackson [Fri, 27 Jun 2014 15:51:46 +0000 (16:51 +0100)]
standalone-config-example: Provide a setting of Repos

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agostandalone: Use DAILY_BRANCH_POSTMAKE_HOOK
Ian Jackson [Mon, 30 Jun 2014 10:12:08 +0000 (11:12 +0100)]
standalone: Use DAILY_BRANCH_POSTMAKE_HOOK

DAILY_BRANCH_PREEXEC_HOOK is after we have made half the prospective
email, but this is not useful for standalone users.

Reported-by: Ian Campbell <Ian.Campbell@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch in v2 of the series.

10 years agostandalone: Honour OSSTEST_CONFIG
Ian Jackson [Fri, 27 Jun 2014 15:51:27 +0000 (16:51 +0100)]
standalone: Honour OSSTEST_CONFIG

If OSSTEST_CONFIG is set in the environment, do not default it
to ~/.xen-osstest/config.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agosubmodulefixup: Tell git cache [fetch=try]
Ian Jackson [Fri, 27 Jun 2014 10:42:31 +0000 (11:42 +0100)]
submodulefixup: Tell git cache [fetch=try]

Make git_massage_url take an optional %xopts list, currently with
only GitFetchBestEffort as a possible key.

Use that in submodulefixup, unconditionally.  Often the submodule
update will work fine with an older version of the submodule, and if
the submodule we get is out of date then the submodule update will
fail anyway.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Do not always reclone
Ian Jackson [Fri, 27 Jun 2014 14:24:18 +0000 (15:24 +0100)]
adhoc-revtuple-generator: Do not always reclone

Instead, try setting the `origin' remote's URL and fetching.  If that
fails it may be that the local repo is corrupted, so then we try
recloning.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Avoid an undefined variable warning
Ian Jackson [Fri, 27 Jun 2014 13:44:58 +0000 (14:44 +0100)]
adhoc-revtuple-generator: Avoid an undefined variable warning

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Include $treename in some error messages flight-27575 flight-27576 flight-27578 flight-27579 flight-27580 flight-27582 flight-27587 flight-27774 flight-27775 flight-27776 flight-27777 flight-27778 flight-27779 flight-27781 flight-27782 flight-27783 flight-27784 flight-27785 flight-27786 flight-27787 flight-27788 flight-27789 flight-27790 flight-27791 flight-27792 flight-27793 flight-27794 flight-27797 flight-27798 flight-27799 flight-27800 flight-27801 flight-27802 flight-27803 flight-27804 flight-27805 flight-27806 flight-27807 flight-27808 flight-27809 flight-27810 flight-27811 flight-27812 flight-27814 flight-27815 flight-27816 flight-27817 flight-27818 flight-27819 flight-27820 flight-27821 flight-27822 flight-27823 flight-27824 flight-27826 flight-27827 flight-27828 flight-27829 flight-27831 flight-27832 flight-27833 flight-27834 flight-27836 flight-27837 flight-27838 flight-27840 flight-27841 flight-27842 flight-27843 flight-27845 flight-27846 flight-27847 flight-27848 flight-27849 flight-27850 flight-27851 flight-27853 flight-27854 flight-27855 flight-27856 flight-27857 flight-27858 flight-27860 flight-27861 flight-27867 flight-27868 flight-27869 flight-27870
Ian Jackson [Fri, 27 Jun 2014 11:30:35 +0000 (12:30 +0100)]
adhoc-revtuple-generator: Include $treename in some error messages

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Cope with parentless git commits
Ian Jackson [Fri, 27 Jun 2014 11:30:13 +0000 (12:30 +0100)]
adhoc-revtuple-generator: Cope with parentless git commits

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agots-xen-install: Correct mlog to logm flight-27471 flight-27483 flight-27484 flight-27490 flight-27491 flight-27508 flight-27525 flight-27538 flight-27546 flight-27553 flight-27569
Ian Campbell [Mon, 16 Jun 2014 08:31:03 +0000 (09:31 +0100)]
ts-xen-install: Correct mlog to logm

Only logm exists.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: Print messages to stderr.
Anthony PERARD [Mon, 9 Jun 2014 15:16:50 +0000 (16:16 +0100)]
mg-debian-installer-update: Print messages to stderr.

This fix standalone-reset which rely on the output of this script.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Dario Faggioli <dario.faggioli@citrix.com>
10 years agomg-hosts: Usage documentation.
Ian Jackson [Wed, 25 Jun 2014 14:05:34 +0000 (15:05 +0100)]
mg-hosts: Usage documentation.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomg-hosts: Allow spaces in setflagexpr exprs
Ian Jackson [Wed, 25 Jun 2014 14:05:21 +0000 (15:05 +0100)]
mg-hosts: Allow spaces in setflagexpr exprs

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-for-branches: clean up tmp/
Ian Jackson [Wed, 25 Jun 2014 09:33:01 +0000 (10:33 +0100)]
cr-for-branches: clean up tmp/

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoImplement direct IPMI power control. flight-27153 flight-27158 flight-27159 flight-27160 flight-27161 flight-27164 flight-27165 flight-27166 flight-27168 flight-27169 flight-27171 flight-27173 flight-27201 flight-27203 flight-27218 flight-27248 flight-27249 flight-27280 flight-27282 flight-27293 flight-27300 flight-27315 flight-27316 flight-27321 flight-27330 flight-27352 flight-27357 flight-27358 flight-27380 flight-27391 flight-27392 flight-27393 flight-27394 flight-27395 flight-27396 flight-27401 flight-27402 flight-27403 flight-27404 flight-27405 flight-27406 flight-27407 flight-27408 flight-27409 flight-27410 flight-27411 flight-27412 flight-27413 flight-27414 flight-27415 flight-27416 flight-27417 flight-27418 flight-27419 flight-27420 flight-27421 flight-27422 flight-27423 flight-27424 flight-27425 flight-27426 flight-27427 flight-27428 flight-27429 flight-27430 flight-27431 flight-27432 flight-27433 flight-27434 flight-27435 flight-27436 flight-27437 flight-27438 flight-27439 flight-27440 flight-27441 flight-27442 flight-27443 flight-27444 flight-27445 flight-27446 flight-27447 flight-27448 flight-27449 flight-27450 flight-27452 flight-27456 flight-27462 flight-27463 flight-27464 flight-27465 flight-27466 flight-27467 flight-27468 flight-27469 flight-27473 flight-27474 flight-27475 flight-27476 flight-27477 flight-27478 flight-27479 flight-27480 flight-27481 flight-27482
Ian Campbell [Mon, 16 Jun 2014 08:51:21 +0000 (09:51 +0100)]
Implement direct IPMI power control.

The xenuse module currently used to control some machines (specifically the ARM
marilith machines) is unreliable and subject to breakages on the xenrt server
side.

Since xenuse/xenrt is just wrapping IPMI for these machines implement direct
control via a new Osstest::PDI::impi module.

This is distinct from the existing impiextra (arbitrary ipmi commands on power
status change) because IPMI power control is not idempotent, therefore it is
necessary to query the current status before turning the machine on or off.

After this change the correct power method for a marilith node is:
    ipmiextra on marilith-n0-mgmt.uk.xensource.com <user> <pass> chassis bootdev pxe;ipmi marilith-n0-mgmt.uk.xensource.com <user> <pass>

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Ian, when we discussed this on IRC you expressed a preference for using
freeipmitools, but since the existing ipmiextra module uses ipmitool I figured
it would be better to be consistent.

10 years agocrontab: reduce frequency of linux-next flight-27064 flight-27066 flight-27067 flight-27068 flight-27070 flight-27072 flight-27073 flight-27074 flight-27075 flight-27077 flight-27079 flight-27080 flight-27081 flight-27082 flight-27083 flight-27084 flight-27086 flight-27087 flight-27088 flight-27089 flight-27090 flight-27092 flight-27093 flight-27094 flight-27095 flight-27096 flight-27097 flight-27098 flight-27099 flight-27100 flight-27101 flight-27102 flight-27103 flight-27104 flight-27105 flight-27106 flight-27107 flight-27109 flight-27110 flight-27111 flight-27112 flight-27113 flight-27114 flight-27116 flight-27117 flight-27118 flight-27119 flight-27120 flight-27121 flight-27122 flight-27123 flight-27124 flight-27126 flight-27127 flight-27128 flight-27129 flight-27130 flight-27132 flight-27133 flight-27138 flight-27141 flight-27142 flight-27144 flight-27145 flight-27146 flight-27147 flight-27148 flight-27149 flight-27151 flight-27152 flight-27154 flight-27155 flight-27156 flight-27157
Ian Jackson [Mon, 9 Jun 2014 14:29:52 +0000 (15:29 +0100)]
crontab: reduce frequency of linux-next

10 years agomake-flight: Do not filter the qemu-upstream-* jobs
Ian Jackson [Mon, 9 Jun 2014 14:20:05 +0000 (15:20 +0100)]
make-flight: Do not filter the qemu-upstream-* jobs

When "qemu-xen-traditional" was the default, this filtering was
correct.  But now, "qemu-xen" is the default, and that version may
also be used in other tests for ancillary functions (eg, in dom0 for
the bootloader).

So simply do all of the jobs when testing qemu-upstream-*.

Retain the old behaviour for qemu-xen-4.2-testing, which has
qemu-xen-traditional as the default.  (There is no
qemu-xen-4.1-testing, and 4.3 has the change to the default.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agobuild_clone: When guessing vcs from url suffix, strip cache info flight-26977 flight-26997 flight-27012 flight-27031 flight-27032 flight-27033 flight-27034 flight-27038 flight-27042 flight-27043 flight-27044 flight-27045 flight-27047 flight-27048 flight-27049 flight-27050 flight-27051 flight-27052 flight-27055 flight-27056 flight-27057 flight-27060 flight-27061 flight-27062 flight-27063
Ian Jackson [Thu, 5 Jun 2014 10:15:46 +0000 (11:15 +0100)]
build_clone: When guessing vcs from url suffix, strip cache info

We assume that anything not containing `/'s after `%20' is
uninteresting.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoap-common: Use besteffort_repo for rumpsrc flight-26968
Ian Jackson [Wed, 4 Jun 2014 13:47:54 +0000 (14:47 +0100)]
ap-common: Use besteffort_repo for rumpsrc

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocri-common, ap-common: abstract out besteffort_repo
Ian Jackson [Wed, 4 Jun 2014 13:46:49 +0000 (14:46 +0100)]
cri-common, ap-common: abstract out besteffort_repo

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-daily-branch: Fix URLs used for trees which should be using push gate output
Ian Campbell [Wed, 4 Jun 2014 12:49:16 +0000 (13:49 +0100)]
cr-daily-branch: Fix URLs used for trees which should be using push gate output

For several trees we were using the upstream URL for tests of all branches when
we should have been using the push gate output on xenbits (to help isolate us
from external failures). i.e. tree_libvirt was git://libvirt.org/libvirt.git
for every branch but for branches other than libvirt it should be
git://xenbits.xen.org/libvirt.git.

This is down to a missing third argument to determine_version. Fix this for all
but osstest (which shouldn't be touched).

Tested with:
 $ BRANCHES="xen-unstable libvirt rumpuserxen seabios qemu-mainline qemu-xen-unstable linux-next linuxfirmware osstest"
 $ for b in $BRANCHES ; do f=T${b//-}; ./standalone make-flight -f $f $b; done
 $ check() { \
     f=T${1//-} ; \
     echo $1 ; \
     ./mg-show-flight-runvars $f | \
         awk '$2 ~ /^tree_/ && $3 !~ /^$|xenbits.xen.org/ { printf "    %-28s %s\n", $2, $3 }' | \
         sort -u; \
   }
 $ for b in xen-unstable libvirt rumpuserxen seabios qemu-mainline qemu-xen-unstable linux-next osstest ; do check $b; done

Which produces:

xen-unstable
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
libvirt
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_libvirt                 git://libvirt.org/libvirt.git
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
rumpuserxen
    tree_rumpuserxen             https://github.com/rumpkernel/rumpuser-xen
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
seabios
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
    tree_seabios                 git://git.seabios.org/seabios.git
qemu-mainline
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_qemuu                   git://git.qemu.org/qemu.git
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
qemu-xen-unstable
linux-next
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_linux                   git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
linuxfirmware
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
osstest
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src

The presence of tree_gnulib_libvirt is expected while the
tree_rumpuserxen_rumpsrc is erroneous (I think) but not intended to be fixed
here.

Note that linuxfirmware is broken since tree_linuxfirmware should be
$GIT_KERNEL_ORG/pub/scm/linux/kernel/git/firmware/linux-firmware.git. This is
also not addressed here.

For comparison the xen-unstable check output prior to this fix was:

xen-unstable
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_libvirt                 git://libvirt.org/libvirt.git
    tree_rumpuserxen             https://github.com/rumpkernel/rumpuser-xen
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src

Note the incorrect tree_libvirt and tree_rumpuserxen.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com

10 years agomake-flight: include cri-common before ap-common
Ian Campbell [Wed, 4 Jun 2014 11:05:56 +0000 (12:05 +0100)]
make-flight: include cri-common before ap-common

The latter wants cached_repo which the former provides.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com

10 years agostandalone: ensure log dir exists for all users of with_logging
Ian Campbell [Wed, 4 Jun 2014 11:00:02 +0000 (12:00 +0100)]
standalone: ensure log dir exists for all users of with_logging

By moving the check into that function. Previously at least make-flight was
failing to all ensure_logs

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com

10 years agots-logs-capture: collect /proc/partitions
Ian Jackson [Wed, 4 Jun 2014 11:29:36 +0000 (12:29 +0100)]
ts-logs-capture: collect /proc/partitions

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agots-logs-capture: collect resize2fs.log
Ian Jackson [Wed, 4 Jun 2014 11:28:48 +0000 (12:28 +0100)]
ts-logs-capture: collect resize2fs.log

This is generated by ts-xen-build-prep but previously we didn't
collect it.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agots-xen-build-prep: Run the backgrounded resize2fs with ionice
Ian Jackson [Wed, 4 Jun 2014 11:24:06 +0000 (12:24 +0100)]
ts-xen-build-prep: Run the backgrounded resize2fs with ionice

We wrap this in
  ionice -cBest-Effort -n7

This is an attempt to stop resize2fs from starving builds of IO
bandwidth.  We have been seeing that some IO-intensive build steps
(notably, configure) can take unreasonable amounts of time and we
suspect resize2fs as the culprit.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoREADME.dev: more info about mg-blockage flight-26941
Ian Jackson [Tue, 3 Jun 2014 15:08:49 +0000 (16:08 +0100)]
README.dev: more info about mg-blockage

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTolerate outages of git://git.sv.gnu.org/gnulib.git
Ian Campbell [Tue, 20 May 2014 10:54:36 +0000 (11:54 +0100)]
Tolerate outages of git://git.sv.gnu.org/gnulib.git

gnulib is a bit special in that it isn't part of a push gate and
therefore doesn't have an output repo on xenbits. This means that if
git.sv.gnu.org is down for some reason then any flight which uses it
will potentially fail, even ones which are not specifically trying to
test gnulib.git. Normally unrelated flights are isolated from 3rd
party failures because they use the relevant push gates' outputs.

The git-proxy which we use supports passing [fetch=try] as an option
which will cause it to try and fetch from upstream and fall back to
the cached version if upstream does not respond. This is exactly what
we want for gnulib.git. Proxy options in the URL are separate with a
space which we encode here as %20 to avoid having to go around adding
quotes to all the uses of shell variables which contain a URL.

Provide a helper shell function cached_repo which conditionally
arranges for the given repo to be cached with the given options only
if the osstest config sets GitCacheProxy, so as not to break uses of
osstest outside of the production environment where a cache may not be
available (of course such uses will then be subject to outages of 3rd
party repos)

It was also necessary to make repo_tree_rev_fetch_git idempotent WRT
the proxy prefix to avoid trying to clone
git://proxy/git://proxy/git://some.real.domain/foo.git.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
10 years agocri-args-hostlists, etc.: Send rumpuserxen reports to rumpkernel-builds
Ian Jackson [Tue, 3 Jun 2014 14:59:51 +0000 (15:59 +0100)]
cri-args-hostlists, etc.: Send rumpuserxen reports to rumpkernel-builds

There is now a list, rumpkernel-builds@lists.sourceforge, for this
kind of information relating to rumpuser-xen.

Provide a facility in cri-args-hostlists for start_email to look for
$OSSTEST_EMAIL_HEADER--$branch.

Introduce two new files
  daily-cron-email-real--rumpuserxen
  daily-cron-email-real-bisectcomplete--rumpuserxen
which override the general settings.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocs-bisection-step: Remove a bit of duplication
Ian Jackson [Tue, 3 Jun 2014 14:52:18 +0000 (15:52 +0100)]
cs-bisection-step: Remove a bit of duplication

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocs-bisection-step: Cope with url-less trees flight-26918 flight-26920 flight-26922 flight-26924 flight-26925 flight-26926 flight-26927 flight-26929 flight-26931 flight-26933 flight-26934 flight-26935 flight-26936 flight-26937 flight-26938 flight-26940 flight-26942 flight-26944 flight-26945 flight-26946 flight-26948 flight-26949 flight-26950 flight-26951 flight-26952 flight-26956 flight-26964 flight-26967 flight-26969 flight-26971 flight-26972 flight-26973 flight-26979 flight-26980 flight-26981 flight-26982 flight-26984 flight-26985 flight-26986 flight-26988 flight-26989 flight-26990 flight-26991 flight-26992 flight-26995 flight-26996 flight-26998 flight-26999 flight-27000 flight-27002 flight-27003 flight-27004 flight-27006 flight-27007 flight-27008 flight-27009 flight-27011 flight-27013 flight-27014 flight-27015 flight-27016 flight-27017 flight-27018 flight-27019 flight-27020 flight-27022 flight-27023 flight-27026 flight-27027 flight-27028 flight-27029 flight-27030
Ian Jackson [Mon, 2 Jun 2014 10:09:17 +0000 (11:09 +0100)]
cs-bisection-step: Cope with url-less trees

Since osstest 64a9c064, some flights have a tree with an empty
tree_<foo> variable.  This breaks the bisector.  Make it more robust
as follows:

When searching for the basis flight, ignore not only trees with no url
specified but also ones with an empty url.

When preparing the set of trees we care about, eliminate any where the
latest flight has no url (or an empty url).

(It would still be better for the flight constructor to leave
tree_<foo> unset, and for the ts-* build script to set it in as a part
of the post-build data collection.  That would allow the bisector to
work on that tree where relevant.  But doing this is a task for
another day - if for no other reason than that we have to decide
whether to put it in a new variable built_tree_<foo>.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoap-qemu-revision: Make it actually work flight-26456 flight-26471 flight-26493 flight-26494 flight-26495 flight-26496 flight-26497 flight-26498 flight-26499 flight-26500 flight-26501 flight-26503 flight-26504 flight-26508 flight-26512 flight-26515 flight-26530 flight-26549 flight-26553 flight-26554 flight-26556 flight-26565 flight-26566 flight-26567 flight-26569 flight-26626 flight-26657 flight-26663 flight-26681 flight-26701 flight-26738 flight-26740 flight-26764 flight-26773 flight-26774 flight-26775 flight-26777 flight-26797 flight-26828 flight-26866 flight-26871 flight-26879 flight-26884 flight-26907 flight-26908
Ian Jackson [Fri, 23 May 2014 09:55:51 +0000 (10:55 +0100)]
ap-qemu-revision: Make it actually work

This is used (only) for obtaining lists of the corresponding qemu
revisions (for the report email) when doing a Xen push.  It has never
worked because we didn't provide a type argument to git-cat-file.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-daily-branch: Only set TREE_SEABIOS for seabios flights
Ian Campbell [Fri, 23 May 2014 09:26:26 +0000 (10:26 +0100)]
cr-daily-branch: Only set TREE_SEABIOS for seabios flights

For all other flights we want to use the Config.mk settings.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoDefault vcs to git flight-26417 flight-26419 flight-26420 flight-26421 flight-26422 flight-26423 flight-26425 flight-26427 flight-26428 flight-26429 flight-26430 flight-26431 flight-26433 flight-26434 flight-26437 flight-26449 flight-26450 flight-26451 flight-26454 flight-26455 flight-26457 flight-26458 flight-26460 flight-26461 flight-26462 flight-26464 flight-26465 flight-26466 flight-26468 flight-26469 flight-26470 flight-26472 flight-26473 flight-26475 flight-26476 flight-26477 flight-26478 flight-26479 flight-26480 flight-26481 flight-26482 flight-26485 flight-26489 flight-26490 flight-26491
Ian Jackson [Thu, 22 May 2014 11:24:19 +0000 (12:24 +0100)]
Default vcs to git

adhoc-revtuple-generator and cs-bisection-step try to guess the VCS
from the URL, by looking at the end of the URL for a suitable file
extension.  If none was found, things would break.

This relies on a convention about tree URLs which is sadly no longer
being widely observed.  In particular many projects have git URLs
which do not mention git.

Instead, have these scripts assume that urls which don't end in .git
or .hg are for git.

This renders hg into a second-class VCS in osstest, which is (I think)
tolerable.  If it isn't, then someone ought to come up with some
better way of ensuring that the vcs is explicitly specified to
adhoc-revtuple-generator and cs-bisection-step.  That would probably
be doable but is too much work right now.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agosg-run-job: Do blocked check globally, in run-job flight-26374
Ian Jackson [Wed, 21 May 2014 11:36:02 +0000 (12:36 +0100)]
sg-run-job: Do blocked check globally, in run-job

Only run-job can fully abandon the whole job.  With the previous
arrangements (in bc63a901), a blocked build job would go on to try to
run ts-logs-capture and become broken (rather than blocked).

The downside of this is that it is not possible in the future to
introduce a kind of job which can selectively carry on if some of its
dependencies fail.  This notion is IMO an acceptable casualty.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTestSupport::broken: Update status of `preparing' jobs
Ian Jackson [Wed, 21 May 2014 11:34:01 +0000 (12:34 +0100)]
TestSupport::broken: Update status of `preparing' jobs

If the current job is preparing, and we call broken, then we should
update the status to that supplied by broken's caller.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomake-flight: rumpuserxen amd64 too
Ian Jackson [Wed, 21 May 2014 10:54:10 +0000 (11:54 +0100)]
make-flight: rumpuserxen amd64 too

Build and test amd64 rump kernel.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocrontab: Rearrange actually-daily branches
Ian Jackson [Wed, 21 May 2014 10:37:48 +0000 (11:37 +0100)]
crontab: Rearrange actually-daily branches

Make them start at roughly the same time, by using cr-for-branches for
them all in one go.  That increases the extent to which build hosts
will be reused (saving on reinstallations), and hence also the extent
to which the build hosts' ccache will be effective (saving on cpu).

Set them off at 04:18 UK time (currently equivalent to 21:18 Pacific
time), in the hope that at least some of them might have produced
answers during following the UK working day.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorumpuserxen: Increase build timeout for grievous bodge
Ian Jackson [Wed, 21 May 2014 10:42:37 +0000 (11:42 +0100)]
rumpuserxen: Increase build timeout for grievous bodge

This involves copying 90Mby of data.  If the test machine is busy (in
the worst case, running the kernel side of resize2fs which due to
Linux's poor IO scheduling hogs the entire IO system) it can take a
long time.  In flight 26359 it was still running after 280s, when the
machine was released.

Increase the timeout to 2000s.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-publish-flight-logs: Provide ref path for osstest branch push flight-26353 flight-26360 flight-26364 flight-26365 flight-26373
Ian Jackson [Mon, 19 May 2014 10:58:47 +0000 (11:58 +0100)]
cr-publish-flight-logs: Provide ref path for osstest branch push

Without this, the push fails if the osstest tree is in detached head
state.  (Which is a situation which can occur during by-hand testing.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoAdd my developer/deployment notes as README.dev
Ian Campbell [Mon, 19 May 2014 11:08:09 +0000 (12:08 +0100)]
Add my developer/deployment notes as README.dev

These are mostly targetted at people setting up things on the production
infra.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoproduction-config: Update to pickup latest Debian Installer
Ian Campbell [Mon, 19 May 2014 11:32:16 +0000 (12:32 +0100)]
production-config: Update to pickup latest Debian Installer

Pickup the changes made in
     mg-debian-installer-update: Updates for newer armhf kernels from backports.
     mg-debian-installer-update: grab Xen PV domU capable images too

I have run

    for i in i386 amd64 armhf ; do
        OSSTEST_CONFIG=production-config ./mg-debian-installer-update wheezy $i 'firmware-bnx2'
    done

on the osstest vm. Point the config to these new versions.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agosg-run-job: check for prerequisite builds when running build jobs
Ian Campbell [Thu, 15 May 2014 10:30:39 +0000 (11:30 +0100)]
sg-run-job: check for prerequisite builds when running build jobs

For example build-$ARCH-libvirt depends on build-$ARCH for the Xen bits.
Performing this check early saves allocating, installing, preparating etc a
host just to find the prerequisite is missing.

Rename ts-xen-build-check to ts-build-check while we are there. The script
checks all prerequisite builds, not just Xen.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agostandalone: allow making a baseline flight.
Ian Campbell [Thu, 15 May 2014 10:30:38 +0000 (11:30 +0100)]
standalone: allow making a baseline flight.

While there make the existing --lvextend argument actually work (by passing it
to getopt) and document both.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agots-xen-build: Move Xen build tree from xen-unstable to just xen
Ian Campbell [Thu, 15 May 2014 10:30:37 +0000 (11:30 +0100)]
ts-xen-build: Move Xen build tree from xen-unstable to just xen

It's not always xen-unstable we are testing, and I had to check that it was
the right generic name, so reduce this confusion.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agots-xen-build: stash the Xen and SeaBIOS .config's
Ian Campbell [Thu, 15 May 2014 10:30:36 +0000 (11:30 +0100)]
ts-xen-build: stash the Xen and SeaBIOS .config's

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoAdd a flight to test seabios.org's master branch
Ian Campbell [Thu, 15 May 2014 10:30:35 +0000 (11:30 +0100)]
Add a flight to test seabios.org's master branch

Most of this is the usual stuff for adding a new branch.

cr-daily-branch needed to set REVISION_SEABIOS for the seabios branch and leave
it blank for others (so that Config.mk's version will be used). ts-xen-build
already consumes the resulting version runvar but needed to be taught about the
tree URL runvar.

make-flight and mfi-common are updated to only build+test seabios on x86.

The new branch is added to cr-daily-branches.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoAdd a flight to test qemu.org's ("mainline") master branch.
Ian Campbell [Thu, 15 May 2014 10:30:34 +0000 (11:30 +0100)]
Add a flight to test qemu.org's ("mainline") master branch.

The naming here is slightly confused because we call our branches of qemuu on
xenbits "qemu-upstream-foo", so I went with qemu-mainline to be distinct.

I've introduced $qemuubranch along the lines of $xenbranch, it is normally set
to qemu-upstream-{X.Y,unstable} but for the qemu-mainline flight it instead
points to qemu-mainline.

I've run up to the make-flight bit of cr-daily-branch and it differs only in
the expected ways from a standard xen-unstable branch (i.e. uses xen.git#master
and qemu.git#master).

I did consider causing make-flight:job_create_test_filter_callback to omit any
test which didn't use qemuu but I decided not to because it is used for PV
qdisk backends too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: grab Xen PV domU capable images too
Ian Campbell [Mon, 28 Apr 2014 10:41:24 +0000 (11:41 +0100)]
mg-debian-installer-update: grab Xen PV domU capable images too

Some test cases are going to want this, so they should be grabbed and
versioned here.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoBuild against liblzma to support .xz compressed kernels
Ian Campbell [Mon, 28 Apr 2014 10:41:23 +0000 (11:41 +0100)]
Build against liblzma to support .xz compressed kernels

Install the library at runtime.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocr-external-linux: remove
Ian Campbell [Mon, 28 Apr 2014 10:41:22 +0000 (11:41 +0100)]
cr-external-linux: remove

Seems to be unused.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocr-daily-branch: Only build oldkern for xen-unstable flights
Ian Campbell [Thu, 15 May 2014 10:35:44 +0000 (11:35 +0100)]
cr-daily-branch: Only build oldkern for xen-unstable flights

This is just supposed to be a build test, it doesn't actually get used.
There's no need to be doing this for every flight, doing it for xen-unstable
is enough

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocrontab: Bisect the libvirt branch
Ian Campbell [Wed, 14 May 2014 08:14:16 +0000 (09:14 +0100)]
crontab: Bisect the libvirt branch

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: Updates for newer armhf kernels from backports.
Ian Campbell [Mon, 28 Apr 2014 09:18:38 +0000 (10:18 +0100)]
mg-debian-installer-update: Updates for newer armhf kernels from backports.

In newer kernels sd_mod.ko requires crypto symbols, so include
kernel/crypto/*.ko in the installer overlay cpio.

Also prefer the newest versions of packages pulled in from backports, by
reversing the sort. Anchor the linux-image package name to we only pick up the
-armmp flavour and not -armmp-lpae (which is unnecessary in an installer
environment).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorump kernels: Create a "branch" for rump kernel tests flight-26348 flight-26351 flight-26352 flight-26354 flight-26355 flight-26356 flight-26357 flight-26359 flight-26361
Ian Jackson [Fri, 16 May 2014 16:22:10 +0000 (17:22 +0100)]
rump kernels: Create a "branch" for rump kernel tests

This is a traditional push gate.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorump kernels: Provide two test jobs
Ian Jackson [Fri, 16 May 2014 15:57:16 +0000 (16:57 +0100)]
rump kernels: Provide two test jobs

10 years agorump kernels: Provide a build job
Ian Jackson [Thu, 15 May 2014 15:48:56 +0000 (16:48 +0100)]
rump kernels: Provide a build job

Provide:
  - test script ts-rumpuserxen-build
  - recipe build-rumpuserxen
  - in flights, job build-i386-rumpuserxen
  - in ap-common, TREE and TREEVCS for RUMPUSERXEN and RUMPUSERXEN_RUMPSRC

mfi-common also honours REVISION_RUMPUSERXEN, but provides no way to
specify the revisions of the subtrees (the two submodules and
rumpsrc).  The upstream rumpuser-xen git repository contains the
commit hashes of those.

We had to do some pretty unpleasant things to get the rumpuser-xen
build system to work with rumpsrc we cloned ourselves.

During bisection the bisector must choose a unification of the various
trees' revision graphs, into a single graph.  But we do not here teach
adhoc-revtuple-generator about these interrelationships.  As a result
the bisector may choose a unification inconsistent with the subtree
references in rumpuser-xen.git.  This might impair the performance of
the bisector in the presence of cross-tree version incompatibilities,
but the results should be tolerable for now.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomfi-common: Honour REVISION_LIBVIRT=disable
Ian Jackson [Fri, 16 May 2014 16:56:54 +0000 (17:56 +0100)]
mfi-common: Honour REVISION_LIBVIRT=disable

This makes it possible for (for example) cr-daily-branch to disable
the libvirt build, when it is not needed.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoprepareguest: Tolerate $mb=undef
Ian Jackson [Fri, 16 May 2014 15:51:59 +0000 (16:51 +0100)]
prepareguest: Tolerate $mb=undef

Allow callers to pass undef for $mb, to indicate that there is no need
for prepareguest to do anything about storage.

No functional change for any of the existing callers.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoguest_umount_lv: Tolerate lack of volume group
Ian Jackson [Fri, 16 May 2014 15:27:25 +0000 (16:27 +0100)]
guest_umount_lv: Tolerate lack of volume group

Check if the guest has a volume group, and if not do nothing.

We are going to introduce a kind of guest with no significant storage.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agotarget_jobdir: Create the directory
Ian Jackson [Fri, 16 May 2014 15:53:43 +0000 (16:53 +0100)]
target_jobdir: Create the directory

Callers of target_jobdir can now rely on the directory existing after
it returns.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTestSupport: Break out target_extract_jobdistpath_subdir
Ian Jackson [Fri, 16 May 2014 15:26:26 +0000 (16:26 +0100)]
TestSupport: Break out target_extract_jobdistpath_subdir

This is the meat of what was Osstest::BuildSupport::xendist, which now
becomes a simple convenience wrapper.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTestSupport: Break out target_jobdir
Ian Jackson [Fri, 16 May 2014 15:20:12 +0000 (16:20 +0100)]
TestSupport: Break out target_jobdir

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoBuildSupport: Provide some support for git submodules
Ian Jackson [Thu, 15 May 2014 15:01:13 +0000 (16:01 +0100)]
BuildSupport: Provide some support for git submodules

* Provide a new "submodulefixup" function which, given a cloned
  tree which has submodules:
    - massages the .gitmodules file according to the tree_*
      runvars and massage_url
    - arranges for the submodules to be fetched and checked out
    - stores the revisions of the submodules, as necessary
    - honours any revision_ tags for the submodules
    - allows the caller to give the submodules names which are
      suitable for use in runvars etc. (and might differ from
      the names or paths used in the supermodule)

* In a submodule, .git is actually a file containing a reference
  to a subdirectory of the main git tree.  Cope with that.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoBuildSupport, ts-*-build: Remove some clone-and-hack
Ian Jackson [Wed, 14 May 2014 13:30:54 +0000 (14:30 +0100)]
BuildSupport, ts-*-build: Remove some clone-and-hack

Introduce Osstest::BuildSupport containing selectbuildhost,
builddirsprops and prepbuilddirs.  The corresponding replicated code
from the various ts-*-build scripts is abolished.

Also break out xendist, which we are going to want to reuse.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agogit_massage_url: Make idempotent
Ian Jackson [Thu, 15 May 2014 15:47:46 +0000 (16:47 +0100)]
git_massage_url: Make idempotent

If called more than once, it should not produce a url starting with
multiple copies of the git cache proxy prefix.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTestSupport: Provide target_editfile
Ian Jackson [Thu, 15 May 2014 15:00:57 +0000 (16:00 +0100)]
TestSupport: Provide target_editfile

... as well as target_editfile_root.  No callers yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
11 years agomake-flight etc.: More Debian HVM testcases flight-26236 flight-26255 flight-26288 flight-26315 flight-26333 flight-26334 flight-26335 flight-26336 flight-26337 flight-26340 flight-26341 flight-26342 flight-26344 flight-26345 flight-26346 flight-26347 flight-26349
Wei Liu [Wed, 23 Apr 2014 17:03:28 +0000 (18:03 +0100)]
make-flight etc.: More Debian HVM testcases

ts-debian-hvm-install was written to work as general HVM testcase. It
can be tuned to easily use another BIOS other than OVMF.

Modify make-flight and friends to create more Debian HVM 64bit testcases
to run with SeaBIOS and ROMBIOS.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
11 years agoadhoc-people-touched-filter: Include patch authors
Ian Jackson [Mon, 12 May 2014 13:36:10 +0000 (14:36 +0100)]
adhoc-people-touched-filter: Include patch authors

The git "Author:" tag of each commit should be included.

(We used to include only special pseudo-headers from the patch
comment, because the old hg practice used to be to put the committer
in the hg commit author.  But git separates committer and author.)

This involves a slight refactoring.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
11 years agolibvirt: Fixes for older versions of Xen
Ian Campbell [Fri, 2 May 2014 13:08:04 +0000 (14:08 +0100)]
libvirt: Fixes for older versions of Xen

libvirt supports Xen 4.2 onwards. However:
  - We were not correctly setting the libvirtbuildjob runvar on 4.3 and
    earlier.
  - Xen 4.2 installs to /usr not /usr/local so the CFLAGS/LDFLAGS used to build
    libvirt were wrong.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
11 years agoDrop unnecessary TREEVCS settings for libvirt and gnulib-libvirt
Ian Campbell [Tue, 6 May 2014 08:05:23 +0000 (09:05 +0100)]
Drop unnecessary TREEVCS settings for libvirt and gnulib-libvirt

This will be correctly inferred from the URL.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
11 years agostandalone: Export BUILD_LVEXTEND_MAX when running make-flight.
Ian Campbell [Fri, 2 May 2014 10:23:04 +0000 (11:23 +0100)]
standalone: Export BUILD_LVEXTEND_MAX when running make-flight.

This is stops the build taking over the entire VG and allows tests to be run
without reinstalling the host.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
11 years agostandalone: Use WebspaceLog for apache log check
Ian Campbell [Thu, 1 May 2014 16:10:55 +0000 (17:10 +0100)]
standalone: Use WebspaceLog for apache log check

Also print the result in the error message for cut-and-paste convenience.

While there I noticed that run-test was setting OSSTEST_CONFIG twice.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
11 years agoTestSupport: stash installer files etc.
Ian Jackson [Fri, 9 May 2014 11:04:19 +0000 (12:04 +0100)]
TestSupport: stash installer files etc.

Stash the pxeboot config file, the preseed file, et al.  This is done
centrally in file_link_contents.

We don't stash things whose $contents is actually a coderef because
(a) doing so would be inconvenient and (b) many of them are large and
not very interesting.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
11 years agots-xen-build: collect built seabios and ovmf versions flight-26109 flight-26124 flight-26125 flight-26126 flight-26127 flight-26129 flight-26130 flight-26131 flight-26132 flight-26133 flight-26134 flight-26135 flight-26136 flight-26137 flight-26138 flight-26139 flight-26140 flight-26141 flight-26142 flight-26143 flight-26145 flight-26146 flight-26147 flight-26148 flight-26149 flight-26150 flight-26151 flight-26152 flight-26153 flight-26154 flight-26155 flight-26156 flight-26157 flight-26158 flight-26160 flight-26161 flight-26162 flight-26163 flight-26164 flight-26165 flight-26166 flight-26167 flight-26168 flight-26169 flight-26170 flight-26171 flight-26172 flight-26173 flight-26174 flight-26176 flight-26177 flight-26178 flight-26179 flight-26180 flight-26181 flight-26182 flight-26183 flight-26187 flight-26188 flight-26189 flight-26190 flight-26191 flight-26192 flight-26193 flight-26194 flight-26195 flight-26196 flight-26198 flight-26199 flight-26200 flight-26201 flight-26202 flight-26203 flight-26204 flight-26206 flight-26207 flight-26208 flight-26209 flight-26210 flight-26212 flight-26213 flight-26214 flight-26215 flight-26216 flight-26217 flight-26220 flight-26221 flight-26222 flight-26223 flight-26224 flight-26225 flight-26226 flight-26228 flight-26233 flight-26234 flight-26235 flight-26237 flight-26238 flight-26239 flight-26241 flight-26245 flight-26246 flight-26248 flight-26249 flight-26250 flight-26251 flight-26252 flight-26253 flight-26254 flight-26256 flight-26257 flight-26258 flight-26259 flight-26260 flight-26261 flight-26263 flight-26268 flight-26269 flight-26271 flight-26273 flight-26274 flight-26275 flight-26276 flight-26277 flight-26278 flight-26279 flight-26280 flight-26281 flight-26282 flight-26283 flight-26284 flight-26285 flight-26286 flight-26287 flight-26289 flight-26293 flight-26301 flight-26302 flight-26303 flight-26304 flight-26305 flight-26306 flight-26307 flight-26309 flight-26317 flight-26318 flight-26319 flight-26320 flight-26321 flight-26322 flight-26323 flight-26324 flight-26325
Ian Campbell [Fri, 25 Apr 2014 16:11:08 +0000 (17:11 +0100)]
ts-xen-build: collect built seabios and ovmf versions

The seabios path was incorrect and ovmf was not present.

Take the opportunity to rename collectversion_qemu to collectversions.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
11 years agocri-common: Only use git proxy for git, http and https URLs
Ian Campbell [Fri, 25 Apr 2014 15:15:20 +0000 (16:15 +0100)]
cri-common: Only use git proxy for git, http and https URLs

For testing I had ap-fetch-version-old returning a file:// URL to a local repo.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
11 years agots-debian-hvm-install: increase boot_timeout to 200s flight-26093 flight-26094 flight-26095 flight-26096 flight-26097 flight-26098 flight-26099 flight-26100 flight-26101 flight-26102 flight-26103 flight-26104 flight-26105 flight-26106 flight-26107 flight-26110 flight-26111 flight-26112 flight-26113 flight-26114 flight-26115 flight-26117 flight-26118 flight-26119 flight-26120 flight-26121 flight-26122 flight-26123 flight-26128 flight-26144 flight-26159 flight-26175 flight-26197 flight-26205 flight-26227 flight-26232 flight-26262 flight-26292 flight-26314 flight-26326 flight-26339 flight-26343 flight-26350 flight-26358 flight-26372 flight-26459 flight-26483 flight-26506 flight-26551 flight-26664 flight-26885 flight-26932 flight-26947 flight-26970 flight-26994 flight-27005 flight-27021 flight-27035 flight-27058 flight-27069 flight-27085 flight-27115 flight-27140 flight-27150
Ian Campbell [Tue, 29 Apr 2014 12:16:34 +0000 (13:16 +0100)]
ts-debian-hvm-install: increase boot_timeout to 200s

Tianocore can be slow to post, especially on large guests. We've been hitting
the 100s timeout on lake-frog and fire-frog. Even when succeeding the duration
is 90s+.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
11 years agoproduction-config: Update di version again.
Ian Campbell [Mon, 28 Apr 2014 07:55:06 +0000 (08:55 +0100)]
production-config: Update di version again.

This contains the current version for amd64 and i386 (actually the same as
yesterday) which picks up the latest point release but for armhf this is a link
to the old (2013-12-12) image since armhf uses a backports kernel and is
unaffected by the point release but does seem to have a deendency on some newer
infrastructure (depmod et al, I think) which needs sorting out.

11 years agoproduction-config: Update to newer debian installer snapshot.
Ian Campbell [Sun, 27 Apr 2014 11:01:12 +0000 (12:01 +0100)]
production-config: Update to newer debian installer snapshot.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
11 years agocrontab: Add a daily run of the libvirt branch.
Ian Campbell [Fri, 25 Apr 2014 09:07:56 +0000 (10:07 +0100)]
crontab: Add a daily run of the libvirt branch.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
11 years agoAdd libvirt build and test jobs.
Ian Campbell [Wed, 23 Apr 2014 10:07:01 +0000 (11:07 +0100)]
Add libvirt build and test jobs.

I have run this through a "cr-daily-branch --real" and the push gate machinery
seem functional. The test cases run through to starting a guest which fails
because virsh support has not been implemented yet.

The libvirt.git initscript is horribly redhat specific, and the debian package
one is large and complex, so ts-libvirt-build inserts its own simple
initscript.

I have observed the autogen.sh phase taking 34 minutes on army (arm build
cluster) hence the surprisingly large timeout of an hour. The build took around
20 minutes on the same system, but I used a timeout of an hour anyway to be
safe.

The standalone wrapper is updated to handle path_libvirt in set-paths.

libvirt.git's autogen.sh involves initialising/cloning a git submodule, which
doesn't use osstest's git cache and introduces unpredicability into what is
built. Therefore ap-* are taught about a new libvirt-gnulib tree (prefixed
libvirt incase we end up with other users of gnulib in the future). However
this doesn't use the usual push gate mechanism, since this was deemed to be
overkill and just always uses the tip of gnulib. The results are recorded
though so the bisector can still operate over this tree in the event of
breakage.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jim Fehlig <jfehlig@suse.com>
Cc: Dario Faggioli <dario.faggioli@citrix.com>
11 years agocrontab: Enable linux-next branch
Ian Campbell [Fri, 25 Apr 2014 08:37:13 +0000 (09:37 +0100)]
crontab: Enable linux-next branch

linux-next is published week days in the early hours GMT.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
11 years agocr-*, ap-*: Facilities for testing linux-next flight-25976 flight-25986 flight-25987 flight-25988 flight-25989 flight-25991 flight-25992 flight-25993 flight-25994 flight-25995 flight-25996 flight-25997 flight-25998 flight-25999 flight-26000 flight-26001 flight-26002 flight-26003 flight-26004 flight-26005 flight-26006 flight-26007 flight-26008 flight-26009 flight-26011 flight-26012 flight-26013 flight-26014 flight-26015 flight-26016 flight-26017 flight-26018 flight-26019 flight-26021 flight-26022 flight-26023 flight-26024 flight-26025 flight-26026 flight-26027 flight-26029 flight-26030 flight-26031 flight-26032 flight-26034 flight-26035 flight-26036 flight-26037 flight-26038 flight-26039 flight-26040 flight-26041 flight-26043 flight-26045 flight-26046 flight-26048 flight-26049 flight-26051 flight-26054 flight-26055 flight-26056 flight-26057 flight-26058 flight-26059 flight-26060 flight-26061 flight-26062 flight-26063 flight-26065 flight-26066 flight-26067 flight-26068 flight-26070 flight-26071 flight-26072 flight-26074 flight-26075 flight-26076 flight-26077 flight-26078 flight-26079 flight-26080 flight-26081 flight-26083 flight-26084 flight-26085 flight-26087 flight-26088 flight-26089 flight-26090 flight-26091
Ian Jackson [Thu, 10 Apr 2014 17:20:18 +0000 (18:20 +0100)]
cr-*, ap-*: Facilities for testing linux-next

* Define the new branch's git source repo in ap-common.

* Provide a branch-settings file which disables running additional
  baseline tests, listing changesets in flight reports, pushing,
  and which arranges for the baseline revision for the flight
  report to be determined "late", ie after the to-be-tested
  versions of everything have been decided.

* When the baseline revision is to be determined "late", run
  the new script ap-fetch-version-baseline-late.  This is another
  collection of per-branch rules.  We introduce currently one
  per-branch rule, for linux-next:

* The baseline version for a test of linux-next is
    - the version tested by the most recent flight on the
      linux-linus branch;
    - provided that said version is an ancestor of the linux-next
      version under test;
    - otherwise we try an earlier linux-linus flight.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
11 years agosg-check-tested: support --git-ancestor-of
Ian Jackson [Thu, 10 Apr 2014 17:18:54 +0000 (18:18 +0100)]
sg-check-tested: support --git-ancestor-of

This will be used for the linux-next branch, whose baseline is the
most recently tested linux-linus which is an ancestor of the
being-tested linux-next.

We do not provide this for any VCS other than git.  Hopefully we will
never need to.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
11 years agosg-check-tested: add "use strict"
Ian Jackson [Thu, 10 Apr 2014 17:18:01 +0000 (18:18 +0100)]
sg-check-tested: add "use strict"

This revealed no bugs so no other changes needed.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
11 years agots-xen-install: nodhcp: Make idempotent flight-25959 flight-25964 flight-25974 flight-25975 flight-25977 flight-25978 flight-25979 flight-25980 flight-25981 flight-25982 flight-25983 flight-25984 flight-25985
Ian Jackson [Thu, 17 Apr 2014 15:06:05 +0000 (16:06 +0100)]
ts-xen-install: nodhcp: Make idempotent

If run more than once, this function would erroneously determine that
xenbr0 was the primary physical interface name and try to enroll it
into the bridge (also named xenbr0).

Instead, spot the "bridge_ports" line and use it to extract the
physical interface name.  Eliminate from consideration any interfaces
matching xenbr\d+.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Campbell <ian.campbell@citrix.com>
11 years agots-xen-install: nodhcp: Invert sense of interfaces loop
Ian Jackson [Thu, 17 Apr 2014 15:05:02 +0000 (16:05 +0100)]
ts-xen-install: nodhcp: Invert sense of interfaces loop

We are going to want to process other kinds of lines too.  So change
this loop around by inverting the condition and not using "next".

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Campbell <ian.campbell@citrix.com>
11 years agots-xen-install: nodhcp: Fix iface regexp
Ian Jackson [Thu, 17 Apr 2014 15:03:43 +0000 (16:03 +0100)]
ts-xen-install: nodhcp: Fix iface regexp

This regexp would never match because of the spurious $.  Remove the
$ and also the preceding unnecessary \s*.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Campbell <ian.campbell@citrix.com>
11 years agots-hosts-allocate-Executive: improve debugging slightly
Ian Jackson [Wed, 23 Apr 2014 10:04:03 +0000 (11:04 +0100)]
ts-hosts-allocate-Executive: improve debugging slightly

I spotted a logfile where an allocation had failed (later worked OK)
with $allocatable=0 here.  Provide more info next time.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
11 years agocr-daily-branch: Print better message re lack of push
Ian Jackson [Thu, 17 Apr 2014 15:32:33 +0000 (16:32 +0100)]
cr-daily-branch: Print better message re lack of push

wantpush might be false because
 - OSSTEST_PUSH is false (only set to true in daily-cron-settings-real)
 - baseline untested (what we had here)
 - "$tree" is unknown (although I think this would fail elsewhere)
 - input and output revisions are the same

Print the message "Published tested tree is already up to date" only
if that's actually the case.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Campbell <Ian.Campbell@citrix.com>
11 years agopreseed_base: drop obsolete references to NetGateway and NetNetmask
Ian Jackson [Thu, 17 Apr 2014 11:00:36 +0000 (12:00 +0100)]
preseed_base: drop obsolete references to NetGateway and NetNetmask

NetGateway and NetNetmask are not honoured (their lines in the preseed
file are commented out) and I'm not aware of anyone who sets them.
They have been there from the dawn of time.

Removing them gets rid of this warning (which appears twice)
 Use of uninitialized value in concatenation (.) or string at Osstest/Debian.pm line 432.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
11 years agomg-allocate: Fix an "uninitialized value" warning
Ian Jackson [Thu, 17 Apr 2014 10:37:14 +0000 (11:37 +0100)]
mg-allocate: Fix an "uninitialized value" warning

If $ok is 0, $got_shareix could be undef but would still be used
for Info in the return value.  Initialise it to "x".

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>