John Garbutt [Tue, 7 Feb 2017 19:12:50 +0000 (19:12 +0000)]
Stop failed live-migrates getting stuck migrating
When there are failures in driver.cleanup, we are seeing live-migrations
that get stuck in the live-migrating state. While there has been a patch
to stop the cause listed in the bug this closes, there are other
failures (such as a token timeout when talking to cinder or neutron)
that could trigger this same failure mode.
When we hit an error this late in live-migration, it should be a very
rare event, so its best to just put the instance and migration into an
error state, and help alert both the operator and API user to the
failure that has occurred.
Matt Riedemann [Wed, 5 Apr 2017 20:27:41 +0000 (16:27 -0400)]
Perform old-style local delete for shelved offloaded instances
This fixes a regression from some local delete code added for cells v2
where it assumed that if an instance did not have a host, it wasn't
scheduled to a cell yet. That assumption misses the fact that the
instance won't have a host if it was shelved offloaded. And to be
shelved offloaded, the instance had to have first been built on a host
in a cell.
So we simply duplicate the same check as later in the _delete() method
for instance.host or shelved-offloaded to decide what the case is.
Obviously this is all a giant mess of duplicate delete path code that
needs to be unwound, and that's the plan, but first we're fixing
regressions and then we can start rolling this duplication all back
so we can get back to the single local delete flow that we know and love.
Matt Riedemann [Fri, 24 Mar 2017 16:06:07 +0000 (12:06 -0400)]
Set size/status during image create with FakeImageService
This is needed for an upcoming change which introduces
a functional test which shelves a server. Shelving a server
creates a snapshot image and in the real world, glance sets
the size and status attributes on the image when it's created
in glance. Our FakeImageService wasn't doing that, so tests
that are running at the same time with the same fake noauth
credentials are listing images and picking up the shelve
snapshot image which doesn't have size or status set and
that produces a KeyError in the API code.
Matt Riedemann [Sat, 1 Apr 2017 01:08:55 +0000 (21:08 -0400)]
Commit usage decrement after destroying instance
This fixes a regression in Ocata where we were always
decrementing quota usage during instance delete even
if we failed to delete the instance. Now the reservation
is properly committed after the instance is destroyed.
The related functional test is updated to show this working
correctly now.
Matt Riedemann [Fri, 31 Mar 2017 23:56:14 +0000 (19:56 -0400)]
Add regression test for quota decrement bug 1678326
This was spotted from someone validating the fix for
bug 1670627. They reported that even though they failed
to delete an instance in ERROR state that was in cell0,
the quota usage was decremented.
This is because we committed the quota reservation
to decrement the usage before actually attempting to destroy
the instance, rather than upon successful deletion.
The rollback after InstanceNotFound is a noop because of
how the Quotas.rollback method noops if the reservations
were already committed. That is in itself arguably a bug,
but not fixed here, especially since the counting quotas
work in Pike will remove all of the reservations commit and
rollback code.
Matt Riedemann [Wed, 5 Apr 2017 19:12:41 +0000 (15:12 -0400)]
Short-circuit local delete path for cells v2 and InstanceNotFound
When we're going down the local delete path for cells v2 in the API
and instance.destroy() fails with an InstanceNotFound error, we are
racing with a concurrent delete request and know that the instance
is alread deleted, so we can just return rather than fall through to
the rest of the code in the _delete() method, like for BDMs and
console tokens.
Matt Riedemann [Fri, 24 Mar 2017 02:07:03 +0000 (22:07 -0400)]
Do not attempt to load osinfo if we do not have os_distro
We get a warning logged every time we try to load up osinfo
with an image metadata that does not have the 'os_distro'
property set. We should be smarter and just not try to load
osinfo at all if we know we cannot get results.
Matt Riedemann [Tue, 21 Mar 2017 17:18:08 +0000 (13:18 -0400)]
libvirt: conditionally set script path for ethernet vif types
Change I4f97c05e2dec610af22a5150dd27696e1d767896 worked around
a change introduced in libvirt 1.3.3 where the script path on
a LibvirtConfigGuestInterface could not be the emptry string
because libvirt would literally take that as the path and couldn't
resolve it, when in fact it used to indicate to libvirt that the
script path is a noop. This has been fixed in libvirt 3.1.
On Ubuntu with libvirt<1.3.3, if the script path is None then
it defaults to /etc/qemu-ifup which is blocked by AppArmor.
So this change adds a conditional check when setting the script
path value based on the libvirt version so we can straddle releases.
Starting from version 1.3.5, Libvirt allows to set a vlan tag for macvtap
passthrough mode on SR-IOV VFs. Libvirt also removes any vlan tags that
has been set externally, by the ip link command.
In order to support the older libvirt versions, this code will make
the behaviour backward compatible by checking the libvirt version.
This can be completely removed once the minimum libvirt version will increase.
Evgeny Antyshev [Mon, 6 Mar 2017 14:27:06 +0000 (14:27 +0000)]
get_model method missing for Ploop image
Image.get_model is called in partition injection code,
and now inject partition attempt fails unconditionally.
This patch makes use of disk/api.py inject_data failure tolerance:
it doesn't fail unless injected data is mandatory.
Balazs Gibizer [Fri, 17 Mar 2017 10:24:49 +0000 (11:24 +0100)]
do not include context to exception notification
The wrap_exception decorator optionally emited a notification.
Based on the code comments the original intention was not to include the
context to that notification due to security reasons. However the
implementation did included the context to the payload of the legacy
notification.
Recently we saw circural reference errors during the payload serialization
of this notification. Based on the logs the only complex data structure
that could cause circural reference is the context. So this patch
removes the context from the legacy exception notification.
The versioned exception notification is not affected as it does not
contain the args of the decorated function.
ShunliZhou [Fri, 10 Mar 2017 06:05:57 +0000 (14:05 +0800)]
Add populate_retry to schedule_and_build_instances
When boot an instance and failed on the compute node, nova will
not retry to boot on other host.
Since https://review.openstack.org/#/c/319379/ change the create
instance workflow and called schedule_and_build_instances which
not populate the retry into filter properties. So nova will not
retry when boot on compute fail. This patch populate retry to
instance properties when call schedule_and_build_instances.
melanie witt [Thu, 16 Mar 2017 18:24:23 +0000 (18:24 +0000)]
Fix functional regression/recreate test for bug 1671648
There are a couple of issues with the test:
1. It doesn't consider both hosts from the two compute services
during scheduling.
2. There is a race where sometimes claims.Claim.__init__ won't
be called because if the RT instance_claim runs before
update_available_resource has run, it will create a
claims.NopClaim instead.
This adds the RetryFilter to enabled_filters, adds set_nodes() calls
to set the nodenames of each compute service to match its host,
resulting in consideration of both hosts for scheduling, and stubs
resource_tracker.ResourceTracker.instance_claim instead of
claims.Claim.__init__.
NOTE(mriedem): The conflict is due to this patch coming after cb4ce72f5f092644aa9b84fa58bcb9fd89b6bedc in Pike. Since this
is a fix for the functional test that the bug fix builds on,
we actually want this to come *before* the bug fix backport.
Matt Riedemann [Wed, 15 Mar 2017 20:58:11 +0000 (16:58 -0400)]
Add a functional regression/recreate test for bug 1671648
This adds a test which recreates the regression bug introduced
in Ocata where build retries are not populated when creating
instances in conductor for cells v2.
The change that fixes the bug will go on top of this and modify
the test to show the bug is fixed.
Matt Riedemann [Tue, 14 Mar 2017 16:34:59 +0000 (12:34 -0400)]
Add release notes for 15.0.1 Ocata bug fix release
There are several high severity, high impact bug fixes going
into the upcoming 15.0.1 Ocata release. This change adds release
notes highlighting the most important fixes along with a known
issue for another regression bug that is not yet fixed in Ocata,
but should be shortly.
Matt Riedemann [Thu, 9 Mar 2017 02:51:07 +0000 (21:51 -0500)]
Decrement quota usage when deleting an instance in cell0
When we fail to schedule an instance, e.g. there are no hosts
available, conductor creates the instance in the cell0 database
and deletes the build request. At this point quota usage
has been incremented in the main 'nova' database.
When the instance is deleted, the build request is already gone
so _delete_while_booting returns False and we lookup the instance
in cell0 and delete it from there, but that flow wasn't decrementing
quota usage like _delete_while_booting was.
This change adds the same quota usage decrement handling that
_delete_while_booting performs.
NOTE(mriedem): This change also pulls in some things from
I7de87dce216835729283bca69f0eff59a679b624 which is not being
backported to Ocata since in Pike it solves a slightly different
part of this quota usage issue. In Pike the cell mapping db_connection
is actually stored on the context object when we get the instance
from nova.compute.api.API.get(). So the fix in Pike is slightly
different from Ocata. However, what we need to pull from that Pike
change is:
1. We need to target the cell that the instance lives in to get the
flavor information when creating the quota reservation.
2. We need to change the functional regression test to assert that
the bug is fixed.
The code and tests are adjusted to be a sort of mix between both
changes in Pike without requiring a full backport of the 2nd
part of the fix in Pike.
This adds a functional regression test for bug 1670627.
This is the recreate scenario. Patches that are proposed to
fix the bug will build on top of this and change it's assertions
to know when it's properly fixed.
Balazs Gibizer [Thu, 9 Mar 2017 16:28:02 +0000 (17:28 +0100)]
Fix missing instance.delete notification
The I8742071b55f018f864f5a382de20075a5b444a79 introduced cases when an
instance object is destroyed without the instance.delete notification
being emitted.
This patch adds the necessary notification to restore legacy
behaviour.
Ionuț Bîru [Fri, 10 Mar 2017 15:22:22 +0000 (17:22 +0200)]
Correctly set up deprecation warning
In the current state, no warning was output in the logs.
With this fix, a warning is output in the logs and the value from
[DEFAULT] is used correctly.
Stephen Finucane [Tue, 21 Feb 2017 20:34:50 +0000 (15:34 -0500)]
conf: Fix invalid rST comments
These were breaking the openstack-docs-tools build, which is used to
build the central config guide. We should probably add a unit tests to
ensure all config options use valid (or at least, error-free) rST, but
that's a job for another day.
Matt Riedemann [Tue, 7 Mar 2017 01:25:55 +0000 (20:25 -0500)]
Remove unused placement_database config options
The placement_database config options were added in Newton
but the actual code to use the options was reverted and is
not used. The help text actually has a typo (Ocata was 15.0.0,
not 14.0.0) and, more importantly, the help text makes it sound
like you should really be setting up a separate database for the
placement service but we don't actually use these options, we use
the api_database options for all of the placement data models.
To avoid confusion until this is actually supported, let's just
remove the options which should have been cleaned up as part of 39fb302fd9c8fc57d3e4bea1c60a02ad5067163f anyway.
Matt Riedemann [Tue, 7 Mar 2017 00:00:31 +0000 (19:00 -0500)]
libvirt: pass log_path to _create_pty_device for non-kvm/qemu
log_path is required in _create_pty_device if:
1. serial consoles are disabled
2. libvirt/qemu are new enough that they support virtlogd
This was working fine for kvm and qemu since _create_consoles_s390x
and _create_consoles_qemu_kvm pass in the log_path, but for the
non-kvm/qemu cases, like xen, the log_path wasn't provided.
This wasn't caught by the XenProject CI since it's using libvirt
1.3.1 which does not have virtlogd support so this path was
not exercised and apparently not unit tested either.
A release note is provided since this is a pretty severe bug if
you're running new enough libvirt/qemu and not using kvm/qemu as
the virt type because CONF.serial_console.enabled is False by
default so you're going to have failed server creates immediately
upon upgrading to Ocata.
Stephen Finucane [Wed, 22 Feb 2017 18:02:47 +0000 (13:02 -0500)]
libvirt: Handle InstanceNotFound exception
In 'ad1c7ac2', we stopped returning NovaException from certain libvirt
operations in favour of more specific exception types. Unfortunately, as
part of this changeover we missed an exception type. Correct this
oversight.
m4cr0v [Tue, 13 Dec 2016 08:17:34 +0000 (16:17 +0800)]
Fix spice channel type
nova generate wrong spice channel type 'pty', which should be 'spicevmc'
Closes-Bug: #1634495
Change-Id: I58e9a8df9f40f900eeabd9d40429663cbbedb8d6
(cherry picked from commit f4be97d8cf8811a64a2d9f7d990e79d45cdf0d62)
Matt Riedemann [Fri, 17 Feb 2017 18:31:41 +0000 (13:31 -0500)]
Only create vendordata_dynamic ksa session if needed
We're logging a warning about vendordata dynamic auth
not being configured every time we create a server with
a config drive. The dynamic vendordata v2 stuff is all
optional and controlled via configuring:
CONF.api.vendordata_dynamic_targets
This change only attempts to create the ksa session
when we try to make a request, which would only happen
if CONF.api.vendordata_dynamic_targets is configured.
Ensure that instance directory is removed after success migration/resize
Nova recreates instance directory on source host after successful migration/resize.
This patch removes directory of migrated instance from source host.
Jordan Pittier [Fri, 27 Jan 2017 14:22:25 +0000 (15:22 +0100)]
Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin
When an admin calls /v2.1/servers/detail?tenant_id=XX, then the
`get_all` method of nova.compute.api.API is called with 2
conflicting search options: {'tenant_id': XX, 'project_id': YY}.
But because, latter on, in that `get_all` method we define a dict
called filter_mapping, on which we iter upon, which value takes
precedence depends on the order in which the dict is iterated upon.
As the order in unpredictable and varies between Py2 and Py3, this
is problematic. Especially, on Python 2 hash randomization is
disabled by default but it's enabled by default on Py3 (unless
the PYTHONHASHSEED env var is set to a fixed value) [0]
The (unreliable) order we iterate on items of that `filter_mapping`
dict is why the Tempest test_list_servers_by_admin_with_specified_tenant
test randomly fails on Py35.
This patch ensures that, if the all_tenants search option is
not set, then the `tenant_id` search option is ignored. Note that
this *is* the current behavior on Py27, as documented in lp:#1185290
and tested by Tempest here [1].
Dan Smith [Mon, 27 Feb 2017 15:52:29 +0000 (07:52 -0800)]
Ignore deleted services in minimum version calculation
When we go to detect the minimum version for a given service, we
should ignore any deleted services. Without this, we will return
the minimum version of all records, including those that have been
deleted with "nova service-delete". This patch filters deleted
services from the query.
Use the keystone session loader in the placement reporting
Using load_session_from_conf_options has the advantage that it honors
session settings like cafile and insecure, to make use of non-system TLS
certificates (or disable certificate checks at all). Also client
certificates and timeout values can be specified, too.
melanie witt [Fri, 17 Feb 2017 17:27:57 +0000 (17:27 +0000)]
Skip soft-deleted records in 330_enforce_mitaka_online_migrations
The 330_enforce_mitaka_online_migrations migration considers
soft-deleted records as unmigrated (the blocker migration uses the
select function from sqlalchemy), but the online migrations only
migrate non-deleted records (the migrations use the model_query
function which defaults to read_deleted='no'). So even after running
all of the online migrations, operators can get stuck until they can
hard delete any soft-deleted compute_nodes, aggregates, and
pci_devices records they have.
Matt Riedemann [Thu, 16 Feb 2017 17:15:12 +0000 (12:15 -0500)]
Deprecate xenserver.vif_driver config option and change default
There are two in-tree options for the xenserver.vif_driver,
the bridge driver and the ovs driver. The XenAPI subteam has
confirmed that the bridge driver is for nova-network (which is
deprecated) and the ovs driver is for Neutron, and that's how
things are tested in CI.
Since we changed the default on use_neutron to be True for Ocata
we need to change the default on the vif_driver to be the ovs
driver so it works with the default config, which is Neutron.
We're deprecating the option though since we can use the use_neutron
option to decide which vif driver to load - which will make
deploying and configuring nova with xen as the backend simpler.
Huan Xie [Sun, 22 Jan 2017 11:08:40 +0000 (03:08 -0800)]
Fix live migrate with XenServer
Live migration with XenServer as hypervisor failed with xapi
errors "VIF_NOT_IN_MAP". There are two reasons for this
problem:
(1) Before XS7.0, it supports VM live migration without
setting vif_ref and network_ref explicitly if the destination
host has same network, but since XS7.0, it doesn't support
this way, we must give vif_ref and network_ref mapping.
(2) In nova, XenServer has introduced interim network for
fixing ovs updating wrong port in neutron, see bug 1268955
and also interim network can assist support neutron security
group (linux bridge) as we cannot make VIF connected to
linux bridge directly via XAPI
To achieve this, we will add {src_vif_ref: dest_network_ref}
mapping information, in pre_live_migration, we first create
interim network in destination host and store
{neutron_vif_uuid: dest_network_ref} in migrate_data, then in
source host, before live_migration, we will calculate the
{src_vif_ref: dest_network_ref} and set it as parameters to
xapi when calling VM.migrate_send. Also, we will handle the
case where the destination host is running older code that
doesn't have this new src_vif_ref mapping, like live migrating
from an Ocata compute node to a Newton compute node.
Corey Bryant [Wed, 15 Feb 2017 21:43:50 +0000 (16:43 -0500)]
Enable defaults for cell_v2 update_cell command
Initialize optional parameters for update_cell() to None and
enable getting the transport_url and db_connection from
nova.conf if not specified as arguments.
Matt Riedemann [Mon, 13 Feb 2017 20:48:43 +0000 (15:48 -0500)]
Cleanup some issues with CONF.placement.os_interface
This change fixes a few things with the recently added
"os_interface" option in the [placement] config group.
1. It adds tests for the scheduler report client that
were missing in the original change that added the
config option.
2. It uses the option in the "nova-status upgrade check"
command so it is consistent with how the scheduler
report client uses it.
3. It removes the restrictive choices list from the
config option definition. keystoneauth1 allows an
"auth" value for the endpoint interface which means
don't use the service catalog to find the endpoint
but instead just read it from the "auth_url" config
option. Also, the Keystone v3 API performs strict
validation of the endpoint interface when creating
an endpoint record. The list of supported interfaces
may change over time, so we shouldn't encode that
list within Nova.
4. As part of removing the choices, the release note
associated with the new option is updated and changed
from a 'feature' release note to simply 'other' since
it's not really a feature as much as it is a bug fix.
Dan Smith [Fri, 10 Feb 2017 15:37:51 +0000 (07:37 -0800)]
Remove straggling use of main db flavors in cellsv1 code
This remaining use of the flavor query routine from the cellsv1 code
still looks at the main database. This patch converts it to use the
object which looks in the right place.
Andreas Jaeger [Sun, 5 Feb 2017 14:56:27 +0000 (15:56 +0100)]
Prepare for using standard python tests
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.
This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.
See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html
Update devref for this.
Needed-By: Iea42a0525b2c5a5cdbf8604eb23a6e7b029f6b48
Change-Id: Ie9bae659077dbe299eea131572117036065bdccf
(cherry picked from commit d60dffc6bef4d2de32b9509a62f894136c434c3c)
John Garbutt [Mon, 6 Feb 2017 17:42:59 +0000 (17:42 +0000)]
Default live_migration_progress_timeout to off
live_migration_progress_timeout aims to timeout a live-migration well
before the live_migration_completion_timeout limit, by looking for when
it appears that no progress has been made copying the memory between the
hosts. However, it turns out there are several problems with the way we
monitor progress. In production, and stress testing, having
live_migration_progress_timeout > 0 has caused random timeout failures
for live-migrations that take longer than live_migration_progress_timeout
One problem is that block_migrations appear to show no progress, as it
seems we only look for progress in copying memory. Also the way we query
QEMU via libvirt breaks when there are multiple iterations of memory
copying.
We need to revisit this bug and either fix the progress mechanism or
remove the all the code that checks for the progress (including the
automatic trigger for post-copy). But in the mean time, lets default to
having no timeout, and warn users that have overridden this
configuration by deprecating the live_migration_progress_timeout
configuration option.
For users concerned about live-migration timeout errors, I have
cleaned up the configuration option descriptions, so they have a better
chance of stopping the live-migration timeout errors they may come
across.
Matt Riedemann [Wed, 8 Feb 2017 01:28:13 +0000 (20:28 -0500)]
Allow None for block_device_mapping_v2.boot_index
The legacy v2 API allowed None for the boot_index [1]. It
allowed this implicitly because the API code would convert
the block_device_mapping_v2 dict from the request into a
BlockDeviceMapping object, which has a boot_index field that
is nullable (allows None).
The API reference documentation [2] also says:
"To disable a device from booting, set the boot index
to a negative value or use the default boot index value,
which is None."
It appears that with the move to v2.1 and request schema
validation, the boot_index schema was erroneously set to
not allow None for a value, which is not backward compatible
with the v2 API behavior.
This change fixes the schema to allow boot_index=None again
and adds a test to show it working.
This should not require a microversion bump since it's fixing
a regression in the v2.1 API which worked in the v2 API and
is already handled throughout Nova's block device code.
Dan Smith [Tue, 7 Feb 2017 16:43:42 +0000 (08:43 -0800)]
Add an update_cell command to nova-manage
In case a cell record was incorrectly created, or needs to be updated
to point to a new database or mq endpoint, we need a way for users to
be able to update a CellMapping record. Since deleting and re-creating
is definitely not an option, this patch adds an update_cell which
allows updating the mutable fields if necessary.
Lee Yarwood [Tue, 31 Jan 2017 18:39:15 +0000 (18:39 +0000)]
libvirt: Remove redundant bdm serial mangling and saving during swap_volume
During an initial swap_volume call the serial of the original volume was
previously stashed in the connection_info of the new volume by the
compute layer. This was used by I19d5182d11 to allow the virt driver to
lookup and update the existing BDM with the new volume's connection_info
after it had been used by connect_volume.
Future calls to swap_volume in the compute layer would not
update the serial found in the old volume connection_info. This would
result in an invalid serial being copied into the new volume
connection_info and used to preform a lookup of a BDM that didn't exist.
To correct this we now explicitly set the serial of the new volume to
that of the new volume id. While the correct serial id should already be
present in the connection_info provided by most backend Cinder volume
drivers the act of updating this dict is required by our own functional
tests to invoke a failure case :
https://git.io/vDmRE
The serial is updated once more to match the volume id returned
by os-migrate-volume-completion prior to the BDM being updated in the
compute layer.
The BDM lookup and save from the virt layer is also removed as the
compute layer retains a reference to new_cinfo and will update the BDM
with this, including any modifications, at the end of swap_volume.
Finally, the associated Tempest admin test is also extended by the
following change to now attempt a second volume swap to verify these
changes :
Dan Smith [Fri, 3 Feb 2017 17:20:11 +0000 (09:20 -0800)]
Update the upgrades part of devref
This removes some references to "online schema migrations", as well
as some references to in-progress things that have been (long since)
completed. It also clarifies some of the upgrade steps, and unifies
the notion of "offline" and "live" upgrades, calling out only a couple
places where the process differs.
This came from me explaining the document to someone and calling out
things that were no longer accurate.
EdLeafe [Thu, 2 Feb 2017 18:48:35 +0000 (18:48 +0000)]
Delete a compute node's resource provider when node is deleted
Currently when a compute node is deleted, its record in the cell DB is
deleted, but its representation as a resource provider in the placement
service remains, along with any inventory and allocations. This could
cause the placement engine to return that provider record, even though
the compute node no longer exists. And since the periodic "healing" by
the resource tracker only updates compute node resources for records in
the compute_nodes table, these old records are never removed.
This patch adds a call to delete the resource provider when the compute
node is deleted. It also adds a method to the scheduler report client
to make these calls to the placement API.
projects / osstest / openstack-nova.git / log