Bugzilla – Bug 8879
xVM hvm guests fail to pxe boot properly against osol_0906 111a AI image
Last modified: 2009-05-19 01:47:28 UTC
You need to log in before you can comment on or make changes to this bug.
This sounds like a regression of Bug 6850, but we're seeing again that hvm guests fail to pxe boot properly when installing from the 111a AI image. I was trying this from a rather old dom0 (nv_106) which I knew worked with the fix for bug 6850. The virt-install command line I used was: # zfs create -V 1g rpool/guests/ai_guest # virt-install -n osol -v --ram 1024 -f /dev/zvol/dsk/rpool/guests/ai_guest --vnc --pxe As before, the symptoms are that we get the grub menu.lst, bring up the OS, start to run filesystem/root:live-media then timeout trying to download the zlib files from the install server, dropping to maintenance mode. Logging in as root, we can see that xnf0 isn't plumbed, which explains why we can't progress with the installation. To work around, in the guest vnc window, login as root, the do: * ifconfig xnf0 plumb * ifconfig xnf0 dhcp * umount /tmp * svcadm clear filesystem/root:live-media * svcadm clear dns/multicast * svcadm enable auto-installer * ^D The fix for bug 6850 holds though - there is a /reconfigure in the miniroot. Kudos to Drew Fisher for spotting this bug when testing the ongoing xVM 3.3 port - that it's reproducible in xVM 3.1 as well suggests it's not our fault.
So what's failing to plumb the interface then? Is either instance of svc:/network/physical is online after the failure? That appears to be the service that ends up calling net_reconfigure(). Also, is svc:/network/datalink-management:default online after the failure?
Marking imcomplete as we're waiting for information from Tim.
Is there any difference between hvm and pv guests regarding this issue? In reproducing 8883, which is a pv guest, its xnf0 interface is plumbed just fine when booted into a 111a or 111b AI image.
Adding this to the blocker list while we continue triaging it.
I'm still investigating, but in the miniroot: * network-physical:default is online * datalink-management:default is online assuming the contents of the logfiles in /etc/svc/volatile are correct (they both report that the start methods exited with status 0) I'll try to build a new miniroot with some useful tools (like svcs - bug 6851 :-) and investigate further. Yep, pv boots are different wrt. enabling networking, as we're able to pass parameters to the kernel from virt-install, namely "-B network-interface=xnf0,xpv-hcp=dhcp,livemode=text". We can't do this for hvm guests - indeed in this case, we're using the menu.lst file obtained via tftp and pxe.
Tim, when you say "there is a /reconfigure in the miniroot.", are you seeing that by inspecting an unbooted AI image bootroot archive, or are you seeing that in a booted miniroot in your hvm guest?
(In reply to comment #6) Both - there's a /reconfigure in the unbooted miniroot, as well as in the booted guest after live-fs-root has dropped to maintenance. (which is odd, I thought something removed that file having found it and set the smf property accordingly?)
(In reply to comment #7) > (In reply to comment #6) > > Both - there's a /reconfigure in the unbooted miniroot, as well as in the > booted guest after live-fs-root has dropped to maintenance. (which is odd, I > thought something removed that file having found it and set the smf property > accordingly?) Yes, startd consumes it and removes it normally. That might be the failure here. Can you use svccfg to see if the reconfigure property in the restarter service is set?
I'd thought /sbin/rcS does that? But anyway, no the SMF property isn't set.
And the file system is read/write, correct?
Yes, all the filesystems are mounted read/write in the guest when dropped to maintenance.
Yes, all the filesystems are mounted read/write in the guest when dropped to maintenance. Sorry, I also need to correct a previous statement: the SMF property *is* set correctly to "true" (I was looking at the return code from svcprop -q and interpreting incorrectly)
I have a 111b AI image running now where I'm seeing the same behaviour. root@usuki-10:/tmp/a# ls -al /space/osol-0906-111b-ai-x86.iso -rw-r--r-- 1 root root 168400896 2009-05-09 01:59 /space/osol-0906-111b-ai-x86.iso
The install process appears to presume that a NIC used for PXE boot will be plumbed in the IP stack when it attempts to downloads the additional components. For PCI devices this happens as a result of code in strplumb which searches the device tree for an "Ethernet Controller" device with the MAC address reported by the PXE process. xnf devices are not inserted into the device tree until xpvd enumerates xenstore. This is happening too late for strplumb to find the device and plumb it in the HVM kernel (perhaps because it expects to tell xpvd to enumerate via an impl_bus_probe, but the probe is added after such probes have already been run). timf tested this theory by removing the pv_rtls driver from the miniroot loaded over PXE. This allows the normal rtls driver to bind to the PCI device emulated by qemu-dm and the installation proceeds successfully (well, it downloads the additional components). One solution would be to have some user-level component plumb xnf0 if it is not already present. This could be added to net-physical, live-devices-local or live-fs-root. However, this would only address the case where a system is booted via PXE but has a non-network root filesystem (i.e. hard disk or ramdisk). For systems with an NFS root it is required that strplumb be able to plumb the NIC before the root filesystem is mounted. It may be possible to achieve this by having the pv_rtls driver cause xpvd to enumerate devices much earlier. mrj suggested that this could be done by building pv_rtls with -Nxpvd, causing xpvd to load when pv_rtls is loaded. This sounds promising, but has not yet been tested.
dme suggested above an experiment to break at rtls`_init and modload xpvd: that didn't appear to work unfortunately - the guest still had no plumbed interfaces on boot. A workaround for this bug though, is to do exactly what we do for pv guests, by adding "-B network-interface=xnf0,xpv-hcp=dhcp" arguments to the kernel. This would need to be written the menu.lst entry stored in the tftpboot server for that AI service: any hvm guests the user wants to install would need to use this particular service. This would mean needing two AI services if you wanted to install both hvm and baremetal guests, configuring your dhcp macros for clients appropriately so that the right service is set for a given mac address, so it's not a perfect solution, but does allow hands-free installs, once configured on the AI server and doesn't require any code changes, but should be in the release notes. [ I remain unsure how guests were able to install after bug 6850 was fixed - perhaps a timing issue ? ] Does this seem reasonable? I have a hvm guest AI-installing at the moment.
(In reply to comment #15) > [ I remain unsure how guests were able to install after bug 6850 was fixed - > perhaps a timing issue ? ] When 6850 was fixed, I don't think hvm guests worked with AI yet, and hence was never known to have worked with that solution. pv guests were working with AI with that fix, as they still do today.
(In reply to comment #15) > dme suggested above an experiment to break at rtls`_init and modload xpvd: that > didn't appear to work unfortunately - the guest still had no plumbed interfaces > on boot. > > A workaround for this bug though, is to do exactly what we do for pv guests, by > adding "-B network-interface=xnf0,xpv-hcp=dhcp" arguments to the kernel. This > would need to be written the menu.lst entry stored in the tftpboot server for > that AI service: any hvm guests the user wants to install would need to use > this particular service. An alternative here, if you don't want to dedicate an AI service as an "hvm guest only" service, is to do client specific adds for each hvm guest using installadm create-client. It has a -b option to specify boot properties. > > Does this seem reasonable? I have a hvm guest AI-installing at the moment. Sounds like a reasonable workaround to me. So who owns this bug now? From dme's comment it sounds like its more something specific to hvm than something we need to do in the installer microroot environment?
We've decided to publish a release note documenting one of the work-arounds for this particular issue for 2009.06.
(In reply to comment #16) > When 6850 was fixed, I don't think hvm guests worked with AI yet I'm pretty sure they did, which was why I filed this. The pv AI support was to specify the xpv-hcp bootargs via virt-install, the hvm fix was to touch /reconfigure in the install microroot. Either way, it sounds like that alone wasn't enough to really fix the underlying problem for hvm guests, it just worked at the time. (In reply to comment #17) > An alternative here, if you don't want to dedicate an AI service as an > "hvm guest only" service, is to do client specific adds for each hvm > guest using installadm create-client. It has a -b option to specify > boot properties. Yep, whatever option makes the most sense for administrators: my understanding is, with a separate AI service (hence separate dhcp macro), users would need to configure which clients use that macro on the dhcp server, with client-specific 'installadm -b' options, they would do instead on the AI server. Would you guys (the install team) make the call on which is most user-friendly, either solution sounds good to me, I'm leaning towards the installadm route. > Sounds like a reasonable workaround to me. So who owns this bug now? > From dme's comment it sounds like its more something specific to hvm > than something we need to do in the installer microroot environment? I think the xVM team own it. Will file a bug against guest networking to have it do the right thing.