Bugzilla – Bug 4294
du isn't accurate for calculating the size of UFS-based archives
Last modified: 2009-03-20 16:51:47 UTC
You need to log in before you can comment on or make changes to this bug.
Archives for bootroot and pkg.zlib, and the .image file, all rely on du to return an accurate size of the directory tree they will contain/describe. When the directory tree in question has zfs compression turned on, du reports the compressed size. Since the archives require enough space to hold uncompressed files, they are sized too small and DC fails. There are many possible solutions: 1) Unconditionally fail when an encountered ZFS file system is compressed. 2) Perhaps multiply the size of a compressed archive by its compressratio to get a more accurate size. This may require setting things up so that in all cases when a ZFS file system is sized up, it contains only the contents which will be put into the archive. (This way the ratio is not affected by files which won't be part of the archive.)
This bug is not deemed a stopper, as the workaround (not using compressed filesystems) can be documented (and is already the default anyway).
A simpler option is to just force uncompressed for the file systems that DC creates and cares about. It's just a property you set per-dataset.
Changing the synopsis to account for the new problems seen when builds are done on UFS file systems. In the UFS file system case, there appears to be more overhead / metadata associated. For example, reported directory sizes are multiples of 512 bytes, whereas in ZFS the reported directory sizes are much smaller (can be in the single digits if that's all that is needed). So when du sizes things up in a zfs file system, the overall size will be smaller than what is needed to put those same files into a proper ufs file system. This seems to be substantiated by the fact that when the size/padding of the bootroot is made large enough to accommodate the extra overhead, the bootroot archive build completes.
*** Bug 5274 has been marked as a duplicate of this bug. ***
Even with compression turned off, du reports slightly different sizes depending on whether our build area is on ufs or zfs. The following statistics for the bootroot are reported by du while building a build 109 image: du -sk for bootroot in zfs build area: 210mb du -sk for bootroot in ufs build area: 196mb In bootroot_archive.py, we add 10mb padding to the size returned by du -sk to be the size we use for creating the bootroot. While using ZFS as the build area, it works because du -sk for zfs somehow returned a slightly larger size. But the same algorithm for UFS build areas don't work, as reported in bug 5274. du -sk for a build area that's on a compressed ZFS dataset also doesn't work, as reported in this bug, because du reports the actual size of the file stored on disk for ZFS (see http://opensolaris.org/os/community/zfs/faq/#whydusize) Though forcing people to use ZFS dataset without compression can solve the problem with with ZFS not reporting the right size when it is compressed, it doesn't solve the problem with ufs. To make it work consistently in all the cases, I am taking the approach the /boot/solaris/bin/root_archive script used for calculating size of the boot archives. Instead of using du, a function will be introduced in osol_install.install_utils. Given a directory as argument, this function will traverse all the directories/files under it, and add up the sizes for all the directories and files, and return the value in bytes. If the size for files and directories are not in multiple of 1024 bytes, it will be rounded up to the next multiple of 1024 bytes. With the above algorithm, I am able to get a consistent size regardless what I use as the build area. For the bootroot for b109, I get 202mb. If the existing algorithm to add 10mb padding is used, there's no error on the bootroot getting filled up during image construction time. However, when the live cd is booted up, I get an error about not being able to initialize stuff because / is full. So, instead of adding a fixed padding size to the bootroot like we do now, I am going to add 10% of the size returned by the function to take care of the UFS overhead. So, by default, the bootroot calculation will add 10% to the calculated size. There will not be any additional padding. The ability to specify additional padding will be left in DC so if additional padding is needed to be specified for testing/experimenting purposes, it can be used. The size store in the .image_info file will also be calculating by this algorithm instead of "du -sk".
Fixed in changeset: 480:def536ad9850