Author: Kai

Compressing VMDK and VHDX disk images with qemu-img convert

/ Kai / Leave a comment

This is actually a kind of anniversary post. 10 years ago I wrote a howto about compressing Virtualbox VDI disk images by zeroing the unused space. Today, although the hypervisor, the OS, even my whole build environment has changed, I’m still using the same technique to keep virtual disk images as small as possible.

This time I’m reducing the size of VMDK disk images during a CI/CD pipeline build using the qemu-img tool.

The basic idea is the same as 10 years ago: create and setup the disk image; as soon as you are finished delete unnecessary files and zero-fill the free space. Then use ‘qemu-img’ to either compact the disk image (if this is supported) or create a copy of it minus the free space.

This works because most hypervisors support dynamically allocated virtual disk image formats. They grow every time a disk sector is written for the first time until the virtual disk reaches its maximum defined capacity. A big advantage because it allows the efficient storage of rather empty virtual image files (eg. right after their initial setup).

But their size is not automatically reduced when files are deleted. This is where the ‘zeroing the empty space’ step kicks in… by zeroing these sectors, they can be marked as ’empty’ in the virtual disk image file. Next time when ‘compacting’ the image or when creating a copy of it, they will not be copied over to the new image.

In case of a freshly installed Linux system you can fill up the empty space by creating a temporary file on the root partition, fill it with zeros until all remaining disk space is used up, then delete it and shut down the system.

dd if=/dev/zero of=/target/temp.img bs=4M; sync; rm /target/temp.img

I’m assuming here that the target partition on the virtual disk you want to compress is mounted as /target. The first part (disk dump) will write zeros to a temporary file and use up all disk space available on the partition. After synchronizing the file system (just to be safe everything was written), the temporary file is deleted again.

I’m then using qemu-img to create a copy of the disk image. Here I’m converting a VMDK image again into a VMDK:

qemu-img convert -p -f vmdk -O vmdk /path/to/src.vmdk /path/to/dest.vmdk

The option -p stands for an optional progress meter, -f for the source format, and -O for the destination format. The last two parameters are the input and output filenames.

If you prefer qcow2 formatted disk images you can make use of the compress option to even further reduce the image size. The source format here is again VMDK; the option -c stands for compress:

qemu-img convert -c -p -f vmdk -O qcow2 /path/to/src_image.vmdk /path/to/dest_image.qcow2

So in the end, by simply creating a copy and thereby getting rid of empty space, the resulting image can be reduced to only a bit more than the overall size of the files in the image.

Replacing the WiFi Card on an Asus ROG Strix B450-I Motherboard

/ Kai / Leave a comment

I’ve recently got so fed up with the Linux driver support of the Realtek RTL8822BE network adapter on my motherboard (especially its poor Bluetooth support) that I seriously considered a hardware fix. In the end I’ve replaced it with a much better supported card, an Intel® Wi-Fi 6 AX200.

The Realtek linux driver in Ubuntu Desktop 20.04 up to 21.10 is crappy, to say the least – I’ve constantly had WiFi connection issues and connecting to my stereo via Bluetooth was not even working out of the box. Only after manually extracting the firmware from the Windows driver, copying it to the right location, and adapting a few settings, it worked. Poorly.

Network controller: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
Subsystem: ASUSTeK Computer Inc. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
Flags: bus master, fast devsel, latency 0, IRQ 74, IOMMU group 15
I/O ports at c000 [size=256]
Memory at fc500000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
Kernel driver in use: rtw_8822be
Kernel modules: rtw88_8822be

Having already dealt with these issues before on my Dell notebook by, surprise, replacing the Realtek card with an Intel AX200 about a year ago, I was eager to try the same on my Mini-ITX motherboard.

Asus Wi-Fi GO! Card Socket

I haven’t found much information about the Asus Wi-Fi GO! card on the board. But I’ve seen quite a few of them on eBay and some were shown in a disassembled state. So I was quite confident that the RF shield could be opened and inside would be a standard PCIe M.2 WiFi card that could be replaced.

The RF shield is held on the board by two screws (from the bottom side) and it can be pulled off the PCIe connector after removing these screws. Another screw holds the two halves of the shield together.

Opened Wi-Fi GO! Card Shield
Removed Realtek RTL8822BE Card

Replacing the PCIe WiFi card should then be pretty straight forward. In my case I’ve bought an Intel AX200 card for under 20,- € (shipping included) as replacement. It should be possible to replace the PCIe card without completely disassembling the the two SMA connectors; I’ve done it for convenience. Also there’s a black rubber block stuck onto the card that you might want to reuse (peel it off carefully).

The reassembly with a new card is pretty much doing the same steps in reverse order, so I’m not describing or showing it here. In the end, it probably took me longer to get the motherboard out of the Mini-ITX case and back in again than disassembling the Wi-Fi card; less than half an hour for the actual card disassembly and replacement.

Backing up and restoring a bootable USB Stick

/ Kai / Leave a comment

I recently needed to create a backup of a bootable USB stick. My usual approach is to simply use dd (disk dump) on a Linux system to create a full copy of the stick into an image file. This time I wanted to play around a bit and see if I could store only the relevant data, to restore the stick without storing empty sectors or – even worse – remnants of old, potentially sensitive data from previous uses.

The USB stick in this case was a 8 GByte Sandisk Cruzer Blade with a bootable FAT32 partition on it:

# fdisk -l /dev/sda
Disk /dev/sda: 7.45 GiB, 8004304896 bytes, 15633408 sectors
Disk model: Cruzer Blade    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x01234567

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1  *     2048 15632383 15630336  7.5G  c W95 FAT32 (LBA)

Step 1: Backing up the Partition Layout

I started with creating a backup of the partition layout. sfdisk comes handy here as it is easily scriptable (not required in this case). You can pipe its output into a text file and use the same file to re-create the partition layout. 

# sfdisk -d /dev/sda 
label: dos
label-id: 0x01234567
device: /dev/sda
unit: sectors
sector-size: 512

/dev/sda1 : start=        2048, size=    15630336, type=c, bootable


# sfdisk -d /dev/sda > partition-layout-sda.sfdisk

Step 2: Copying the Boot Sector and Hidden Data

You probably won’t need this step if your USB stick is not bootable. In my case it was a recovery stick so it contained a bootable MBR FAT32 partition. Backing up the boot sector is easy: create a disk dump of the first sector (512 bytes). This sector contains the master boot record (the first 446 bytes), partition table entry points (64 bytes), and two magic bytes (0x55, 0xAA) at the end.

# dd if=/dev/sda of=bootsect-sda.img bs=512 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.00350891 s, 146 kB/s

The Master Boot Record only consists of the first 446 bytes. I’ve taken a shortcut here and stored the whole first sector. When restoring the MBR I will only restore the first 446 bytes of it.

Some disk images can contain additional (secret) data in the gap between the boot sector and the first sector of the first partition. The GRUB2 boot loader for example stores additional data in this area. I haven’t created a backup of these sectors in my test; but if you would need to, the command would look something like this…
(In my case the first partition starts at sector 2048, so there could be about 1 MByte of additional hidden data.) 

# dd if=/dev/sda of=bootsect-and-additional-sectors.img bs=512 count=2048
2048+0 records in
2048+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616555 s, 17.0 MB/s

Step 3: Creating a backup of the files on the stick

So now we’re getting to the interesting part: creating a copy of the files on the USB stick. And only of the files. A really handy tool for this is partclone. It supports various partition formats and only stores the files without the ‘free’ space.

This can be slightly improved by adding a layer of compression. In my case most of the contents was already compressed, so this step wasn’t a big improvement (only 0.1 GBytes or about 4%):

# partclone.fat32 -c -s /dev/sda1 | gzip -c -9 > sda1.img.gz
Partclone v0.3.13 http://partclone.org
Starting to clone device (/dev/sda1) to image (-)
Reading Super Block
Calculating bitmap... Please wait... 
Elapsed: 00:00:01, Remaining: 00:00:00, Completed: 100.00%                      
Total Time: 00:00:01, 100.00% completed!
done!
File system:  FAT32
Device size:    8,0 GB = 15630336 Blocks
Space in use:   4,6 GB = 8976976 Blocks
Free Space:     3,4 GB = 6653360 Blocks
Block size:   512 Byte
Elapsed: 00:06:10, Remaining: 00:00:00, Completed: 100.00%, Rate: 745,33MB/min, 
current block:   15630336, total block:   15630336, Complete: 100.00%           
Total Time: 00:06:10, Ave. Rate:  745,3MB/min, 100.00% completed!
Syncing... OK!
Partclone successfully cloned the device (/dev/sda1) to the image (-)
Cloned successfully.

Schrödinger’s Backup, or: Restoring the USB Stick

Well, it’s always nice to have a backup; but you need to verify that it can be restored.

The condition of any backup is unknown until a restore is attempted.

Schrödinger’s Backup

My approach was to take another USB stick, erase its contents, and then try to restore the files and images I had created earlier. I probably could have combined these steps into a shell script, but as this was a one-off I simply typed them in.

The first step is to erase the old partition layout – if there is one as in my case.

# fdisk -l /dev/sda
Disk /dev/sda: 7.45 GiB, 8004304896 bytes, 15633408 sectors
Disk model: Cruzer Blade    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xfedcba98

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1        2048 15632383 15630336  7.5G  b W95 FAT32


# dd if=/dev/zero of=/dev/sda bs=1M count=1; sync
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114768 s, 9.1 MB/s

Next I’ve recreated the partition layout by piping the ‘partition-layout-sda’ file back into sfdisk.

# sfdisk /dev/sda < partition-layout-sda.sfdisk 
Checking that no-one is using this disk right now ... OK

Disk /dev/sda: 7.45 GiB, 8004304896 bytes, 15633408 sectors
Disk model: Cruzer Blade    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x01234567.
/dev/sda1: Created a new partition 1 of type 'W95 FAT32 (LBA)' and of size 7.5 GiB.
Partition #1 contains a vfat signature.
/dev/sda2: Done.

New situation:
Disklabel type: dos
Disk identifier: 0x01234567

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1  *     2048 15632383 15630336  7.5G  c W95 FAT32 (LBA)

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

After that I recreated the MBR (the first 446 bytes of the boot sector).

# dd if=bootsect-sda.img of=/dev/sda bs=446 count=1
1+0 records in
1+0 records out
446 bytes copied, 0.00312617 s, 143 kB/s

The last step was to restore the files with partclone.

# gzip -c -d sda1.img.gz | partclone.fat32 -r -o /dev/sda1 
Partclone v0.3.13 http://partclone.org
Starting to restore image (-) to device (/dev/sda1)
Calculating bitmap... Please wait...
done!
File system:  FAT32
Device size:    8.0 GB = 15630336 Blocks
Space in use:   4.6 GB = 8976976 Blocks
Free Space:     3.4 GB = 6653360 Blocks
Block size:   512 Byte
Elapsed: 00:03:20, Remaining: 00:00:00, Completed: 100.00%, Rate:   1.38GB/min, 
current block:   15630336, total block:   15630336, Complete: 100.00%           
Total Time: 00:03:20, Ave. Rate:    1.4GB/min, 100.00% completed!
Syncing... OK!
Partclone successfully restored the image (-) to the device (/dev/sda1)
Cloned successfully.

It’s alive

The new stick is bootable. Checksum comparisons during boot did not report any errors. And I was able to significantly reduce the stored data. (I probably would have saved even more on larger USB sticks.)

So in the end, was it worth the time to go through all these steps? Maybe. It definitely was fun to do.

I will continue to simply create disk dumps via dd. But it is good to know that you can reduce the size of such a bootable USB stick if you, for example, need to distribute personalized images to customers and ‘every byte counts’. For everyday use cases I would recommend Clonezilla though. It not only covers all the steps from above, but is also far better field-tested and adapted to various different disk and partition layouts.

Serious Error in the Sharp PC-1500 Technical Reference Manual

/ Kai / 4 Comments

There’s an error in all of the the Sharp PC-1500 Technical Reference Manual versions I have come across so far: in the pinout of the 60-pin connector a pin is documented as being VBAT instead of F-GND. Your Sharp PC-1500 can be ‘killed’ if you short VBAT (Battery Voltage) to F-GND (Frame Ground).

If you look at the traces (here: taken from the Radio Shack PC2 Technical Reference Manual) you can see that pin 44 is connected to F-GND instead of VBAT. It is easy to verify this with a multimeter in continuity mode.

I have to admit that I also came across this ‘by accident’… so keep in mind to verify the documentation whenever possible. The corrected version of the 60-pin connector pinout should look like this:

LED Grow Light Station for Chilis and Tomatos

/ Kai / 4 Comments

Days are getting longer, and in the afternoons the temperature rises above 10°C… but if I’ve learned a lesson the last two years then that growing chilis and tomatos in our living room (although located on the south side of our house) is still far from enough for these seedlings…

So I’ve taken a few hours and built an LED grow light station for them. 😀

Two LED panels (bought on Amazon for ~ 23 € each, hanging above two tubs with seedlings.
These photos show the seedlings two weeks after sowing the tomato and chili seeds.

3D Printed Raspberry Pi Rack Mount with Heat-sink (Passive Cooling)

/ Kai / 3 Comments

With now 6 Raspberry Pis lying (hanging) around inside and around my server rack I’ve started to look for a nice and clean way to mount them in my server rack. On thingiverse I’ve seen a few 3D printed Raspberry Pi brackets and I immediately liked the idea; but as I wanted a simple design with the ability to add a heat sink, I had to start from scratch. Also, at the time I started the design, there weren’t any brackets for the newest Raspberry Pi 4. So, the result of my effort can be seen here:

After a about half a dozen iterations over the last two months, the following 3D print is ready to be equipped with heat sinks and mounted in my server rack:

I’ve prepared two versions of the RasPi Rack holder, one for the the older Raspberry Pi 3 Model and one for the Rasperry Pi 4. The STL files for 3D printing are available for download and licensed under CC-BY-SA 4.0: RasPi_RackMount.zip

The heat sink was bought on ebay (link) and is the upper half of a passive cooled aluminium Raspberry Pi case.

The 2 HE rack panel was bought at Musicstore.de, but I’ve also seen them on ebay and Amazon. They also sell small panels to cover 1 or 2 empty slots, if necessary. The 2 HE panel can take up to 10 modules (i.e. Raspberry Pis).

The RasPi’s are currently all powered from the back side, each with their own power supply. I’m currently planning on inserting a 5x USB charger into one of the module slots to power up to five at the same time.