Emulating large disks in Linux with VDO

Some time ago. I needed to write a script that would partition and format disks for a big data project. If the disk size was large enough, I needed to use a GUID Partition Table (GPT) partition, otherwise use a Master Boot Record (MBR) partition. The problem was that I did not have access to systems with large disks to test this script.

I remembered that Red Hat Enterprise Linux (RHEL) 7.5 was shipping with a new feature called the Virtual Disk Optimizer (VDO), which enabled on-disk compression and deduplication. I decided to see if this new feature would help my situation. Could I take a small disk, say 10GB, and make it look like a 10TB disk to the OS?

Obviously, no compression technology would allow me actually store 10TB on a 10GB disk, but in my case, I just needed to logically emulate a 10TB disk. Would this work?

Let’s see.

Example

In this example, I use Red Hat Enterprise Linux 8.1 under VirtualBox to test, but this procedure should work with any version of Linux that includes VDO. To accomplish this, I:

1. Installed the vdo and kmod-vdo packages:

[root@defiant ~]# yum install vdo kmod-kvdo

2. Created a 10GB disk and attached it to my VM:

[root@defiant ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda             8:0    0   32G  0 disk 
├─sda1          8:1    0    1G  0 part /boot
└─sda2          8:2    0   31G  0 part 
  ├─rhel-root 253:0    0 27.8G  0 lvm  /
  └─rhel-swap 253:1    0  3.2G  0 lvm  [SWAP]
sdb             8:16   0   10G  0 disk 
sr0            11:0    1 1024M  0 rom  

3. Created the VDO volume:

[root@defiant ~]# vdo create --name=vdo-disk --device=/dev/sdb --vdoLogicalSize=10T
Creating VDO vdo-disk
Starting VDO vdo-disk
Starting compression on VDO vdo-disk
VDO instance 0 volume is ready at /dev/mapper/vdo-disk

4. Created a GPT partition:

[root@defiant ~]# parted /dev/mapper/vdo-disk mklabel gpt mkpart p1 xfs 1MB 10TB
Information: You may need to update /etc/fstab.

5. Displayed the results:

[root@defiant ~]# parted /dev/mapper/vdo-disk print
Model: Linux device-mapper (vdo) (dm)
Disk /dev/mapper/vdo-disk: 11.0TB
Sector size (logical/physical): 4096B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End      Size     File system  Name  Flags
 1      1049kB  10000GB  10000GB               p1

6. Created a filesystem and mounted it:

[root@defiant ~]# mkdir /data

7. Formatted the disk (this part will probably take a long time):

[root@defiant ~]# mkfs.xfs /dev/mapper/vdo-disk1 
meta-data=/dev/mapper/vdo-disk1  isize=512    agcount=10, agsize=268435455 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=2441405952, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

8. Mounted the disk:

[root@defiant ~]# mount /dev/mapper/vdo-disk1 /data
[root@defiant ~]# df -h
Filesystem             Size  Used Avail Use% Mounted on
devtmpfs               395M     0  395M   0% /dev
tmpfs                  411M     0  411M   0% /dev/shm
tmpfs                  411M  5.7M  406M   2% /run
tmpfs                  411M     0  411M   0% /sys/fs/cgroup
/dev/mapper/rhel-root   28G  1.8G   27G   7% /
/dev/sda1             1014M  291M  724M  29% /boot
tmpfs                   83M     0   83M   0% /run/user/1000
/dev/mapper/vdo-disk1  9.1T   65G  9.1T   1% /data

If you followed along, you now have emulated a 10TB disk. Note that, with overhead, it’s only showing 9.1TB, but you can experiment with sizes to get the size you need. Obviously you can’t actually store large amounts of data on this disk, but for the purpose of logically emulating a 10TB disk, this setup has been useful to me.

Want to try out Red Hat Enterprise Linux? Download it now for free.