Linux disk management (2) - Logical Volume Manager

In Linux world, disk as storage device, it is detected by Linux kernel typically as “/dev/sda”, “/dev/hdd”. The tools like “fdisk” is used to create partions, and create filesystem on the partition, then mount the partition to the root file system. The disk space is ready to be used.

There is a serious problem though: if the files stored in the disk keep growing, the space will be all used eventually, disk becomes full. Before LVM, the only solution is to add another disk, and mount to a different mount point in the file system. This could be a problem, if the current running program is not aware of new mounting point.

Logical Volume Manager (LVM) is a device mapper target that provides logical volume management for the Linux kernel. It adds a layer between physical disk and file system, and supports the dynamic growing of LVM.

Linux LVM

As shown in the above diagram, there are three key enties in LVM:

Physical Volume (PV)

Physical Volume can be either hard disks, hard disk partitions, or Logical Unit Numbers (LUNs) of an external storage device. Volume management treats each PV as being composed of a sequence of chunks called physical extents (PEs). It can be cretaed by using command “pvcreate”. “pvdisplay” can be used to show the current PVs.

Volume Group (VG)

Volume Group is a set of same-sized PVs which act similarly to hard disks in a RAID1 array. VGs are usually laid out so that they reside on different disks and/or data buses for maximum redundancy. It can be created by using “vgcreate”, and “vgdisplay” can be used to show the current VGs. Also there is tool like “vgextend” to expand VG.

Logial Volume (LV)

Logical Volume is created from VG, and can be acting raw block devices just like disk partitions: creating mountable file systems on them, or using them as swap storage. It can be created by using “lvcreate”, “lvextend” to expand the size and “lvdisplay” can be used to show the current LVs.

Another way to look the relations:

Linux LVM-relation

Rcently my little server hosting cross reference search, its disk usage is getting close to full. So I added one 1T hard drive. See below steps I took to provision the new disk into service:

First check what are existing VG, LVs.

weng@weng-ubuntu:/scratch$ sudo vgdisplay
[sudo] password for weng:
  --- Volume group ---
  VG Name               weng-ubuntu-vg
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               2.27 TiB
  PE Size               4.00 MiB
  Total PE              596104
  Alloc PE / Size       579961 / 2.21 TiB
  Free  PE / Size       16143 / 63.06 GiB
  VG UUID               tOmV6f-q8jm-jcf8-J9bX-CONX-wUSN-DdRozv

weng@weng-ubuntu:/scratch$ sudo lvdisplay
  --- Logical volume ---
  LV Name                /dev/weng-ubuntu-vg/root
  VG Name                weng-ubuntu-vg
  LV UUID                WfqdYQ-655I-SDGT-eA7r-cXzL-lMvF-K94sP6
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                2.20 TiB
  Current LE             577918
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0

  --- Logical volume ---
  LV Name                /dev/weng-ubuntu-vg/swap_1
  VG Name                weng-ubuntu-vg
  LV UUID                bly5P8-Y4Pj-kQnf-2sB8-SHBF-ikDn-yUUBuC
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                7.98 GiB
  Current LE             2043
    Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:1

weng@weng-ubuntu:/scratch$

Check the new disk /dev/sdb

weng@weng-ubuntu:/scratch$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Create LVM partition on /dev/sdb and create PVs

weng@weng-ubuntu:/scratch$ sudo fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xdd30f6ba.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-1953525167, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1953525167, default 1953525167):
Using default value 1953525167

Command (m for help): p

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdd30f6ba

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  1953525167   976761560   83  Linux

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1 80  Old Minix
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdd30f6ba

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  1953525167   976761560   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
weng@weng-ubuntu:/scratch$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
81 heads, 63 sectors/track, 382818 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xdd30f6ba

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  1953525167   976761560   8e  Linux LVM
eng@weng-ubuntu:/scratch$ sudo pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created
weng@weng-ubuntu:/scratch$ 

Add PVs into VG to Extend VG

weng@weng-ubuntu:/scratch$ sudo vgdisplay
  --- Volume group ---
  VG Name               weng-ubuntu-vg
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               2.27 TiB
  PE Size               4.00 MiB
  Total PE              596104
  Alloc PE / Size       579961 / 2.21 TiB
  Free  PE / Size       16143 / 63.06 GiB
  VG UUID               tOmV6f-q8jm-jcf8-J9bX-CONX-wUSN-DdRozv
  
weng@weng-ubuntu:/scratch$ sudo vgextend weng-ubuntu-vg /dev/sdb1
  Volume group "weng-ubuntu-vg" successfully extended
weng@weng-ubuntu:/scratch$ sudo vgextend weng-ubuntu-vg /dev/sdb1

weng@weng-ubuntu:/scratch$ sudo vgdisplay
  --- Volume group ---
  VG Name               weng-ubuntu-vg
  System ID
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               3.18 TiB
  PE Size               4.00 MiB
  Total PE              834571
  Alloc PE / Size       579961 / 2.21 TiB
  Free  PE / Size       254610 / 994.57 GiB
  VG UUID               tOmV6f-q8jm-jcf8-J9bX-CONX-wUSN-DdRozv

weng@weng-ubuntu:/scratch$ 

Extend LVM

weng@weng-ubuntu:/scratch$ sudo lvextend -L+950G /dev/weng-ubuntu-vg/root
  Extending logical volume root to 3.16 TiB
  Logical volume root successfully resized^M
weng@weng-ubuntu:/scratch$ sudo lvextend -L+45G /dev/weng-ubuntu-vg/root
  Extending logical volume root to 3.17 TiB
  Logical volume root successfully resized
weng@weng-ubuntu:/scratch$ sudo lvdisplay
  --- Logical volume ---
  LV Name                /dev/weng-ubuntu-vg/root
  VG Name                weng-ubuntu-vg
  LV UUID                WfqdYQ-655I-SDGT-eA7r-cXzL-lMvF-K94sP6
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                3.17 TiB
  Current LE             831358
  Segments               4
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0

  --- Logical volume ---
  LV Name                /dev/weng-ubuntu-vg/swap_1
  VG Name                weng-ubuntu-vg
  LV UUID                bly5P8-Y4Pj-kQnf-2sB8-SHBF-ikDn-yUUBuC
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                7.98 GiB
  Current LE             2043
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:1


weng@weng-ubuntu:/scratch$  df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/weng--ubuntu--vg-root  2.2T  2.0T  131G  94% /
udev                               3.9G  4.0K  3.9G   1% /dev
tmpfs                              1.6G  764K  1.6G   1% /run
none                               5.0M     0  5.0M   0% /run/lock
none                               3.9G     0  3.9G   0% /run/shm
cgroup                             3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                          228M   28M  189M  13% /boot
weng@weng-ubuntu:/scratch$ 

Resize LVM

weng@weng-ubuntu:/scratch$ sudo resize2fs /dev/weng-ubuntu-vg/root
resize2fs 1.42 (29-Nov-2011)
Filesystem at /dev/weng-ubuntu-vg/root is mounted on /; on-line resizing required
old_desc_blocks = 142, new_desc_blocks = 203
The filesystem on /dev/weng-ubuntu-vg/root is now 851310592 blocks long.

weng@weng-ubuntu:/scratch$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/weng--ubuntu--vg-root  3.2T  2.0T  1.1T  66% /
udev                               3.9G  4.0K  3.9G   1% /dev
tmpfs                              1.6G  764K  1.6G   1% /run
none                               5.0M     0  5.0M   0% /run/lock
none                               3.9G     0  3.9G   0% /run/shm
cgroup                             3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                          228M   28M  189M  13% /boot

weng@weng-ubuntu:/scratch$