How to Shorten Software Deployment Time

Disk subsystem is a bottleneck for most servers, except for large data centers with SAN storages based on SSD. Every disk IO operation takes a few milliseconds, while CPU and RAM perform millions of operations during the same period. But there’s a simple way for development teams to deploy large datasets that consist of small files. The following tips will help you save software deployment time significantly, making the whole process dozens of times faster.

Test storage server characteristics

Our test system has three Intel i3 cores and 4GB of RAM memory. It is still stable and works comfortably even if these parameters are reduced to 2GB of RAM and 2 processor cores.

What about our testing environment?

All the test machines are based on Debian7, hypervisor host is based on Windows7. In real environment you can use any hypervisor you like.
Dataset consists of more than 2 million small files and weighs about 5GB with no directory tree based optimisation. Tar packaging without compression takes about 4 hours considering minimal disk usage priority.

The development dataset is based on backups from the Production server. It’s a massive of real data, which consist of daily full database backup, daily files delta backup and weekly full files backup. All the data is packed in tarball on the Production server and uploaded to the storage server.

As soon as data is unpacked, we get a Big Directory. This process lasts about half an hour, and full directory listing is even slower. To copy tarball you need about 10 minutes and about an hour to copy the data unpacked.

In this case, dataset deployment for 4 developers takes 40 minutes on each computer, working at the highest IO loading speed. If the developer damages his data, the whole process starts again. Using this model you spend a lot of precious developers’ time.

The main idea is to make deploy process simple, cheap, and save software deployment process time.

Our deploy scheme is based on Oracle ZFS filesystem. It has a lot of kill features, such as:

• Read-only snapshots wiсh reduce storage space loss,
• Read-write clones,
• Ease of use,
• Failure tolerance, built-in RAID analog from the box. RAIDZ technology, make working without RAID controllers and software RAID subsystems being installed.

You can find tips on how to install this environment on ZFS on Linux homepage.

So, let the deployment begin!

1. Add repository, install ZFS utilities and create the storage

To install ZFS on Debian you can use ZoL repo.

$ su - 
# wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_5_all.deb 
# dpkg -i zfsonlinux_5_all.deb 
# wget http://zfsonlinux.org/4D5843EA.asc -O - | apt-key add - 
# apt-get update 
# apt-get install debian-zfs

You need three virtual hard drives to create RAIDZ array and one virtual hard drive for your OS. Our system is installed on /dev/sdd drive.

root@debian-7-64-storage:/# parted -l 
Model: VBOX HARDDISK (scsi) 
Disk /dev/sda: 51.5GB 
Sector size (logical/physical): 512B/512B 
Partition Table: gpt 
Number Start End Size File system Name Flags 
 1 2097kB 51.5GB 51.5GB zfs primary 
Model: VBOX HARDDISK (scsi) 
Disk /dev/sdb: 51.5GB 
Sector size (logical/physical): 512B/512B 
Partition Table: gpt 
Number Start End Size File system Name Flags 
 1 2097kB 51.5GB 51.5GB zfs primary 
Model: VBOX HARDDISK (scsi) 
Disk /dev/sdc: 51.5GB 
Sector size (logical/physical): 512B/512B 
Partition Table: gpt 
Number Start End Size File system Name Flags 
 1 2097kB 51.5GB 51.5GB zfs primary 
Model: ATA VBOX HARDDISK (scsi) 
Disk /dev/sdd: 51.5GB 
Sector size (logical/physical): 512B/512B 
Partition Table: msdos 
Number Start End Size Type File system Flags 
 1 1049kB 49.4GB 49.4GB primary ext4 boot 
 2 49.4GB 51.5GB 2137MB extended 
 5 49.4GB 51.5GB 2137MB logical linux-swap(v1)

Create the storage space:

root@debian-7-64-storage:~# zpool create main-storage raidz /dev/sda1 /dev/sdb1 /dev/sdc1

root@debian-7-64-storage:~# zpool list 
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT 
main-storage 143G 232K 143G 0% 1.00x ONLINE - 
root@debian-7-64-storage:~# zfs create main-storage/dev-storage -o mountpoint=/dev-storage 
root@debian-7-64-storage:~# zfs list 
NAME USED AVAIL REFER MOUNTPOINT 
main-storage 195K 93.8G 38.6K /main-storage 
main-storage/dev-storage 38.6K 93.8G 38.6K /dev-storage

2. Deploy dataset initial test

Extract the initial copy of your data:

root@debian-7-64-storage:/dev-storage# time tar -xvf ./dataset.tar | wc -l 
1038756 
real 29m27.212s 
user 5m17.373s 
sys 3m25.046s

Once you’re finished, make a few copies:

root@debian-7-64-storage:/# ls -1l /dev-storage/ 
total 14 
... 
drwxr-xr-x 6 root root 6 Mar 27 19:19 stage1 
drwxr-xr-x 6 root root 6 Mar 27 19:19 stage2

At this moment each directory contains a lot of small files – approximately 2 million in total.

3. Deploy developers’ datasets

Create the initial snapshot.

root@debian-7-64-storage:~# zfs snapshot main-storage/dev-storage@initial

root@debian-7-64-storage:~# zfs list -t snapshot 
NAME USED AVAIL REFER MOUNTPOINT 
main-storage/dev-storage@initial 0 - 8.37G -

This snapshot is a read-only copy of your data. To change it, clone the initial snapshot.

root@debian-7-64-storage:~# mkdir -p /snapshots/dev-{1,2} 
root@debian-7-64-storage:~# zfs clone main-storage/dev-storage@initial main-storage/dev-1 -o mountpoint=/snapshots/dev-1 
root@debian-7-64-storage:~# zfs clone main-storage/dev-storage@initial main-storage/dev-2 -o mountpoint=/snapshots/dev-2

root@debian-7-64-storage:~# zfs list 
NAME USED AVAIL REFER MOUNTPOINT 
main-storage 8.37G 85.4G 38.6K /main-storage 
main-storage/dev-1 128K 85.4G 8.37G /snapshots/dev-1 
main-storage/dev-2 1.33K 85.4G 8.37G /snapshots/dev-2 
main-storage/dev-storage 8.37G 85.4G 8.37G /dev-storage

Thereafter, take the clones’ snapshots. This way you make data reset possible in the case of damage.

root@debian-7-64-storage:~# zfs snapshot main-storage/dev-1@initial

root@debian-7-64-storage:~# zfs snapshot main-storage/dev-2@initial

root@debian-7-64-storage:~# zfs list -t snapshot 
NAME USED AVAIL REFER MOUNTPOINT 
main-storage/dev-1@initial 1.33K - 8.37G - 
main-storage/dev-2@initial 0 - 8.37G - 
main-storage/dev-storage@initial 131K - 8.37G -

We couldn’t set quota restrictions for our test storage, but it is possible if you use a setting attribute “quota=<quotasize>” by zfs utility.

4. Deploy data to developers sandboxes

Finally, you have two mounted virtual filesystems, which means you can export the writable copy to developer’s machine.

root@debian-7-64-storage:~# mount 
….. 
main-storage on /main-storage type zfs (rw,relatime,xattr,noacl) 
main-storage/dev-storage on /dev-storage type zfs (rw,relatime,xattr,noacl) 
main-storage/dev-1 on /snapshots/dev-1 type zfs (rw,relatime,xattr,noacl) 
nfsd on /proc/fs/nfsd type nfsd (rw,relatime) 
main-storage/dev-2 on /snapshots/dev-2 type zfs (rw,relatime,xattr,noacl)

We’ve chosen NFS to transfer data between storage and developer’s computer. It is simple, fast enough and requires minimum of the system resources.

root@debian-7-64-storage:~# ip addr show eth0 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 
 link/ether 08:00:27:a3:f7:15 brd ff:ff:ff:ff:ff:ff 
 inet 172.16.0.111/16 brd 172.16.255.255 scope global eth0

root@debian-7-64-dev:~# ip addr show eth0 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 
 link/ether 08:00:27:e9:e5:ba brd ff:ff:ff:ff:ff:ff 
 inet 172.16.0.105/16 brd 172.16.255.255 scope global eth0

You can activate Jumbo Frames to improve the performance:

root@debian-7-64-storage:~# ifconfig eth0 mtu 9000 
root@debian-7-64-dev:~# ifconfig eth0 mtu 9000

root@debian-7-64-storage:~# netstat -i 
Kernel Interface table 
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg 
eth0 9000 0 2367100 0 0 0 2702231 0 0 0 BMRU

Check its work.

root@debian-7-64-storage:~# ping 172.16.0.105 -M do -s 8972 
PING 172.16.0.105 (172.16.0.105) 8972(9000) bytes of data. 
8980 bytes from 172.16.0.105: icmp_req=1 ttl=64 time=0.611 ms 
8980 bytes from 172.16.0.105: icmp_req=2 ttl=64 time=0.413 ms 
8980 bytes from 172.16.0.105: icmp_req=3 ttl=64 time=0.284 ms

Set up the sharing. To simplify the process I haven’t adjusted the additional authentication and access rights subsystems. Log in at the host address.

root@debian-7-64-storage:~# cat /etc/exports 
/snapshots/dev-1 172.16.0.105/32(rw,nohide,insecure,no_subtree_check,async,no_root_squash,fsid=0) 
/snapshots/dev-2 172.16.0.62/32(rw,nohide,insecure,no_subtree_check,async,no_root_squash,fsid=0)

Launch service.

root@debian-7-64-storage:~# service nfs-kernel-server start 
[ ok ] Exporting directories for NFS kernel daemon.... 
[ ok ] Starting NFS kernel daemon: nfsd mountd.

Mount new filesystem to the developer’s machine. I disable metadata files and increase packet size for better performance.

root@debian-7-64-dev:~# mount -t nfs 172.16.0.111:/snapshots/dev-1 /srv/data -o noacl,nocto,noatime,nodiratime,rsize=8192,wsize=8192

root@debian-7-64-dev:~# mount 
172.16.0.111:/snapshots/dev-1 on /srv/data type nfs (rw,noatime,nodiratime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,nocto,noacl,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.111,mountvers=3,mountport=42230,mountproto=udp,local_lock=none,addr=172.16.0.111)

At this stage, the exported directories are available:

root@debian-7-64-dev:/srv/data# ls -1l 
total 14 
drwxr-xr-x 6 root root 6 Mar 27 20:19 stage1 
drwxr-xr-x 6 root root 6 Mar 27 20:19 stage2

Repeat the same actions for the rest of developer’s computers. If you don’t want to use the command line, you can write a simple helper script for developers’ computers that provides all the dataset management functions via ssh.

Following this scheme, you will definitely save loads of time for your team and become a deployment guru.