Virtual Infrastructures Which is Faster NFS or iSCSI
Part 1. The setup
In my current position I am responsible for the support and management of all aspects of my employer's IT infrastructure. This ranges from firewalls and network switches to servers and printers to phones and desktops. While my employer is a not-for-profit agency we have been able to implement an enterprise-class infrastructure thanks in part to several grants and donations we have received. It is my hope to make these enterprise-class methods & best-practices available to small-businesses and educational institutions at a fraction of the cost through the use of user-friendly open-source software.
At the core of these best-practices is the use of virtualization technology which has numerous benefits such as:
· consolidate under-utilized physical servers to reduce hardware, electricity, and cooling cost
· isolate applications to individual “virtual machines” to prevent incompatibility issues
· run multiple operating systems utilizing a single hardware platform
If you have only one virtual machine host server then local storage is sufficient however if you have several servers working in conjunction with each other then you need some sort of centralized storage to utilize those servers to their fullest potential. Centralized storage usually consists of a SAN or NAS that makes disk space available to the servers using the NFS, iSCSI, or Fiber-Channel protocols.
I have not yet had the opportunity to work with Fiber-Channel due to the costs associated but for the past few years I have having been using the NFS & iSCSI protocols in virtual infrastructures. There are numerous discussions on the Internet stating that one is better than the other but few have gone into the details as to why.
Recently I have been involved in several discussions with some of my peers regarding the different filesystems (EXT3, EXT4, UFS, ZFS) available in today's mainstream Linux distributions. Each has its own set of requirements, benefits, and drawbacks. As a result of these discussions I decided that I wanted to see for myself which protocol and filesystem provided the best centralized storage performance for virtualization host servers.
For hardware, my test environment consists of a Supermicro Xeon 3450 based server with 16GB ram and a 500GB hard drive connected via cross-over cable to an HP Proliant N40L Microserver with 8GB ram, a 250Gb hard-drive, and two 1.5TB Western Digital Black hard-drives. For software, the Supermicro server is running VMWare vSphere 5.0 and the HP server will serve as the centralized storage running FreeNAS, CentOS, Ubuntu, & OpenFiler. The servers are connected via dedicated network cards and a cross-over cable to eliminate the possibility of a network switch's backplane bandwidth affecting performance.
After a bit of research I came up with four procedures/applications that I wanted to use that would consistently test each of the protocols and filesystems on the different distributions. The applications are installed in a Windows XP virtual machine hosted on the Supermicro's local datastore but ran on a secondary thin-provisioned 80GB virtual hard-disk that is hosted on the HP's datastore.
1. Use the vSphere client to copy a 3.05GB ISO file from the Supermicro's local datastore to the HP's datastore (and vice-versa) and time how long it takes.
2. Run ATTO Disk Benchmark v2.47 inside the virtual machine with all of the default settings. This program transfers data ranging from .5KB to 8MB both to (write) and from (read) the secondary virtual disk and measures the transfer rate.
3. Run Intel NAS Performance Test v1.7.1 inside the virtual machine with all of the default settings. This program is capable of running around 12 different benchmark tests on the secondary virtual disk.
· HD Video Playback, 2x HD Playback, 4x HD Playback
· HD Video Record
· HD Playback & Record
· Content Creation
· Office Productivity
· File copy to/from NAS
· Dir copy to/from NAS
· Photo Album
4. Run Iometer v inside the virtual machine using the procedure and config file provided on http://technodrone.blogspot.com/2010/06/benchmarking-your-disk-io.html. Iometer creates a 50GB file on the secondary virtual disk and the following tests are ran for 5 minutes each with a 2 minute lead up in an attempt to produce more “real world” scenarios.
· 4K; 100% Read; 0% random (Regular NTFS Workload 1)
· 4K; 75% Read; 0% random (Regular NTFS Workload 2)
· 4K; 50% Read; 0% random (Regular NTFS Workload 3)
· 4K; 25% Read; 0% random (Regular NTFS Workload 4)
· 4K; 0% Read; 0% random (Regular NTFS Workload 5)
Ideally each of the above tests would be ran a minimum of three times and the results averaged to get the final results for each test for each OS, filesystem, & protocol. However doing so would have taken way longer than I can spare for this project so each test was ran just one time.
I'm not going to go into considerable detail about each of the operating systems I'm testing as each could be a full article in itself. Instead I'm going to just cover the basic setup of each and focus more on the results.
Part 2 – FreeNAS 8
FreeNAS 8.0.4 x64 ZFS NFS
The first operating system tested was FreeNAS 8.0.4 x64. I installed the operating system to the 250GB hard-drive and created a ZFS mirror of the two 1.5TB hard-drives. A secondary subnet IP address is assigned to a dedicated NIC that can be accessed from the storage NIC on the ESXi host and a NFS share is created on the ZFS mirror. The NFS share is mounted inside of ESXi and the first set of tests are began.
1. Copying the 3.05GB ISO file to the NFS datastore took 16m 35s for a throughput of 3.04 MB/s. Copying the ISO from the NFS datastore took 2m 16s for a throughput of 22.71 MB/s.
2. Screen capture of the ATTO benchmark results:
3. Screen capture of the Intel NASPT benchmark results:
4. Summary of the Iometer results after taking approximately 4 hours to create the 50GB test file:
It would appear that the overall results show good read performance but slow write performance for FreeNAS 8 running NFS via a ZFS mirror.
FreeNAS 8.0.4 x64 ZFS iSCSI
Next the NFS share is removed from ESXi and FreeNAS and replaced with an iSCSI extent. Unfortunately FreeNAS 8 doesn't support device extents on a ZFS mirror so a file extent had to be used. A file extent is simply a single file that is shared and looks like a single disk to the ESXi host. This is not recommended in production environments because if something happens to that file all of the virtual machines could be lost. All other settings remain the same as the first test.
1. Copying the 3.05GB ISO file to the iSCSI datastore took 49s for a throughput of 61.63 MB/s. Copying the ISO from the NFS datastore took 2m 10s for a throughput of 23.23 MB/s.
2. Screen capture of the ATTO benchmark results:
3. Screen capture of the Intel NASPT benchmark results:
4. Summary of the Iometer results after taking approximately 30 min to create the 50GB test file:
It would appear that the overall results show a slight drop in read performance but considerable improvement in write performance for FreeNAS 8 running iSCSI via a ZFS mirror.
FreeNAS 8.0.4 x64 UFS NFS
Next the iSCSI file extent and the ZFS mirror are deleted and replaced with a UFS mirror and NFS share.
1. Copying the 3.05GB ISO file to the NFS datastore took 58m 28s for a throughput of .86 MB/s. Copying the ISO from the NFS datastore took 3m 4s for a throughput of 16.41 MB/s.
2. Screen capture of the ATTO benchmark results:
3. Screen capture of the Intel NASPT benchmark results:
4. Summary of the Iometer results after taking over 24 hours and crashing FreeNAS to create only 30GB of the 50GB test file:
FreeNAS 8.0.4 x64 UFS iSCSI
Next the NFS share is removed from ESXi and FreeNAS and replaced with an iSCSI extent. Again, since FreeNAS 8 doesn't support device extents on a ZFS mirror a file extent had to be used.
1. Copying the 3.05GB ISO file to the iSCSI datastore took 1m 26s for a throughput of 35.12 MB/s. Copying the ISO from the NFS datastore took 2m 14s for a throughput of 22.54 MB/s.
2. Screen capture of the ATTO benchmark results:
3. Screen capture of the Intel NASPT benchmark results:
4. Summary of the Iometer results after taking approximately 30 min to create the 50GB test file:
FreeNAS 8 Summary of results
In order to get a better look at each of the test results I put them all into LibreOffice Calc so that I could see each test side by side and bolden/underline the top performer of each:
1. Summary of the copy ISO test for each filesystem and protocol:
2. Summary of the ATTO test for each filesystem and protocol:
3. Summary of the Intel NASPT test for each filesystem and protocol:
4. Summary of the Iometer test for each filesystem and protocol:
Based upon each of these summaries it would appear that ZFS is definitely the winner between the two filesysems. It is interesting to note that if the need for a datastore with faster reads is greater then NFS is the way to go but if faster writes are needed then iSCSI is the way to go. However in a production environment I can not recommend the use of iSCSI in FreeNAS unless a hardware raid solution is used for the ZFS drivespace so that file extents are not used.
[Tutorial] All your favorite downloaders on FreeNAS 8 (amd64, ZFS)
What's up people? Long time no see. Yeah I know. Have been running FreeNAS 7 for over a year, everything very stable. Few weeks ago I thought it would be time to finally take the step and upgrade to FreeNAS 8. So I bought the hardware, set everything up and started searching the interwebs on how to get my favorite downloaders working on the new FreeNAS 8. Few weeks later I got (most of it) running.
The bad news: auto startup and shutdown. Can't seem to get them working. So for now, you have to use a startup script, every time you boot your server.
It's kinda like tethered booting your jailbroken iPhone 4.
Anyways. If you're like me and just want to get everything running and don't reboot your NAS every hour, grab a beer and let's go!
UPDATE 7 DECEMBER 2011
- Modified startup.sh and shutdown.sh a bit. and made them available for download only. Copy/pasting would truncate the code.
[NOTES]
- This tutorial is a simplified version of the tutorial on the FreeNAS forums created by "asforme", which again is a modified version of "delicatepc"'s script. Many thanks to these guys.
- amd64 means support for 64-bit processors. Intel CPU's with 64-bit are supported!
- Like the previous tutorials, we're going to use "Zeta" as our disk mount. This is a ZFS mount. Of course create and/or use your own mount every time this name is mentioned.
- Obviously, change the IP address in the tutorial to your own IP address.
- Turn on SSH by going to your FreeNAS webinterface -> services -> SSH. Make sure "login as Root with password is checked".
- Copy and paste commands which are more than 1 row long, into a text editor first to make sure everything gets executed as it should.
- Create a share (AFP, CIFS, whatever..) to access your disk mount, we're gonna need it later on.
1. Open a SSH connection to your FreeNAS box using Terminal (OSX) or Putty (Windows). In OSX this is done by opening Terminal and typing:
ssh -l root 192.168.1.250
(login with your admin password)
2. We're going to create a directory to put our downloaders in.
mkdir /mnt/Zeta/downloaders
And move into that directory:
cd /mnt/Zeta/downloaders
3. Next we need to download all needed packages to get our downloaders working.
Yes, this is one big command. If you watch closely, you'll see the packages get installed.
If you're running the i386 build of FreeNAS 8, you could try this tutorial by changing "amd64" to "i386" in the above command.
If you get "broken pipe" or "file not found" errors, check if your network configuration is set in FreeNAS. If the error still occors, it could be that paths have changed at the FreeBSD server. Try to find out what the correct paths are by FTP'ing to ftp.freebsd.org/pub/FreeBSD/....
4. Extract the downloaded packages using the foreach command. That way, we don't have to use the extract command for each one of the downloaded packages. Way faster.
Besides that, we're extracting only the bin and lib folders so no unneeded files like README's get copied.
foreach fn ( *.tbz )
tar -xjf $fn bin lib
end
If you get any error's here, don't worry, that's normal.
5. Cheetah seems to be weirdly packaged so we need to do some extra stuff here.
At the moment of writing, 0.6.10 is the latest SABnzbd version. If you're reading this tutorial at a later time, make sure you grab the latest package. Just change "0.6.10" to the newest version number.
Next, unpack SABnzbd.
tar -xvf SABnzbd-0.6.10-src.tar.gz
Move it to a new folder called "SABnzbd".
mv SABnzbd-0.6.10 SABnzbd
And remove the compressed package:
rm *.gz
Create a new directory for the SABnzbd config file:
mkdir sabconfig
7. Next up is CouchPotato, our Movie watchman.
Fetch the latest version of CouchPotato:
9. Next up, a fairly new downloader: Headphones. Does the same thing as Sick Beard and CouchPotato, only for music. Try it!
Fetch the latest version of Headphones:
Next up, we're going to create the startup.sh file. We're gonna use this file to fire up our downloaders and later on (in an updated tutorial) set this file as a startup when we run our FreeNAS server.
There were some problems when the startup.sh file was created using nano. Thing was; users seemed to just copy the rows of text into nano, which leaded to truncated text. So if you want to do it right, I advise to mount your share (Zeta) in Windows/OSX/Linux/whatever and create the startup.sh file in your /mnt/Zeta/downloaders/ directory from there.
10. Stay in your ssh session and set permissions for your downloaders folder.
Go one folder down:
cd..
And change permissions:
chmod 777 downloaders
Now you can freely add files to your downloaders directory. Create a file named "startup.sh".
Download the file from here.
11. Launch your startup script by typing the following:
bash /mnt/Zeta/downloaders/startup.sh
That seems to be it guys. If someone manages to get the startup working, please let me know.
I'm working on a shutdown script. this is the script for now and should be working for the downloaders. Run this script when you're planning to reboot your NAS.
