Using NFS for Networked Backups
by Glenn Graham03/06/2003
Arguments abound between system administrators as to the correct way to back up a network of Unix hosts. Some argue that tapes are the answer, while others lean toward more modern means, such as rewritable CD-ROMs. No matter the method, the end goal remains: to back up hosts over a network, in a manner that is conveniently indexed and easily retrieved.
During the late 80s, I decided to cut costs by using a spare server running NFS along with a single 500MB tape drive loaded with ¼" DC6525 tape cartridges. The idea was simple and performed well. I dedicated three tapes per host and performed a revolving backup of each host three times a week. The only problem was that the media was cumbersome and, in many cases, unreliable. Gone are the days of DC6525s; they've been replaced by more expensive, complex devices.
Over the past year, I've been revamping my network of 24 hosts. Before I began, I considered writable CD-ROMs and other commercial products. After comparing the cost of commercial products to mass storage drives, I concluded it would be more economical to construct a central NFS server running Linux. Based on my original model, I replaced the tape drive with two hot-swappable SCSI drives running RAID. This allows for real-time backup, removable data, and future expansion.
Throughout this article I will discuss some very basic methods used for NFS. Most are common knowledge, but are frequently put aside as being too complicated, antiquated, or difficult to implement. We will touch only the tip of the iceberg. These examples are only common guidelines, as NFS is infinitely customizable.
|
Related Reading 
Essential System Administration |
Generally speaking, performing any type of backup requires some thought,
planning, and in-depth knowledge of the files, peripherals, and role of each
host. Within a network, no two backups are alike. The core backup consists of
configuration files found under /etc, various cron files, and system-specific binaries. In the case of a server, we would back up
/var/mail, web directories, or /var/db, depending on
the server's role.
The first step in planning your backup is to carefully examine the target system(s) by mapping system-dependent files and including them in your scripts.
NFS Basics
Before we begin, let's introduce some basic NFS concepts. NFS is supported
on UDP and was first introduced by Sun Microsystems in March 1989 under RFC1094. Many
consider NFS insecure and vulnerable to a variety of exploits, especially the
RPC portmapper. Despite its insecurities, NFS is still an accepted method of
mounting and backing up remote file systems, especially across local networks.
That said, when using NFS on an open network it is important to understand
firewalling techniques. At a minimum, use TCP wrappers or whitelists provided
under most /etc/hosts.allow configurations. Never run NFS
unprotected, and always to kill mountd and rpc.portmap
while not in use.
NFS operates on the principle of a remote mount and runs the Unix
portmapper and mountd (the server used for NFS mount
requests) daemons on each host. /etc/exports controls access
to the server, containing a list of authorized hosts, along with user
permissions and allowable directories.
From man mountd:
When mountd is started, it loads the export host addresses and options into the kernel using the mount system call. After changing the exports file, a hangup signal should be sent to the mountd daemon to get it to reload the export information. After sending the SIGHUP, check the syslog output to see if mountd logged any parsing errors in the exports file. If mountd detects that the running kernel does not include NFS support, it will attempt to load a loadable kernel module containing NFS code.
By default, most BSD kernels have NFS built in and configurable under
/etc/rc.conf, whereas Linux requires a loaded module or kernel
recompilation.
NFS Backup Principles
The theory of NFS backup is relatively simple: mount each host to the NFS server, write a .tar backup script, and check that the following permission sets conform on both machines:
- Read Write access.
- Mount permissions.
- GID/UID.
Exports and Permissions
The /etc/exports file controls various options allowed by the
NFS server. In this example, jack.foo.com would be mounted on NFS
under /jack, and allowed to read and write to
/jack.
# File /etc/exports on NFS Server
# When backing up an entire system, including root files (UID 0)
# you would use the no_root_squash option as outlined below *
# For files belonging to jack, the only option required is 'rw'
/jack jack.foo.com(rw,no_root_squash) The UID/GID set can be somewhat confusing, especially if the NFS server and
hosts are different Unix flavors, or if they use different password schemas.
The owner and group are UID/GID dependent, and must be identical on the client
and server. If user jack has UID 2020 on client jack,
he must also have UID 2020 on the NFS server.
In case UID 2020 already exists on the NFS server as say, user sam, writing
UID 2020 from the client to the server would appear on the server as
sam. Also, if UID 2020 doesn't already exist, the files would
simply appear as owned by UID 2020, without user identification. (We often see
this scenerio when untarring a file from a remote FTP download where user
joe, UID 1010, may inadvertently be another user on our system, so
the file untars and appears to be owned by that user.)
When restoring, it's important to ensure the UID/GID matches the restoration on the target host for the same reason stated above.
From man exports:
Very often, it is not desirable that the root user on a client machine be treated as root when accessing files on the NFS server. To this end, uid 0 is normally mapped to a different id: the so-called anonymous or nobody uid. This mode of operation (called "root squashing") is the default, and can be turned off with no_root_squash. Use caution when configuring this option.
Be sure to back up password and group files as a standard procedure. Any time you add or delete a user from the client, update the group and password files.
Two Simple NFS Server Scripts
In this example, we start rpc.portmap, rpc.mountd,
and rpc.nfsd and write out a PID for each process to
nfs.pid.
#!/bin/bash
# Script to start NFS
/usr/sbin/rpc.portmap
sleep 1
/usr/sbin/rpc.mountd
sleep 1
/usr/sbin/rpc.nfsd
/sbin/pidof /usr/sbin/rpc.portmap > /var/run/nfs.pid
/sbin/pidof /usr/sbin/rpc.nfsd >> /var/run/nfs.pid
/sbin/pidof /usr/sbin/rpc.mountd >> /var/run/nfs.pid
echo "NFS READY"
# EOFTo stop NFS, we use the following script to kill everything listed in
nfs.pid, killing all three processes:
#!/bin/bash
# Script to stop NFS
kill `cat /var/run/nfs.pid`
/bin/rm /var/run/nfs.pid
echo "NFS HALTED"The Synchronized Backup
We can synchronize our backups by setting a cron job on each host and
server to start, stop, and use NFS at a specific time. After a backup
window, each host shuts down its NFS services. In order to synchronize
properly, each host should be timed to a
central NTP server for accuracy.
The NFS server is initiated from cron by a script similar to
the one shown above, and remains active for a specified period of time, depending
on the period set for a full backup. For a large backup, 30 minutes is usually
fine. Next, each client mounts a drive on the server, writes its files, then
umounts. This example uses tar. Other methods include
dump.
#!/bin/sh
# File NFSBackup Script for jack.foo.com
echo 'Initialize NFS Backup'
mount nfs.foo.com:/jack /mnt/jack
df
echo "Is nfs.foo.com mounted?"
read ans;
echo "Full Backup of Jack to follow"
sleep 4
cd /mnt/jack
tar -cvvf jack.tar /vmlinuz /System.map /root /mydocs /etc
/bin /sbin /var/local /var/spool /usr/sbin /usr/bin home
/usr/home/glenn /usr/home/ifconfig
sleep 5
umount /mnt/jack
echo "finished"Restoring Files
In almost every instance, restoration is a manual process, and even today, a full restore can be confusing, especially where the target system is used as a multipurpose host. Under typical circumstances, I restore a system by mounting the host to the NFS server, and perform the backup by reading from the remote filesystem.
Mount jack.foo.com to nfs.foo.com using the
mount command:
mount nfs.foo.com:/jack /mnt/jack; then cd to /mnt/jack.
To restore, simply untar jack.tar from
nfs.foo.com:
tar -xvf jack.tarCopy the necessary files from /mnt/jack to the appropriate
locations.
This is only one of many available options.
Summary
Here are some basic rules to follow:
- In the case of a complete re-installation, make sure the freshly-installed OS version is the same as the original. This is especially important to ensure that all configuration files match.
- Ensure all users have the same UID/GID as before. Check these IDs against
the original password and group files. It is important to note each UID/GID
corresponding to the individual remote system(s). It's not enough to simply
restore everything as
root. - Remember to restore the original kernel during a restoration. In the case
of a Linux system, for example, run
liloorgrubto load the kernel. - Remember to archive the backups and store them in a safe place, preferably off-site. Redundancy can save your data in case of a physical disaster.
Throughout my career working with Unix, I've seen fifty methods of backup
that include sophisticated hardware, media, and variations of GUIs and Web-based
interfaces. While some still exist, many have faded into the background. In
my opinion, combining NFS with tar (or dump) is still
the best option. It takes some knowledge to configure an NFS backup properly,
but in the end, it is definitely one of the most reliable means of protecting
valuable data.
Glenn Graham has been working with telecommunications since 1977.
Return to the Linux DevCenter.
You must be logged in to the O'Reilly Network to post a talkback.
Showing messages 1 through 25 of 25.
-
backing from an extra machine with rar
2003-08-10 10:56:38 anonymous2 [Reply | View]
I use an extra machine as a backup pc, and since it does nothing else, it can be a rather cheap one. Therefore, read-only mounted shares of any kind (nfs preferred, but samba is ok, too) are sufficient, and no silly user can sweep anything he doesn't own.
To avoid problems with 2 GB limit, I use RAR, which is a good program under linux as well as under windows...
regards
Nihad
-
Amanda
2003-04-01 12:51:18 anonymous2 [Reply | View]
I still beleive tapes are the best storage medium for backups. Also, Amanda does an excellent job of backing up multiple hosts and keeping a restore as simple as possible.
-
Free NFS client for Windows
2003-03-19 11:33:34 anonymous2 [Reply | View]
SAMBA, give me a break. There is no way I'm going to load the Microsoft SMB protocol onto my UNIX or Linux boxes. Make the Windows junk work like a UNIX box and load NFS onto it.
Free NFS client for Windows. I don't know of any that were ever worth a flip. I've tried Hummingbirds product, the old FTP Software product, and Intergraph's product. Out of those, I liked the Intergraph product the best (it had the best integration and least problems).
As far as using NFS for backups, I'm like everybody else on this board; don't do it. NFS is great for what it was designed to do and backing up directories wasn't its calling. Others on this board have listed great ideas. I would suggesting following those ideas.
-
remote backups
2003-03-18 02:43:31 anonymous2 [Reply | View]
A year ago I had this same itch (doing remote backups). This resulted in a little tool called hdup
which can do all this and more (encryped, remote
backups and restores).
Yes... this is a sameless plug for my program...anyway you can find it here:
http://www.miek.nl/projects/hdup16/hdup16.html
-
A small error?
2003-03-15 17:30:03 anonymous2 [Reply | View]
Aren't you in the wrong directory when you restore. Instead you need to be in the root directory?
cd /
tar xvf /mnt/jack/jack.tar
Jim
-
re: tar size limits
2003-03-15 12:41:08 anonymous2 [Reply | View]
Surprisingly, you can split files with split. man split for further info.
MfG shurdeek
-
NFS for backups?
2003-03-15 12:39:42 anonymous2 [Reply | View]
NFS for backups is one of the stupidest ideas I've heard, and someone doing this for a job should be fired.
- sends data unencrypted, anyone can spoof. Basically the same as if you gave anyone a readonly root access to all your computers.
- impossible to set up securely, if you give ANY write privileges, with a simple IP spoof attacker can delete/change anything
- no checksums, if errors occur in transfer, you're screwed.
A very simple workaround is to pipe the results of tar or dump or whatever into ssh. This solves the mentioned problems.
MfG shurdeek -
NFS for backups?
2006-06-08 06:56:51 ArturAnj [Reply | View]
Sometimes it's not stupid at all: in a controled network environment, sometimes it's very easy to had an extra network card to each server, and...
(It still have the checksum problem, but the beneficts are great - keep it simple) -
NFS for backups?
2003-03-19 09:27:35 simon_hibbs [Reply | View]
It's not inherently more stupid than using NFS at all, at least from a security standpoint. Sure they get all the data rather than dribs and drabs, but still...
On your second point - why would you make a backup share writeable? Duh!
Finaly, ok on checksums you have a point but this is just down to limitations in tar and the article does suggest other tools such as dump. Ultimately he wasn't writing an article aimed at being the be-all and end-all of backups.
Still, piping backups through ssh. That's quite neat, I usualy pipe remote backups through compress, but if I'm working in a nonsecure network I'll remember that.
Simon Hibbs
-
tar size limits?
2003-03-14 17:16:31 anonymous2 [Reply | View]
I tried to do something similar with tar using a samba share. The issue I had was the 2Gb file size limit or tar (8Gb with newer versions I think). What enhancements to your method can split the tar file into 2Gb files when there is more than 2Gb of data to backup? -
tar size limits?
2003-03-28 08:01:17 anonymous2 [Reply | View]
Instead of tar'ing into a tarball, you could:
tar -cf - $PATH1 | tar -xvpf - -C $PATH2
where $PATH1 is path to files you want to backup and $PATH2 is path to save files to.
But i haven't had any size problems with the bsd's - i have some 14gb tarfiles lying around. I thought recent RH had no problems too?
-
Don't use NFS for backup.
2003-03-14 11:25:49 anonymous2 [Reply | View]
It's just too scary to have a cracker cleaning up is mess with
rm -r /*
while your backup is NFS mounted no_root_squash.
For backup I recommend
dirvish
rsync
or
storeBackup
-
Have a look at rdiff-backup
2003-03-14 08:49:04 tarvin [Reply | View]
I have tried backup setups like the ones discribed in the article. It can certainly be made to work.
However, I have since found a better way:
rdiff-backup
- See http://rdiff-backup.stanford.edu/
It combines several nice features:
- May use already running services (on my servers, SSH is normally already running, while NFS is most often _not_)
- Easy access to most recent backup data.
- Access to previous backups: The incremental history may extend as far as space on the backup host allows. Unfortunately, the restore syntax is rather difficult to express.
- Super-efficient network use. Means that the backup server may actually be placed far from the backed up hosts == better data security in case of fire, water damage, theft, etc.
Currently, rdiff-backup in reality needs to run as root on the backup server, as well as on the backed up server. However, the software is currently being improved in this regard: Soon, the software doesn't need to be run as root on the backup server.
Written in Python 2.2. Good support-climate (well-working mailing list).
Really worth a try.
-
Re: Free NFS Windows clients
2003-03-14 06:57:01 anonymous2 [Reply | View]
I am aware of no free (or good) free NFS Windows client software, but when/if you want to share files between UNIX and Windows boxes, I've *always* found samba (www.samba.org) to be much more useful. -
Re: Free NFS Windows clients
2003-04-25 08:45:53 anonymous2 [Reply | View]
If you want a solid performer with great support, you won't find it for free. As far as SAMBA, find the SAMBA mailing list and look at all the problems - but that's what you get with a reverse engineered product (unpredictability).
Why don't you just search for windows and nfs on goolge and eval some products - they aren't that expensive.
IMO
-
simple locking
2003-03-11 12:55:47 anonymous2 [Reply | View]
Backups can sometimes take more than their alotted time, and they will often take less.
Suggestion:
1) In an area on the NFS server that all clients can read, create a "flag area" where the backup server creates a file corresponding to the name of the client who should now be backing up files.
2) Each client monitors the flag area for their name. When they see it, they begin dumping backup data to the server.
3) When a client completes, along with the backup file, they create a "completed" file (e.g. "touch `hostname`.completed"). They can shut down NFS if desired.
4) The NFS server watches for the "completed" file. When it sees it (or if nothing happens for a while, as would happen if the machine was down), it marks that host as finished and goes to the next host on the list.
5) When all hosts are done, shut down the NFS server if desired.
-
and, of course....
2003-03-11 11:18:33 flucifredi [Reply | View]
What simon suggested works as well with ssh, giving you an encrypted, secure backup channel even over unsecured networks (and without you needing to re-enable the dreaded r* services).
I am up for it, too. I considered the NFS choice for my network but I decided against it because the ssh+tar approach gives me absolute portability (I am even using it for a few windows machines).
-Federico -
and, of course....
2003-08-21 21:17:27 anonymous2 [Reply | View]
Over time I've tried several backup options and in the end I found using tar to a backup drive and then ssh to a backup server with both disk and tape backup was the least worry path. I now have four levels of failure before loosing data. (RAID to local IDE to backup server IDE and TAPE.)
Using NFS was an idea until I reviewed the security track of NFS - too much can go wrong. Only if there was no Internet access would I consider it. It's bad enough that users browse on windows boxes. I don't need that headache.
-
Free Windows NFS clients?
2003-03-10 10:47:13 anonymous2 [Reply | View]
Does anyone know of any free Windows NFS clients? Something that can mount NFS shares on a Windows XP box and transfer files between them?
-
Why not use tar and rsh?
2003-03-10 04:27:55 simon_hibbs [Reply | View]
nice article - it's surprisiong how often people forget about checking UID/GIDs.
I'm not clear what benefit you get from using NFS, as against streaming the backup directly across the network. From the tar man page :
tar cvfb - 20 files | rsh host dd of=/dev/rmt/0 obs=20b
You can also accomplish a similar result using dump. I prefer 'sucking' the data, rather than 'squirting' it as you do with tar above, because you can run the backup set entirely from the backup server without scheduling jobs on multiple machines.
Backup script pseudocode :
FOR each machine to backup -
IF (ping machine to backup) THEN
suck data with a compression filter
NEXT
Simon Hibbs
http://wiki.zmanda.com/