Archives and Archiving Files and Documents

An archive is an organizational strategy for data. It's a structure into which data can be stored in a way that makes it easy to retrieve the data in the future.

There are a few different ways to organize information. To use some computer terms: "tables", "time", and "hierarchy".

Tables refers to database tables, where data is organized into records and fields (or rows and columns). A record is a unit of data, like a row in a list. A field is information about the data, or the data itself, like the columns in a row. The useful property of a table is that every row has the same columns, so you can sort and group by columns.

A hierarchy is like a filing system of folders.

Chronological organization is to organize information by time, so you can retrieve the data from a specific time period.

The computer's file system uses all three methods of organization. Each file has common fields, like the modification time, size, and usually a file extension.

The files are stored in a hierarchy, and people typically name the folders uniformly. This uniform naming breaks up the filename into fields, so it's easier to sort through the files.

For more info, see the file naming convention articles below.

The file system generally lacks the ability to add extra fields of data. For example, it would be useful to be able to attach major and minor version numbers to every file. While there are some ways to do this, there isn't a simple way that exposes itself through the user interface, easily.

Consequently, the folder hierarchy is usually used instead of extra fields. It's not a bad or good thing - it's just how we do it. For some examples of this, see the folder organizing articles below.

Good archiving can assist backups by breaking the file system into parts. For example, if the folders are organized by client, you can split up the backups by client. Then, you can direct archives for old clients onto specific media, which might be kept offline for offsite. With very little work, you can cut down the time required to backup adequately -- and that translates into a greater capacity for the entire backup system.

Backup Laptops with a Dock

If you have a laptop that you travel with, consider getting a dock for your office desk. If your laptop isn't dockable (because it's a "home" laptop computer), then, get a universal dock. A universal dock is a dock with a USB connector, and an internal USB hub. (You might call it a glorified USB hub.)

To the hub, attach a USB hard drive or USB flash drive. The USB flash is better, because it uses less power.

Get some "sync" software that synchronizes folders. Some will initiate a sync when the drive is connected. Windows users may use a tool like Allway Sync or FolderClone. Set it up to backup the My Documents folder and perhaps the Desktop as well.

Every time you dock and log in, the software should sync and backup your important documents to the USB flash drive.

See the article backup external hard disk or usb flash drive for more ideas.

Backup Tapes

Backup tapes are a popular backup medium, but recently has become more expensive than disk. It's cheaper to use hard disks for backup.

Backup tapes have some advantages. They are smaller than disks, so you can pack more into a box, and send it to an archival location. Usually, backup tapes are stored in a cool, dry room. They are more durable, in that a shock to the backup tape won't cause failure, whereas disks may have a head-crash.

There are many different types of backup tapes, ranging from the old 9mm, serpentine, the Travan, and the DAT. The main backup tapes out there are 4mm that are used with DAT drives.

Enterprises (meaning businesses with scale and money) still buy backup tapes.

Consumers (meaning everyone else) has moved on to disk-based backups. Backup tapes cost more per megabyte than disks, unbelievably.

I guess there are a lot of enterprises overpaying for their data.

Backup to CD-R and CD-RW

Backup to CD shares a lot of problems that backup to DVD has, with some interesting differences.

The main difference is that CDs ware 1/5th the size of DVDs, so you can't backup as much data. Consequently, the backup is "faster" because the data set is smaller.

So, CDs are basically not good for backing up your system, but, are a great way to make archival snapshots of your work-in-progress.

For example, if you wanted to retain 7 days of your past work, you can purchase 7 CD-RWs, and label them "Monday", "Tuesday", "Wednesday" and so forth. Put them in jewel cases, and then into a CD box.

Each day, either at the start or end of a day, run a backup of your work, and then store it. It won't take more than 10 minutes.

For your effort, you are rewarded with an archive of your most important data, at your fingertips.

Backup to DVD

Creating backups on DVD-R or DVD-RW allows you to store up to 4.8 gigabytes of data (or 2.4 if you use single layder DVDs).

The main advantages:

• low cost
• archival, by default
• widely supported, and readers are common

The main disadvantages:

• slow write speeds
• limited capacity
• data is easily damaged

If you are going to backup to DVD, get an SATA DVD burner.

Chances are, you're only going to do a data backup, so, make duplicates of all your installer CDs and DVDs first. Make a disc with all the downloaded installers, and all the serial codes.

Then, backup the data. You may need to partition your data on the disk, and set up different backup jobs, to spread the backup across multiple DVDs. Prepare to spend a lot of time waiting.

Another disadvantage is that you can't always choose backup software, because the burner may not work with generic DVD burning software.

That all said, a DVD is very light, and easy to mail. It's a great way to make a weekly backup of a large project that can be sent off-site "just in case". It also gives your client something solid in exchange for paying their weekly invoice.

Backup to External Hard Disk or USB Flash Drive

A simple, transparent way to backup a personal computer is with an external hard drive or USB flash drive.

You don't need special software to do this - just copy the files.

The real issue is getting your files organized so all your documents are saved to the disk in one simple motion. (See organizing your files.)

Also, if you're a data-completist, you'll want to save the settings files (the .dotfiles in unix, and the hidden Application Settings in Windows).

If you wish to automate the process, some of the best software to use is "sync" software that compares the copies to the originals, and updates the copies automatically. The program I use is Allway Sync. There are others as well, but I found the interface to Allway Sync easiest to comprehend.

External hard disks have two risks. One is that the power adapter may fail. Another is that, because the drive is in a mobile case, you can drop the disk and have a head crash.

USB flash disks are less prone to damage, but it's possible to put them in your pocket, forget about them, and toss your pants into the washing machine, destroying the device.

USB flash drives also tend to be fragile because they stick out of the USB port. If you want to install it permanently, get a cheap usb extension cord at the dollar store, and tape the disk to your case.

A good backup solution for someone who isn't computer savvy is sync software, a USB flash memory drive, and the aforementioned extension cord. Set it up for them, and tell them to store their documents in only one folder.

Backup to Floppy

Backing data up to floppy went out with the 1990s. Hardly any computers have floppy disk drives anymore.

That's why it's important to gather your floppy disk based backups and move the data onto a hard disk.

When you do this, you'll be shocked at how much data's been lost to floppy disk degredation.

You may need to clean your floppy disk drive heads if you are a smoker. If you can't find one of those cleaning disks, you can fake it by taking a floppy disk that you aren't going to need and pouring a little alcohol on both sides of the disk. Pop it in for three or four seconds while the disks slide across the surface, then pop it out. The risk here is that the diskette will shed material before the heads get cleaned - so use a new diskette.

Floppy disks are still useful for doing system installations or restorations, so you still see them on some back-office systems.

Cloud Computing Backups

With more and more work being done "in the cloud" with web-based applications that store data on a remote server, edited through a web browser (or specialized client application), you'll want to backup the remote data locally.

The way to do this is to export the data using a tool that automates the process. For example, Google Docs Backup.

One of the nice things about Application Service Providers is that they save you from installing and updating software. The big risk is that they'll upgrade and leave your older documents unusable.

Legacy data in traditional backup scenarios is managed in two ways: one is pickling, where an entire system and software stack is retained to read the data. Another way to manage legacy data is to convert it to newer, more useful formats, or to older generic formats.

Cloud computing leaves you only the latter option. So, applications like Google Docs Backup try to convert the data to something generic.

#! /bin/bash

echo
echo Dumping db to db.mysql.
echo Type your password.
ssh launion@webhost.net mysqldump -u uname '-p--remotepw---' database_name  > db.mysql
echo Loading
mysql -u root '-p----password----' database_name < db.mysql

Database Backup

The correct way to backup a database is to use a "mirror" or "replica" of the database. That's a server that's running a duplicate of the database, and, perhaps also acting as a load-balancing server.

These two servers are connected by a network, and as requests come in to the main server, they are either performed on the main server, or passed on to the mirror. Update and delete operations are carried out on the main server, then executed on the mirror.

A typical scenario is to reserve a single IP address for hosting the service (on LAN). The main database has this address. It also has a second IP address for inter-database communications. The second datbase has only the inter-db communications IP address. If the main database fails, the second machine "takes over" the IP address.

This failover is combined with database replication. The free MySQL server has this feature, and is described in replication.

This provides maximum fault-tolerance.

A simpler backup that doesn't have the advantages of a mirror, is to dump the contents of a database to text files, and back those up. This is obviously a lot cheaper than purchasing a second database server.

If you opt for the latter method, make sure that you can build a database server and load the data quickly.

For archival purposes, you may want to make a database dump regularly, compress it, and have it backed up with the rest of the files.

If you're backing up a database on a website, see the article
How to Backup MySQL on a Website.

Failure

Computers fail, eventually.

There are hardware failures, and software failures.

Hardware is nice, and tends to fail one part at a time. So, if your system breaks, you can replace the part, and be operational again.

That's assuming that you can still purchase the part. You may have a spare, but, does it still work? Some components, like electrolytic capacitors, can age and fail.

Software failures are harder to detect, and sometimes, software failures are invisible when they happen, but manifest symptoms later, with bad data being revealed.

Full Backup

A full backup is a backup of all the files. It's used in contrast to the incremental backup, which is a backup of files changed since the last backup.

A common problem with full backups of networks or large file servers is that they take a long time. Backing up 300 gigabytes of data can take over half a day (over a 1gigabit ethernet network, to a SATA 3, RAID 5 NAS box).

So, full backups are typically scheduled to run over the weekend, when fewer people are using the network.

If there's too much data, a full backup may not be possible. The only solution is to split the file system into separate branches, and backup the branches on different days.

Full backups are performed in conjunction with incremental backups, usually scheduled to run once a day in the evening. A typical schedule is to perform one full backup each month, and then perform an incremental backup each evening.

Generally, it's bad to schedule full backups that fall on the 1st, last, and 15th day of the month, because those are "paydays" and it's possible that accountants may need to use the computers. (I think that mean the second and third weekends are best.) That said, backups are important enough to run even if someone's working on the weekend.

Incremental Backups

An incremental backup is a backup of all the files that have changed since the last backup. Typically, you perform a full backup, then a series of incremental backups.

You can perform full and incremental backups using tools like BackupExec or NTBACKUP.EXE. All commercial backups can do incrementals.

Incremental backups take far less space than full backups, and also take a lot less time to perform. In some situations, it's feasible to run backups during the workday.

Restoration of files from an incremental backup are performed by restoring the latest version of a file. This is done automatically.

Generally, previous versions can also be restored, so incremental backups also serve as a way to archive changes to the file system.

In many cases, incremental backups are better for archives than full backups for archives. For one, files that are created, and then later deleted, in the interval between full backups, are not stored in the archival backup. The problem is, basically, size - because a full backup is the same size as all the files. Keeping incrementals as well requires the full size, plus space for all the incrementals.

Incremental backups after periodic full backups are the preferred way to perform backups.

Inverted, Multiple Backups with USB Flash Memory

When you have multiple computers, you might want to put your files on a USB flash disk (aka a thumb drive or jump drive), and backup data to your computer's disks.

Create a folder called "backups", and a folder within it called "usb", and copy your files into there. If you have automatic sync software, you can set it to backup the data frequently. Hard disks are so fast you won't even notice the backups.

Set this up on all your computers, and your risk of data loss is nearly zero.

Just let the data "settle down" and force a file copy before removing the USB flash memory.

You could also set up a similar scheme with a USB external disk. The only issue is that moving hard disks tends to damage them and lead to data loss.

If you have important data, consider using encryption software. Losing the USB flash drive with your vital data would be bad.

Secondary Backups

It's a good idea to run two sets of backups. For one, it's possible that the backup software can fail, leaving some data unsaved. It's also difficult to check the backups every day, so it's likely that some minor glitch could lead to several days without backups. You could run out of space on the backup device. A device may go offline and stay offline for no discernable reason.

Going more than one or two days without a functioning incremental backup is unacceptable. As more work is lost, there's a "network effect" where people depend on other people's work, and you have to involve more people in the disaster recovery effort.

A cheap way to avoid this problem is to run two sets of backups - a primary and a secondary backup - with the full backups staggered, and with longer runs of incremental backups on the secondary backup. Store the secondary backup on a different device (a hard disk in your PC is a good place).

This way, if the system fails on the primary, you can use the secondary to recover. If the secondary fails, you have the primary.

In my experience, you can run two backups and check them twice a week, and there is never a situation where both backups are failing, but there's occasionally a problem with one of the backups.

Subversion (svn) as a Backup

If you're programming (or managing programmers), you can use Subversion or any other revision control system as a backup.

If you're not using a revision control system at all, it's a good idea to start immediately.

Not only does the repository (the system where the code is stored) a backup of the code - it's also a way to roll back changes to your code. All the popular systems also enable team programming over the internet.

The few hours spent learning the system pay off many times over in saved time and mitigated risk.