Belgium
Canada
Germany
Italy
France
Spain
United KingdomOpen Files and BackupAssist
How BackupAssist handles open files in Windows NT, 2000, XP and 2003
| Contents: |
|
|---|
Important Notice
The information and instructions contained within this document are specific to BackupAssist v3.x.x. While some of this information will be valid for other versions of BackupAssist we do not support the application of any steps or suggestions on versions other than v3.x.x
CHAPTER 1 - Introduction and Overview
1.1 Open File Support in BackupAssist
In order to maintain a reliable and recoverable backup strategy it is essential that your backup solution provides support for backing up open files. Without having open file protection in your backup solution mission critical data files can be missed. Consequently, your backup data sets will be unreliable and incomplete, all because certain application files were in use or open during the time of the backup process.
BackupAssist provides two different technologies for backing up open files in Windows environments:
- Snapshot Technology - Using the Microsoft® Volume Shadow Copy Service in Windows XP and 2003, BackupAssist is able to perform a backup of all open files by taking a live snapshot of your data before it is backed up. The Microsoft® VSS snapshot will only work, however, if the application data being backed up is compliant with the Microsoft® Shadow Copy Service framework.
As an alternative, BackupAssist offers an optional add-on to BackupAssist, the Advanced Open Files Add-on. This Add-on provides complete open file support for Windows NT, 2000, XP, 2003 and SBS using the BackupAssist Snapshot Manager, which performs the point-in-time snapshot of open file data. The advanced add-on supports backing up open data files for applications that are not VSS compliant.
- Conventional File Copy - by enhancing the functionality of NTBackup, the standard license of BackupAssist is able to backup all files that are not exclusively locked on Windows NT, 2000, XP, 2003 and SBS. This method is also utilized for backing up data on network shares, where snapshot technology cannot be applied. Both Snapshot technology and Conventional File Copy can be used together with BackupAssist to ensure that you have complete open file data protection. The two technologies act to cover the range of open file backup requirements that small and medium businesses may have, as well even supporting more larger enterprise organizations.
BackupAssist's dual technologies for handling open file backups allow businesses to choose a cost-effective solution that is tailored towards their backup requirements and budgets. The following chapters explain how each technology works, where they apply, and what the advantages and disadvantages are of each method of open file support.
CHAPTER 2 - Snapshot Technology
2.1 Introduction to Snapshot technology
Issues with applications and open files
Active applications will often keep their files open continuously. This can present a real problem for backup applications as it can prevent them from accessing and copying these files to the required backup media (tape, hard-disk, DVD/CD, etc). The Conventional File Copy method (described in Chapter 3) can work for some open files but it cannot be used to backup exclusively locked files such as application or database files.
Additionally, even when applications do not have open files, the data backed up from these applications may not be consistent with state of the original data at that particular point in time, due to the time needed to open, backup and close a file during the backup process.
Even more important than this, servers run critical applications such as databases which often process and record massive flows of transactions and data messages and swap this data constantly between memory and disk. This makes it very difficult to distinguish which data needs to be backed up in order to maintain consistency and reliability.
While it is possible to simply disable these applications during the backup process to ensure that the data is in a stable state, this results in unacceptable amounts of downtime, particularly for applications that must be running 24 hours a day. Additionally, if you are continuously starting and stopping applications, you run the risk of the application not re-starting on request every time, which will greatly hinder the productivity of your business operations.
For these open file problems, Snapshot technology can be used to take a snapshot or 'picture' of the data at a particular instant in time. A backup application can then backup from this snapshot or point-in-time copy, rather than the original data files that may be open and inaccessible to backup requests.
The BackupAssist Snapshot options
BackupAssist currently gives you the ability to use either Microsoft® VSS (Windows XP / 2003 only) or its own BackupAssist Snapshot Manager to perform a point-in-time snapshot.
You can choose your preferred Snapshot Manager within the Options tab of BackupAssist, as shown below:
2.2 Using Microsoft® VSS with BackupAssist
On Windows XP and 2003, files on a local drive that are currently in use can be backed up completely as if a snapshot image was taken at the time of the backup. This is done through a feature known as volume shadow copy or VSS. This feature ensures data integrity and reliable backups.
VSS is a storage management framework specifically designed for Windows to create point-in-time copies or snapshots of a hard disk volume. VSS uses a combination of requestors (backup applications), writers (business applications) and providers (snapshot technology) to perform the snap-shotting process, and works as follows:
- The backup application or requestor launches to perform a backup and sends a request to VSS to perform the snapshot.
- VSS communicates with running applications or writers to complete existing transactions and halt new ones.
- VSS then takes the snapshot using the snapshot technology or provider.
- Once the snapshot is complete, VSS communicates with the running applications so they can resume normal operations.
- The backup application then performs a backup of the snapshot created, where no open files are present, which means that no file are skipped.
- Once the backup operation on the snapshot completes, VSS removes the snapshot from the system.
To enable VSS within BackupAssist, simply choose Microsoft® Volume Shadow Copy Service as your open file snapshot manager during the setup wizard:
If you are not using Windows XP, 2003 or SBS 2003, then the option for Shadow Copy will not be available:
2.3 The BackupAssist Snapshot Manager - advantages over VSS and its usage
VSS Limitations
While the Microsoft® VSS framework does provide an excellent solution for storage management and point-in-time snapshots to cater for backing up applications with open files, it does have certain limitations:
- VSS is currently only available through Windows XP and Windows 2003.
This means that businesses running Windows 2000 or later servers cannot use VSS to backup their open files. - VSS is a framework that only supports VSS compliant applications.
Applications must be VSS compliant in order to interface with the VSS framework. Currently, many business applications are not VSS compliant, which includes a large number of legacy applications.
This means that a large number of applications will not be protected by VSS as they are not compliant with the framework.
When applications are not VSS compliant
For applications that are not VSS compliant, the VSS framework only provides crash consistency on files that are open or in use during the backup procedure. Crash consistency only guarantees file system integrity; it does not guarantee transaction integrity. This means that if an application transaction is in progress at the time of a snapshot, the actual snapshot may only contain a partial record of that transaction; consequently the data backed up is incomplete.
For an example of this inconsistency we can relate to a typical banking transaction, involving the following two actions:
The transaction above involves two simple actions: a withdraw of a sum of money from one bank account and deposit of the same sum to another bank account. These two actions must be recorded within the one transaction or the accounts will be out of balance.
Now, we can take a look at a scenario using a VSS snapshot to backup a non-VSS compliant application:
In this scenario, because the application being backed up by VSS is not VSS compliant, the VSS framework has no indication of any transactions in progress. In this particular case the snapshot has been taken in the middle of a transaction and as a consequence the backup of this snapshot will only be of the withdraw action and not the deposit; the accounts will be out of balance.
This can obviously cause major issues when it comes time to perform a restoration because your backup may be missing vital actions within various transactions.
The BackupAssist Snapshot Manager advantage
To overcome these limitations, users of BackupAssist can opt for the BackupAssist Advanced Open Files Add-on which involves the use of the BackupAssist Snapshot Manager.
The BackupAssist Snapshot Manager is an advanced open file management solution which ensures that you have complete protection for all your mission critical data and actually supports legacy and other custom applications that are not compliant with the VSS framework.
To overcome the VSS issue of performing a snapshot during an open or unfinished transaction, the BackupAssist Snapshot manager will monitor write operations occurring across all of your active applications. Once the Snapshot Manager has determined a point in time where significant time has elapsed since the last write operation, it will perform the snapshot. The time that the Snapshot Manager waits until it performs the snapshot is known as the Write Inactivity Period (WIP) and for mass storage systems a safe WIP is usually from 2 to 5 seconds.
Basically when the WIP elapses, it is assumed that the application data is consistent (no write operations) and that it is safe to perform the snapshot.
When to use the BackupAssist Snapshot Manager
You should use the BackupAssist Snapshot Manager under the following two conditions:
- You are running Windows 2000 Server or earlier where Microsoft® VSS is not available
- You are running applications under any Windows environment that are not compliant with the VSS framework
How to use the BackupAssist Snapshot Manager
To enable the BackupAssist Snapshot Manager, ensure that when you first install BackupAssist that you select the Advanced Open Files Add-on as part of the install:
During the BackupAssist Configuration Wizard, to enable the BackupAssist Snapshot Manager, simply choose Use the BackupAssist Snapshot Manager when selecting your open file snapshot manager:
If you did not install the Advanced Open Files Addon during the install process then you will need to run the BackupAssist upgrader, which can be downloaded from the product website or obtained through your distributor.
After running the BackupAssist upgrade Wizard, the third step of the upgrader will allow you to install the Advanced Open Files Add-on module:
You can then access the Options tab in BackupAssist to select the BackupAssist Snapshot Manager for your open file backups.
Once a backup runs, the backup report will also indicate the status of the Snapshot that was taken before the backup commenced.
The following screenshot demonstrates a typical report with the Advanced Open File Add-on installed and the BackupAssist Snapshot Manager enabled:
The BackupAssist Event Viewer will also indicate the processes involved in the Snapshot:
2.4 Useful References
Microsoft® VSS - Microsoft Technet: How Volume Shadow Copy Works
Snapshot Technology - Search Storage Articles
Chapter 3 will now cover how BackupAssist can use the Conventional File Copy Method to further ensure that you have a secure backup of all your open files, including backups of network share data files that do not have exclusive locks.
CHAPTER 3 - Conventional File Copy
3.1 How Conventional File Copy Works
When using the Windows Backup Utility, NTBackup, (which BackupAssist uses to perform the backup operation) files that are in use or open during the backup process are skipped and as a consequence, not backed up. BackupAssist uses a multi-phase backup process to workaround the issue of skipped files during the backup.
Phase One
Your selected drives and directories are backed up. Open files are skipped by NTBackup. BackupAssist will detect these skipped files and then attempt to copy them to a temporary directory.
Phase Two
Phase Two of the BackupAssist backup process involves a backup of this temporary location where the open files were previously copied to. For open files that are locked with a shared lock or no lock at all are successfully backed up using this method. Examples of these open files include Outlook .pst files.
Files that have exclusive locks, however, cannot be copied, and as a consequence cannot be backed up using this method. These open files are typically from database applications.
For specific databases such as Microsoft Exchange or SQL Server that have exclusively locked files, you can use the respective BackupAssist modules (built-in Exchange module and the SQL Add-on) to backup these resources. All BackupAssist modules are also executed during Phase Two. This multi-phase process is unique to BackupAssist, and extends the capabilities of Windows Backup (NTBackup).
Overcoming exclusively locked file issues
For all other application data with exclusively locked files, you would need to use either VSS or the BackupAssist Snapshot Manager, as mentioned in the previous chapter.
VSS (Windows XP/2003 only) and the BackupAssist Snapshot Manager both overcome the issue of not being able to backup exclusively locked files by taking a live snapshot/point-in-time copy of the files before the backup commences and then backing up from this snapshot rather than the original open files.
3.2 When Does Conventional File Copy Apply?
Conventional File Copy for open file backups can apply in two separate situations:
- When no snapshot technology is available - either VSS or the BackupAssist Snapshot Manager)
- When you are attempting to backup data from a network share - you will require the Conventional File Copy method when you are attempting to backup open files residing on a network share as current snapshot technology does not support snapping network volumes.
3.3 Utilizing Conventional File Copy
To utilize Conventional File Copy for backing up open files, simply choose your preferred snapshot manager in the Options tab and then ensure that Re-attempt backup of open files is checked, as shown in the following screenshot:
You will also need to set the Open Files working directory. This is the temporary location that any open files detected in phase one of the backup process are copied to, before being backed up during phase two.
Please note: the open files working directory must be a specific directory that is only be used for the BackupAssist open file backup operations - the entire directory will be purged once a backup is complete.
3.4 A practical example of how the Conventional File Copy Method works
This following simple example illustrates how BackupAssist handles open file backups using the Conventional File Copy Method.
Example details
The following system has been used for this example:
Operating System: Windows NT 4.0 Server
Directory to backup: C:\TestDir
Open files in this directory (as shown below):
Operating System: Windows NT 4.0 Server Directory to backup: C:\TestDir Open files in this directory: C:\TestDir\Open file - shared lock.txt (locked with a shared lock) C:\TestDir\Open file - exclusive lock.txt (locked with an exclusive lock)
Backup procedure
When BackupAssist first runs NTBackup to backup your files in phase one of the backup process, all open files are skipped by NTBackup. There were two open files in the backup directory, so we expect both files to be skipped and not backed up. BackupAssist will then try to copy the open files skipped by NTBackup to the Open Files Working Directory, which has been set to:
C:\Program Files\BackupAssist\OpenFiles
Only the shared lock file can be copied. The exclusively locked file cannot be copied, because the Windows Operating System prevents programs from accessing it.
BackupAssist will then run NTBackup again for phase two of the backup process, and backup files from the Open Files Working Directory. From this directory, BackupAssist recreates the full directory structure of the open files. The table below provides instructive examples of how this works:
| Original Location of Open File | BackupAssist copies this file to... (OFWD = Open Files Working Directory) |
|---|---|
| c:\TestDir\Open file - shared lock.txt | OFWD\c\TestDir\Open file - shared lock.txt |
| d:\hello\there.txt | OFWD\d\hello\there.txt |
| \\servername\sharename\abc.exe | OFWD\servername\sharename\abc.exe |
The backup report also indicates what files were skipped and summarizes which files could be backed up using the Conventional File Copy Method, and which files could not be backed up using this method (i.e. files that had exclusive locks), as shown below:
This means that you can deduce the original location of an open file from its location in the Open Files Working Directory.
The email report (shown below) summarizes all the open files that could and couldn't be copied.
3.5 Restoring Conventional Open File Backup
To perform a restoration of open files, run NTBackup. On Windows NT, you should insert your tape into the tape drive; on Windows 2000 and later you should locate your tape from the Restore tab and insert it when prompted.
Set 1 corresponds to the first backup phase - where files and directories are backed up. Any open files would have been skipped during this phase and as a result do not appear in this set.
Set 2 corresponds to the second backup phase - where open files are copied and backed up (along with additional data such as Exchange Server, and any add-on modules such as SQL Server data).
Phase One Backup
During the Phase One backup of c:\TestDir, only one file was closed (ie. not open) - Normal file.txt.
In the view of the tape (shown below), you can see that file is available on the tape. To restore, select that file and click the Restore button.
Phase Two Backup
During the Phase Two backup, the open files were copied to the Open Files Working Directory (in this case - c:\Program Files\BackupAssist\OpenFiles).
The file, Open file - shared lock.txt was copied, and as shown in the screenshot below, is available for restoration.
When you restore this file, you can restore it to its original location (in the Open Files Working Directory). You'll then manually copy this file back to its original location (in this case, C:\TestDir).
