Supporting Perforce: Backup and Recovery

The Perforce service stores two kinds of data: versioned files and metadata.

  • Versioned files are files submitted by Perforce users. Versioned files are stored in directory trees called depots.

    There is one subdirectory under the server's root directory for each depot in your Perforce installation. The versioned files for a given depot are stored in a tree of directories beneath this subdirectory.

  • Database files store metadata, including changelists, opened files, client workspace specifications, branch mappings, and other data concerning the history and present state of the versioned files.

    Database files appear as db.* files in the top level of the server root directory. Each db.* file contains a single, binary-encoded database table.

Backup and recovery concepts

Disk space shortages, hardware failures, and system crashes can corrupt any of the Perforce server's files. That's why the entire Perforce root directory structure (your versioned files and your database) should be backed up regularly.

The versioned files are stored in subdirectories beneath your Perforce server root, and can be restored directly from backups without any loss of integrity.

The files that constitute the Perforce database, on the other hand, are not guaranteed to be in a state of transactional integrity if archived by a conventional backup program. Restoring the db.* files from regular system backups can result in an inconsistent database. The only way to guarantee the integrity of the database after it's been damaged is to reconstruct the db.* files from Perforce checkpoint and journal files:

  • A checkpoint is a snapshot or copy of the database at a particular moment in time.

  • A journal is a log of updates to the database since the last snapshot was taken.

The checkpoint file is often much smaller than the original database, and it can be made smaller still by compressing it. The journal file, on the other hand, can grow quite large; it is truncated whenever a checkpoint is made, and the older journal is renamed. The older journal files can then be backed up offline, freeing up more space locally.

Both the checkpoint and journal are text files, and have the same format. A checkpoint and (if available) its subsequent journal can restore the Perforce database.

Warning

Checkpoints and journals archive only the Perforce database files, not the versioned files stored in the depot directories!

You must always back up the depot files (your versioned file tree) with the standard OS backup commands after checkpointing.

Because the information stored in the Perforce database is as irreplaceable as your versioned files, checkpointing and journaling are an integral part of administering Perforce, and should be part of your regular backup cycle.

Checkpoint files

A checkpoint is a file that contains all information necessary to re-create the metadata in the Perforce database. When you create a checkpoint, the Perforce database is locked, enabling you to take an internally consistent snapshot of that database.

Versioned files are backed up separately from checkpoints. This means that a checkpoint does not contain the contents of versioned files, and as such, you cannot restore any versioned files from a checkpoint. You can, however, restore all changelists, labels, jobs, and so on, from a checkpoint.

To guarantee database integrity upon restoration, the checkpoint must be as old as, or older than, the versioned files in the depot. This means that the database should be checkpointed, and the checkpoint generation must be complete, before the backup of the versioned files starts.

Regular checkpointing is important to keep the journal from getting too long. Making a checkpoint immediately before backing up your system is good practice.

Creating a checkpoint

Checkpoints are not created automatically; someone or something must run the checkpoint command on the Perforce server machine. To create a checkpoint, invoke the p4d program with the -jc (journal-create) flag:

p4d -r server_root -jc

You can create a checkpoint while the Perforce service (p4d) is running. The checkpoint is created in your server root directory (that is, P4ROOT if no server_root is specified).

To make the checkpoint, p4d locks the database and then dumps its contents to a file named checkpoint.n in the P4ROOT directory, where n is a sequence number.

Before unlocking the database, p4d also copies (on UNIX where the journal is uncompressed, renames) the journal file to a file named journal.n-1 in the P4ROOT directory (regardless of the directory in which the current journal is stored), and then truncates the current journal. The MD5 checksum of the checkpoint is written to a separate file, checkpoint.n.md5, and the lastCheckpointAction counter is updated to reflect successful completion.

Note

When verifying the MD5 signature of a compressed checkpoint, the checkpoint must first be uncompressed into a form that reflects the line ending convention native to the system that produced the checkpoint. (That is, a compressed checkpoint generated by a Windows server should have CR/LF line endings, and a compressed checkpoint generated on a UNIX system should have LF line endings.)

This guarantees that the last checkpoint (checkpoint.n) combined with the current journal (journal) always reflects the full contents of the database at the time the checkpoint was created.

The sequence numbers reflect the roll-forward nature of the journal; to restore databases to older checkpoints, match the sequence numbers. That is, you can restore the database reflected by checkpoint.6 by restoring the database stored in checkpoint.5 and rolling forward the changes recorded in journal.5. In most cases, you're only interested in restoring the current database, which is reflected by the highest-numbered checkpoint.n rolled forward with the changes in the current journal.

To specify a prefix or directory location for the checkpoint and journal, use the -jc option. For example, you might create a checkpoint with:

p4d -jc prefix

In this case, your checkpoint and journal files are named prefix.ckp.n and prefix.jnl.n respectively, where prefix is as specified on the command line and n is a sequence number. If no prefix is specified, the default filenames checkpoint.n and journal.n are used. You can store checkpoints and journals in the directory of your choice by specifying the directory as part of the prefix.

You can prespecify the prefix for a server with:

p4 configure set journalPrefix=prefix

When the journalPrefix configurable is set, the configured prefix takes precedence over the default filenames. This behavior is particularly useful in multiserver and replicated environments.

To create a checkpoint without being logged in to the machine running the Perforce service, use the command:

p4 admin checkpoint [-z | -Z] [prefix]

Running p4 admin checkpoint is equivalent to p4d -jc. You must be a Perforce superuser to use p4 admin.

You can set up an automated program to create your checkpoints on a regular schedule. Be sure to always check the program's output to ensure that checkpoint creation was started. Compare the checkpoint's actual MD5 checksum with that recorded in the .md5 file, and back up the .md5 file along with the checkpoint. After successful creation, a checkpoint file can be compressed, archived, or moved onto another disk. At that time or shortly thereafter, back up the versioned files stored in the depot subdirectories.

To restore from a backup, the checkpoint must be at least as old as the files in the depots, that is, the versioned files can be newer than the checkpoint, but not the other way around. As you might expect, the shorter this time gap, the better.

If the checkpoint command itself fails, contact Perforce technical support immediately. Checkpoint failure is usually a symptom of a resource problem (disk space, permissions, and so on) that can put your database at risk if not handled correctly.

Note

You can verify the integrity of a checkpoint using the p4d -jv command.

Journal files

The journal is the running transaction log that keeps track of all database modifications since the last checkpoint. It's the bridge between two checkpoints.

If you have Monday's checkpoint and the journal that was collected from then until Wednesday, those two files (Monday's checkpoint plus the accumulated journal) contain the same information as a checkpoint made Wednesday. If a disk crash were to cause corruption in your Perforce database on Wednesday at noon, for instance, you could still restore the database even though Wednesday's checkpoint hadn't yet been made.

Warning

By default, the current journal filename is journal, and the file resides in the P4ROOT directory. However, if a disk failure corrupts that root directory, your journal file will be inaccessible too.

We strongly recommend that you set up your system so that the journal is written to a filesystem other than the P4ROOT filesystem. To do this, specify the name of the journal file in the environment variable P4JOURNAL or use the -J filename flag when starting p4d.

To restore your database, you only need to keep the most recent journal file accessible, but it doesn't hurt to archive old journals with old checkpoints, should you ever need to restore to an older checkpoint.

Journaling is automatically enabled on all Windows and UNIX platforms. If P4JOURNAL is left unset (and no location is specified on the command line), the default location for the journal is $P4ROOT/journal.

The journal file grows until a checkpoint is created; you'll need make regular checkpoints to control the size of the journal file. An extremely large current journal is a sign that a checkpoint is needed.

Every checkpoint after your first checkpoint starts a new journal file and renames the old one. The old journal is renamed to journal.n, where n is a sequence number, and a new journal file is created.

By default, the journal is written to the file journal in the server root directory (P4ROOT. Because there is no sure protection against disk crashes, the journal file and the Perforce server root should be located on different filesystems, ideally on different physical drives. The name and location of the journal can be changed by specifying the name of the journal file in the environment variable P4JOURNAL or by providing the -J filename flag to p4d.

Warning

If you create a journal file with the -J filename flag, make sure that subsequent checkpoints use the same file, or the journal will not be properly renamed.

Whether you use P4JOURNAL or the -J journalfile option to p4d, the journal filename can be provided either as an absolute path, or as a path relative to the server root.

Example 2. Specifying journal files

Starting the service with: 

$ p4d -r $P4ROOT -p 1666 -J /usr/local/perforce/journalfile

Perforce Server starting...

requires that you either checkpoint with: 

$ p4d -r $P4ROOT -J /usr/local/perforce/journalfile -jc

Checkpointing to checkpoint.19...
Saving journal to journal.18...
Truncating /usr/local/perforce/journalfile...

or set P4JOURNAL to /usr/local/perforce/journalfile and use the following command: 

$ p4d -r $P4ROOT -jc

Checkpointing to checkpoint.19...
Saving journal to journal.18...
Truncating /usr/local/perforce/journalfile...

If your P4JOURNAL environment variable (or command-line specification) doesn't match the setting used when you started the Perforce service, the checkpoint is still created, but the journal is neither saved nor truncated. This is highly undesirable!

Checkpoint and journal history

You can use the p4 journals command to display the history of checkpoint and journal activity for the server. This history includes information about the following events: the server takes a checkpoint, journal rotation, journal replay, checkpoint scheduling. For detailed information about command output and options, see the description of the p4 journals command in the P4 Command Reference.

Verifying journal integrity

You can verify the integrity of a checkpoint using the p4d -jv command.

Disabling journaling

To disable journaling, stop the service, remove the existing journal file (if it exists), set the environment variable P4JOURNAL to off, and restart p4d without the -J flag.

Versioned files

Your checkpoint and journal files are used to reconstruct the Perforce database files only. Your versioned files are stored in directories under the Perforce server root, and must be backed up separately.

Versioned file formats

Versioned files are stored in subdirectories beneath your server root. Text files are stored in RCS format, with filenames of the form filename,v. There is generally one RCS-format (,v) file per text file. Binary files are stored in full in their own directories named filename,d. Depending on the Perforce file type selected by the user storing the file, there can be one or more archived binary files in each filename,d directory. If more than one file resides in a filename,d directory, each file in the directory refers to a different revision of the binary file, and is named 1.n, where n is the revision number.

Perforce also supports the AppleSingle file format for Macintosh. These files are stored in full and compressed, just like other binary files. They are stored in the Mac's AppleSingle file format; if need be, the files can be copied directly from the server root, uncompressed, and used as-is on a Macintosh.

Because Perforce uses compression in the depot file tree, do not assume compressibility of the data when sizing backup media. Both text and binary files are either compressed by p4d (and are denoted by the .gz suffix) before storage, or they are stored uncompressed. At most installations, if any binary files in the depot subdirectories are being stored uncompressed, they were probably incompressible to begin with. (For example, many image, music, and video file formats are incompressible.)

Backing up after checkpointing

In order to ensure that the versioned files reflect all the information in the database after a post-crash restoration, the db.* files must be restored from a checkpoint that is at least as old as (or older than) your versioned files. For this reason, create the checkpoint before backing up the versioned files in the depot directory or directories.

Although your versioned files can be newer than the data stored in your checkpoint, it is in your best interest to keep this difference to a minimum; in general, you'll want your backup script to back up your versioned files immediately after successfully completing a checkpoint.

Backup procedures

To back up your Perforce installation, perform the following steps as part of your nightly backup procedure.

  1. Verify the integrity of your server:

    p4 verify //...

    You might want to use the -q (quiet) option with p4 verify. If called with the -q option, p4 verify produces output only when errors are detected.

    By running p4 verify before the backup, you ensure the state of the archives and that this information is stored as part of the backup you're about to make.

    Regular use of p4 verify is good practice not only because it enables you to spot any corruption before a backup, but also because it gives you the ability, following a crash, to determine whether or not the files restored from your backups are in good condition.

    Note

    For large installations, p4 verify might take some time to run. Furthermore, the server is under heavy load when p4 verify is verifying files, which can impact the performance of other Perforce commands. Administrators of large sites might choose to perform p4 verify on a weekly basis, rather than a nightly basis.

    For more about the p4 verify command, see Verifying files by signature.

  2. Make a checkpoint by invoking p4d with the -jc (journal-create) flag, or by using the p4 admin command. Use one of:

    p4d -jc

    or:

    p4 admin checkpoint

    Because p4d locks the entire database when making the checkpoint, you do not generally have to stop the Perforce service during any part of the backup procedure.

    Note

    If your site is very large (gigabytes of db.* files), creating a checkpoint might take a considerable length of time.

    Under such circumstances, you might want to defer checkpoint creation and journal truncation until times of low system activity. You might, for instance, archive only the journal file in your nightly backup and only create checkpoints and roll the journal file on a weekly basis.

  3. Ensure that the checkpoint has been created successfully before backing up any files. (After a disk crash, the last thing you want to discover is that the checkpoints you've been backing up for the past three weeks were incomplete!)

    You can tell that the checkpoint command has completed successfully by examining the error code returned from p4d -jc or by observing the truncation of the current journal file.

  4. Confirm that the checkpoint was correctly written to disk by comparing the MD5 checksum of the checkpoint with the .md5 file created by p4d -jc.

    The checksum in the .md5 file corresponds to the checksum of the file as it existed before any compression was applied, and assumes UNIX-style line endings even if the service is hosted on Windows. (If your checkpoint file was created with the -z compression option, you may need to decompress it and account for line ending differences.)

  5. Once the checkpoint has been created successfully, back up the checkpoint file, its .md5 file, the old journal file, and your versioned files. (In most cases, you don't actually need to back up the journal, but it is usually good practice to do so.)

    Note

    There are rare instances (for instance, users obliterating files during backup, or submitting files on Windows servers during the file backup portion of the process) in which your versioned file tree can change during the interval between the time the checkpoint was taken and the time at which the versioned files are backed up by the backup utility.

    Most sites are unaffected by these issues. Having Perforce available on a 24/7 basis is generally a benefit worth this minor risk, especially if backups are being performed at times of low system activity.

    If, however, the reliability of every backup is of paramount importance, consider stopping the Perforce service before checkpointing, and restart it only after the backup process has completed. Doing so will eliminate any risk of the system state changing during the backup process.

    You never need to back up the db.* files. Your latest checkpoint and journal contain all the information necessary to re-create them. More significantly, a database restored from db.* files is not guaranteed to be in a state of transactional integrity. A database restored from a checkpoint is.

    Windows

    On Windows, if you make your system backup while the Perforce service is running, you must ensure that your backup program doesn't attempt to back up the db.* files.

    If you try to back up the db.* files with a running server, Windows locks them while the backup program backs them up. During this brief period, Perforce is unable to access the files; if a user attempts to perform an operation that would update the file, the server can fail.

    If your backup software doesn't allow you to exclude the db.* files from the backup process, stop the server with p4 admin stop before backing up, and restart the service after the backup process is complete.

  6. If you have used the p4 serverid command to identify your server with a server.id file, the server.id file (which exists in the server's root directory) must be backed up.

Recovery procedures

If the database files become corrupted or lost either because of disk errors or because of a hardware failure such as a disk crash, the database can be re-created with your stored checkpoint and journal.

There are many ways in which systems can fail. Although this guide cannot address all failure scenarios, it can at least provide a general guideline for recovery from the two most common situations, specifically:

  • corruption of your Perforce database only, without damage to your versioned files

  • corruption to both your database and versioned files.

The recovery procedures for each failure are slightly different and are discussed separately in the following two sections.

If you suspect corruption in either your database or versioned files, contact Perforce technical support.

Database corruption, versioned files unaffected

If only your database has been corrupted, (that is, your db.* files were on a drive that crashed, but you were using symbolic links to store your versioned files on a separate physical drive), you need only re-create your database.

You will need:

  • The last checkpoint file, which should be available from the latest P4ROOT directory backup. If, when you backed up the checkpoint, you also backed up its corresponding .md5 file, you can confirm that the checkpoint was restored correctly by comparing its checksum with the contents of the restored .md5 file.

  • The current journal file, which should be on a separate filesystem from your P4ROOT directory, and which should therefore have been unaffected by any damage to the filesystem where your P4ROOT directory was held.

You will not need:

  • Your backup of your versioned files; if they weren't affected by the crash, they're already up to date

To recover the database

  1. Stop the current instance of p4d:

    p4 admin stop

    (You must be a Perforce superuser to use p4 admin.)

  2. Rename (or move) the database (db.*) files:

    mv your_root_dir /db.* /tmp

    There can be no db.* files in the P4ROOT directory when you start recovery from a checkpoint. Although the old db.* files are never used during recovery, it's good practice not to delete them until you're certain your restoration was successful.

  3. Verify the integrity of your checkpoint using a command like the following: 

    p4d -jv my_checkpoint_file

    The command tests the following:

    • Can the checkpoint be read from start to finish?

    • If it's zipped can it be successfully unzipped?

    • If it has an MD5 file with its MD5, does it match?

    • Does it have the expected header and trailer?

    Use the -z flag with the -jv flag to verify the integrity of compressed checkpoints.

  4. Invoke p4d with the -jr (journal-restore) flag, specifying your most recent checkpoint and current journal. If you explicitly specify the server root (P4ROOT), the -r $P4ROOT argument must precede the -jr flag. Also, because the p4d process changes its working directory to the server root upon startup, any relative paths for the checkpoint_file and journal_file must be specified relative to the P4ROOT directory:

    p4d -r $P4ROOT -jr checkpoint_file journal_file

    This recovers the database as it existed when the last checkpoint was taken, and then applies the changes recorded in the journal file since the checkpoint was taken.

Note

If you're using the -z (compress) option to compress your checkpoints upon creation, you'll have to restore the uncompressed journal file separately from the compressed checkpoint.

That is, instead of using:

p4d -r $P4ROOT -jr checkpoint_file journal_file

you'll use two commands:

p4d -r $P4ROOT -z -jr checkpoint_file.gz

p4d -r $P4ROOT -jr journal_file

You must explicitly specify the .gz extension yourself when using the -z flag, and ensure that the -r $P4ROOT argument precedes the -jr flag.

Check your system

Your restoration is complete. See Ensuring system integrity after any restoration to make sure your restoration was successful.

Your system state

The database recovered from your most recent checkpoint, after you've applied the accumulated changes stored in the current journal file, is up to date as of the time of failure.

After recovery, both your database and your versioned files should reflect all changes made up to the time of the crash, and no data should have been lost. If restoration was successful, the lastCheckpointAction counter will indicate "checkpoint completed".

Both database and versioned files lost or damaged

If both your database and your versioned files were corrupted, you need to restore both the database and your versioned files, and you'll need to ensure that the versioned files are no older than the restored database.

You will need:

  • The last checkpoint file, which should be available from the latest P4ROOT directory backup. If, when you backed up the checkpoint, you also backed up its corresponding .md5 file, you can confirm that the checkpoint was restored correctly by comparing its checksum with the contents of the restored .md5 file.

  • Your versioned files, which should be available from the latest P4ROOT directory backup.

You will not need:

  • Your current journal file.

The journal contains a record of changes to the metadata and versioned files that occurred between the last backup and the crash. Because you'll be restoring a set of versioned files from a backup taken before that crash, the checkpoint alone contains the metadata useful for the recovery, and the information in the journal is of limited or no use.

To recover the database

  1. Stop the current instance of p4d:

    p4 admin stop

    (You must be a Perforce superuser to use p4 admin.)

  2. Rename (or move) the corrupt database (db.*) files:

    mv your_root_dir /db.* /tmp

    The corrupt db.* files aren't actually used in the restoration process, but it's safe practice not to delete them until you're certain your restoration was successful.

  3. Compare the MD5 checksum of your most recent checkpoint with the checksum generated at the time of its creation, as stored in its corresponding .md5 file.

    The .md5 file written at the time of checkpointing holds the checksum of the file as it existed before any compression was applied, and assumes UNIX-style line endings even if the service is hosted on Windows. (If your checkpoint file was created with the -z compression option, you may need to decompress them and account for line ending differences.)

  4. Invoke p4d with the -jr (journal-restore) flag, specifying only your most recent checkpoint:

    p4d -r $P4ROOT -jr checkpoint_file

    This recovers the database as it existed when the last checkpoint was taken, but does not apply any of the changes in the journal file. (The -r $P4ROOT argument must precede the -jr flag. Also, because the p4d process changes its working directory to the server root upon startup, any relative paths for the checkpoint_file must be specified relative to the P4ROOT directory.)

    The database recovery without the roll-forward of changes in the journal file brings the database up to date as of the time of your last backup. In this scenario, you do not want to apply the changes in the journal file, because the versioned files you restored reflect only the depot as it existed as of the last checkpoint.

To recover your versioned files

  1. After you recover the database, you then need to restore the versioned files according to your system's restoration procedures (for instance, the UNIX restore(1) command) to ensure that they are as new as the database.

Check your system

Your restoration is complete. See Ensuring system integrity after any restoration to make sure your restoration was successful.

Files submitted to the depot between the time of the last system backup and the disk crash will not be present in the restored depot.

Note

Although "new" files (submitted to the depot but not yet backed up) do not appear in the depot after restoration, it's possible (indeed, highly probable!) that one or more of your users will have up-to-date copies of such files present in their client workspaces.

Your users can find such files by using the following Perforce command to examine how files in their client workspaces differ from those in the depot. If they run...

p4 diff -se

...they'll be provided with a list of files in their workspace that differ from the files Perforce believes them to have. After verifying that these files are indeed the files you want to restore, you may want to have one of your users open these files for edit and submit the files to the depot in a changelist.

Your system state

After recovery, your depot directories might not contain the newest versioned files. That is, files submitted after the last system backup but before the disk crash might have been lost.

  • In most cases, the latest revisions of such files can be restored from the copies still residing in your users' client workspaces.

  • In a case where only your versioned files (but not the database, which might have resided on a separate disk and been unaffected by the crash) were lost, you might also be able to make a separate copy of your database and apply your journal to it in order to examine recent changelists to track down which files were submitted between the last backup and the disk crash.

In either case, contact Perforce Technical Support for further assistance.

Ensuring system integrity after any restoration

After any restoration, use the command:

p4 counter lastCheckpointAction

to confirm that the lastCheckpointAction counter has been updated to reflect the date and time of the checkpoint completion.

You should also run p4 verify to ensure that the versioned files are at least as new as the database:

p4 verify -q //...

This command verifies the integrity of the versioned files. The -q (quiet) option tells the command to produce output only on error conditions. Ideally, this command should produce no output.

If any versioned files are reported as MISSING by the p4 verify command, you'll know that there is information in the database concerning files that didn't get restored. The usual cause is that you restored from a checkpoint and journal made after the backup of your versioned files (that is, that your backup of the versioned files was older than the database).

If (as recommended) you've been using p4 verify as part of your backup routine, you can run p4 verify after restoration to reassure yourself that the restoration was successful.

If you have any difficulties restoring your system after a crash, contact Perforce Technical Support for assistance.