Journal files

The current journal A file containing a record of every change made to the metadata since the time of the last checkpoint. This file grows as each transaction is logged. is the running transaction log that keeps track of all the Helix Core Server database modifications since the previous checkpoint A backup copy of the underlying metadata at a particular moment in time. A checkpoint can recreate db.user, db.protect, and other db.* files. See also metadata. or journal. Journal file rotation is an incremental metadata backup, and the checkpoint is a full metadata backup.

Tip

Journals can become large. Frequent journal rotation reduces the size of each journal file. Consider daily journal rotation and weekly checkpoints.

To restore the server database to the most recent point in time, you need:

The most recent checkpoint.
Any journal files created since that checkpoint.
The journal file that was active at the time the server was stopped.

For example, suppose it is Wednesday and you have:

The weekly checkpoint file that you create before your users start work on Monday.
The daily journal files for Monday and Tuesday.
The currently running journal file of Wednesday.

If a disk crash occurs on Wednesday that corrupts the server database, you can restore the server database by using Monday's checkpoint file, the rotated journal files for Monday and Tuesday, and Wednesday's journal file.

Journal file names

The journal file grows until it is rotated by a checkpoint or a journal rotation command, such as p4 admin journal. If the current journal is extremely large, a checkpoint is needed.

Every checkpoint, including your first checkpoint, starts a new journal file and renames the old journal file. We call this journal rotation The process of renaming the current journal to a numbered journal file as part of a checkpoint..

Rotated journals are in the P4ROOT directory and named journal.n, where

journal. represents the value of the journalPrefix configurable with the suffix .jnl.
n is the incrementing journal number

For example, if the journalPrefix is /p4/1/logs/master, the first three rotated journals are stored as

/p4/1/logs/master.jnl.1

/p4/1/logs/master.jnl.2

/p4/1/logs/master.jnl.3

For an example of making a checkpoint and rotated journal file, see the steps in the Backup procedure topic.

Journal rotation and filsys.atomic.rename

On Windows Server 2016 and newer, journal rotation can be performed using an atomic rename instead of a copy and truncate operation. To enable this feature, see the filesys.atomic.rename configurable in the Helix Core Command-Line (P4) Reference.

Location

Journaling is automatically enabled on all Windows and Linux platforms. By default, the current journal filename is journal, and the file resides in the P4ROOT directory. However, if a failure corrupts that root directory, your journal file will be inaccessible.

Warning

To reduce risk, make sure that the journal file and the Helix Core Server root are located on different file systems, ideally on different physical drives.

To change the name and location of the journal, do one of the following:

Specify the name and path of the journal file in the P4JOURNAL environment variable.
Include the -J filename option when starting p4d and make sure that subsequent checkpoints use the same file, or the journal will not be properly renamed

Tip

To restore your database, you only need to keep the most recent journal file accessible. Consider also archiving old journals with old checkpoints. This makes it possible to restore to an older checkpoint.

Whether you use P4JOURNAL or the -J journalfile option to p4d, the journal filename can be provided either as an absolute path, or as a path relative to the server root.

Example Specify journal files

Starting the service with:

p4d -r $P4ROOT -p 1666 -J /usr/local/perforce/journalfile
Perforce Server starting...

requires that you either checkpoint with:

p4d -r $P4ROOT -J /usr/local/perforce/journalfile -jc
Checkpointing to checkpoint.19...
Saving journal to journal.18...
Truncating /usr/local/perforce/journalfile...

or set P4JOURNAL to /usr/local/perforce/journalfile and use the following command:

p4d -r $P4ROOT -jc
Checkpointing to checkpoint.19...
MD5(checkpoint.19)=48769A82387B04987568309823E784C9
Rotating /usr/local/perforce/journalfile to journal.18

Important

Make sure your P4JOURNAL environment variable (or command-line specification) matches the setting used when you started the Perforce service. Otherwise the journal is neither saved nor truncated.

Checkpoint and journal history

You can use the p4 journals command to display the history of checkpoint and journal activity for the server. This history includes information about the following events:

The server takes a checkpoint.
Journal rotation.
Journal replay.
Checkpoint scheduling.

To learn more, see the description of command output and options in the p4 journals command in the Helix Core Command-Line (P4) Reference.

Verify integrity

You can verify the integrity of a checkpoint using the p4d -jv command.

Automate maintenance

See Triggering on journal rotation.

Disable journaling

To disable journaling, stop the service, remove the existing journal file (if it exists), set the environment variable P4JOURNAL to off, and restart p4d without the -J option.

Journal notes

The presence of journal notes in journals and checkpoints allows later users of the journal or checkpoint to examine the notes and use the information. For example, replica servers might take action based on the journal notes they process.

Journal note fields

Journal notes have three common fields: type, date, and version.

type = Journal note type.

date = Timestamp indicating when the note was written.

version = Server version that wrote the note. To learn more, see Releases in the Helix Core Server Schema Documentation.

Journal notes have additional information based on the type of journal note. This information is in the form of a set of integer (i1-i5) and string (s1-s5) flags.

For example:

@nx@ type date @version@ i1 i2 i3 i4 i5 @s1@ @s2@ @s3@ @s4@ @s5@

where @nx@ is additional metadata for the benefit of journal replication tools.

Journal Note Types

Type	Name	Flags
0	Checkpoint Header	i1 = case-handling and unicode settings s1 = database root s2 = journal file name
1	Checkpoint Trailer
2	Journal Header	i1 = case-handling and unicode settings i2 = journal type s1 = database root s2 = journal file name s3 = server version - of server that created the journal dump (might be blank) s4 = serverid - blank unless identifying the server that this journal dump applies to journal type field (i2) shows how the journal file was produced: 0 = normal journal 1 = p4d -jd 2 = p4d -jds 3 = p4d -xx (jnl.fix) 4 = journaldbchecksums -u
3	Journal Trailer
4	Table Summary	i1 = table version i2 = old version i3 = table checksum i4 = table bad-unlock count s1 = table name
5	Server Upgrade	i1 = start value of upgrade counter i2 = destination value of upgrade counter
6	Table Upgrade	i1 = upgrade being performed s1 = name of the upgrade function called
7	Server Startup	s1 = server port s2 = server root s3 = server name s4 = serverid
8	Server Shutdown	s1 = server port
9	Unicode Enabled	used in journal file only
10	Table Patched
11	Journal Replayed	i1 = flags in effect during journal replay s1 = name of the journal/checkpoint replayed
12	Table Checksum	i1 = table version i2 = table checksum i3 = table bad-unlock count i4 = table generation s1 = table name
13	Unload Header	i1 = domain type (client/label/stream) s1 = domain name s2 = unload filename
14	Unload Trailer
15	Change Checksum	i1 = change number i2 = number of revisions in change i3 = written-by: journaldbchecksums (0), submit (1), populate (2), fetch/push/unzip (3) s1 = change checksum
16	Table Dump	i1 = table version i2 = table checksum i3 = number of records i4 = isCompressed s1 = table name s2 = dump file name
17	Block Checksum	i1 = table version i2 = block number i3 = block size i4 = dump version s1 = table name s2 = first key in block s3 = last key in block s4 = block checksum
18	Master Up	s1 = port s2 = name (might be blank) s3 = serverid (might be blank)
19	Master Stopping	s1 = port s2 = name (might be blank) s3 = serverid (might be blank)
20	Master Transition	s1 = port s2 = name (might be blank) s3 = serverid (might be blank)
21	Standby Up	s1 = port s2 = name (might be blank) s3 = serverid (might be blank)
22	Workspace Up	s1 = port s2 = name (might be blank) s3 = serverid (might be blank)
23	Replica Transaction	n s1 = serverid of this replica s2 = current statefile position in this replica i1 = last pid seen in @ex@ or @mx@ record from target i2 = last timestamp seen in @ex@ or @mx@ record from target
24	User Rename	s1 = old user name s2 = new user name
25	User Delete	s1 = user name s2 = user name
26	Seed Header	s1 = serverid of the replica being seeded i1 = database flags i2 = last checkpoint number i3 = seed journal number
27	Seed Trailer
28	Storage upgrade	Used during 2019.1 upgrade process.
29	Background archive transfer	i1: `1` turned on i1: `0` turned off
30	Restricted standby status	i1 = `1` failback was successful, and the former restricted standby is no longer a restricted standby. i1 = `0` failback failed after a note was sent already with i1 = 1, so the standby remains a restricted standby.
31	Remove partitioned database table	s1: table name
32	Replica file revision status request	s1: depotFile s2: depotRev s3: change s4: expected digest i1: VerifyDepotCode 0 == None, 1 == archive, 2 == unload, 3 == shelf
33	Notify edge servers about `p4 obliterate`	i1: `1` = `-h` option of `p4 obliterate`, which on 2024.2 and later purges the `db.have` table on edge servers if `-h` is not selected. i1: `2` = `-l` option of `p4 obliterate`, which was introduced in 2024.2.

Usage Examples

Each checkpoint contains a checkpoint header note:

@nx@ 0 1296517759 @30@ 2 0 0 0 0 @/p4root@ @journal@ @@ @@ @@

and a checkpoint trailer note:

@nx@ 1 1296517767 @30@ 0 0 0 0 0 @@ @@ @@ @@ @@

The checkpoint header note includes the case-handling and unicode flags in effect when the checkpoint was taken. This allows Helix Core Server to ensure that the same flags are in effect when the checkpoint is restored:

p4d -r /p4root -jr checkpoint.9
Perforce db files in '.' will be created if missing...
Recovering from checkpoint.9...
Perforce server error:
Journal file 'checkpoint.9' replay failed at line 1!
Bad transaction marker!
Case-handling mismatch: server uses Windows-style (-C1) but journal flags are Unix-style (-C0)!

Using the server version in the checkpoint header note, a warning is issued when restoring a checkpoint created by a newer version of server:

p4d -r /p4root -jr checkpoint.9
Recovering from checkpoint.9...
Perforce server info:
Server version 30 is replaying a version 31 journal.

Journal notes are independent of the other contents of a checkpoint or journal. In case of a problem, journal notes can be removed. This example uses the grep utility to remove all lines that have @nx@ at the beginning.

grep -v '^@nx@' checkpoint.N > checkpoint.N.new

An example of a checkpoint that did not complete successfully due to an error:

p4 journals -m 3
... start 1715090494
... startDate 2024/05/07 14:01:34
... end 1715090495
... endDate 2024/05/07 14:01:35
... pid 2045216
... type checkpoint
... flags  (admin checkpoint)
... jnum 85
... jfile /.ckp.85
... jdate 0
... jdateDate 1970/01/01 00:00:00
... jdigest D41D8CD98F00B204E9800998ECF8427E
... jsize -1
... jtype text
... failed 1
... errmsg open for write: /.ckp.85: Permission denied

To determine how long it took to take the checkpoint, subtract the timestamps of the checkpoint header and trailer. In this example, the endDate is 20 minutes after the startDate:

... start 1713781651
... startDate 2024/04/22 10:30:01
... end 1713781651
... endDate 2024/04/22 10:50:01
... pid 1611629
... type checkpoint
... flags  (admin checkpoint)
... jnum 82
... jfile /home/giles/depots/depot23200/Journals/commit.ckp.82
... jdate 1713781651
... jdateDate 2024/04/22 10:50:01
... jdigest C89CC8CF4E185A8468AA2D63E851D41A
... jsize 1643548
... jtype text