Standby and forwarding-standby server

Helix Core server 2019.1 introduced replication improvements with a standby server (or a forwarding standby server), which provide advantages over using a Read-only replica:

no transactions are lost if the master fails
the p4 pull command no longer needs to acquire database locks
all journal records are pulled before the p4 pull command goes to sleep

A standby server splits the pull command process into the p4 journalcopy and p4 pull -L commands.

The p4 journalcopy thread:

writes journal records to a local numbered journal file. As a result, no database tables are locked
uses a file in the server root directory, statejcopy, to track the last position read in the source server journal
updates the statejcopy file in standby server

In replicas other than standby servers, the pull thread updates the state file.

The standby journal:

is a complete copy of the master journal
always appears as a numbered journal, such as journal.41, even for the current journal

Creating a Standby Server

Prerequisite: A server from which a checkpoint can be taken. A standby server must be created by using a checkpoint of its master. Conversion of a replica of any other type to a standby is not supported. In what follows, we use the example of a commit server to act as the master for a newly-created standby server.

Preparing the commit server

On the commit server, add a service user (if needed) and assign its Type to be service. The service user name can be any legal Perforce user name. (See Service users)
p4 user -f serviceuser [...] Type: Service
Save the user specification and close the editor.
Provide the serviceuser with a password:

p4 passwd serviceuser
Create a service group, add the serviceuser to the group, and set the Timeout field to unlimited:
```
p4 group service
[...]
Timeout: unlimited
[...]
Users:
    serviceuser
```
Save the group specification and close the editor to save the user.
Add the service group to the protections table with super capabilities:
```
p4 protect
[...]
super group service * //...
```

Set up the configuration for the standby server using a server spec form. For example:

ServerID:     commit-standby
Type:         server
Address:      {standbyserver host}:{port number}
Services:     standby
Options:      nomandatory
ReplicatingFrom:     {commit-server-ID}
Description:    Standby server for {commit-server-ID}.
DistributedConfig:
	any#auth.default.method=ldap
	any#auth.ldap.order.1=popeye
	any#auth.ldap.userautocreate=1
	any#db.monitor.shared=8192
	...
	any#server.allowpush=2
	P4TARGET={server:port}
	P4TICKETS={/path/to/p4tickets-file}
	P4LOG={/path/to/logfile}
	db.replication=readonly
	lbr.replication=readonly
	journalPrefix={/path/to/rotated/journal/commit-standby}
	monitor=1
	rpl.journalcopy.location=1
	serviceUser={serviceuser-name}
	startup.1=journalcopy -i 1
	startup.2=pull -L -i 1
	startup.3=pull -u -i 1

Note
Under DistributedConfig:
the values in bold and {italics} represent values specific to your server
entries starting any# are configurables that have already been set on the commit server to apply to all servers. These entries cannot be changed in this form.
The Options: field must be set to nomandatory until the standby is running and up to date. This field can then be changed to mandatory. See the Failover topic.

Take a checkpoint of the commit server. You will it use to create the standby server that can become the commit during a Failover process.

Preparing the standby server

Run the following command to restore metadata from the specified checkpoint:
p4d -r /standby/p4root -jr commit.ckp.12345
Set the server ID for the newly-seeded standby server:

p4d -r /standby/p4root -xD commit-standby
Log the service user into the commit server from the machine the standby server is running on:
p4 -E P4TICKETS={/path/to/p4tickets-file} -u {serviceuser-name} -p {commit-server:port} login
Start the commit standby replica.
Confirm the server status with the command:
p4 -p {standbyserver:port} -Ztag info
Confirm that serverServices is set to standby and that replica is showing the server and port number of the commit server.

p4 -ztag info [...] ... ServerID commit-standby [...] ... replica commit-server:1666
Important
If you want the standby server to be capable of becoming the new master during Failover, fill out a Duplicate Server Request form at https://www.perforce.com/support/duplicate-server-request. Having the license ready reduces the risk of delay during a failover process.

Monitoring a Standby Server

Use p4 servers -J to check the replication status of all standby replicas. For example:

p4 servers -J
commit-standby '2019/09/20 12:49:16' standby 1234/8779 1234/8779 wadL/1 1

If you want verbose output, use the -Ztag option:

p4 -Ztag servers -J
... ServerID commit-standby
... Updated 2019/09/20 12:49:16
... ServerType standby
... ServerOptions nomandatory
... PersistedJournal 1234
... PersistedSequence 8779
... AppliedJournal 1234
... AppliedSequence 8779
... JAFlags wadL/1
... IsAlive 1

Use p4 journalcopy -l to determine the current copy position as detected by the replica:

p4 journalcopy -l
Current replica persisted journal state is: Journal 1234, Sequence 8779.

To check the status of the replica metadata replication, use the p4 pull -lj command:

p4 pull -lj Current replica journal state is: Journal 1234, Sequence 8779. Current master journal state is: Journal 1234, Sequence 8779. The statefile was last modified at: 2019/09/20 12:49:16. The replica server time is currently: 2019/09/20 13:20:11 -0700 PDT