Failback after failover

After a Failover with master server participation, failback restores the previous role of that master server. After a successful failback,

  • the pre-failover master server is restored to its original role as the master server

  • (optional) the pre-failover standby server is restored to its original role of standby server

Prerequisites for using the failback command

  • Both the master and the standby servers must be version 2022.1 or later.

  • The original p4 failover command must have been performed with master participation.

  • The p4 failback command must be performed with master participation.

  • When the master is an edge server, the failback command requires commit server participation.

  • The server ID for the failed-over server must not be manually altered prior to performing the Procedure for using the failback command.

  • Journals created after failover or failback must still exist in their expected location before the standby-to-be is started as a standby.

  • We recommend that a DNS alias point to the IP address of the master server. This allows the same DNS alias to point the new master server (former standby server).

  • We recommend that a DNS alias point to the IP address of the standby server.

Procedure for using the failback command

1.

 

On the machine where the pre-failover master was running, convert the pre-failover master server to a standby.

  1. Preview the p4d -Fm command described at Failback options:
    p4d -r p4root -Fm masterServerID standbyServerID
    where
    masterServerID and standbyServerID refer to the pre-failover server IDs of the master and standby, and
    the p4root location is the P4ROOT location of the pre-failover master server.

  2. If the preview looks acceptable, include the -y option to perform the operation:
    p4d -r p4root -y -Fm masterServerIDstandbyServerID

Note

This command changes this server to a restricted standby, which allows it to ignore configuration settings that would interfere with its operation as the target for the failback command.
2. Start the newly converted standby server and wait for its replication to catch up with the master.
3. (Optional) Change the standby server spec to make it a mandatory standby.

4.

 

After the completion of a successful failover,

  1. Verify that the former standby (or forwarding standby) has been restarted as the new master by issuing the p4 info command and checking the ServerID to ensure that it is the same ServerID that the previous master server used.

  2. Site-specific changes might be needed to use the new master server. It might be necessary to make DNS changes so that users and replicas can connect to the new master server. For example,

If you have a DNS alias set up for the master server If you do not have a DNS alias set up for the master server

Update the IP address of that DNS alias to point to the IP address of its new location.

After this is done, users will be able to connect to the master using the same P4PORT as before.

The DNS alias for the standby must be changed to point to the IP address of the master before failover.

 

Update the P4TARGET environment variable and server specifications to use the correct host. The port number should remain the same, but the host name must be changed to use its new location.

  • Change your P4PORT to point to the new master host and the same port number so that you can connect to the new master.

  • On the new master server, change the P4TARGET for each replica or edge server by issuing the p4 configure show allservers command and then issuing the p4 configure set "replica-name#P4TARGET=new-master-server:port-number" command.

  • Update each replica's own P4TARGET by issuing the p4d -r $P4ROOT "-cset replica-name#P4TARGET=new-master-server:port-number"command.

  • Update your server specifications with the proper hostname and port number by issuing the p4 server servername command.

  • Inform your users if they need to update their P4PORT to connect to the new host location of the master. The port number should remain the same as before, and your users can now issue new commands.

 
5.

(Optional) On the machine where the pre-failover standby was running, restore the pre-failover standby to its former status of standby for the restored master.

  1. Preview the p4d -Fs command described at Failback options:
    p4d -r p4root -Fs masterServerID standbyServerID
    where
    masterServerID and standbyServerID refer to the pre-failover server IDs of the master and standby, and
    the p4root location is the P4ROOT location of the pre-failover standby.

  2. If the preview looks acceptable, include the -y option to perform the operation:
    p4d -r p4root -y -Fs masterServerIDstandbyServerID

Note

This command changes this server to a restricted standby. When this server discovers from replication that p4 failback has been successfully run, it will function as an unrestricted standby for the restored master.
6. (Optional) Change the standby server spec to make it a mandatory standby.

If p4 failback cannot be used

Failback might still be possible even if the Prerequisites for using the failback command are not fully met.

Prerequisites if p4 failback cannot be used

Ensure that the original commit or master server is:

  • reconfigured as a standby before starting it

  • assigned the serverid of a standby server, and that this serverid is different from the serverid of the current commit or master server

Reseeding the original commit or master server

If p4 failback cannot be used, it is a best practice to reseed the original commit or master from the new commit or master before performing the Failback steps if p4 failback cannot be used. Consider the following:

If the original commit or master did not participate in the failover ... If the original commit or master participated in the failover ...
... the old commit or master's metadata might contain transactions that did not make it to the original standby at the time of failover. Those transactions could reappear at failback. To avoid this possibility, reseed the original commit or master from the new commit or master before performing the failback steps. ... reseeding might not be necessary. However, if you have any doubts about metadata integrity, the safest option is to reseed the original commit or master from the new commit or master before performing the failback steps.

Failback steps if p4 failback cannot be used

At the new commit or master server

  1. Verify that the standby is pulling from new commit or master server by issuing p4 servers -J
  2. Check the result, which might be something like:

    commit '2019/07/09 16:41:36' commit-server 40/13642 40/13642 wadL/1 1

    standby '2019/07/09 16:41:31' standby 40/10000 40/10000 wAdl/4 1

    where 10000 is lower than 13642, which indicates that the standby is not yet fully caught up with the commit or master server.

  3. Wait a moment, then reissue p4 servers -J to verify that standby is fully caught up with the commit or master server. For example:

    commit '2019/07/09 16:41:36' commit-server 40/13642 40/13642 wadL/1 1

    standby '2019/07/09 16:41:36' standby 40/13642 40/13642 wAdl/4 1

At the new standby that was the original commit or master server

  1. Issue the failover command: p4 failover
  2. Follow the steps at Failover.