Using parallel processing for submits, syncs, and shelves

Submits and syncs

You can configure the server to allow parallel file transfer for submit and sync processing. Parallel processing is most effective with long-haul, high-latency networks, or with other network configurations that prevent the use of available bandwidth with a single TCP flow. In addition, parallel processing between a client running on a multi-core machine and the server allows the uncompression of large compressed binary files to happen in parallel on the client.

• Use the net.parallel.max configurable to:

  • Transfer files in parallel during the submit process.
  • Speed up sync processing by having the p4 sync command transfer files using multiple threads. You do this by setting the net.parallel.max configuration variable to a value greater than 1 and by using the --parallel option to the p4 sync command. See Parallel processing in the p4 sync comand in Helix Core Command-Line (P4) Reference.
• Use the net.parallel.submit.threads configurable to specify the number of threads for sending files in parallel for each submit (P4V 2017.3 and later). For details about setting up a service user, P4TRUST, and ExternalAddress for the commit server to pull files from the edge server, see Parallel submits the p4 submit command in Helix Core Command-Line (P4) Reference. For information about service users, see Types of users in Helix Core Command-Line (P4) Reference.
• Use the net.parallel.threads configurable to turn on parallel sync in a server. This parameter specifies the number of independent network connections that can be used for syncing files concurrently for each sync. When this parameter is set, parallel sync is automatically enabled in P4V as well (P4V 2017.3 and later).
• To reduce lock contention during parallel syncs, set the client.sendq.dir configurable.

Shelves

A p4 shelve command might execute more rapidly by transferring multiple files in parallel.

To enable automatic parallel shelving, set the net.parallel.shelve.threads configurable to a value that is less than or equal to the value of the net.parallel.max configurable.

Optionally, set the net.parallel.shelve.min configurable to change the minimum number of files in a parallel shelve, which, by default, is 9.

Optionally, set the net.parallel.shelve.batch configurable to change the number of files in a parallel shelve, which, by default, is 8.

To disable automatic parallel shelving, unset the net.parallel.shelve.threads configurable.