Perforce 2005.2 System Administrator's Guide


<< Previous Chapter Scripting Perforce: Triggers and Daemons	Table of Contents Index Perforce on the Web	Next Chapter >> Perforce and Windows

Chapter 7
Tuning Perforce for Performance

Your Perforce server should normally be a light consumer of system resources. As your installation grows, however, you may wish to revisit your system configuration to ensure that it is configured for optimal performance.

The following chapter briefly outlines some of the factors that can affect the performance of a Perforce server, provides a few tips on diagnosing network-related difficulties, and offers some suggestions on decreasing server load for larger installations.

Tuning for Performance

The following variables can affect the performance of your Perforce server.

Memory

Server performance is highly dependent upon having sufficient memory. Two bottlenecks are relevant: the first can be avoided by ensuring that the server doesn't page when running large queries, and the second by ensuring that the db.rev table (or at least as much of it as practical) can be cached in main memory.

Determining memory requirements for large queries is fairly straightforward: the server requires about 1KB/file of RAM to avoid paging; 10,000 files will require 10MB of RAM.
To cache db.rev, the size of the db.rev file in an existing installation can be observed and used as an estimate. New installations of Perforce can expect db.rev to require about 150-200 bytes per revision, and roughly 3 revisions per file, or about 0.5KB of RAM per file.

Thus, if there is 1.5KB of RAM available per file, or 150MB for 100,000 files, the server will not page, even when performing an operation involving all files. It is still possible that multiple large operations will be performed simultaneously and thus require more memory to avoid paging. On the other hand, the vast majority of operations will only involve a small subset of files.

For most installations, a system with enough RAM for 1.5KB per file in the depot will suffice.

Filesystem performance

Perforce is judicious with regards to its use of disk I/O; its metadata is well-keyed and accesses are mostly sequential scans of limited subsets of the data.

The only disk-intensive activity is file check-in, where the Perforce server must write and rename files in the archive. Server performance depends heavily upon the operating system's filesystem implementation, and in particular, whether directory updates are synchronous.

Although Perforce does not recommend any specific filesystem, Linux servers are generally fastest (owing to Linux's asynchronous directory updating), but may have poor recovery if power is cut at the wrong time. The BSD filesystem (also used in Solaris) is relatively slow, but much more reliable. NTFS performance falls somewhere in between these two ranges. The filesystems used by IRIX and OSF have demonstrated an excellent combination of both speed and robustness.

Performance in systems where database and versioned files are stored on NFS-mounted volumes is typically dependent on the implementation of NFS in question and/or the underlying storage hardware. Perforce has been tested and is supported under the Solaris implementation of NFS.

Under Linux and FreeBSD, database updates over NFS can be an issue as file locking is relatively slow; if the journal is NFS-mounted on these platforms, all operations will be slower. In general (but in particular on Linux and FreeBSD) we recommend that the Perforce database, depot, and journal files be stored on disks local to the machine running the Perforce server process.

These issues affect only the Perforce Server process (p4d). Perforce client programs, (such as p4, the Perforce Command-Line Client) have always been able to work with client workspaces on NFS-mounted drives (for instance, workspaces in users' home directories).

Disk space allocation

Perforce disk space usage is a function of three variables:

Number and size of client workspaces
Size of server database
Size of server's archive of all versioned files

All three variables depend on the nature of your data and how heavily you use Perforce.

The client file space required is the size of the files that your users will need in their client workspaces at any one time.

The server's database size can be calculated with a fair level of accuracy; as a rough estimate, it requires 0.5KB per user per file. (For instance, a system with 10,000 files and 50 users will require 250MB of disk space for the database). The database can be expected to grow over time as histories of the individual files grow.

The size of the server's archive of versioned files depends on the sizes of the original files stored and grows as revisions are added. For most sites, allocate space equivalent to at least three times the aggregate size of the original files.

If you anticipate your database growing into the gigabyte range, you should ensure that your platform has adequate support for large filesystems. See "Allocate disk space for anticipated growth" on page 20.

The db.have file holds the list of files opened in client workspaces, and tends to grow more rapidly than other files in the database. If you are experiencing issues related to the size of your db.have file and are unable to quickly switch to a server with adequate support for large files, deleting unused client workspace specifications and reducing the scope of client workspace views can help alleviate the problem.

Network

Perforce can run over any TCP/IP network. Although we have not yet seen network limitations, the more bandwidth the better. Presumably FDDI would be better than 10Mb/s Ethernet, but some users have reported that using a T1 (1.5 Mb/s) provides response times comparable to using Perforce locally. Perforce employees work successfully over ISDN (64 Kb/s) lines.

Perforce uses a TCP/IP connection for each client interaction with the server. The server's port address is defined by P4PORT, but the TCP/IP implementation picks a client port number. After the command completes and the connection is closed, the port is left in a state called TIME_WAIT for two minutes. While the port number ranges from 1025 to 32767, generally only a few hundred or thousand can be in use simultaneously. It is therefore possible to occupy all available ports by invoking a Perforce client command many times in rapid succession, such as with a script.

Before release 99.2, both the server and client side of the connection remained in TIME_WAIT, which meant that a script running on one user's machine could deprive other users of service by tying up all available ports on the server side. As of Release 99.2, only the client side goes into TIME_WAIT, leaving the Perforce server free to handle other clients.

CPU

Perforce is based on a client/server architecture. Both the client and server are lightweight in terms of CPU resource consumption. By way of example, a server supporting 80 users on a low-end (140 MHz) SPARC Ultra server can use as little as 7 CPU-minutes per day, or about 0.5% of available processing power. Weighting this for peak use and headroom, such a server could support upwards of 800 users.

In general, CPU power is not a major consideration when determining the platform on which to install a Perforce server.

Diagnosing Slow Response Times

Perforce is normally a light user of network resources. While it is possible that an extremely large user operation could cause the Perforce server to respond slowly, consistently slow responses to p4 commands are usually caused by network problems. Any of the following may cause slow response times:

misconfigured domain name system (DNS)
misconfigured Windows networking
difficulty accessing the p4 executable on a networked file system

A good initial test is to run p4 info. If this does not respond immediately, then there is a network problem. Although solving network problems is beyond the scope of this manual, here are some suggestions for troubleshooting them.

Hostname vs. IP address

On a client machine, try setting P4PORT to the server's IP address instead of its hostname. For example, instead of using

P4PORT=host.domain:1666

try using:

P4PORT=1.2.3.4:1666

with your site-specific IP address and port number.

On most systems, you can determine the IP address of a host by invoking:

ping hostname

If p4 info responds immediately when you use the IP address, but not when you use the hostname, the problem is likely related to DNS.

Try p4 info vs. P4Win

If you are using P4Win, you can compare the response time of P4Win's "Show Connection Info" (Help -> Show Connection Info) with the response time from the p4 info command.

If P4Win returns the information quickly, but p4 info is slow, you have a DNS-related problem. (When the Perforce server receives a p4 info request from either the command line or from P4V, the server performs a reverse name lookup in order to send back both the client and server hostnames along with other configuration information. When the server receives a P4Win"Show Connection Info" request, however, it simply returns the IP addresses.)


Note	This test is only valid for Release 99.1 and newer servers. Perforce Servers prior to 99.1 always perform a reverse name lookup.

Windows wildcards

In some cases, p4 commands using unquoted filepatterns with a combination of depot syntax and wildcards, such as:

p4 files //depot/*

can result in a delayed response on Windows. You can prevent the delay by putting double quotes around the file pattern, like so:

p4 files "//depot/*"

The cause of the problem is the p4 command's use of a Windows function to expand wildcards. When quotes are not used, the function interprets //depot as a networked computer path and spends time in a futile search for a machine on the network named depot.

DNS lookups and the hosts file

On Windows, the %SystemRoot%\system32\drivers\etc\hosts file can be used to hardcode IP address-hostname pairs. You may be able to work around DNS problems by adding entries to this file.

The corresponding UNIX file is /etc/hosts.

Location of the "p4" executable

If none of the above diagnostic steps explains the sluggish response time, it's possible that the p4 executable itself is on a networked file system which is performing very poorly. To check this, try running:

p4 -V

This merely prints out the version information, without attempting any network access. If you get a slow response, network access to the p4 executable itself may be the problem. Copying or downloading a copy of p4 onto a local filesystem should improve response times.

Preventing Server Swamp

Generally, Perforce's performance depends on the number of files a user tries to manipulate in a single command invocation, not the size of the depot. That is, syncing a client view of 30 files from a 3,000,000-file depot should not be much slower than syncing a client view of 30 files from a 30-file depot.

The number of files affected by a single command is largely determined by:

p4 command line arguments (or selected folders in the case of GUI operations).

Without arguments, most commands will operate on, or at least refer to, all files in the view.

Client views, branch views, label views, and protections.

Because commands without arguments operate on all files in the view, it follows that the use of unrestricted views and unlimited protections can result in commands operating on all files in the depot.

When the server answers a request, it locks down the database for the duration of the computation phase. For normal operations, this is a successful strategy, as it can "get in and out" quickly enough to avoid a backlog of requests. Abnormally large requests, however, can take seconds, sometimes even minutes. If frustrated users hit CTRL-C and retry, the problem gets even worse; the server consumes more memory and responds even more slowly.

At sites with very large depots, unrestricted views and unqualified commands will make a Perforce server work much harder than it needs to. Users and administrators can ease load on their servers by:

Using "tight" views
Assigning protections
Limiting maxresults
Writing efficient scripts
Using compression efficiently

Using tight views

The following "loose" view is trivial to set up but could invite trouble on a very large depot:


`//depot/... //workspace/...`

In the loose view, the entire depot was mapped into the client workspace; for most users, this can be "tightened" considerably. The following view, for example, is restricted to specific areas of the depot:


`//depot/main/srv/devA/... //workspace/main/srv/devA/... //depot/main/drv/lport/... //workspace/main/dvr/lport/... //depot/rel2.0/srv/devA/bin/... //workspace/rel2.0/srv/devA/bin/... //depot/qa/s6test/dvr/... //workspace/qa/s6test/dvr/...`

Client views, in particular, but also branch views and label views, should also be set up to give users just enough scope to do the work they need to do.

Client, branch, and label views are set by a Perforce administrator or by individual users with the p4 client, p4 branch, and p4 label commands respectively.

Two of the techniques for script optimization (described in "Using branch views" on page 123 and "The temporary client workspace trick" on page 124) rely on similar techniques. By limiting the size of the view available to a command, fewer commands need to be run, and when run, the commands require fewer resources.

Assigning protections

Protections (see "Administering Perforce: Protections" on page 73) are actually another type of Perforce view. Protections are set with the p4 protect command and control which depot files can be affected by commands run by users.

Unlike client, branch, and label views, however, the views used by protections can be set only by Perforce superusers. (Protections also control read and write permission to depot files, but the permission levels themselves have no impact on server performance.) By assigning protections in Perforce, a Perforce superuser can effectively limit the size of a user's view, even if the user is using "loose" client specifications.

Protections can be assigned to either users or groups. For example:


`write user sam * //depot/admin/... write group rocketdev * //depot/rocket/main/... write group rocketrel2 * //depot/rocket/rel2.0/...`

Perforce groups are created by superusers with the p4 group command. Not only do they make it easier to assign protections, but they provide useful fail-safe mechanisms in the form of maxresults and maxscanrows, described in the next section.

Limiting database queries

Each Perforce group has an associated maxresults and maxscanrows value. The default for each is "unlimited", but a superuser can use p4 group to limit it for any given group.

Users in such groups are unable to run any commands which affect more database rows than the group's maxresults limit. (For most commands, the number of database rows affected is roughly equal to the number of files affected.)

Like maxresults, maxscanrows prevents certain user commands from placing excessive demands on the server. (For most commands, the number of rows that could be scanned is roughly equal to the number of files affected, multiplied by the average number of revisions per file in the depot.)

To set these limits, fill in the Maxresults: or Maxscanrows: field in the p4 group form. If a user is listed in multiple groups, the highest of the maxresults (or maxscanrows) limits (but not including the default "unlimited" setting) for those groups is taken as the user's maxresults (or maxscanrows) value.

Example:

Effect of setting maxresults and maxscanrows:

As an administrator, you wish members of the group rocketdev to be limited to operations of 20,000 files or less, and to scan no more than 100,000 revisions:

Group: rocketdev Maxresults: 20000 Maxscanrows: 100000 Timeout: 43200 Subgroups: Users: bill ruth sandy

`Group: rocketdev Maxresults: 20000 Maxscanrows: 100000 Timeout: 43200 Subgroups: Users: bill ruth sandy`

Suppose that Ruth has an unrestricted ("loose") client view. When she types:

p4 sync

her sync command is rejected if the depot contains more than 20,000 files. She can work around this limitation either by restricting her client view, or, if she needs all of the files in the view, by syncing smaller sets of files at a time, as follows:

p4 sync //depot/projA/... p4 sync //depot/projB/...

Either method enables her to get her files, but without tying up the server to process a single extremely large command.

If Ruth tries a command that scans every revision of every file, such as:

p4 filelog //depot/projA/...

and there are less than 20,000 files, but more than 100,000 revisions (perhaps the projA directory contains 8000 files, each of which has 20 revisions), the maxresults limit does not apply, but the maxscanrows limit does.

To remove any limits on the number of result lines processed (or database rows scanned) for a particular group, set the Maxresults: or Maxscanrows: value for that group to unlimited.

As these limitations can make life difficult for your users, do not use them unless you find that certain operations are slowing down your server. The Maxresults: value should never be less than 10,000, since certain operations performed by P4Win, the Perforce Windows Client, may require a Maxresults: value of between 5,000 and 8,000. Similarly, Maxscanrows should rarely need to be set below 50,000.

For more information, including a comparison of Perforce commands and the number of files they affect, type:

p4 help maxresults p4 help maxscanrows

from the command line.

Maxresults and maxscanrows for users in multiple groups

As mentioned earlier, if a user is listed in multiple groups, the highest maxresults limit of all the groups a user belongs to is the limit that affects the user. The default value of "unlimited" is not a limit; if a user is in a group where maxresults is set to "unlimited", he or she is still limited by the highest maxresults (or maxscanrows) limit of the other groups of which he or she is a member. A user's commands are truly unlimited only when the user belongs to no groups, or when all of the groups of which the user is a member have their maxresults set to "unlimited"

A side effect of this is that you can't create a group that assigns "unlimited" maxresults values to superusers, because if any of the users in such a group were to belong to another group, the "unlimited" limit from the superuser group would also apply to them. You can get around this by assigning a very high maxresults limit to your superusers group.

For example:


`Group: superusers Maxresults: 10000000 Maxscanrows: 100000000`

(The largest possible maxresults or maxscanrows limit is platform-dependent; on most platforms, this is a 32-bit integer.)

Scripting efficiently

The Perforce Command-Line Client, p4, supports the scripting of any command that can be run interactively. The Perforce server can process commands far faster than users can issue them, so in an all-interactive environment, response time is excellent. However, p4 commands issued by scripts -- triggers, review daemons, or command wrappers, for example -- can cause performance problems if you haven't paid attention to their efficiency. This is not because p4 commands are inherently inefficient, but because the way one invokes p4 as an interactive user isn't necessarily suitable for repeated iterations.

This section points out some common efficiency problems and solutions.

Iterating through files

Each Perforce command issued causes a connection thread to be created and a p4d subprocess to be started. Reducing the number of Perforce commands your script runs is the first step to making it more efficient.

To this end, scripts should never iterate through files running Perforce commands when they can accomplish the same thing by running one Perforce command on a list of files and iterating through the command results.

For example, try an approach like this:

for i in `p4 diff2 path1/... path2/...` do [process diff output] done

Instead of this:

for i in `p4 files path1/...` do p4 diff2 path1/$i path2/$i [process diff output] done

Using list input files

Any Perforce command that accepts a list of files as a command line argument can also read the same argument list from a file. Scripts can make use of the list input file feature by building up a list of files first, then passing the list file to p4 -x.

For example, if your script currently does something like:

for components in header1 header2 header3 do p4 edit ${component}.h done

...a more efficient alternative would be:

for components in header1 header2 header3 do echo ${component}.h >> LISTFILE done p4 -x LISTFILE edit

The -x flag instructs p4 to read arguments, one per line, from the named file. If the file is specified as "-" (a dash), the standard input is read.

Using branch views

Branch views can be used with p4 integrate or p4 diff2 to reduce the number of Perforce command invocations. For example, if you have a script that runs:

p4 diff2 pathA/src/... pathB/src/... p4 diff2 pathA/tests/... pathB/tests/... p4 diff2 pathA/doc/... pathB/doc/...

you can make it more efficient by creating a branch view that looks like this:


`Branch: pathA-pathB View: pathA/src/... pathB/src/... pathA/tests/... pathB/tests/... pathA/doc/... pathB/doc/...`

and replacing the three commands with one:

p4 diff2 -b pathA-pathB

Limiting label references

Repeated references to large labels can be particularly costly. Commands that refer to files using labels as revisions will scan the whole label once for each file argument. To keep from hogging the Perforce server, your script should get the labeled files from the server, then scan the output for the files it needs.

For example, this:

p4 files path/...@label | egrep "path/f1.h|path/f2.h|path/f3.h"

will impose a lighter load on the Perforce server than either this:

p4 files path/f1.h@label path/f1.h@label path/f3.h@label

or this:

p4 files path/f1.h@label p4 files path/f2.h@label p4 files path/f3.h@label

The "temporary client workspace" trick described below may also reduce the number of times you have to refer to files by label.

The temporary client workspace trick

Most Perforce commands can process all the files in the current workspace view with a single command line argument. By making use of a temporary client workspace with a view that contains only the files on which you want to work, you may be able to reduce the number of commands you have to run, and/or to reduce the number of file arguments you need to give each command.

For instance, suppose your script runs these commands:

p4 sync pathA/src/...@label p4 sync pathB/tests/...@label p4 sync pathC/doc/...@label

You can combine the command invocations and reduce the three label scans to one by using a client spec that looks like:


`Client: XY-temp View: pathA/src/... //XY-temp/pathA/src/... pathB/tests/... //XY-temp/pathB/tests/... pathC/doc/... //XY-temp/pathC/doc/...`

and running:

p4 -c XY-temp sync @label

Using compression efficiently

By default, revisions of files of type binary are compressed when stored on the Perforce server.

Some file formats (for example, .GIF and .JPG images, .MPG and .AVI media content, files compressed with .gz and .ZIP compression) include compression as part of the file format. Attempting to compress such files on the Perforce Server results in the consumption of server CPU resources with little or no savings in disk space.

To disable server storage compression for these file types, specify such files as type binary+F (binary, stored on the server in full, without compression) either from the command line or from the p4 typemap table.

For more about p4 typemap, including a sample typemap table, see "Defining filetypes with p4 typemap" on page 48.

Checkpoints for Database Tree Rebalancing

Perforce's internal database stores its data in structures called Bayer trees, more commonly referred to as B-trees. While B-trees are a very common way to structure data for rapid access, over time the process of adding and deleting elements to and from the trees can eventually lead to imbalances in the data structure.

Eventually, the tree may become sufficiently unbalanced that performance is negatively affected. The Perforce checkpoint and restore processes (see "Backup and Recovery Concepts" on page 25) re-create the trees in a balanced manner, and consequently, you may see some increase in server performance following a backup, a removal of the db.* files, and the re-creation of the db.* files from a checkpoint.

Rebalancing the trees is normally only useful if the database files have become more than about 10 times the size of the checkpoint. Given the length of time required for the trees to become unbalanced during normal Perforce use, we expect that the majority of sites will never need to restore the database from a checkpoint (that is, rebalance the trees) for performance reasons.