System resources

The Helix Core Server has the ability to monitor the CPU pressure and memory usage. In addition, the server can reduce the amount of work the server accepts when resources become limited by

• pausing incoming commands when medium thresholds are reached

• terminating incoming commands when high thresholds are reached

The server thereby regulates resource usage and maintains more consistent performance. Such resource monitoring can prevent large spikes of resource usage because the server spreads the load over a longer period of time.

Servers with a high number of concurrently-running commands benefit the most. Configure your server monitoring so that the thresholds are above the day-to-day ceiling of load. This results in the least amount of performance change from the baseline. Setting the thresholds too low will result in resource under-utilization and unnecessary slowdowns.

For additional information and advice about how to set the limits, see the output of p4 help server-resources.

Configurables

When any resource has exceeded its configured threshold, the server begins pausing incoming commands until resource usage has dropped below the threshold.

pausing

Configurable	Meaning
sys.pressure.max.pause.time	Number of seconds a command is able to spend in the paused state before the server returns an error to the client. Setting this configurable to 0 disables pausing commands entirely. Tip Prior to enabling the full resource monitoring configuration, run in preview-mode for a while to gauge the effect of the configured thresholds. Preview-mode is when the feature is configured, but sys.pressure.max.pause.time=0 so that the p4 admin resource-monitor background task is sampling resources and setting pressure levels but commands are not subject to pausing.
sys.pressure.max.paused	Maximum number of concurrently-paused client commands on the server. New incoming commands above this threshold will be rejected with an error. Some administrative and replication commands are not subject to pausing.

A Helix Core Server Extension could register a "pressure-pause" hook to be called periodically while a command is paused. Such an Extension could continue the pause, unpause, or defer the choice to the server.

A paused command is visible with p4 monitor show. The time spent in the paused state is recorded in the tracking entries of the server log files. See paused state in P4LOG in Helix Core Server Administrator Guide.

percentage-based memory

Percentage-based memory, ranged 0-100, are based on the ratio of total system memory vs memory available to use without swapping:

Configurable	Meaning
sys.pressure.mem.medium	When commands begin pausing.
sys.pressure.mem.high	The operating system is about to thrash into swap or become unstable, such as having processes subject to the Out Of Memory Killer. Existing commands that request more memory while the server is above the 'high' threshold might be canceled and return an error to the client.

OS-supplied resource pressure thresholds

Where available, the server also uses more accurate operating system resource interfaces to get information about high memory usage. This is available on Linux with the cgroup v2 mechanism. See the Linux kernel's definition of Pressure Stall Information (PSI) for further details.

Configurable	Meaning
sys.pressure.os.mem.medium	(Linux) The server tries to keep memory usage below the 'medium' level.
sys.pressure.os.mem.high	(Windows and Linux) Amount of time some processes on the system are spending stalled waiting for the memory. New incoming commands received by the server while at this threshold are rejected. Existing commands that request more memory while the server is above this threshold might be canceled and return an error to the client. When the Helix Core Server limits its work, it does not distinguish between memory used by other processes on the operating system and its own. For example, if a large external process comes and consumes a large amount of memory, the Helix Core Server can throttle itself in response.
sys.pressure.os.cpu.high	(Linux) CPU monitoring is only available if cgroups v2 support is configured. This configurable represents the amount of time some processes on the system are spending stalled waiting for CPU time. For Linux cgroup support, only the system-wide /proc/pressure/* files are considered.

Duration

Number of milliseconds for averaging samples of resource pressure. A larger duration makes the server less sensitive to changes in pressure. The 'high' threshold must be set higher than the corresponding 'medium' threshold. Otherwise, the default values are used.

Configurable	Meaning
sys.pressure.mem.high.duration	Number of milliseconds for averaging sys.pressure.mem.high
sys.pressure.mem.medium.duration	Number of milliseconds for averaging sys.pressure.mem.medium
sys.pressure.os.cpu.high.duration	Number of milliseconds for averaging sys.pressure.os.cpu.high
sys.pressure.os.mem.high.duration	Number of milliseconds for averaging sys.pressure.os.mem.high
sys.pressure.os.mem.medium.duration	Number of milliseconds for averaging sys.pressure.os.mem.medium

Prerequisites

The prerequisites for the server to be able to respond to resource pressure are:

• Operating system support

• Realtime monitoring counters enabled

• The p4 admin resource-monitor server startup command running. See the command-line output of p4 help admin-resource-monitor

• Existing resource usage baselines have been established

• Enough space between the baseline usage and the medium/high thresholds such that the command-pausing is not always on

For example:

p4 serverid $name
p4 configure set rt.monitorfile=$monitor_file
p4 configure set "$name#startup.1=admin resource-monitor"
p4 admin restart

Exceptions to pausing

The following commands are not subject to being paused under pressure:

admin configure counter counters dbstat dbverify depots diskspace export extension failback failover heartbeat info journalcopy journaldbchecksums journals lockstat logappend login login2 logout logparse logrotate logschema logstat logtail monitor passwd ping protect pull serverid servers topology triggers user

Previewing and activating monitoring

Follow these steps to preview monitoring, and then activate monitoring.

1. Set a server ID name with p4 serverid $name

2. Enable real-time monitoring with p4 configure set rt.monitorfile=$monitor_file

3. Enable the resource monitoring background process with p4 configure set "$name#startup.1=admin resource-monitor"

4. Enable preview mode (no pausing) with p4 configure set sys.pressure.max.pause.time=0

5. Restart the server with p4 admin restart

6. To preview what the pauses would be, check the "Server under resource pressure. Pause rate " message in the log entries of the p4 admin resource-monitor background task.

7. Adjust the configurables, if necessary, until you are satisfied with the preview results

8. Activate monitoring with p4 configure unset sys.pressure.max.pause.time

To disable monitoring

To disable monitoring, do any of the following:

• Turn off the p4 admin resource-monitor startup command, either with p4 monitor terminate, or by removing the startup configurable and restarting the server.

• Change the sys.pressure.max.pause.time configurable to 0

• Change the values of the configurable thresholds (sys.pressure.os.*.high) to 0.