Upgrading a tarball installation

The section describes how to upgrade a Swarm tarball installation to a newer release.

The following process attempts to minimize downtime, but a short period of downtime for Swarm users is unavoidable. There should be no downtime for your Helix Core Server. After a successful upgrade, all Swarm users are logged out.

If you are using Swarm in a production environment, we encourage you to test this upgrade process in a non-production environment first.

PHP and Apache 2.4 installation

RHEL does not have PHP 8 and Apache 2.4 by default so you must upgrade your system before you can upgrade Swarm. This process is only required the first time you upgrade to PHP 8. If you have already upgraded to PHP 8 and Apache 2.4, see Configure Redis on the Swarm machine.

Carry out the following steps:

This section describes how to install PHP 8 and Apache 2.4 on your system using a repository .

1. This process is only required the first time you upgrade to PHP 8. Install PHP 8 and migrate the Swarm Apache configuration to use Apache 2.4.

   Follow the instructions for your OS distribution, see RHEL 8 or RHEL 9 below:

   • RHEL 8 (run these commands as root):

     1. Deploy the epel-release-latest-8.noarch.rpm repository configuration package to give Swarm access to EPEL packages:

     dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

     2. Deploy the Remi repository configuration package to give Swarm access to PHP 8 (only required the first time you upgrade to PHP 8):

     dnf install https://rpms.remirepo.net/enterprise/remi-release-8.rpm

     3. Install the yum-utils package to give access to the yum-config-manager command:

     dnf install yum-utils

     4. Enable the optional channel for some dependencies:

     subscription-manager repos --enable=rhel-8-server-optional-rpms

     5. Install the Default/Single version of PHP 8:
        1. Enable the module stream for PHP:

        dnf module reset php

        2. Install PHP.

        If you want to install PHP 8.3, use the following command:

        dnf module install php:remi-8.3

        To install a specific version of PHP, replace the PHP 8.3 reference in the above command.

        3. Run an upgrade for PHP:

        dnf update

     Important

     If you are upgrading from Swarm 2019.3 to Swarm 2021.1 or later, remove the Swarm PHP 7.1 files:

     yum remove $(yum list installed | grep php71 | awk '{print $1}')

     6. If you are upgrading from Apache 2.2, disable your Apache 2.2 HTTP server so that is does not automatically start:

     systemctl disable httpd

     7. If you are upgrading from Apache 2.2, stop your Apache 2.2 HTTP server:

     systemctl stop httpd

     8. If you have any special php.ini configuration options set, copy the php.ini file to the /etc/php.d/ directory, for example:

     cp /etc/opt/rh/rh-php72/php.d/php.ini /etc/php.d/php.ini

     9. Copy the perforce-swarm-site.conf file to the /etc/httpd/conf.d/ directory if it is not already in there, for example:

     cp /opt/rh/httpd24/root/etc/httpd/conf.d/perforce-swarm-site.conf /etc/httpd/conf.d/perforce-swarm-site.conf

     10. Replace the /log/httpd24 filepath with /log/httpd/ in the perforce-swarm-site.conf file using the sed command:

     sed -i "s#/log/httpd24/#/log/httpd/#g" /etc/httpd/conf.d/perforce-swarm-site.conf

     11. Enable your Apache 2.4 HTTP server so that it automatically starts:

     systemctl enable httpd

     12. Start the Apache 2.4 HTTP server:

     systemctl start httpd

     13. Check that SELinux is working correctly for your system. For instructions on configuring SELinux on RHEL, see SELinux configuration.
   • RHEL 9 (run these commands as root):
     1. Run an upgrade for PHP, this will also upgrade the Swarm packages:

     dnf update

     2. If you didn't run dnf update in the previous step, run the Swarm package upgrade now:

     yum install helix-swarm helix-swarm-triggers helix-swarm-optional

     Important

     If you are upgrading from Swarm 2019.3 to Swarm 2021.1 or later, remove the Swarm PHP 7.1 files:

     yum remove $(yum list installed | grep php71 | awk '{print $1}')

     3. Optional, ImageMagick: when ImageMagick is enabled, Swarm can display the following image formats: BMP, EPS, PSD, TGA, TIFF.
        

        1. Install ImageMagick:

        yum install ImageMagick

        2. Restart the web server to so that ImageMagick is picked up.

        systemctl restart httpd

     4. Check that SELinux is working correctly for your system. For instructions on configuring SELinux on RHEL, see SELinux configuration.
2. PHP 8 and Apache 2.4 are now installed, configure the Swarm Redis service on the Swarm machine, see Configure Redis on the Swarm machine.

Configure Redis on the Swarm machine

Follow the instructions for the Swarm version you are upgrading from:

• Upgrade from Swarm 2019.1 and earlier

• Upgrade from Swarm 2019.2 and later

Upgrade from Swarm 2019.1 and earlier

Swarm requires Redis to manage its caches and by default Swarm uses its own Redis server on the Swarm machine. Swarm caches data from the Helix Core Server to improve the performance of common searches in Swarm and to reduce the load on the Helix Core Server. Redis is included in the Swarm Tarball installation.

This section describes how to configure the Swarm Redis service on the Swarm machine. Do not change the default values of the following configuration files if you are using the Swarm Redis server.

Swarm has two Redis binaries in SWARM_ROOT/p4-bin/bin.linux26x86_64/:

• redis-server-swarm
• redis-cli-swarm

1. Configure the Redis cache database, enter the following details in the redis-server.conf file in /opt/perforce/etc/:

bind 127.0.0.1
port 7379
supervised auto
save ""
dir /opt/perforce/swarm/redis

Default values:

Tip
• The default settings are shown below, the redis-server.conf file contains more detailed information about the Redis configuration for Swarm.
• On Swarm systems with a large number of users, groups, and projects, start-up time can be improved by persisting the memory cache. You can persist the memory cache by disabling background saves and enabling append saves, see the redis-server.conf file comments for detailed information.
• bind 127.0.0.1 - Redis server IP address (loopback interface)
• port 7379 - Redis server port number
• supervised auto - detects the use of upstart or systemd automatically to signal that the process is ready to use the supervisors
• save "" - background saves disabled, recommended.
• dir /opt/perforce/swarm/redis - the directory the Redis cache database is stored in.
Tip

To fine-tune your Redis server settings, make your changes in the Laminas Redis adapter. For information about the Laminas Redis adapter, see the Laminas Redis Adapter documentation.

2. The Redis cache database must be running before Swarm starts so we recommend setting it up as a service so that it starts automatically at boot time.

The following example script will:

• run the Redis service as the Perforce user
• use the configuration file in /opt/perforce/etc/redis-server.conf
• assume the database server is installed in /opt/perforce/swarm/p4-bin/bin.linux26x86_64/

To configure Redis as a service, add a script with the following content in /etc/systemd/system

redis-server-swarm.service

[Unit]
Description=Redis Server for Swarm
After=network.target

[Service]
ExecStart=/opt/perforce/swarm/p4-bin/bin.linux26x86_64/redis-server-swarm /opt/perforce/etc/redis-server.conf
ExecStop=/opt/perforce/swarm/p4-bin/bin.linux26x86_64/redis-cli-swarm shutdown
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIndentifier=redis
User=perforce
	Group=perforce
 
[Install]
WantedBy=multi-user.target

3. Create the SWARM_ROOT/data/config.php file if it does not already exist. The redis block of the SWARM_ROOT/data/config.php file contains the password, namespace, host and port details of the Swarm Redis server:

<?php
    // this block should be a peer of 'p4'
    'redis' => array(
        'options' => array(
            'password' => null, // Defaults to null
            'namespace' => 'Swarm',
            'server' => array(
                'host' => 'localhost', // Defaults to 'localhost' or enter your Redis server hostname
                'port' => '7379', // Defaults to '7379 or enter your Redis server port
            ),            
        ),
        'items_batch_size' => 100000,
        'check_integrity' => '03:00', // Defaults to '03:00' Use one of the following two formats: 
                                      // 1) The time of day that the integrity check starts each day. Set in 24 hour format with leading zeros and a : separator
                                      // 2) The number of seconds between each integrity check. Set as a positive integer. Specify '0' to disable the integrity check.
        'population_lock_timeout' => 300, // Timeout for initial cache population. Defaults to 300 seconds. 
    ),

Configurables:

• password: Redis server password. Defaults to null and should be left at default if using the Swarm Redis server.
• namespace: the prefix used for key values in the Redis cache. Defaults to Swarm and should be left at default if using the Swarm Redis server.
Note

If you have multiple Swarm instances running against a single Redis server, each Swarm server must use a different Redis namespace. This enables the cache data for the individual Swarm instances to be identified. The namespace is limited to ≤ 128 characters.

If one or more of your Swarm instances is connected to multiple Helix Core Servers, the Redis namespace includes the server label and the character limit is reduced to ≤ 127 characters, see Multiple Helix Core Server instances.

• host: Redis server hostname. Defaults to localhost and should be left at default if using the Swarm Redis server.
• port: Redis server port number. Defaults to 7379. Swarm uses port 7379 as its default to avoid clashing with other Redis servers that might be on your network. It should be left at default if using the Swarm Redis server. The default port for a non-Swarm Redis server is 6379.
• items_batch_size: Maximum number of key/value pairs allowed in an mset call to Redis. Sets exceeding this will be batched according to this maximum for efficiency. Defaults to 100000.
Note

The default value of 100000 was chosen to strike a balance between efficiency and project data complexity. This value should not normally need to be changed, contact support before making a change to this value.

• check_integrity: In some circumstances, such as when changes are made in the Helix Core Server when Swarm is down or if errors occur during updates, the Redis cache can get out of sync with the Helix Core Server. Swarm can run a regular integrity check to make sure that the Redis caches and Helix Core Server are in sync. If an integrity check finds an out of sync cache file, Swarm automatically updates the data in that cache.

The check_integrity configurable specifies when the Redis cache integrity check is run. Set as a specific time of day (24 hour format with leading zeros) or a number of seconds (positive integer) between checks. Disable the integrity check with '0'. Defaults to '03:00'.

• population_lock_timeout: specifies the timeout, in seconds, for initial cache population. If you have a large Swarm system, increase this time if the initial cache population times out. Defaults to 300 seconds.

Now that the Swarm Redis service is configured for the Swarm machine, start your Swarm upgrade, see Upgrade Swarm.

Upgrade from Swarm 2019.2 and later

Swarm requires Redis to manage its caches and by default Swarm uses its own Redis server on the Swarm machine. Swarm caches data from the Helix Core Server to improve the performance of common searches in Swarm and to reduce the load on the Helix Core Server. Redis is included in the Swarm Tarball installation.

Swarm has two Redis binaries in SWARM_ROOT/p4-bin/bin.linux26x86_64/:

• redis-server-swarm
• redis-cli-swarm

If the Redis binaries supplied with your new version of Swarm have been updated, your Redis server is automatically upgraded to the new Redis server version. Your original Redis configuration files will not be changed. Do not change the default values of the Redis configuration files if you are using the Swarm Redis server.

Restart the Redis server:

If the Redis binaries have been updated and you are running Redis as a service (recommended), you must restart the Redis server to use the new Redis server version.

1. Run the following command to restart the Redis server:

systemctl restart redis-server-swarm

2. You are now running the updated Redis server version.
Tip

On Swarm systems with a large number of users, groups, and projects, start-up time can be improved by persisting the memory cache. You can persist the memory cache by disabling background saves and enabling append saves, see the redis-server.conf file comments for detailed information.

3. Now that the Swarm Redis service is configured for the Swarm machine, start your Swarm upgrade, see Upgrade Swarm.

Upgrade Swarm

The steps in this section describe how to upgrade Swarm using the provided archive file. SWARM_ROOT refers to the current Swarm installation.

1. Download the new TAR file from Download Helix Swarm.

2. Expand the new swarm.tgz:

   tar -zxf swarm.tgz

   The contents of swarm.tgz are expanded into a top-level folder named swarm-version, where version corresponds to the version downloaded. This directory is identified as SWARM_NEW below.

3. Move SWARM_NEW to be a peer of SWARM_ROOT:

   mv SWARM_NEW SWARM_ROOT/../

4. Copy the SWARM_ROOT/data/config.php file from SWARM_ROOT to SWARM_NEW:

   cp -p SWARM_ROOT/data/config.php SWARM_NEW/data/

5. Create the queue token directory:

   mkdir SWARM_NEW/data/queue

6. Copy the existing trigger token(s):

   sudo cp -pR SWARM_ROOT/data/queue/tokens SWARM_NEW/data/queue/

7. Configure the ImageMagick policy.xml file to allow Swarm to create thumbnails if it is not already configured:

1. Open the ImageMagick policy.xml file for editing:

/etc/ImageMagick-6/policy.xml

2. Disable lines with the pattern below for the following <filetype>s: PS, PS2, EPS, PDF, or XPS:

<policy domain="coder" rights="none" pattern="<filetype>" />

3. Save the policy.xml file.

8. Assign correct ownership to the new Swarm's data directory:

   sudo chown -R www-data SWARM_NEW/data

   Note

   The www-data user above is an example of what the web server user name might be, and can vary based on distribution or customization. For example, the user is typically apache for Red Hat and www-data for Debian/Ubuntu.

9. Update your triggers, see Update your triggers .

Update your triggers

1. Copy the new Swarm trigger script to your Helix Core Server machine. The trigger script is SWARM_ROOT/p4-bin/scripts/swarm-trigger.pl, and requires installation of Perl 5.08+ (use the latest available) on the Helix Core Server machine. If Swarm is using SSL, then the triggers also require the IO::Socket::SSL Perl module.

   Warning

   Do not overwrite any existing trigger script at this time. Give the script a new name, for example: swarm-trigger-new.pl.

2. Configure the Swarm trigger script by creating, in the same directory on the Helix Core Server machine, swarm-trigger.conf. It should contain:

   Note

   If you already have a swarm-trigger.conf file, no additional configuration is required.

   # SWARM_HOST (required)
   # Hostname of your Swarm instance, with leading "http://" or "https://".
   SWARM_HOST="http://my-swarm-host"
   
   # SWARM_TOKEN (required)
   # The token used when talking to Swarm to offer some security. To obtain the
   # value, log in to Swarm as a super user and select 'About Swarm' to see the
   # token value.
   SWARM_TOKEN="MY-UUID-STYLE-TOKEN"
   
   # ADMIN_USER (optional) Do not use if the Workflow feature is enabled (default)
   # For enforcing reviewed changes, optionally specify the normal Perforce user
   # with admin privileges (to read keys); if not set, will use whatever Perforce
   # user is set in environment.
   ADMIN_USER=
   
   # ADMIN_TICKET_FILE (optional) Do not use if the Workflow feature is enabled (default)
   # For enforcing reviewed changes, optionally specify the location of the
   # p4tickets file if different from the default ($HOME/.p4tickets).
   # Ensure this user is a member of a group with an 'unlimited' or very long
   # timeout; then, manually login as this user from the Perforce server machine to
   # set the ticket.
   ADMIN_TICKET_FILE=				
   										
   # VERIFY_SSL (optional)
   # If HTTPS is being used on the Swarm web server, then this controls whether
   # the SSL certificate is validated or not. By default this is set to 1, which
   # means any SSL certificates must be valid. If the web server is using a self
   # signed certificate, then this must be set to 0.
   # set the ticket.
   VERIFY_SSL=1

   Fill in the required SWARM_HOST and SWARM_TOKEN variables with the configuration from any previous Swarm trigger script, typically swarm-trigger.pl.

   Tip

   The ADMIN_USER and ADMIN_TICKET variables were used by the 'enforce triggers' in Swarm 2019.1 and earlier. They can be removed unless you are explicitly disabling workflow and using the deprecated 'enforce triggers'.

   Note

   Swarm 2015.4 and earlier: Swarm trigger script files were available as shell scripts in these earlier Swarm versions, typically swarm-trigger.sh.

   Swarm must now use a Perl trigger script file, typically swarm-trigger.pl.

3. On Linux: ensure that the script is executable:

   sudo chmod +x swarm-trigger-new.pl

4. Rename the new trigger script:

   On Linux:

   mv swarm-trigger-new.pl swarm-trigger.pl

   On Windows:

   ren swarm-trigger-new.pl swarm-trigger.pl

5. Update the triggers in your Helix Core Server.

   Warning

   • The swarm.shelvedel shelve-delete trigger line was added to Swarm in version 2018.1 and updated in version 2020.1.

     • Upgrading from Swarm 2017.4 and earlier: add the swarm.shelvedel shelve-delete trigger line to the Helix Core Server trigger table if it is not already present, see Update the Helix Core Server triggers table to run the trigger script.
     • Upgrading from Swarm 2018.x and 2019.x: replace the existing swarm.shelvedel shelve-delete trigger line in the Helix Core Server trigger table with the one supplied in the Swarm version you are upgrading to.

   • Workflow feature:

     The Workflow feature is enabled by default in Swarm 2019.2 and later. The trigger lines required when workflow is enabled are different to those required when workflow is disabled:

     • Workflow feature enabled (default):
       • Comment out the swarm.enforce.1, swarm.enforce.2, swarm.strict.1, and swarm.strict.2 trigger lines in the Helix Core Server trigger table if they are present, see Update the Helix Core Server triggers table to run the trigger script.
       • Add the swarm.enforce change-submit, swarm.strict change-content, and swarm.shelvesub shelve-submit trigger lines to the Helix Core Server trigger table if they are not already present, see Update the Helix Core Server triggers table to run the trigger script.
     • Workflow feature disabled:

       Comment out the swarm.enforce change-submit, swarm.strict change-content, and swarm.shelvesub shelve-submit trigger lines in the Helix Core Server trigger table if they are present, see Update the Helix Core Server triggers table to run the trigger script.

   1. Run the Swarm trigger script to capture (using Ctrl+C on Windows and Linux) the trigger lines that should be included in the Perforce trigger table:

      On Linux:

      ./swarm-trigger.pl -o

      On Windows:

      path/to/perl swarm-trigger.pl -o

   2. As a Perforce user with super privileges, update the Perforce trigger table by running p4 triggers command and replacing any swarm.* lines with the previously captured trigger line output (using Ctrl+V on Windows and Linux).
   Important

   If you previously customized the Swarm trigger lines, perhaps to apply various Trigger options, be sure to repeat those customizations within the updated trigger lines.

Replace your old Swarm instance with your new Swarm instance

Replace your old Swarm instance with your new Swarm instance. Run the following command:

sudo apache2ctl stop; mv SWARM_ROOT SWARM.old; mv SWARM_NEW SWARM_ROOT; sudo apache2ctl start

Reload the Swarm config cache

As an admin user, delete and reload the Swarm configuration cache from the Cache Info tab of the System Information page. Navigate to the User id dropdown menu, select System Information, select the Cache Info tab, and select Reload Configuration. For more information, see Cache Info.

Validate your upgrade

Check that your new Swarm instance is working correctly by doing the following:

1. Create a new changelist that:
   1. Contains at least one modified file
   2. Contains the #review keyword in the changelist description
2. Right click on the new changelist in P4V and click Shelve Files...
Important

Do not select Request New Swarm Review... because this method uses the API and will not fully test the Swarm Helix Core Server extension.

This is also true if you are using Swarm Helix Core Server Triggers instead of the Swarm Helix Core Server extension.

3. Check that a new Swarm review is created for the changelist.
• If a Swarm review is created, the Swarm Helix Core Server extension is working. If you are using Swarm Helix Core Server Triggers instead of the Swarm Helix Core Server extension and the review is created, the triggers are working.
• If a Swarm review is not created, see Review not created.

Swarm index upgrade

If you are upgrading from Swarm 2017.2 or earlier you should run the index upgrade, this ensures that the review activity history is displayed in the correct order on the Dashboard, and Reviews list pages.

The index upgrade process can be configured to suit your Swarm system specifications. See Upgrade index for details.

Run the upgrade as an Admin user by visiting the following URL:

http://SWARM-HOST/upgrade

Secure your Swarm installation

To make your Swarm installation more secure apply the following recommendations for HTTP and Swarm implementation through security groups.

HTTP

Here is a list of best practices to use when port 80 is exposed for HTTP traffic:

• Redirect to HTTPS: If Port 80 needs to be open to support legacy systems or specific use cases, ensure that all HTTP traffic is redirected to HTTPS to encrypt data in transit.

• Use HSTS (HTTP Strict Transport Security) headers: Implement HSTS headers to force browsers only to use secure HTTPS connections when interacting with your server.

• Close port 80: If there is no requirement to use HTTP, Port 80 must be closed entirely to prevent any unencrypted data transmission.

• Implement SSL/TLS (secure sockets layer and transport layer security) certificates: Ensure that your server is configured with a valid SSL/TLS certificate to enable secure HTTPS connections.

• Firewall configuration: Configure firewalls to block or filter access to Port 80, particularly from untrusted networks.

• Continuous monitoring and auditing: Regularly monitor network traffic and audit server configurations to ensure that unnecessary ports are not exposed and that data is transmitted securely.

When you implement HTTPS, you must make the following changes:

1. Modify your cron job for the Swarm workers.

   Edit the cron configuration file to point to your HTTPS URL, for example, https://HOSTNAME/. For more information about how to edit the cron configuration file, see Set up a recurring task to spawn workers.

   To verify if the cron configuration file points to your HTTPS URL, run the following curl statement:

   curl https://myswarm.host/queue/worker

2. Modify the Swarm Extension or Trigger configuration.

   If you are using the Swarm extension run the following command and change ExtConfig’s Swarm URL to be your new HTTPS URL:

   p4 extension --configure Perforce:helix-swarm

   If you are using triggers, edit swarm-trigger.pl configuration file and set your SWARM_HOST to be https.

3. Edit the external_url in the SWARM_ROOT/data/config.php file’s environment block to point to your HTTPS URL. This URL is used in emails, Jira links, and Swarm test’s pass-and-fail outgoing URL parameters.

   Tip

   If you make a configuration change, Swarm will not use it until the configuration cache has been reloaded, this forces Swarm to use the new configuration. You must be an admin or super user to reload the Swarm config cache. Navigate to the User id dropdown menu, select System Information, click the Cache Info tab, and click the Reload Configuration button.

4. Modify the Helix Core Server's P4.Swarm.URL property. For more information about Helix Core Server integration, see Client integration.

   If your Apache server is listening on both HTTPS and HTTP in perforce-swarm-site.conf file, you must set the auto_register_urlconfigurable in the p4 block to false and correctly configure the P4.Swarm.URL property .

   If your Apache server is listening only on HTTPS and if the auto_register_urlconfigurable in the p4 block is set to true (default value), an Apache restart will correct the property.

   To get all your current values for P4.Swarm.URL property, run:

   p4 -Ztag property -A -l -n P4.Swarm.URL

   Ensure that the P4.Swarm.URL property points to your HTTPS URL.

5. Modify the URL of all applications. Any other applications that reference the URL should be switched to using the HTTPS URL.

Swarm implementation through security groups

Here is a list of best practices for implementation using security groups or the user's preferred setup:

• Use a trusted proxy: Ensure to only use a trusted proxy, such as allow lists, Content Delivery Networks (CDN), and API Gateways.

• Backend servers and other proxies or load balancers should be disabled: Ensure that direct access to backend servers and other proxies or load balancers is disabled, except through the trusted proxy mentioned above. This will prevent unauthorized access while ensuring that all requests are filtered through the trusted proxy.

• Continuous monitoring and logging of the X-Forwarded-For header: Implement monitoring and logging on the X-Forwarded-For header to track and identify any suspicious activities. This can help in identifying and preventing potential malicious activity or security threats.

• Use a secure protocol: Implement a secure protocol such as HTTPS to encrypt the communications between the client and the load balancers, and between the load balancer and backend server to prevent eavesdropping or tampering with the X-Forwarded-For header.

• Configure X-Forwarded-For header: Configure the processing mode of the X-Forwarded-For header (append, preserve, or remove) based on specific technical or security requirements.

All done!