Blog
December 13, 2018
Enterprises with a large number of server instances, supporting thousands of users, need a server upgrade plan. This can minimize downtime and reduce the risk that of something going wrong.
Routinely updating your servers delivers benefits to your teams. Being on a recent version is often a key requirement for support. Each release also has numerous small enhancements and addresses issues. Depending on what version you are on, you could be missing a lot of great new features that could improve your performance.
Most installations are mission critical. A server upgrade plan needs to involve a lot of moving parts with integrations, tooling, and environment settings. With the right plan, surprises can be all but eliminated.
We are going to focus on tips for large enterprise customers who have more complex environments. For smaller teams, you may not need to follow all these tips, but you will likely benefit from doing so.
Server Upgrades in 30 Seconds
Developer productivity and high-performance automation are the keys to getting the most out of your investment. Depending on your version control system, you may have limitations on what you can implement.
One of the main reasons people choose Helix Core — version control from Perforce — is because it supports large, diverse teams no matter where they are located. And when it comes to upgrades, the actual software metadata version upgrade, on average, less than 30 seconds to complete on a single server. Steps include:
- Stop the service.
- Swap binaries for new version.
- Execute a command similar to p4d -r $P4ROOT -J $P4JOURNAL -xu (with appropriate values for P4ROOT and P4JOURNAL).
- Restart the service.
For customers with multiple servers, updating your entire infrastructure can be quick. Many customers use tools like Ansible, Puppet, and Chef to speed deployments. This saves our customers time when they need it most. They can focus on automating and developing.
Code Refactoring? How You Can Save Time >>
Pushing out the server upgrades is the easy part. It’s ensuring everything else still works that can make things complicated. But with preparation, it’s possible to avoid issues. To successfully upgrade, we recommend auditing, testing, and configuring your system with an eye toward present and future needs.
Although the technical details apply to Helix Core, you can use the follow steps as an outline for your own upgrade.
Back to topStep 1: Map Your Topology
If you haven’t already, take inventory of your entire development environment. Start by recording configs and versions of everything, by location, in a spreadsheet. Check your licenses. What are the clients for users and automation, where are they, and what are they doing? Which ones are the most important for your organization?
You want to look at where activity is occurring. Check who or what is interacting and against what servers, replicas, edge servers, or proxies.
As an example, the following command on Linux analyzes standard server logs for different client versions:
grep -a -A1 "Perforce server info:" log | grep -a -vE "Perforce server info:|^\-\-$|\t\S+ \S+ pid [0-9]+ (compute end|completed)" | perl -ne 'print "$1\n" if /^[^\[]+\[(.+)\] /' | sort |uniq -c | sort -k 1,1 -n
Example output:
218 p4/2016.2/LINUX26X86_64/1468155
276 CommitTrigger/v81
376 P4V/NTX64/2017.3/1590419/v83
481 unknown/1.50.6550.21600
1681 P4VS/2017.2.159.0319
1743 Builder/1.0.0.0
1789 SWARM/2017.1/1517929
2242 Pipeline.Automation/1.0.0.0
We can see some standard Helix client programs and custom tools with version strings such as “Builder” and “Pipeline.Automation” have been developed in the organization. It’s easy to find out IP addresses and, thus, the machines from which they are being run.
Tracking server interactions can help you determine what needs to be tested. It helps ensure you’re not missing any components. And you can use this information to map your environment.
This takes some time and planning, but it’s a vital step for things going smoothly. Once you know how things relate, you can start making decisions about how to upgrade.
Obviously, if you are monitoring things, examining logs, and/or have dashboards, you already know much of what is going on. Share this information with your infrastructure team. It is extremely useful for troubleshooting issues during operations and essential to interacting with Perforce support, too.
Back to topStep 2: Test Clients and Integrations
The majority of upgrade challenges tend to occur with custom client programs and other systems interacting with the server. Issues can arise, for example, when a custom client, trigger, or tool runs a command and doesn’t get back what the data it is expecting.
For example, there may be a new field in the output of a command due to server enhancements. Sometimes, even a simple thing like a version mismatch between a server and an API could cause automation to break. Occasionally, an old installed p4 client on a slightly neglected system can be many releases behind the installed server.
The more complicated the environment, the greater the risk for client issues. And although this type of issue is rare and usually easy to fix, it is easily avoided.
Having a test instance of your server allows you to check that integrations are compatible between upgrades. Custom built integrations that were installed to match the server may need to be updated along with Helix Core.
To perform a test:
- Copy a production checkpoint and restore to a test instance.
- Copy across as much of the depot/archive files as you need for testing the upgrade (typically a small subset of the full repository).
- Test the various tools and integrations, including custom configurations. This should include triggers and other workflow enhancements.
- If required, fix issues in the client programs before proceeding with the upgrade.
Testing the new version lets you discover problems on a smaller scale. When you’re upgrading, you’ll avoid surprises, which means minimal downtime for your teams.
Back to topStep 3: Automate Server Upgrades
Once you have mapped the environment and tested clients and integrations, you should be ready to proceed with your upgrade.
But Helix Core also has another option that can help save you time when upgrading. To help enterprises of all sizes efficiently upgrade and minimize server downtime, the open-source Server Deployment Package (SDP) is available from Perforce. SDP includes a standard set of scripts that admins use, and it can help provide consistency of management across all of your instances and servers.
Once installed and configured, it also allows enterprises to easily upgrade. This automates upgrades for multiple servers. The traditional advice was to start on the outside edges of your environment and work inward (replicas before master). But we now recommend upgrading everything at the same time.
Your Server Upgrade Plan with Helix Core
Some customers are cutting-edge adopters and are always on the latest release. Other customers that too busy, and they only update every few years. For both of these enterprises, SDP can be an integral part of your solution.
SDP provides proven basic management and best practice features, including init scripts and checkpoint scripts. The package includes:
- Resources to plan, deploy, and manage sophisticated topologies.
- Standards for developing custom triggers.
- Best practice configurables and other product settings.
- Shell environment management standards.
- Supplemental maintenance scripts for admins – including cleanup tasks, removing old/unused workspaces, and deleting labels.
Proper installation can be a differentiator for live product and eval/PoC deployments at any scale. SDP is mature and sophisticated enough to meet complex needs of many demanding environments with no customization.
How Does SDP Work?
When using the SDP Package for an upgrade, you only need to complete two steps for an instance on a machine:
- Download the new version of P4 and P4D.
- Run upgrade.sh <instance>, e.g.
/p4/common/bin/upgrade.sh 1
Behind the scenes, SDP does the work. It stops the server, provides a clean “upgrade point” via a journal rotation, updates the database on the live and offline directories, and restarts the server.
But upgrading is just one aspect of SDP.
Back to topImprove Your Environment; Make Your Server Upgrades Easy
Upgrading more often — while requiring the allocation of time perhaps twice a year for major upgrades — will mean that the magnitude of changes and variables will be less with each instance. Plus, keeping on top of current server usage (e.g., the variety of custom clients in use) and running a standard test installation, makes upgrading a low-risk activity.
And as technologies change, so does the SDP. We encourage our Perforce community to communicate and make changes. It is continuously receiving and incorporating input from Perforce Consulting, Support, and Engineering.
Now you’re ready to start your server upgrade plan. Check out the latest releases of Helix Core.