Pipeline script from SCM setup

Pipeline supports fetching the DSL (Domain Specific Language) script from the SCM. Typically called Jenkinsfile and located in the root of the project. For the P4 Plugin to operate in this mode you need to provide a Perforce Credential and Workspace mapping for the location of the Jenkinsfile.

Credentials

P4 Plugin makes use of the Jenkins Credential store, making it easier to manage the Helix Core Server (P4D) connection for multiple Jenkins jobs. For details about creating a Perforce Password Credential or Perforce Ticket Credential, see Add a credential.

Add a new Pipeline

  1. From the Jenkins dashboard, click New item from the sidebar menu.
  2. Enter a name for the new job in the Enter an item name field at the top of the page.
  3. Click Pipeline.
  4. Click OK to start configuring your new Pipeline job.
  5. Description (optional): enter a description for the pipeline in the Description field.
  6. Now configure the pipeline on the pipeline page, see below.

Pipeline configuration

Example Pipeline Script from SCM Page

Definition

To configure the pipeline to use a pipeline script that is stored in the SCM, select Pipeline script from SCM from the Definition dropdown.

SCM

To configure the project to use the Perforce Helix Core Server, select Perforce Software from the dropdown.

Perforce Credentials

Select the credentials for the project from the Perforce credentials dropdown.

Workspace behavior

Important
  • Use a unique workspace name for the Jenkinsfile that will not get reused by the sync steps in the script. A postfix of -script will help to identify the workspace's use and make it unique from code sync steps.
  • If you need more than one sync task in a script you MUST use a different workspace name. For instructions on how to do this, see Multiple Syncs.
  • Only map the Jenkinsfile (and perhaps Pipeline libraries) in the workspace view. Jenkins may create an @script directory on the master and you don't want to unnecessarily sync code to an area not used for the actual build.
Note

Many of the workspace fields can include environment variables to help define their value. We recommend that variables are used in the Workspace name to enable you to identify builds. For more information about environment variables, see Variable Expansion.

Perforce workspaces support the following behaviors:

  • Manual (custom view): Manually define the workspace view, and sync options. Creates the workspace if it does not already exist or updates the workspace spec by setting the various options if it already exists. Jenkins will fill out the workspace root and may override the clobber option. For details about manually configuring your workspace, see Manual workspace.
  • Static (static view, master only): Uses a predefined workspace, the workspace must have a valid view and must already exist. The plugin user must either own the workspace or the spec must be unlocked allowing the plugin user to make edits. The workspace view remains static, but Jenkins will update other fields such as the workspace root, and clobber option. For details about using a static predefined workspace, see Static workspace.
  • Template (view generated for each node): Uses a specified template to generate the workspace view. The name of the workspace is generated using the Workspace Name Format field. This makes it an ideal choice for Matrix builds. For details about using a template to generate the workspace, see Template workspace.
  • Streams (view generated by Perforce for each node): Uses a specified stream to generate the workspace view. The name of the workspace is generated using the Workspace Name Format field. This makes it an ideal choice for Matrix builds. For details about using a stream to generate the workspace, see Streams workspace.
  • Spec File (load workspace spec from file in Perforce): Loads the workspace configuration from a depot file containing a client workspace spec. This is the same output as p4 client -o and the Spec depot .p4s format. The name of the Workspace must match the name of the client workspace spec. For details about using a spec file for your workspace, see Spec File workspace.

Populate options

Perforce populates the workspace with the file revisions needed for the build, the options are:

  • Automatic Cleanup and Sync: Use when the Perforce Helix Core Server controls the files in the workspace. This is an efficient option that cleans and syncs file revisions in the workspace. Extra (non versioned files) are removed, missing and modified files are re-added. This option is best for clean builds. For details about the Automatic Cleanup and Sync options, see Automatic Cleanup and Sync.
  • Flush Workspace: Use when files in the workspace are not added by the Perforce Helix Core Server. No file Sync or cleanup is attempted and no files are removed, but the workspace "have list" is updated. This is effectively the same as using the p4 sync -k command. For details about the Flush Workspace options, see Flush Workspace.
  • Forced Clean and Sync: Not recommended because this option is very inefficient. Use if you do not trust the content of your workspace. Removes all of the files in the workspace root, including the build history, and then force syncs the required files. For details about the Forced Clean and Sync options, see Forced Clean and Sync.
  • Graph Force Clean and Sync: Perforce Git Connector required. Use for Perforce Graph and Hybrid only. Removes all of the files in the workspace root, including the build history, and then force syncs the required files. For details about the Graph Force Clean and Sync options, see Graph Force Clean and Sync.
  • Preview Check Only: No file sync or cleanup is attempted, and the Workspace "have list" is not updated. This is effectively the same as using the p4 sync -n command. For details about the Preview Check Only options, see Preview Check Only.
  • Sync Only: Use if you have a hand-crafted build environment that you don't want to lose any files from. No cleanup is attempted and the sync will update all files (as CLOBBER is set) to the required revisions. This option is best for incremental builds and is mostly used for Pipeline builds. For details about the Sync Only options, see Sync Only.

Parallel Sync

Parallel sync enables concurrent transfer of files between Jenkins and the Helix Core Server. Parallel sync can greatly reduce the amount of time it takes to update a workspace, especially when connecting across a high latency network. Parallel sync must be enabled on the Helix Core Server (2014.1 or later). For more information on parallel sync, see Using parallel processing for submits and syncs in  Helix Core Server Administrator Guide. Parallel sync is disabled by default, to enable parallel sync:

  1. Click Advanced.
  2. Select Enabled.
  3. The parallel sync settings will depend on the capabilities of the Jenkins build machine and the Helix Core Server:
    • Threads: Set the number of threads available for parallel sync.
    • Minimum Files: If the number of files being sync'ed is less than this value, parallel sync is not used.
    • Minimum Bytes: If the total size of files being sync'ed is less than this value, parallel sync is not used.

Polling build filters

When polling is used, changes can be filtered to not trigger a build. If the build has more than one filter, the filters are processed starting from the top of the list of filters. Filters can be dragged up or down the list to change the order they are processed in. Click Add a new build filter and select a filter from the dropdown list, options are:

  • Exclude changes from depot path: Changes can be filtered to not trigger a build if all of the files within a change match the specified path. For examples, see Exclude Changes From Depot Path.
  • Exclude changes from user: Changes owned by the Perforce Helix Core Server user specified in the filter will not be built.
  • Exclude changes from outside Java pattern: Changes can be filtered to not trigger a build if none of the files within a change match the Java patterns (regular expression) listed. For examples, see Exclude Changes From Outside Java Pattern.
  • Exclude changes outside view mask: Changes can be filtered to not trigger a build if none of the files within a change are contained in the view mask. For examples, see Exclude Changes Outside view mask.
  • Polling per Change: A build is carried out for every change that is submitted. The polling event will only return the oldest unbuilt change, resulting in incremental builds.

  • Polling latest Change: A build is carried out for the latest change found during polling.

Repository browser

Repository browsing allows Jenkins to use an external browser to navigate files and changes associated with a Jenkins build. The Helix Swarm browser is recommended.
To enable the feature, select the browser from the Repository browser dropdown list, and provide the full URL to the browser.

Repository Browser

Example of a link to a change in Swarm:

Example Change in Swarm

Script Path

The script for the Jenkinsfile is generated automatically based on the details you entered for your pipeline.

Enter the Perforce depot location you want to save the Jenkinsfile script to, for example:

//depot/main/Jenkinsfile

Note

Folders must be separated with / for all platforms.

Lightweight checkout

Pipeline script from SCM would normally use a Perforce Sync to fetch the versioned Jenkinsfile, however if the workspace view includes more than just the Jenkinsfile this operation could be expensive. Jenkins has introduced a Lightweight checkout option that navigates the SCM and fetches files as required. Enabling the option will use a p4 print to fetch the Jenkinsfile avoiding the need for syncing the script files.

Tip

Lightweight checkout can only be used with Pipeline scripts, that is pipeline{...} and not the earlier node(){...} scripts. If you are using lightweight checkout and there are a lot of files in your workspace you might want to disable implicit checkout to avoid syncing all of the files in the workspace.

To disable implicit checkout use: options { skipDefaultCheckout() }

To use this option, select Lightweight checkout.

Build Triggers

If no triggers are set, Jenkins will build based on the Polling build filters in use.

  • Perforce triggered build: A build trigger can be added to the job after you have tested the job and proved that it works correctly. For instructions about configuring a build trigger, see Automatic build using a Perforce triggered build.
  • Poll SCM: If you enable Poll SCM, ensure that your Auto build filters are configured to suit regular builds and that the polling frequency is longer than the time the build takes to complete.

    • Note

    It is generally better to use the Perforce triggered build option trigger a build based on an specific event, such as a submitted change.

Save

When you have finished configuring the job, click Save.

Building

Now that you have configured your job it is ready to be built. For information about building jobs, see Building Jobs.