Sparse streams

P4V supports the creation of Sparse Streams. Sparse streams are immediately available for use because a file is not branched from the parent stream until the user checks out the file. Sparse streams have significant advantages over Task streams.

	Sparse Streams	Task Streams
Performance	Lightweight: no need to wait a long time for the initial populate command, and no need to delete or unload to maintain performance.	Only lightweight after integrating changes from a task stream, and then deleting or unloading the task stream. Otherwise the accumulation of database metadata The data stored by Helix Core Server that describes the file revisions in the depot, where they get their content from (see lazy copy), and the current state of client workspaces, protections, groups, users, labels, streams, and branches. Metadata is stored in the server database and is separate from the archive files that users submit. can affect performance.
Flexibility	`sparsedev` can be converted to `development` `sparserel` can be converted to `release` See the `p4 stream` command for `convertsparse`.	`task` can be converted to `development`
Control	Supports setting a configurable limit on the number of branched (noun) A set of related files that exist at a specific location in the Helix Core depot as a result of being copied to that location, as opposed to being added to that location. A group of related files is often referred to as a codeline. To associate code reviews in Helix Swarm with the projects they are part of, add the 'branch' paths in the Swarm project. (verb) To create a codeline by copying another codeline with the 'p4 integrate', 'p4 copy', or 'p4 populate' command. files to preserve good performance. See `dm.stream.sparse.branchmax` in Helix Core Command-Line (P4) Reference.	Retroactive control by deleting or unloading task streams to reduce metadata load.
Code Review	Works with Helix Swarm	Does not work with Helix Swarm
Virtual Stream	Can be the parent of a virtual stream.	Does not support virtual streams.
Private editing	Supports both private and public editing of the stream spec. Use cases include setting the ChangeView number downwards, adding import paths (or changing paths in general), adding custom fields, and changing the description. To learn more, see Edit stream specs.	Does not support private editing of stream spec.

Reduction of metadata load

Files in sparse streams are only branched as needed, so less metadata needs to be maintained, stored, and sent to downstream servers.

Productivity

Sparse streams allow you to work with a new stream quickly.

If your organization has a "branch per bug" workflow, sparse streams might accelerate productivity.

Suppose you only need to edit one file in a parent stream with one million files. Instead of waiting several hours for the development stream to be created and populated, a sparse stream is ready in less than a minute.

The benefit of being able to start your work quickly might be particularly dramatic in a multi-server environment. If your client workspace is on a remote edge server, network latency to populate a local development stream from the commit serve increases the amount of time it takes for your local client workspace to be ready for you to begin your work.

Lightweight

Sparse streams are lightweight because:

Branching only occurs when a file needs to be changed.
A client workspace of a sparse stream has view mappings for both the parent and child stream depot paths. Most files that are visible in a sparse stream reside solely in the depot path of the parent stream.
When the sparse stream is created, the stream is not populated, parent files within a share path are not branched, and the sparse stream contains no depot files. Instead, when you are working in a sparse stream and Check Out a file from the parent stream, that file is branched into the sparse stream, sync'd locally, and opened for edit.

sparsedev and sparserel

A sparse stream can be a child of a mainline, development, release, or virtual stream.

The two types of sparse streams are sparsedev and sparserel.

In terms of flow rules:

sparsedev is similar to development
sparserel is similar to release

You can convert a sparsedev stream to development, and a sparserel stream to release. See Convert a sparse stream to a non-sparse stream.

You can use either type by itself or in combination with the other type.

Workflow example

This scenario describes a workflow that combines the use of sparserel and sparsedev.

A sparsedev stream avoids a lengthy populate operation from the mainline stream, and is well suited for developing a feature involving a relatively small number of files.
When the small feature is ready, you can copy up to the mainline stream.
When the mainline stream is complete and stable, you can create a release stream to publish a new product version.
Later, if that new version of the product needs a patch or hot fix, you might use a sparserel stream to avoid a lengthy populate operation from the release stream. A workspace client connected to the sparserel stream could make the hotfix build, then merge its changes down to the release stream. Any import of the release stream depot path would then pick up the fix.

Do not use a sparse stream depot path as an import path in another stream because only a small number of files are resident in the sparse stream depot path and the files of the parent stream depot path are missing.

A stream component constructs a complete stream view, but sparse streams cannot be Stream components.

Overlay mapping and ChangeView number

Sparse streams streamline your work through the combination of:

An overlay mapping (+) view into the parent stream to avoid a lengthy populate operation (see Stream path syntax)
A ChangeView number that isolates your sparse stream work from changes in the parent stream until you choose to merge, copy, or integrate.

Together, the overlay mapping and the ChangeView number allow a sparse stream to remain lightweight as edits proceed and changelist number An integer that identifies a changelist. Submitted changelist numbers are ordinal (increasing), but not necessarily consecutive. For example, 103, 105, 108, 109. A pending changelist number might be assigned a different value upon submission.s increment, provided the edits are on a small number of files.

An overlay mapping view into the parent stream

Suppose a sparsedev1 stream named //tools/sparsedev is the child of a mainstream stream named //tools/main. In the //tools/sparsedev1 stream spec, the View field contains

//tools/main/... ...
+//tools/sparsedev1/... ...

where + indicates an overlay mapping. To learn more, see Map different depot locations to the same workspace location.

A ChangeView number

The ChangeView field of the //tools/sparsedev1 stream spec might look like:

//tools/main/...@3

where the @3 ChangeView number isolates your work in the sparse stream from any changes in the parent stream.

The ChangeView number is either the change number immediately before the sparse stream was created, or the last change prior to a merge, copy, or integrate into the sparse stream.

This ChangeView number automatically appears in the Paths field. For instance:

share ...@3

If you add new share paths, the ChangeView number is appended to them.

The ChangeView number can only be changed by integrating.

Create a sparse stream

Before you create a sparse stream, decide whether you want "development (sparse)" or "release (sparse)" and a which stream will be the parent.

Stream Graph method to create a sparse stream

On the P4V Stream Graph tab, right-click the stream that will be the parent of your sparse stream and click Create New Stream from ...
In the Set Up New Stream dialog box, provide a Stream name and for Stream type choose either -- development (sparse) or -- release (sparse).

Note that Branch files from parent on stream creation checkbox cannot be selected.

A sparse stream only branches a file from the parent stream if you check out a file from the sparse stream's view of the parent file.
Click OK.

Shared versus branched

Files that reside in the parent stream of a sparse stream are considered "shared". In this example, only one file, memtypes.qrc, has been branched into the sparse stream.

Integrating to and from sparse streams

A sparsedev stream should follow the same flow of change as a development stream.

A sparserel stream should follow the same flow of change as a release stream.

The main difference is that when integrating to a sparse stream, the sparse stream spec is checked out, the spec changeview number is updated, and the sparse stream spec should be submitted.

After syncing the sparse stream, any new versions of the files that were changed on the parent are now visible in the sparse stream.

If the same file is changed in the parent and in the sparse stream, this file is branched to the sparse stream and scheduled for a resolve. In this case, resolve the files and submit the stream spec and the resolved files.

When copying up from a sparsedev stream to its parent, a subsequent merge down from the parent is necessary for a copy up to be allowed again. This is because the copy up operation increases the spec changeview number of the parent. The changeview number needs to be merged down from the parent to the sparsedev stream so that the sparsedev stream is in sync with its parent.

Manage growth

Sparse streams are well suited for use cases in which you only need to make changes to a relatively small number of files, such as to develop a small bug fix or to release a small bug fix in a patch or hotfix.

As the number of branched files in a sparse stream increases, the risk of a performance degradation also increases.

Limit with configurable

To mitigate that risk, the administrator can control the number of branched files with the dm.stream.sparse.branchmax configurable. If an operation would exceed that limit, the operation is blocked and an error message appears. An administrator with the super access level can change the limit at any time. If the value of the configurable is reset to a value that is less than the number of branched files in any existing sparse stream, that excessively-branched sparse stream will continue to exist, but its files cannot be edited.

Convert to a non-sparse stream

If the number of files that need changes exceed the threshold imposed by the dm.stream.sparse.branchmax configurable, you can Convert a sparse stream to a non-sparse stream.

Tip

Editing a parent file branches the file into the sparse stream. Therefore, even if the first edit of a parent file is subsequently reverted, the file remains branched into the sparse stream.

Restrictions

Requires that P4V be connected to Helix Core Server 2024.1 or later.
Must be a child stream of a parent stream. Typically, sparsedev is a child of a development stream and sparserel is a child of a release stream.
No reparenting, and no unshelving from a different stream.
No undo
The only revision limit allowed is @<=change because the parent depot paths are pinned to a specific change number. The changeview number is automatically updated during integrations into the sparse child. No editing of the changeview number is allowed after any files are populated or added into the sparse stream. The changeview number can be updated only through integration.
Each sparse stream must have at least one share path and ParentViews are always inherit.
Sparse streams can consume Stream components but cannot be consumed as components of other streams.
Sparse streams can have virtual stream children, but virtual children of sparse streams cannot have children and cannot be integration targets. The only type of stream that can be a child of a sparse stream is a virtual stream.
Non-sparse streams cannot be converted to sparse streams, even if the stream was originally sparsedev or sparserel prior to a conversion to development or release.
Like task streams, sparse streams are not supported for use with DVCS or graph depots A depot of type graph that is used to store Git repos managed by Helix Core Server. See also Git Connector and classic depot..
Whether you submit changes to the sparse stream local file, or revert the file, that file remains resident in the sparse stream.

Convert a sparse stream to a non-sparse stream

If you find that your stream needs more files than allowed (see Manage growth), you can convert:

a sparsedev stream to a development stream
a sparserel stream to a release stream

This conversion combines the files of the parent stream with the files of the sparse stream and removes the ChangeView number.

Warning

The conversion is permanent and cannot be reversed. Also, do not revert the conversion because the resulting sparse stream would have poor performance.

The steps are:

On the Stream Graph tab, right-click the sparse stream and click Check Out and Convert Sparse Stream.
In the Convert from sparse stream to non-sparse stream dialog box, click Convert stream.
In the dialog box that says The edited stream and branched files are in the change, click OK.
On the Pending tab, select the changelist and Submit.

The stream name remains the same, but its type has changed. When the second step, p4 submit, has fully completed, the converted stream will be fully branched to contain the metadata for all the files from the parent stream, and the client spec contains no ChangeView and no overlay mappings.

Limit on branched files

By default, the number of files branched into a sparse stream is limited and controlled by your Helix Core Server administrator.

Unless your Helix Core Server administrator has removed the limit, you might encounter the limit when you attempt to check out files. If so, the Files exceed the file maximum allowed in a sparse stream dialog box appears. If this occurs, you can cancel the operation or convert the stream.

You might also see a message that says Are you sure you want to check out xx files?" when you check out more files than the threshold specified by Preferences > Behavior > Warn before checking out files, if the number of files exceeds number. If you click Check Out, you might still encounter the limit on branched files that is controlled by your Helix Core Server administrator.