Sparse streams

Sparse streams have significant advantages over Task streams.

	Sparse Streams	Task Streams
Performance	Lightweight: no need to wait a long time for the initial populate command, and no need to delete or unload to maintain performance.	Only lightweight after integrating changes from a task stream, and then deleting or unloading the task stream. Otherwise database metadata The data stored by Helix Core Server that describes the file revisions in the depot, where they get their content from (see lazy copy), and the current state of client workspaces, protections, groups, users, labels, streams, and branches. Metadata is stored in the server database and is separate from the archive files that users submit. accumulation can affect performance.
Flexibility	`sparsedev` can be converted to `development` `sparserel` can be converted to `release` See the `p4 stream` command for `convertsparse`.	`task` can be converted to `development`
Control	Supports setting a configurable limit on the number of branched (noun) A set of related files that exist at a specific location in the Helix Core depot as a result of being copied to that location, as opposed to being added to that location. A group of related files is often referred to as a codeline. To associate code reviews in Helix Swarm with the projects they are part of, add the 'branch' paths in the Swarm project. (verb) To create a codeline by copying another codeline with the 'p4 integrate', 'p4 copy', or 'p4 populate' command. files to preserve good performance. See `dm.stream.sparse.branchmax` in the Helix Core Command-Line (P4) Reference.	Retroactive control by deleting or unloading task streams to reduce metadata load.
Code Review	Works with Helix Swarm	Does not work with Helix Swarm
Virtual Stream	Can be the parent of a virtual stream.	Does not support virtual streams.
Private editing	Supports both private and public editing of the stream spec. Use cases include setting the ChangeView limit downwards, adding import paths (or changing paths in general), adding custom fields, and changing the description. To learn more, see Private editing of streams in the Helix Core Command-Line (P4) Reference.	Does not support private editing of stream spec.

Reduction of metadata load

Files in sparse streams are only branched as needed, so less metadata needs to be maintained, stored, and sent to downstream servers.

Productivity

Sparse streams allow you to work with a new stream quickly.

If your organization has a "branch per bug" workflow, sparse streams might accelerate productivity.

Suppose you only need to edit one file in a parent stream with one million files. Instead of waiting several hours for the development stream to be created and populated, a sparse stream is ready in less than a minute.

The benefit of being able to start your work quickly might be particularly dramatic in a multi-server environment. If your client workspace is on a remote edge server, network latency to populate a local development stream from the commit serve increases the amount of time it takes for your local client workspace to be ready for you to begin your work.

Lightweight

Sparse streams are lightweight because:

Branching only occurs when a file needs to be changed.
A client workspace of a sparse stream has view mappings for both the parent and child stream depot paths. Most files that are visible in a sparse stream reside solely in the depot path of the parent stream.
When the sparse stream is created, the stream is not populated, parent files within a share path are not branched, and the sparse stream contains no depot files. Instead, when you are working in a sparse stream and check out a file from the parent stream, that file is branched into the sparse stream, sync'd locally, and opened for edit.

Important

Do not run p4 populate to branch all parent files on a newly-created sparse stream. A sparse stream should only branch a small number of files from its parent stream.

sparsedev and sparserel

A sparse stream can be a child of a mainline, development, release, or virtual stream.

The two types of sparse streams are sparsedev and sparserel.

In terms of Flow rules :

sparsedev is similar to development
sparserel is similar to release

but see Integrating to and from sparse streams.

You can convert a sparsedev stream to development, and a sparserel stream to release. See Convert a sparse stream to a non-sparse stream.

You can use either type by itself or in combination with the other type.

Workflow example

This scenario describes a workflow that combines the use of sparserel and sparsedev.

In this scenario:

1. A sparsedev stream avoids a lengthy populate operation from the mainline stream, and is well suited for developing a feature involving a relatively small number of files.

2. When the small feature is ready, you can copy up to the mainline stream.

3. When the mainline stream is complete and stable, you can create a release stream to publish a new product version.

4. Later, if that new version of the product needs a patch or hot fix, you might use a sparserel stream to avoid a lengthy populate operation from the release stream. A workspace client connected to the sparserel stream could make the hotfix build, then merge its changes down to the release stream. Any import of the release stream depot path would then pick up the fix. (See import at Stream views and paths.)

Do not use a sparse stream depot path as an import path in another stream because only a small number of files are resident in the sparse stream depot path and the files of the parent stream depot path are missing.

A stream component constructs a complete stream view, but sparse streams cannot be Stream Components.

Overlay mapping and ChangeView pinning

Sparse streams streamline your work through the combination of:

An overlay mapping view into the parent stream to avoid a lengthy populate operation

and
A ChangeView number acts as limit that isolates your sparse stream work from changes in the parent stream until you choose to merge, copy, or integrate.

Together, the overlay mapping and the limiting ChangeView allow a sparse stream to remain lightweight as edits proceed and changelist number An integer that identifies a changelist. Submitted changelist numbers are ordinal (increasing), but not necessarily consecutive. For example, 103, 105, 108, 109. A pending changelist number might be assigned a different value upon submission.s increment, provided the edits are on a small number of files.

An overlay mapping view into the parent stream

Suppose a sparsedev1 stream named //tools/sparsedev is the child of a mainstream stream named //tools/main. In the //tools/sparsedev1 stream spec, the View field contains

//tools/main/... ...
+//tools/sparsedev1/... ...

where + indicates an overlay mapping. To learn more, see Map different depot locations to the same workspace location.

A ChangeView number acts as limit

The ChangeView field of the //tools/sparsedev1 stream spec might look like:

//tools/main/...@3

where @3 changeview number specifies the limiting change. This limit isolates your work in the sparse stream from any changes in the parent stream.

The changeview number is either the change number immediately before the sparse stream was created, or the last change prior to a merge, copy, or integrate into the sparse stream.

This limiting change number automatically appears in the Paths field. For instance:

share ...@3

If you add new share paths, the limiting number is appended to them.

The ChangeView limit can only be changed by integrating.

Integrating to and from sparse streams

A sparsedev stream should follow the same flow of change as a development stream.

A sparserel stream should follow the same flow of change as a release stream.

The main difference is that when integrating to a sparse stream, the sparse stream spec is checked out, the spec changeview number is updated, and the sparse stream spec should be submitted.

After syncing the sparse stream, any new versions of the files that were changed on the parent are now visible in the sparse stream.

If the same file is changed in the parent and in the sparse stream, this file is branched to the sparse stream and scheduled for a resolve. In this case, resolve the files and submit the stream spec and the resolved files.

Overlays and precedence

The section on Map different depot locations to the same workspace location explains that the overlay mapping takes precedence over that same filename in a preceding view mapping. With sparse streams, the sparse stream view is overlayed on top of the parent view. Therefore, if a file exists both in the overlay mapping of the sparse stream and the preceding mapping of parent stream, when a client workspace does a sync, it gets the sparse stream version, not the parent stream version.

In this example, //depot/parent is a development stream that contains both a.txt and b.txt and //depot/child is a sparsedev stream that contains only a.txt.

Sparse stream overlay mappings Non-sparse stream non-overlay mappings

Sparse stream overlay mappings	Non-sparse stream non-overlay mappings
With an overlay mapping, precedence is given to the lower lines, but files in the preceding view lines are not obscured. With the mapping //depot/parent/... ... +//depot/child/... ... a sync gets both `//depot/child/a.txt` and `//depot/parent/b.txt` and both files are mapped to the workspace.	With non-overlay mappings, a view line that takes precedence over preceding lines obscures the preceding lines. With the mapping //depot/parent/... ... //depot/child/... ... a sync gets only `//depot/child/a.txt` because `//depot/parent/b.txt` is inaccessible.
In the case of `//depot/child/a.txt` and `//depot/parent/a.txt`, the sparse stream overlay view takes precedence, so the workspace is mapped to the sparse stream version. The `//depot/parent/b.txt` file is not obscured.

With an overlay mapping, precedence is given to the lower lines, but files in the preceding view lines are not obscured.

With the mapping

//depot/parent/... ...
+//depot/child/... ...

a sync gets both //depot/child/a.txt and //depot/parent/b.txt and both files are mapped to the workspace.

With non-overlay mappings, a view line that takes precedence over preceding lines obscures the preceding lines.

With the mapping

//depot/parent/... ...
//depot/child/... ...

a sync gets only //depot/child/a.txt because //depot/parent/b.txt is inaccessible.

In the case of //depot/child/a.txt and //depot/parent/a.txt, the sparse stream overlay view takes precedence, so the workspace is mapped to the sparse stream version.

The //depot/parent/b.txt file is not obscured.

Manage growth

Sparse streams are well suited for use cases in which you only need to make changes to a relatively small number of files, such as to develop a small bug fix or to release a small bug fix in a patch or hotfix.

As the number of branched files in a sparse stream increases, the risk of a performance degradation also increases.

Limit with configurable

To mitigate that risk, the administrator can control the number of branched files with the dm.stream.sparse.branchmax configurable. If an operation would exceed that limit, the operation is blocked and an error message appears. An administrator with the super access level can change the limit at any time. If the value of the configurable is reset to a value that is less than the number of branched files in any existing sparse stream, that excessively-branched sparse stream will continue to exist, but its files cannot be edited.

Convert to a non-sparse stream

If the number of files that need changes exceed the limit imposed by the dm.stream.sparse.branchmax configurable, you can Convert a sparse stream to a non-sparse stream.

Tip

Editing a parent file branches the file into the sparse stream. Therefore, even if the first edit of a parent file is subsequently reverted, the file remains branched into the sparse stream. You can, however, remove any branched files that have no changes from the sparse stream by using the p4 prune command.

Restrictions

A sparse stream cannot be reparented.
Sparse streams can consume Stream Components, and inherit stream components from a parent, but cannot be consumed as components of other streams.
If a file has become resident in sparse stream, it remains resident, regardless of whether you submit changes to the file or revert the file.
Non-sparse streams cannot be converted to sparse streams, even if the stream was originally sparsedev or sparserel prior to a conversion to development or release.
Unsupported commands and features:
- p4 unsubmit, p4 resubmit, and p4 undo are not supported.
- Like task streams, sparse streams are not supported for use with DVCS or graph depots A depot of type graph that is used to store Git repos managed by Helix Core Server. See also Git Connector and classic depot..
Integrations:
- Each integration must be full, that is, with a file spec of ... because "cherry-picking" a subset of files is not supported.
- The only revision limit allowed is @<=change because the parent depot paths are pinned to a specific change number. The changeview number is automatically updated during integrations into the sparse child. No editing of the changeview number is allowed after any files are populated or added into the sparse stream. The changeview number can be updated only through integration.
Paths:
- Paths of type import and import+ observe the change specified in the import and import+ paths rather than the changeview number in a sparse stream. To make the import path’s files match the sparse stream changeview number, use a change specifier such as import x/... //depot/x/...@N
- Each sparse stream must have at least one share path and ParentViews are always inherit.
Unshelving:
- A sparse stream client cannot unshelve from a different stream, not even from the parent of the sparse stream.
- The -S option of p4 unshelve is not allowed from a client workspace of a sparse stream.
Virtual streams:
- The only type of stream that can be a child of a sparse stream is a virtual stream.
- A virtual child of a sparse streams cannot have children and cannot be the target of an integration.
- A sparse stream cannot be a child of a virtual stream.

Convert a sparse stream to a non-sparse stream

If you find that your stream needs more files than allowed (see Manage growth), you can convert:

a sparsedev stream to a development stream
a sparserel stream to a release stream

The steps are:

p4 stream convertsparse
p4 submit -d "converted to a non-sparse type"

The stream name remains the same, but its type has changed. When the second step, p4 submit, has fully completed, the converted stream will be fully branched to contain the metadata for all the files from the parent stream, and the client spec contains no ChangeView and no overlay mappings.