Using Bytesafe - Upstreams

Want to have an npm proxy between you and the public registry? Want to make sure all teams are using on the same package versions? Want to delegate the responsibility of making releases?

Using a Bytesafe private registry with Upstreams make this possible!

This post aims to guide you through the concept of upstreams. Read on for insight into:

• Push & pull of package versions when using Bytesafe with upstreams
• Advantages of using and configuring upstreams in your organization
• Example upstream configurations to fit your organization’s workflow

What are upstreams?

Upstreams are linked registries that are used both as a destination and source for packages in your private registries.

Upstreams can be:

1. Public npm registry (registry.npmjs.org)
2. Other Bytesafe registry
3. URL to any npm compatible registry
4. Git repository (see documentation for details)

Package versions can be downloaded (“pulled”) from an upstream to a registry (either manually or when running npm install). Packages can also be published (“pushed”) from a Bytesafe registry up to an upstream.

Configuring upstreams with private registries

In the simplest use-case, Bytesafe offers private registries to manage packages for internal use within an organization.

If a registry is configured without any upstreams:

• Developers can publish package versions to the private registry
• Package versions can NOT be published from the private registry to public npm or any other registry
• Installing packages from the registry requires the package version to be available, else the install fails

This is the default scenario when a new empty Bytesafe registry is created by the user.

For some teams this case may be the preferred option, as they use the registry only for internal packages. But most teams also depend on package versions available in the public registry.

Let’s look at what adding the public npm registry as a upstream looks like.

Add npmjs as upstream to the registry

Adding the public npm registry as an upstream adds new options and features for a team’s Bytesafe registry. The Bytesafe registry is still private and only the team members can access its contents.

Package versions pulled by a developer will also be downloaded and stored in the Bytesafe registry, in essence creating a npm proxy (with the security benefits that entails).

If the public npm upstream is configured with a valid npm token, package versions can also be pushed from the bytesafe registry to the public registry.

Pushing and pulling packages

Bytesafe supports two different ways of interacting with a registry, using the regular npm cli or using the Bytesafe cli / web app.

The main difference between using npm and bytesafe commands lies in the scenario where you use them, where npm is typically used by developers interacting with the registry from a project workspace and bytesafe is used for moving versions between registries.

npm publish & bytesafe push

Both commands push package versions to a target (upstream):

• npm publish - publishes from a project workspace to a registry.
• bytesafe push pushes between registries and upstreams (public npm or other registry).

npm install & bytesafe pull

Both commands retrieve packages from a source (upstream):

• npm install - pulls a package (and dependencies) from a upstream (and installs it in your project workspace).
• bytesafe pull - pulls a package (without dependencies) from a upstream into a registry.

Bytesafe is designed to let developers preserve the regular npm workflow.

Installing packages from a registry with upstreams

Installing a package with npm install is a three-step process. First a list of all available versions for the package is downloaded, then a specific version is chosen from the list, and last the actual contents are downloaded.

When a bytesafe registry have one or more upstreams, npm will receive all available versions of the package, including the versions available in the upstream registries.

When the contents of a specific version is then downloaded, bytesafe will pull the contents from an upstream if needed.

The upstreams are searched in order, which can be changed in the registry settings in Bytesafe.

Different upstream configurations for different workflows

Bytesafe allows a lot flexibility to organize registries and upstreams to fit your specific workflows.

Use the following example configurations as an inspiration for whats possible.

Use Case: Chaining registries

For certain organizations it may not be ideal to have a setup where a developer-facing Bytesafe registry is directly connected to the public registry.

Teams might use a dedicated registry but still want to share private package versions, or maybe a new registry is setup for each sprint or release cycle, or a central registry is needed to control the flow of package versions from the outside.

In this case adding one or more layers of registries between developer teams and the public registry may be beneficial.

Each private registry is only configured with the layer above it as an upstream.

This way of structuring the flow of packages between teams and the outside is especially powerful in combination with policies and plugins.

Remember, pulling packages is recursive

So what happens when a developer want to pull new package versions?

As pulls using Bytesafe are recursive, the pull request action will be passed from registry to registry until the package is found (or it fails).

So although the developers registry is only configured with another private registry as a upstream, they are able to continue working using third party packages.

If the same developer want to push a new package version it only has to push it to the layer above, where it can be distributed to other organization members. No need to keep track of credentials for the public registry or who has the right to publish a new package version. That responsibility is handled by the layer above.

Use Case: Registry with multiple upstreams

What if your team or organization want to use packages, both third party and internal, from different sources without setting up a chain of registries?

Bytesafe supports registries configured with multiple upstreams where the priority of the upstreams decides the order in which they are queried for package versions.

If a team uses a Bytesafe registry that has two upstreams configured with the following priority:

• Priority 1: partner registry (e.g. other team, outsourced dev team etc.)
• Priority 2: public npm registry

When a developer installs a package from the private team registry:

• If the private registry contains the package it will be returned immediately
• If not the package will be pulled from the upstream with the highest priority (i.e the partner registry)

This is can be very useful if the partner registry contains a patched version of a public package for example!

Continued reading - Plugins and policies

The concept of upstreams introduce potential for controlling the flow of packages between registries that your organization control and the public npm registry. By understanding upstreams you can already setup a flow where only approved packages are available for downstream registries or control who is allowed to publish to public npm.

Continue reading our series on Using Bytesafe, to see how Plugins & Policies can secure your code supply chain!