Self hosting a rust crate registry with minimal effort

There are many reasons why you, or your team, would want to host a registry for yourself.

The most prominent one is probably that you don’t want to publish proprietary crates on crates.io, either because they contain your intellectual property or it has no use for someone outside your company. If you’re in the situation that you have to support your product for 5 years or even longer you may want to archive your build environment because services aren’t built for eternity.

Using something like artifactory may be overkill for you and running your own crates.io instance is not very well documented.

So why not start (ab)using the static hosting feature of your git server?

Overview

Running your own registry is described on the official rust-lang site.

The first sentence in that paragraph brought this idea to life.

Instead of adding another server to your enterprise environment we can use the already existing Git and CI servers.

What do I need?

• A git server

  • With the feature to serve static pages
  • A git repository for the index, which I’ll be calling index repository
  • Another (or the same on another branch) repository for the crates. Refered to as crate repository

• Optionally a build/CI server which automatically updates the index repository

Basic idea

The user downloads the index from the index repository as he would from the crates.io registry.

cargo clones the index repository and then tries to match your requested packages to the index. If the crate is found in a matching version the binary is retrieved from the crate repository.

To keep the index repository in sync with the crate repository you may want to use some kind of automation. Build/CI servers integrated with git servers are perfect for this kind of task.

For ease of use I will explain the setup by using GitHub and GitHub Actions. You can do the sync/update steps manually if you want to or use any other automation tool you like.

Let’s get to it

Before starting you have to create both repositories on the server.

Prepare the repositories

In the root of the index repository create a config.json for your setup and commit it.

An example config.json looks like this:

{
    "dl": "https://cschlosser.github.io/crate-storage/{crate}-{version}.crate"
}

The dl key has to be in the format of http(s)://<static page URL>/<repository>/{crate}-{version}.crate where <static page URL> is determined by your git server and <repository> is the name of the crate repository. {crate} and {version} will be replaced by cargo.

Clone the crate repository locally and start with an empty commit git commit --allow-empty -m "Initial commit" or a README.md.

Setup your build server so it executes a script like this on each push to the crate repository:

#! /usr/bin/env bash

set -ex

if [ -z $1 ]; then
    echo "Usage: $0 /path/to/index/repository"
    exit 1
fi
if ! command -v cargo-index; then
    echo "cargo-index is not installed. See https://doc.rust-lang.org/cargo/reference/registries.html#index-format how to update your index manually"
    exit 1
fi

REPO_DIR="$PWD"

pushd "$1"

# Add crates to the index
for file in $(git --git-dir "$REPO_DIR"/.git --work-tree="$REPO_DIR" diff HEAD~1 --diff-filter=A --name-only); do
    if [[ $file != *.crate ]]; then
        continue
    fi
    cargo-index index add --index . --crate "$REPO_DIR/$file" --index-url=$(git remote get-url origin)
done

# Remove crates from the index
for file in $(git --git-dir "$REPO_DIR"/.git --work-tree="$REPO_DIR" diff HEAD~1 --diff-filter=D --name-only); do
    if [[ $file != *.crate ]]; then
        continue
    fi
    crate_name=$(sed -nr "s/([A-Za-z_][A-Za-z0-9_\-]*)\-([0-9]\..*)\.crate/\1/p" <<< "$file")
    crate_version=$(sed -nr "s/([A-Za-z_][A-Za-z0-9_\-]*)\-([0-9]\..*)\.crate/\2/p" <<< "$file")
    cargo-index index yank --index . --package "$crate_name" --version "$crate_version"
done

# You may have to configure git before the script is executed or do this in a separate step
git push

popd

Unlike crates.io this registry marks the crate as yanked and the data is deleted from the dl URL.

Download crates from crates.io

There are several ways to get official crates:

1. Download them directly from crates.io by taking the dl URL from the crates.io config.json.

2. Retrieve them with cargo download.

3. If you want to download a package with all dependencies create a dummy project and add the packages you want downloaded to the [dependencies] section in the Cargo.toml.

If you choose the third option you will have to copy the .crate files from your local cache located at $CARGO_HOME/registry/cache/github.com-<hash>/ into your local crate repository.

The files have to be stored with this pattern: <crate name>-<crate-version>.crate. Copy any downloaded crates into the local crate repository, commit them, push the commit and your build server/CI should start.

If you don’t use a CI server now is the time to update the index repository entries manually.

Publish your internal crates

In the directory of the crate you want to publish run cargo package. The crate will be built and if successful placed in the target/package/ directory.

Copy the crate file into the local crate repository, commit and push it.

Using the crates

The configuration can be made accessible for users by putting it into the README.md of the two repositories.

Note: Crates referencing custom registries can not be pushed to crates.io.

A custom registry

Add the custom registry to your .cargo/config:

[registries]
github_com = { index = "ssh://git@github.com:22/cschlosser/crate-index.git" }

and in your Cargo.toml you can reference packages from this registry like this:

[package]
name = "example"
version = "0.1.0"
edition = "2018"

[dependencies]
if_empty = { version = "0.2.0", registry = "github_com" }

Replace the official registry

If you want to replace all package downloads with the ones from the custom registry you have to add these sections to the .cargo/config:

[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = "github_com"

[source.github_com]
registry = "ssh://git@github.com:22/cschlosser/crate-index.git"

With this approach you can omit specifying the registry for dependencies in the Cargo.toml.

Final thoughts

If you have access to a git server which provides static hosting, setting up a custom registry has never been easier.

Like all solutions, this one has its advantages as well as its disadvantages. So I put together a small overview:

Pros	Cons
Very easy to set up	Publishing crates is not possible with cargo/standard mechanisms
No additional permissions system necessary. Just set the correct permissions on the crate repository	No crate ownership concept
Utilize existing infrastructure
Enforce your custom rules on crates

I will gladly extend this table if you send me additional points.

Click here to send feedback to the author

Related projects:

• cargo local-registry: Run a registry for a package on the local filesystem

• cargo index: Manipulate the index of registries

Examples:

• crate repository with GitHub actions configured

• index repository that is automatically updated