Blog
Self hosting a rust crate registry with minimal effort
There are many reasons why you, or your team, would want to host a registry for yourself.
The most prominent one is probably that you don’t want to publish proprietary crates on crates.io, either because they contain your intellectual property or it has no use for someone outside your company.
If you’re in the situation that you have to support your product for 5 years or even longer you may want to archive your build environment because services aren’t built for eternity.
Using something like artifactory may be overkill for you and running your own crates.io instance is not very well documented.
So why not start (ab)using the static hosting feature of your git server?
Overview
Running your own registry is described on the official rust-lang site.
The first sentence in that paragraph brought this idea to life.
Instead of adding another server to your enterprise environment we can use the already existing Git and CI servers.
What do I need?
-
A git server
- With the feature to serve static pages
- A git repository for the index, which I’ll be calling
index repository - Another (or the same on another branch) repository for the
crates. Refered to ascrate repository
-
Optionally a build/CI server which automatically updates the
index repository
Basic idea
The user downloads the index from the index repository as he would from the crates.io registry.
cargo clones the index repository and then tries to match your requested packages to the index.
If the crate is found in a matching version the binary is retrieved from the crate repository.
To keep the index repository in sync with the crate repository you may want to use some kind of automation.
Build/CI servers integrated with git servers are perfect for this kind of task.
For ease of use I will explain the setup by using GitHub and GitHub Actions.
You can do the sync/update steps manually if you want to or use any other automation tool you like.
Let’s get to it
Before starting you have to create both repositories on the server.
Prepare the repositories
In the root of the index repository create a config.json for your setup and commit it.
An example config.json looks like this:
{
"dl": "https://cschlosser.github.io/crate-storage/{crate}-{version}.crate"
}
The dl key has to be in the format of http(s)://<static page URL>/<repository>/{crate}-{version}.crate where <static page URL> is determined by your git server and <repository> is the name of the crate repository.
{crate} and {version} will be replaced by cargo.
Clone the crate repository locally and start with an empty commit git commit --allow-empty -m "Initial commit" or a README.md.
Setup your build server so it executes a script like this on each push to the crate repository:
#! /usr/bin/env bash
set -ex
if [ -z $1 ]; then
echo "Usage: $0 /path/to/index/repository"
exit 1
fi
if ! command -v cargo-index; then
echo "cargo-index is not installed. See https://doc.rust-lang.org/cargo/reference/registries.html#index-format how to update your index manually"
exit 1
fi
REPO_DIR="$PWD"
pushd "$1"
# Add crates to the index
for file in $(git --git-dir "$REPO_DIR"/.git --work-tree="$REPO_DIR" diff HEAD~1 --diff-filter=A --name-only); do
if [[ $file != *.crate ]]; then
continue
fi
cargo-index index add --index . --crate "$REPO_DIR/$file" --index-url=$(git remote get-url origin)
done
# Remove crates from the index
for file in $(git --git-dir "$REPO_DIR"/.git --work-tree="$REPO_DIR" diff HEAD~1 --diff-filter=D --name-only); do
if [[ $file != *.crate ]]; then
continue
fi
crate_name=$(sed -nr "s/([A-Za-z_][A-Za-z0-9_\-]*)\-([0-9]\..*)\.crate/\1/p" <<< "$file")
crate_version=$(sed -nr "s/([A-Za-z_][A-Za-z0-9_\-]*)\-([0-9]\..*)\.crate/\2/p" <<< "$file")
cargo-index index yank --index . --package "$crate_name" --version "$crate_version"
done
# You may have to configure git before the script is executed or do this in a separate step
git push
popd
Unlike crates.io this registry marks the crate as yanked and the data is deleted from the dl URL.
Download crates from crates.io
There are several ways to get official crates:
-
Download them directly from crates.io by taking the
dlURL from the crates.io config.json. -
Retrieve them with cargo download.
-
If you want to download a package with all dependencies create a dummy project and add the packages you want downloaded to the
[dependencies]section in theCargo.toml.
If you choose the third option you will have to copy the .crate files from your local cache located at $CARGO_HOME/registry/cache/github.com-<hash>/ into your local crate repository.
The files have to be stored with this pattern: <crate name>-<crate-version>.crate. Copy any downloaded crates into the local crate repository, commit them, push the commit and your build server/CI should start.
If you don’t use a CI server now is the time to update the index repository entries manually.
Publish your internal crates
In the directory of the crate you want to publish run cargo package. The crate will be built and if successful placed in the target/package/ directory.
Copy the crate file into the local crate repository, commit and push it.
Using the crates
The configuration can be made accessible for users by putting it into the README.md of the two repositories.
Note: Crates referencing custom registries can not be pushed to
crates.io.
A custom registry
Add the custom registry to your .cargo/config:
[registries]
github_com = { index = "ssh://git@github.com:22/cschlosser/crate-index.git" }
and in your Cargo.toml you can reference packages from this registry like this:
[package]
name = "example"
version = "0.1.0"
edition = "2018"
[dependencies]
if_empty = { version = "0.2.0", registry = "github_com" }
Replace the official registry
If you want to replace all package downloads with the ones from the custom registry you have to add these sections to the .cargo/config:
[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = "github_com"
[source.github_com]
registry = "ssh://git@github.com:22/cschlosser/crate-index.git"
With this approach you can omit specifying the registry for dependencies in the Cargo.toml.
Final thoughts
If you have access to a git server which provides static hosting, setting up a custom registry has never been easier.
Like all solutions, this one has its advantages as well as its disadvantages. So I put together a small overview:
| Pros | Cons |
|---|---|
| Very easy to set up | Publishing crates is not possible with cargo/standard mechanisms |
No additional permissions system necessary. Just set the correct permissions on the crate repository |
No crate ownership concept |
| Utilize existing infrastructure | |
Enforce your custom rules on crates |
I will gladly extend this table if you send me additional points.
Click here to send feedback to the author
Related projects:
-
cargo local-registry: Run a registry for a package on the local filesystem
-
cargo index: Manipulate the index of registries
Examples:
-
crate repository with GitHub actions configured
-
index repository that is automatically updated