Blog

Self hosting a rust crate registry with minimal effort

There are many reasons why you, or your team, would want to host a registry for yourself.

The most prominent one is probably that you don’t want to publish proprietary crates on crates.io, either because they contain your intellectual property or it has no use for someone outside your company. If you’re in the situation that you have to support your product for 5 years or even longer you may want to archive your build environment because services aren’t built for eternity.

Using something like artifactory may be overkill for you and running your own crates.io instance is not very well documented.

So why not start (ab)using the static hosting feature of your git server?

Running your own registry is described on the official rust-lang site.

The first sentence in that paragraph brought this idea to life.

Instead of adding another server to your enterprise environment we can use the already existing Git and CI servers.

  • A git server

    • With the feature to serve static pages
    • A git repository for the index, which I’ll be calling index repository
    • Another (or the same on another branch) repository for the crates. Refered to as crate repository
  • Optionally a build/CI server which automatically updates the index repository

basic overview

The user downloads the index from the index repository as he would from the crates.io registry.

cargo clones the index repository and then tries to match your requested packages to the index. If the crate is found in a matching version the binary is retrieved from the crate repository.

To keep the index repository in sync with the crate repository you may want to use some kind of automation. Build/CI servers integrated with git servers are perfect for this kind of task.

For ease of use I will explain the setup by using GitHub and GitHub Actions. You can do the sync/update steps manually if you want to or use any other automation tool you like.

Before starting you have to create both repositories on the server.

In the root of the index repository create a config.json for your setup and commit it.

An example config.json looks like this:

{
    "dl": "https://cschlosser.github.io/crate-storage/{crate}-{version}.crate"
}

The dl key has to be in the format of http(s)://<static page URL>/<repository>/{crate}-{version}.crate where <static page URL> is determined by your git server and <repository> is the name of the crate repository. {crate} and {version} will be replaced by cargo.

Clone the crate repository locally and start with an empty commit git commit --allow-empty -m "Initial commit" or a README.md.

Setup your build server so it executes a script like this on each push to the crate repository:

#! /usr/bin/env bash

set -ex

if [ -z $1 ]; then
    echo "Usage: $0 /path/to/index/repository"
    exit 1
fi
if ! command -v cargo-index; then
    echo "cargo-index is not installed. See https://doc.rust-lang.org/cargo/reference/registries.html#index-format how to update your index manually"
    exit 1
fi

REPO_DIR="$PWD"

pushd "$1"

# Add crates to the index
for file in $(git --git-dir "$REPO_DIR"/.git --work-tree="$REPO_DIR" diff HEAD~1 --diff-filter=A --name-only); do
    if [[ $file != *.crate ]]; then
        continue
    fi
    cargo-index index add --index . --crate "$REPO_DIR/$file" --index-url=$(git remote get-url origin)
done

# Remove crates from the index
for file in $(git --git-dir "$REPO_DIR"/.git --work-tree="$REPO_DIR" diff HEAD~1 --diff-filter=D --name-only); do
    if [[ $file != *.crate ]]; then
        continue
    fi
    crate_name=$(sed -nr "s/([A-Za-z_][A-Za-z0-9_\-]*)\-([0-9]\..*)\.crate/\1/p" <<< "$file")
    crate_version=$(sed -nr "s/([A-Za-z_][A-Za-z0-9_\-]*)\-([0-9]\..*)\.crate/\2/p" <<< "$file")
    cargo-index index yank --index . --package "$crate_name" --version "$crate_version"
done

# You may have to configure git before the script is executed or do this in a separate step
git push

popd

Unlike crates.io this registry marks the crate as yanked and the data is deleted from the dl URL.

There are several ways to get official crates:

  1. Download them directly from crates.io by taking the dl URL from the crates.io config.json.

  2. Retrieve them with cargo download.

  3. If you want to download a package with all dependencies create a dummy project and add the packages you want downloaded to the [dependencies] section in the Cargo.toml.

If you choose the third option you will have to copy the .crate files from your local cache located at $CARGO_HOME/registry/cache/github.com-<hash>/ into your local crate repository.

The files have to be stored with this pattern: <crate name>-<crate-version>.crate. Copy any downloaded crates into the local crate repository, commit them, push the commit and your build server/CI should start.

If you don’t use a CI server now is the time to update the index repository entries manually.

In the directory of the crate you want to publish run cargo package. The crate will be built and if successful placed in the target/package/ directory.

Copy the crate file into the local crate repository, commit and push it.

The configuration can be made accessible for users by putting it into the README.md of the two repositories.

Note: Crates referencing custom registries can not be pushed to crates.io.

Add the custom registry to your .cargo/config:

[registries]
github_com = { index = "ssh://git@github.com:22/cschlosser/crate-index.git" }

and in your Cargo.toml you can reference packages from this registry like this:

[package]
name = "example"
version = "0.1.0"
edition = "2018"

[dependencies]
if_empty = { version = "0.2.0", registry = "github_com" }

If you want to replace all package downloads with the ones from the custom registry you have to add these sections to the .cargo/config:

[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = "github_com"

[source.github_com]
registry = "ssh://git@github.com:22/cschlosser/crate-index.git"

With this approach you can omit specifying the registry for dependencies in the Cargo.toml.

If you have access to a git server which provides static hosting, setting up a custom registry has never been easier.

Like all solutions, this one has its advantages as well as its disadvantages. So I put together a small overview:

Pros Cons
Very easy to set up Publishing crates is not possible with cargo/standard mechanisms
No additional permissions system necessary. Just set the correct permissions on the crate repository No crate ownership concept
Utilize existing infrastructure
Enforce your custom rules on crates

I will gladly extend this table if you send me additional points.

Click here to send feedback to the author

Related projects:

Examples:




Legal Disclosure Information in accordance with Section 5 TMG

Contact Information

Telephone: +4915678518718
E-Mail: impress@schlosser.xyz
Internet address: christoph.schlosser.xyz

Disclaimer

Accountability for content
The contents of our pages have been created with the utmost care. However, we cannot guarantee the contents' accuracy, completeness or topicality. According to statutory provisions, we are furthermore responsible for our own content on these web pages. In this matter, please note that we are not obliged to monitor the transmitted or saved information of third parties, or investigate circumstances pointing to illegal activity. Our obligations to remove or block the use of information under generally applicable laws remain unaffected by this as per §§ 8 to 10 of the Telemedia Act (TMG).

Accountability for links
Responsibility for the content of external links (to web pages of third parties) lies solely with the operators of the linked pages. No violations were evident to us at the time of linking. Should any legal infringement become known to us, we will remove the respective link immediately.

Copyright
Our web pages and their contents are subject to German copyright law. Unless expressly permitted by law, every form of utilizing, reproducing or processing works subject to copyright protection on our web pages requires the prior consent of the respective owner of the rights. Individual reproductions of a work are only allowed for private use. The materials from these pages are copyrighted and any unauthorized use may violate copyright laws.
Privacy Policy

Personal data (usually referred to just as "data" below) will only be processed by us to the extent necessary and for the purpose of providing a functional and user-friendly website, including its contents, and the services offered there.

Per Art. 4 No. 1 of Regulation (EU) 2016/679, i.e. the General Data Protection Regulation (hereinafter referred to as the "GDPR"), "processing" refers to any operation or set of operations such as collection, recording, organization, structuring, storage, adaptation, alteration, retrieval, consultation, use, disclosure by transmission, dissemination, or otherwise making available, alignment, or combination, restriction, erasure, or destruction performed on personal data, whether by automated means or not.

The following privacy policy is intended to inform you in particular about the type, scope, purpose, duration, and legal basis for the processing of such data either under our own control or in conjunction with others. We also inform you below about the third-party components we use to optimize our website and improve the user experience which may result in said third parties also processing data they collect and control.

Our privacy policy is structured as follows:

I. Information about us as controllers of your data
II. The rights of users and data subjects
III. Information about the data processing

I. Information about us as controllers of your data

The party responsible for this website (the "controller") for purposes of data protection law is:

Christoph Schlosser

Telephone: +49 15678 518718

Email: privacy@schlosser.xyz

The controller's data protection officer is:

Christoph Schlosser

II. The rights of users and data subjects

With regard to the data processing to be described in more detail below, users and data subjects have the right

  • to confirmation of whether data concerning them is being processed, information about the data being processed, further information about the nature of the data processing, and copies of the data (cf. also Art. 15 GDPR);
  • to correct or complete incorrect or incomplete data (cf. also Art. 16 GDPR);
  • to the immediate deletion of data concerning them (cf. also Art. 17 DSGVO), or, alternatively, if further processing is necessary as stipulated in Art. 17 Para. 3 GDPR, to restrict said processing per Art. 18 GDPR;
  • to receive copies of the data concerning them and/or provided by them and to have the same transmitted to other providers/controllers (cf. also Art. 20 GDPR);
  • to file complaints with the supervisory authority if they believe that data concerning them is being processed by the controller in breach of data protection provisions (see also Art. 77 GDPR).

In addition, the controller is obliged to inform all recipients to whom it discloses data of any such corrections, deletions, or restrictions placed on processing the same per Art. 16, 17 Para. 1, 18 GDPR. However, this obligation does not apply if such notification is impossible or involves a disproportionate effort. Nevertheless, users have a right to information about these recipients.

Likewise, under Art. 21 GDPR, users and data subjects have the right to object to the controller's future processing of their data pursuant to Art. 6 Para. 1 lit. f) GDPR. In particular, an objection to data processing for the purpose of direct advertising is permissible.

III. Information about the data processing

Your data processed when using our website will be deleted or blocked as soon as the purpose for its storage ceases to apply, provided the deletion of the same is not in breach of any statutory storage obligations or unless otherwise stipulated below.

Server data

For technical reasons, the following data sent by your internet browser to us or to our server provider will be collected, especially to ensure a secure and stable website: These server log files record the type and version of your browser, operating system, the website from which you came (referrer URL), the webpages on our site visited, the date and time of your visit, as well as the IP address from which you visited our site.

The data thus collected will be temporarily stored, but not in association with any other of your data.

The basis for this storage is Art. 6 Para. 1 lit. f) GDPR. Our legitimate interest lies in the improvement, stability, functionality, and security of our website.

The data will be deleted within no more than seven days, unless continued storage is required for evidentiary purposes. In which case, all or part of the data will be excluded from deletion until the investigation of the relevant incident is finally resolved.

Contact

If you contact us via email or the contact form, the data you provide will be used for the purpose of processing your request. We must have this data in order to process and answer your inquiry; otherwise we will not be able to answer it in full or at all.

The legal basis for this data processing is Art. 6 Para. 1 lit. b) GDPR.

Your data will be deleted once we have fully answered your inquiry and there is no further legal obligation to store your data, such as if an order or contract resulted therefrom.

Social media links via graphics

We also integrate the following social media sites into our website. The integration takes place via a linked graphic of the respective site. The use of these graphics stored on our own servers prevents the automatic connection to the servers of these networks for their display. Only by clicking on the corresponding graphic will you be forwarded to the service of the respective social network.

Once you click, that network may record information about you and your visit to our site. It cannot be ruled out that such data will be processed in the United States.

Initially, this data includes such things as your IP address, the date and time of your visit, and the page visited. If you are logged into your user account on that network, however, the network operator might assign the information collected about your visit to our site to your personal account. If you interact by clicking Like, Share, etc., this information can be stored your personal user account and possibly posted on the respective network. To prevent this, you need to log out of your social media account before clicking on the graphic. The various social media networks also offer settings that you can configure accordingly.

The following social networks are integrated into our site by linked graphics:

LinkedIn

LinkedIn Ireland Unlimited Company, Wilton Plaza, Wilton Place, Dublin 2, Irland, a subsidiary of LinkedIn Corporation, 1000 W. Maude Avenue, Sunnyvale, CA 94085 USA.

Privacy Policy: https://www.linkedin.com/legal/privacy-policy

EU-US Privacy Shield https://www.privacyshield.gov/participant?id=a2zt0000000L0UZAA0&status=Active

GitHub

GitHub, Inc., 88 Colin P Kelly Jr Street, San Francisco, California 94107.

Privacy Policy: https://help.github.com/en/github/site-policy/github-privacy-statement

EU-US Privacy Shield https://www.privacyshield.gov/participant?id=a2zt000000001K2AAI

Google Fonts

Our website uses Google Fonts to display external fonts. This is a service provided by Google Ireland Limited, Gordon House, Barrow Street, Dublin 4, Irland (hereinafter: Google).

Through certification according to the EU-US Privacy Shield

https://www.privacyshield.gov/participant?id=a2zt000000001L5AAI&status=Active

Google guarantees that it will follow the EU's data protection regulations when processing data in the United States.

To enable the display of certain fonts on our website, a connection to the Google server in the USA is established whenever our website is accessed.

The legal basis is Art. 6 Para. 1 lit. f) GDPR. Our legitimate interest lies in the optimization and economic operation of our site.

When you access our site, a connection to Google is established from which Google can identify the site from which your request has been sent and to which IP address the fonts are being transmitted for display.

Google offers detailed information at

https://adssettings.google.com/authenticated

https://policies.google.com/privacy

in particular on options for preventing the use of data.

BootstrapCDN

We use the service BootstrapCDN, a content delivery network (“CDN”), on our website to optimise the speed of retrieval, the design and presentation of the contents of our website on different devices.  This is a service of StackPath, LLC, 2021 McKinney Ave.  Suite 1100, Dallas, TX 75201, USA, hereinafter simply referred to as "BootstrapCDN".

In order to be able to deliver the content of our website quickly, the service uses so-called JavaScript libraries.  Corresponding files are loaded from a BootstrapCDN server that records your IP address. 

Through certification according to the EU-US Privacy Shield

https://www.privacyshield.gov/participant?id=a2zt0000000CbahAAC&status=Active

BootstrapCDN guarantees that it will follow the EU's data protection regulations also when processing data in the United States.

At

https://www.stackpath.com/legal/master-service-agreement/#privacy  

Prospectone Sp.z.o.o. offers further data protection information.

The legal basis is Art.  6 para. 1 lit. f) GDPR. Our legitimate interest lies in speeding up loading times and protecting our website, as well as in analysing and optimising our website.

To totally prevent the execution of BootstrapCDN’s JavaScript code, you can install a so-called JavaScript blocker, such as noscript.net or ghostery.com.  If you were to prevent or restrict the execution of the Java Script code, this can mean that for technical reasons not all the content and functions of our website may be available.

jQuery CDN

We use the service jQuery CDN, a Content Delivery Network ("CDN"), on our website to optimise the retrieval speed, the design and presentation of the content of our website on different devices. 

This is a service of the jQuery Foundation, hereinafter referred to as "jQuery". jQuery is distributed for the JS Foundation via the StackPath CDN.

In order to be able to deliver the content of our website quickly, the service uses so-called JavaScript libraries.  Relevant files are loaded from the CDN server if they are not already available in your browser cache when you visit another website.  In the case of a connection to the CDN server, your IP address is recorded.  It cannot be ruled out that a connection will be established to a server outside the EU.

The legal basis is Art.  6 para. 1 lit. f) GDPR. Our legitimate interest lies in speeding up loading times and protecting our website, as well as in analysing and optimising our website.

To totally prevent the execution of jQuery’s JavaScript code, you can install a so-called JavaScript blocker, such as noscript.net or ghostery.com. . If you were to prevent or restrict the execution of the Java Script code, this can mean that for technical reasons not all the content and functions of our website may be available.

CloudFlare

To secure our website and to optimize loading times, we use the CloudFlare CDN (content delivery network). This is a service of Cloudflare Inc., 101 Townsend Street, San Francisco, California 94107, USA, hereinafter referred to as "CloudFlare".

The legal basis for collecting and processing this information is Art. 6 Para. 1 lit. f) GDPR. Our legitimate interest lies in the secure operation of our website and in its optimization.

If you access our website, your queries are forwarded to CloudFlare servers. Statistical access data about your visit to our website is collected and CloudFlare stores a cookie on your terminal device via your browser. Access data includes

- your IP address;

- the page(s) on our site that you access;

- type and version of internet browser you are using;

- your operating system;

- the website from which you came prior to visiting our website (referrer URL);

- your length of stay on our site; and 

- the frequency with which our pages are accessed.

The data is used by CloudFlare for statistical evaluations of the accesses as well as for the security and optimization of the offer.

If you do not agree to this processing, you have the option of preventing the installation of cookies by making the appropriate settings in your browser. Further details can be found in the section about cookies above.

CloudFlare offers further information about its data collection and processing as well your rights and your options for protecting your privacy at this link:

https://www.cloudflare.com/privacypolicy/.

Model Data Protection Statement for Anwaltskanzlei Weiß & Partner