Production deployments

This section is dedicated to the task of deploying an IPFS Cluster and running it in a stable fashion. It describes:

Make sure you are familiar with the Configuration section first.

All the IPFS Cluster peers in a cluster must be running the **minor.major** of `ipfs-cluster-service`: any peer in `0.6.x` will work together, but `0.6.x` will not work with `0.7.x` peers.

Deployment methods

This subsection provides different resources to automate the deployment of an IPFS Cluster:

Help completing this section

We would be very grateful if you have used different methods to deploy IPFS Cluster (docker, kubernetes, puppet etc.) and share your know-how. Let us know in the website repository.

Running IPFS Cluster in production

This subsection provides useful information for running go-ipfs and IPFS Cluster in a stable production environment.

service.json configuration tweaks

The configuration file contains a few options which should be tweaked according to your environment, capacity and requirements:

go-ipfs configuration tweaks

[BACKUP ~/.ipfs]
$ ipfs config profile apply badgerds # or ipfs init --profile=server,badgerds
$ ipfs-ds-convert convert # Make sure you have enough disk space for the conversion.
$ ipfs-ds-convert cleanup # removes the backup data

Make sure you have enough space for the conversion.

Automatically upgrade on restart

For easy and quick upgrades, make sure your system starts and restarts IPFS Cluster and go-ipfs peers as follows:

ipfs-cluster-service daemon --upgrade
ipfs daemon --migrate

Systemd service files

Modifying the peerset: adding and removing peers

This subsection explains how to modify the cluster’s peerset. The peerset is maintained by the consensus implementation, so instructions are specific to the implementation used. Right now, only the raft implementation is available.

Raft consensus

Raft is our default consensus implementation. It provides high availability, protection against network splits and fast state convergence. It is appropiate for small sized clusters (what small means is to be determined, but probably < 20 peers) running in trusted environments.

The downside is that Raft requires strict procedures when updating the cluster peerset in order to assure consistency and correct operations of the consensus. In fact, updating the peerset is a commit operation in Raft, meaning that it always needs a functioning leader (and thus, the majority of peers in the peerset need to be online for it to take effect).

Adding peers

Adding peers should always be performed by bootstrapping as explained here.

Removing peers

Removing a peer is a final operation for that peer. That means, that peer cannot (should not) be started again unless its Raft state is cleaned up (ipfs-cluster-service state clean).

Removing peers can be done using ipfs-cluster-ctl which calls the DELETE /peers/<id> API endpoint:

ipfs-cluster-ctl peers rm <peerID>

A peer ID looks like QmQHKLBXfS7hf8o2acj7FGADoJDLat3UazucbHrgxqisim. Removing a peer has the following effects:

The `leave_on_shutdown` option triggers automatic removal on clean shutdowns.

Monitoring and automatic re-pinning

IPFS Cluster includes a monitoring component which gathers metrics and triggers alerts when a metric is no longer renewed. There are currently two types of metrics:

Every metric carries a Time-To-Live associated with it. This TTL can be configued in the informer configuration section. The ping metric TTL is determined by the cluster.monitoring_ping_interval, and is equal to 2x its value.

Every IPFS Cluster peer broadcasts metrics regularly to all other peers. This happens TTL/2 intervals for the informer metrics and in cluster.monitoring_ping_interval for the ping metric.

When a metric for an existing cluster peer stops arriving and previous metrics have outlived their Time-To-Live, the monitoring component triggers an alert for that metric. monbasic.check_interval determines how often the monitoring component checks for expired TTLs and sends these alerts. If you wish to detect expired metrics more quickly, decrease this interval. Otherwise, increase it.

The IPFS Cluster peer will react to ping metrics alerts by searching for pins allocated to the alerting peer and triggering re-pinning requests for them, unless the cluster.disable_repinning option is true. These re-pinning requests may result in re-allocations if the the CID’s allocation factor crosses the replication_factor_min boundary. Otherwise, the current allocations are maintained.

The monitoring and failover system in cluster is very basic and requires improvements. Failover is likely to not work properly when several nodes go offline at once (specially if the current Leader is affected). Manual re-pinning can be triggered with ipfs-cluster-ctl pin <cid>. ipfs-cluster-ctl pin ls <CID> can be used to inspect the current list of peers allocated to a CID.

Data persistence and backups

Backups are never a bad thing. This subsection explains what IPFS Cluster does to make sure your pinset is not lost in a disaster event, and what further measures you can take.

When we speak of backups, we are normally referring to the ~/.ipfs-cluster/raft folder (state folder), which effectively contains the cluster’s pinset and other consensus-specific information.

When a peer is removed from the cluster, or when the user runs ipfs-cluster-service state clean, the state folder is not removed. Instead, it is renamed to raft.old.X, with the newest copy being raft.old.0. The number of copies kept around is configurable (raft.backups_rotate).

On the other side, raft additionally takes regular snapshots of the pinset (which means it is fully persisted to disk). This is also performed on a clean shutdown of the peers.

When the peer is not running, the last persisted state can be manually exported with:

ipfs-cluster-service state export

This will output the pinset, which can be in turn re-imported to a peer with:

ipfs-cluster-service state import

export and import can be used to salvage a state in the case of a disaster event, when peers in the cluster are offline, or not enough peers can be started to reach a quorum (when using raft). In this case, we recommend importing the state on a new, clean, single-peer, cluster, and bootstrapping the rest of the cluster to it manually.

Next steps: Troubleshooting