Adding and pinning
Cluster usage mostly consists on adding and removing pins. This is usually performed using the
ipfs-cluster-ctl utility or talking to one of the Cluster APIs.
ipfs-cluster-ctl <command> --help
When working with a large number of pins, it is important to keep an eye on the state of the pinset, whether every pin is getting correctly pinned an allocated. This section provides in-depth explanations on how pinning works and the different operations that a cluster peer can perform to simplify and maintain the cluster pinsets.
ipfs-cluster-ctlcommands, but every one of them is using an HTTP REST API endpoint from the cluster peer, so all commands can be performed directly against the API.
- Adding files
- Pinning CIDs
- Replication factors
- The pinning process
- Filtering results
- Automatic syncing and recovering
ipfs-cluster-ctl add myfile.txt
ipfs-cluster-ctl add command is very similar to the
ipfs add command and share most of the same options (such as those that define chunking, the DAG type or which CID-version to use).
However, where the
ipfs add command only adds to the local IPFS daemon, the
ipfs-cluster-ctl add command will add to several Cluster peers at the same time. How many it adds depends on the replication factors you set as command flags pin or the defaults in the configuration file.
This means that when the add process is finished, your file will have been fully added to several IPFS daemons (and not necessarily the local one). For example:
$ ipfs-cluster-ctl add pinning.md added QmarNBnreCx4YtT4ETXxQ4dn2xQpcTGd2PaVM4b2UuyGku pinning.md $ ipfs-cluster-ctl pin ls QmarNBnreCx4YtT4ETXxQ4dn2xQpcTGd2PaVM4b2UuyGku # check pin data QmarNBnreCx4YtT4ETXxQ4dn2xQpcTGd2PaVM4b2UuyGku | | PIN | Repl. Factor: -1 | Allocations: [everywhere] | Recursive $ ipfs-cluster-ctl status QmarNBnreCx4YtT4ETXxQ4dn2xQpcTGd2PaVM4b2UuyGku # request status from every peer QmarNBnreCx4YtT4ETXxQ4dn2xQpcTGd2PaVM4b2UuyGku : > cluster0 : PINNED | 2019-07-26T12:25:18.23191214+02:00 > cluster1 : PINNED | 2019-07-26T10:25:18.234842017Z > cluster2 : PINNED | 2019-07-26T10:25:18.212836746Z > cluster3 : PINNED | 2019-07-26T10:25:18.238415569Z > cluserr4 : PINNED | 2019-07-26T10:25:24.508614677Z
The process of adding this way is however way slower than adding to a local IPFS daemon and it is not recommended for bigger files (i.e. more than 3MB). As an alternative, the
--local flag can be provided to the
add command. In this case, the content will be added to the local IPFS daemon of the peer receiving the request, and then pinned normally. The arrival of the pin will make ipfs retrieve the file faster than it would have been to send each block individually, so this way is more appropiate for larger files.
--local with positive replication-factors may mean that the content is added on a peer that is finally not allocated to store the content and will not end up pinning it. Sometimes, when this is relevant, it can be worked around using the
--allocations flag to force the pin to be allocated to the same peer ID on which it is being added.
Another feature in the
add command that is not available on IPFS is the possibility of importing CAR files (ala.
ipfs dag import). In order to import a CAR file you can do
ipfs-cluster-ctl add --format car myfile.car. CAR files should have a single root, which is the CID that becomes pinned after import.
ipfs-cluster-ctl pin add <cid/ipfs-path>
In many cases, you know what content from the IPFS network you want to add to your Cluster. The
ipfs-cluster-ctl pin add operation is similar to the
ipfs pin add one, but allows to set Cluster-specific flags, such the replication factors or the name associated to a pin. For example:
$ ipfs-cluster-ctl pin add --name cluster-website --replication 2 /ipns/cluster.ipfs.io QmXvQLhK2heNz65fWRabTfbzXwYfaBgEBuTdUJNzp69Xjx : > cluster0 : PINNING | 2019-07-26T12:31:08.180738872+02:00 > cluster1 : REMOTE | 2019-07-26T10:31:08.231791643Z > cluster2 : PINNING | 2019-07-26T10:31:08.223206563Z > cluster3 : REMOTE | 2019-07-26T10:31:08.227396652Z > cluster4 : REMOTE | 2019-07-26T10:31:09.189573842Z $ ipfs-cluster-ctl pin ls QmXvQLhK2heNz65fWRabTfbzXwYfaBgEBuTdUJNzp69Xjx QmXvQLhK2heNz65fWRabTfbzXwYfaBgEBuTdUJNzp69Xjx | cluster-website | PIN | Repl. Factor: 2--2 | Allocations: [12D3KooWGbmjg3MDUYFosLNPbE1jKkv5fzKHD7wyGDa1P95iKMjF QmYY1ggjoew5eFrvkenTR3F4uWqtkBkmgfJk8g9Qqcwy51] | Recursive
As we see, the pin started pinning in two places (replication = 2). When we check the pin object, we see both the peer IDs it was allocated to and that it is called
Pins can be removed at any time with
ipfs-cluster-ctl pin rm.
Every pin submitted to IPFS Cluster carries two replication options:
The cluster configuration sets the default values that apply when the option is not set.
The replication_factor_min value specifies the minimal number of copies that the pin should have in the cluster. If automatic repinning is enabled and the cluster detects that the peers that should be pinning an item are not available, and that the item is under-replicated (the number of peers pinning it is below
replication_factor_min), it will re-allocate the item to new peers. Pinning will fail directly when there are not enough peers to pin something up to
The replication_factor_max value indicates how many peers should be allocated to the pin. On pin submission, the cluster will try to allocate that many peers, but not fail if it cannot find so many, as long as it finds more than
replication_factor_min. Repinnings of an item will try to increase allocations to
replication_factor_max, however automatic repinnings of an item, when enable, will not affect pins that are between the two thresholds.
The recommendation is to use thresholds with some leeway (usually 2-3, or 3-5) when
disable_repinning is set to
false. In this case, without leeway, a cluster peer going down for a few seconds could trigger repinnings and result in an unbalanced cluster, even if the peer comes up fine later and still holds the content (at which point it will be unpinned because it is no longer allocated to it).
The pinning process
Cluster-pinning and unpinning are at the core of the cluster operation and involve multiple internal components but have two main stages:
- The Cluster-pinning stage: the pin is correctly persisted by cluster and broadcasted to every other peer.
- The IPFS-pinning stage: every peer allocated to the pin must ensure the IPFS daemon gets it pinned.
The stages and the results they produce are actually inspectable with the different options for
ipfs-cluster-ctl pin add ...waits 1 second by default and reports
statusresulting from the ongoing IPFS-pinning process.
ipfs-cluster-ctl pin add --no-status ...does not wait and reports
pin lsinformation resulting from the Cluster-pinning process.
ipfs-cluster-ctl pin add --wait ...waits until the IPFS-pinning process is complete.
The Cluster-pinning stage
We consider a
pin add operation has been successful when the cluster-pinning stage is finished. This means the pin has been ingested by Cluster and that things are underway to tell IPFS to pin the content. If IPFS fails to pin the content, Cluster will know, report about it and try to handle the situation. The cluster-pinning stage is relatively fast, but the ipfs-pinning stage can take days. Therefore the second stage happens asynchronously once the cluster-pinning stage is completed.
The process can be summarized as a follows:
- A pin request arrives including certain options.
- Given the options (particularly replication factors), a list of current cluster peers is selected as “allocations” for that pin, based on how much free space is available on each.
- These and other things result in a pin object which is commited and broadcasted to everyone (the how depends on the consensus component).
The IPFS-pinning stage
Once the Cluster-pinning stage is completed, each peer is notified of a new item in the pinset. If the peer is among the allocations for that item, it will proceed to ask ipfs to pin it:
- Peer starts “tracking” CID
- If allocated to it, an “ipfs pin add” operation is triggered
- The tracking process waits for the “ipfs pin add” operation to succeed.
pin ls vs
It is very important to distinguish between
ipfs-cluster-ctl pin ls and
ipfs-cluster-ctl status. Both endpoints provide a list of CIDs pinned in the cluster, but they do it in very different ways:
pin lsshows shows information from the cluster shared state or global pinset which is fully available in every peer. It shows which are the allocations and how the pin has been configured. For example:
QmY7UvWxx2oPBzTpJdHKTrjJbzKYA5GF8Qd8SnGpjXCFAp | | PIN | Repl. Factor: 2--3 | Allocations: [12D3KooWGbmjg3MDUYFosLNPbE1jKkv5fzKHD7wyGDa1P95iKMjF QmSGCzHkz8gC9fNndMtaCZdf9RFtwtbTEEsGo4zkVfcykD QmdFBMf9HMDH3eCWrc1U11YCPenC3Uvy9mZQ2BedTyKTDf] | Recursive
status. however, requests information about the status of each pin on each cluster peer, including whether that CID is PINNED on IPFS, or still PINNING, or errored for some reason:
QmY7UvWxx2oPBzTpJdHKTrjJbzKYA5GF8Qd8SnGpjXCFAp : > cluster0 : PINNED | 2019-07-26T12:18:29.834862706+02:00 > cluster1 : PINNING | 2019-07-26T10:25:18.365068131Z > cluster2 : REMOTE | 2019-07-26T10:25:18.356000031Z > cluster3 : REMOTE | 2019-07-26T10:25:18.374354035Z > cluster4 : PINNING | 2019-07-26T10:28:19.661061918Z
In order to show this information, the
status request must contact every of the peers (only the peers pinning something can return on what state that operation is). Thus, the
status request can be very expensive on clusters with many peers, but provides very useful information on the state of pin. Both commands take an optional
cid to limit the results to a single item.
status commands supports filtering to display only pins which are in a given situation (in at least one of the peers). The following filters are supported:
cluster_error: pins for which we cannot obtain status information (i.e. the cluster peer is down)
pin_error: pins that failed to pin (due to an ipfs problem or a timeout)
unpin_error: pins that failed to unpin (due to an ipfs problem or a timeout)
error: pins in
unexpected_unpinned: pins that are not pinned by ipfs, yet they should be pinned and are not pin_queued or pinning right now.
pinned: pins were correctly pinned
pinning: pins that are currently being pinned by ipfs
unpinning: pins that are currently being unpinned by ipfs
remote: pins that are allocated to other cluster peers (remote means: not handled by this peer).
pin_queued: pins that are waiting to start pinning (usually because ipfs is already pinning a bunch of other things)
unpin_queued: pins that are waiting to start unpinning (usually because something else is being unpinned)
queued: pins in
It is sometimes useful to combine this option with the
--local one. For example:
ipfs-cluster-ctl status --local --filter pinning,queued
will only display CIDs which are still pinning or queued (waiting to start pinning) in the local peer.
ipfs-cluster-ctl status --filter error
will display status information for CIDs which are in error state for some reason (the error message contains more information).
ipfs-cluster-ctl status --helpprovides more information on usage and options
Sometimes an item is pinned in the Cluster but it actually fails to pin on the allocated IPFS daemons because of different reasons:
- The IPFS deamon is down or not responding
- The pin operation times out or errors
- It is manually removed from IPFS.
In these cases, the items will show a status of
PIN_ERROR (equivalently, also
UNPIN_ERROR when removing) or
UNEXPECTEDLY_UNPINNED. This is not a cluster issue and it usually indicates a problem with IPFS (content is not available etc.).
In such cases, the
ipfs-cluster-ctl recover can be used to retrigger a pin or unpin operation against the allocated ipfs daemons as needed, once the problems have been fixed. As explained below,
recover operations are regularly triggered by every cluster peer automatically anyways. Note that pins can also be re-added with
pin add, obtaining a similar effect. The main difference is that
recover happens in sync (waits until done), while
pin add returns immediately.
ipfs-cluster-ctl recover --helpprovides more information on usage and options.
Automatic syncing and recovering
Cluster peers run sync and recover operations automatically, in intervals defined in the configuration:
state_sync_intervalcontrols the interval to make sure that all the items in the global pinset are tracked so that
statuscan be reported on them.
ipfs_sync_intervalcontrols the interval to make sure that the ipfs status matches the current view of it that cluster has (the same as syncing above)
pin_recover_intervalcontrols the interval to trigger recover operations for all pins in error state.