Snarl

Reliable Storage with Entanglements in Swarm

In cryptographic decentralized storage systems, files are split into chunks and distributed across a network of peers. These storage systems encode files using Merkle trees, a hierarchical data structure that provides integrity verification and lookup services. A Merkle tree maps the chunks of a file to a single root whose hash value is the file’s content-address.

A major concern is that even minor network churn can result in chunks becoming irretrievable due to the hierarchical dependencies in the Merkle tree. For example, chunks may be available but can not be found if all peers storing the root fail. Thus, to reduce the impact of churn, a decentralized replication process typically stores each chunk at multiple peers. However, we observe that this process reduces the network’s storage utilization and is vulnerable to cascading failures as some chunks are replicated 10x less than others.

Snarl is a novel storage component that uses a variation of alpha entanglement codes to add user-controlled redundancy to address these problems. Our evaluation shows that Snarl increases storage utilization by 5x in Swarm with improved file availability. File recovery is bandwidth-efficient and uses less than 2x chunks on average in scenarios with up to 50 % of total chunk loss.