The State Growth Problem Facing Blockchains
Much of the discussions around layer 1 chains so far have been about designing new consensus algorithms for better scalability. At the same time, very little has been written about the state growth problem that layer 1 chains face. Addressing state growth is an underexplored yet critical factor for the long-term success of a layer 1 chain.
A blockchain is a distributed ledger that stores state data such as transactions. State growth refers to how quickly the size of the blockchain is growing. State growth is a serious concern for a base layer chain and potentially an existential threat if not solved quickly, for reasons why we will soon see.
Fortunately, there’s been a lot of research and development going on to solve this issue. In this article, we detail the state growth problem and how it relates to the tragedy of the commons problem in economics. We then explain why the state growth problem matters so much and outline various solutions that core developers are working on right now to address state growth.
The tragedy of the (storage) commons
The tragedy of the commons is a situation in which individual users with no restrictions will deplete a system’s shared resource. The term originated in 1833 by British economist William Forster Lloyd to describe overuse of common land. In his example, cattle herders share a common parcel of land on which each is entitled to let their cows graze. If a herder puts more than his allotted number of cattle on the shared land, the herder would receive additional benefits but the whole group shares the damage to the shared land. If all herders made this self-interested decision, overgrazing would cause the shared land to be destroyed to the detriment of all.
In the crypto world, blockchain nodes share their disk space, a common resource, to store transactions and states. Occupied disk space cannot be used by others unless the space is released by its owner. A node will continuously store the occupied disk space but the user of the occupied space does not have to pay rent. (Remember, transaction fees are only one-time payments when users write data to the blockchain.) Afterwards, users essentially have permanent usage rights to a storage system with more availability than Amazon S3. This kind of infinite and permanent storage cost is shared by all full nodes in the blockchain network.
Because many different dApps are built on Ethereum, the tragedy of the commons phenomenon is notable on Ethereum. For example, at block 5700001 (May 30, 2018), the top 5 smart contracts ranked by storage usage are:
- EtherDelta, 5.09%
- IDEX, 4.17%
- CryptoKitties, 3.05%
- ENS, 1.92%
- EOS Sale, 1.73%
The interesting one is the EOS sale. Even though the EOS sale has concluded and EOS tokens are already transacting on the EOS chain, the disk space used by the EOS sale has become a permanent part of an Ethereum full node, and as a result part of a full node’s disk space is continuously being wasted. This cost is not only proportional to the size of the occupied disk space but also to the length of the occupied time.
The “pay once, occupy forever” state storage model gives very little incentive for users to voluntarily clear state, and without management a blockchain’s disk space will be abused. In a well-designed economic system, the user should bear the cost of storage.
State explosion
State growth has recently become a serious issue, especially with Ethereum. The Bitcoin blockchain has grown to 229 GB in 10 years, while the Ethereum blockchain has grown to 262 GB (Geth) / 180 GB (Parity) in just 4 years. [1] Any attempt to scale Ethereum today without a solution to state growth would only exacerbate the problem.
An Ethereum full node running on AWS today costs around $50–70 per month, or less if you run locally with a good Internet connection. If the cost of running a node continues to exceed the rate at which commodity hardware improves and cheapens (a.k.a. Moore’s Law), Ethereum will collapse down to a small, relatively centralized core of nodes run by economically incentivized participants such as Infura and Etherscan.
This decline in Ethereum nodes is beginning to occur in practice. There are multiple reasons for this decline, but the cost and difficulty of running a full node today is certainly a big reason.
Node count is important because it is a measure of decentralization of a crypto network. If only a few nodes are running Ethereum, it becomes easy for them to collude and censor transactions. If that happens, Ethereum loses its censorship resistance which is arguably the most appealing aspect to developers.
Ideally, the cost of running a full node is well within the range of non-professional participants to keep the network maximally decentralized. Keeping this barrier low allows participants of the decentralized network to verify transactions independently without having to trust a third-party. This is fundamentally the reason why public blockchains are valuable.
In the future when blockchain technology gets mass adoption, at what speed will state growth continue when there’s a killer app with hundreds of millions of users writing data to the blockchain every day? Without a solution to state growth, the irony is that only a centralized blockchain with a few well-resourced validators can support a killer app, which defeats the whole purpose of Web 3.
State rent
But it’s not all doom and gloom for Ethereum. A lot of research is being done right now on solving the state growth issue for the base layer chain. The most notable solution is state rent, which is planned to be included in Phase 2 of the Ethereum 2.0 roadmap and likely Ethereum 1.x as well.
State rent, also known as storage rent or state fees, requires users to prepay rent for storing data on the blockchain. The amount of rent paid is proportional to both the size of the data and the amount of time the data is stored on-chain; many different models of charging rent have been suggested with no clear winner. By making the user rather than the full node pay for the cost of state, this prevents state bloat by ensuring that unused information falls out of the state over time.
Among state rent proposals, they differ in terms of user friendliness and implementation complexity. The simplest implementation is a rent mechanism that simply evicts contracts that do not pay rent and does not offer users any way to resurrect their contracts. (This is also ironic given that a blockchain is supposed to be an immutable ledger.) In contrast, a rent mechanism in which users can later resurrect contracts that didn’t pay rent is friendlier to users but more complex to implement.
Eviction doesn’t happen automatically when a user fails to pay rent; someone actually has to create a transaction that interacts with the contract in some way to trigger the eviction mechanism. To resurrect evicted contracts, the eviction doesn’t completely remove the state but instead leaves a “stub,” which is a commitment to the state of the contract before the eviction and allows us to restore the state later on.
As an example, if you have a smart contract wallet with lots of tokens in it but forget to pay rent, the smart contract wallet is gone but you would still be able to rebuild the state of the contract in another contract and use a special opcode to restore the smart contract wallet from the stub. The resurrected contract then comes back to life.
It’s important to note that sharding doesn’t solve the state growth problem. Sharding just divides the entire chain into N smaller chains, but the growth rate for each sharded chain still exceeds Moore’s Law. At first running a full node on each sharded chain is easy, but pretty soon it becomes slow and resource-intensive again.
State pruning
There is also a big concern that state growth is getting out of hand and needs to be bounded in the existing Ethereum 1.0 chain before Ethereum 2.0 “Serenity” launches in the next few years. Some core devs argue that at best, the current Ethereum chain can only sustain state growth for three more years. If some drastic changes are not made before then to reduce state growth, then Ethereum will not survive the transition to 2.0. As a result, one of the upgrade proposals in Ethereum 1.x is reducing and capping the storage space requirements with either state rent or state pruning, or both.
State rent is the hardest and most controversial change within Ethereum 1.x, as it will introduce breaking changes to the existing Ethereum mainnet. It is also technically complex to implement, especially to provide a user-friendly experience.
The good news is that state pruning (delete past blocks and past logs) is an easy, non-controversial change that can be adopted immediately to reduce the required disk space. Currently most of the data a full node downloads is past blocks and past logs. The actual account state is only a fraction of that total data. To be clear, past blocks and past logs would continue to be stored somewhere and be widely available (such as on IPFS), but they would not need to be stored by full nodes that dominate the network. Full nodes would instead only store some recent history of blocks and logs, perhaps a year or so of data.
State pruning would break dApps that expect a full node to index and query all past events. Running those dApps locally would require the user to run a more space-intensive archival node or query an indexing service of past blocks and past logs. For the majority of users though, running a full node would become fast and painless like in the early days of Ethereum.
But it would only be a temporary fix. Say state pruning only stores the past year of data. As blockchain adoption increases and more and more users are adding new state to the chain every day, the amount of historical data stored on-chain would need to gradually shrink from a year down to a few months in order to support the increase in storage utilization. Shrinking the amount of historical data stored on-chain would break more and more dApps.
Nervos: “Cap and trade” state storage
Another approach to solving the state growth issue is to re-architecture a layer 1 chain from first principles so that users bear the cost of storage. Nervos is a next-gen scalable blockchain started by former researchers at Ethereum and Blockstream. It is the strongest team we’ve seen building a layer 1 chain outside of Ethereum, and the project is scheduled to launch in Q4 2019 (testnet is already live). Nervos has an extremely clever cryptoeconomic design that addresses several challenges with implementing state rent on Ethereum.
First, implementing state rent is tricky for account model blockchains like Ethereum. In the account model, assets for all users are stored in a single contract, such as an ERC-20 contract that shows the balances for each owner of a particular token. Without a direct way to split out state for each individual user, it becomes difficult to coordinate all the users to pay the state rent in a fair and efficient way.
While Bitcoin uses the UTXO model and Ethereum uses the account model, Nervos uses the cell model. In the cell model, each user’s digital assets are stored separately in their respective cells. If a user does not want to continue to hold assets, the data in the cell can be released and the storage capacity can be sold or lent to another user. This design makes it easy to attribute state rent to each individual user rather than to smart contracts that then have to figure out how to pass on the costs to the individual users.
Second, the blockchain should always be able to collect state rent from users storing state. Implementing punishments for delinquent users, such as evictions and resurrections, greatly increases the complexity of the system. Plus smart contract composability makes evictions and resurrections more complicated. Even if a particular contract is current on its rent payment, it still may not be fully functional if some of its dependent contracts are behind on their payments.
To solve this problem, the design of Nervos’s state rent mechanism resembles how cap and trade works in environmental policy to limit carbon emissions.
The native token for the Nervos base layer chain is called the “Common Knowledge Byte,” or “CK Byte” for short. The CK Bytes represent cell capacity in bytes, and they give token holders the right to occupy a piece of the blockchain’s overall storage. For example, if you own 1000 CK Bytes, you can create a cell with 1000 bytes in capacity or multiple cells that add up to 1000 bytes in capacity. You can then use the 1000 bytes to store assets, application state, or other types of data.
Since the supply of CK Byte tokens represents the overall state size, the issuance policy of the tokens bounds the state growth. As state storage is bounded and becomes a scarce resource (like bandwidth in Bitcoin and computation in Ethereum), storage can be priced and traded freely on the market. As a result, the cost of adding storage would be higher when the global state capacity is mostly full and lower when it’s mostly empty.
Instead of mandating periodic rent payments from users, Nervos collects rent through inflation. Issuance of new CK Byte tokens is the “inflation tax” to all existing token holders. For users who use their CK Bytes to store state, this recurring inflation tax is how users pay state rent to the miners.
But wait, what about the CK Bytes that aren’t being used to store state? They shouldn’t have to pay state rent as well! As a solution, Nervos allows users to lock up unused CK Bytes in a special contract called the NervosDAO, and token holders in this DAO receive part of the new token issuance, proportional to the amount of tokens they own, to make up for this otherwise unfair dilution. For them it’s as if the inflation doesn’t exist and they’re holding hard-capped tokens like Bitcoin.
As an example, say 60% of all CK Bytes are used to store state, 35% are locked in the NervosDAO, and 5% are trading on exchanges. Then 60% of the issuance goes to the miners, 35% goes to the NervosDAO to be distributed to the locked tokens proportionally, and the remaining 5% gets burned.
The brilliance of this cryptoeconomic mechanism is that demand for storing assets and state on Nervos directly puts demand on owning the native tokens. Thus, the CK Byte tokens effectively capture value from the assets they secure. A higher token price aligns everyone’s incentives and makes the cryptonetwork more robust — users get more security on their assets, developers get more adoption reflected in more assets stored, miners get higher income, and hodlers get price appreciation of their tokens.
Stateless blockchains
The last approach is to shed the concept of full nodes storing state altogether. Is this even possible? Thanks to two new cryptographic primitives — accumulators and vector commitments — described by Stanford researchers Dan Boneh, Benedikt Bünz, and Ben Fisch in December 2018, we can design stateless blockchains. [2]
In a stateless blockchain, full nodes can validate transactions without storing the entire state of the chain but rather only a short commitment to the state. This is different from light clients in that light clients download block headers but have to rely on full nodes to verify the entire transaction history to make sure nothing was double spent. A stateless full node has the same security guarantees as a regular full node without having to store any state.
Cambrian Tech is at the forefront of accumulator technology and has built a proof-of-concept for stateless Bitcoin nodes. The same idea can be applied to an account model blockchain like Ethereum but using a vector commitment instead of an accumulator. Many next-generation blockchains, such as Filecoin, are looking to upgrade their full node architecture to the stateless blockchain model. [3]
Conclusion
There are many other solutions to the state growth problem that don’t involve changing the base layer chain. Analyzing the technical details of other approaches is beyond the scope of this article. Plus since almost all dApps will continue to store state on the base layer chain, the solutions to state growth described in this article seem to be the most practical in the foreseeable future.
Crypto is exciting because it is a very interdisciplinary subject. Because state growth falls under the tragedy of the commons problem that’s widely studied in economics, we can apply lessons from the economics literature to solve state growth in crypto. State rent and cap and trade are just two mechanisms to prevent freeriding of the storage commons; there are many other great proposals such as Nobel Prize Economist Elinor Ostrom’s book Governing the Commons.
Maximalists like to attack other chains for various technical flaws. What they forget is that technology is not static but constantly evolving; research and development is always being done to solve technical challenges. Given both how much crypto relies on prior art in different fields and the pace of R&D in this space, I’m optimistic that the state growth problem will no longer be an existential threat but a thing of the past.
Disclosure: 1confirmation is an early investor in Ethereum, Nervos, and Cambrian Tech.
[1] Ethereum makes the distinction between full nodes and archival nodes. Each time a block is added, state data is added and removed from the latest version of the Ethereum state trie. An archival node keeps all of the pruned state data for each historical block, meaning that you can see what the account balances and smart contract state looked like at any historical block rather than just the most recent block. An archival node currently requires 2.7 TB of storage. That said, archival nodes are mainly just used for research purposes and block explorers like Etherscan, and everyday users can just run a pruned Ethereum full node to verify transactions.
[2] An accumulator functions like a secure decentralized set. It allows full nodes to efficiently reach consensus on a set of values and prove that an element is or isn’t contained in the set without requiring any individual node to store the entire set. Similar to a Merkle tree, an accumulator stores its state commitment in constant space. A notable difference, however, is that its inclusion and exclusion proofs also take up constant space and can be verified in constant time.
In a set, the exact order of elements doesn’t matter; in a vector, the order does matter. A vector commitment is similar to an accumulator, but that a vector commitment commits to not only inclusion (or exclusion) in the vector but also position. In other words, a vector commitment allows full nodes to prove that a certain element exists (or doesn’t exist) in the vector at a certain position.
[3] The difference between the UTXO and the account model is that the order of data stored in an account model blockchain matters. Thus, accumulators are used for UTXO model blockchains like Bitcoin and vector commitments are used for account model blockchains like Ethereum.
About the author: Richard works at 1confirmation, an early-stage crypto venture fund based in San Francisco. Sign up for the 1confirmation newsletter and don’t hesitate to reach out on Twitter.
