Looking before the Scaling Up Leap

Looking before the Scaling Up Leap
Gavin Andresen (1/6/15)
-----------------------------------
I've been busy the last few weeks filling up my hard disk with copies of the main Bitcoin blockchain.

My goal is to prove that it is safe to raise the maximum block size from 1MB to at least 20MB, with the existing Bitcoin Core code (no fancy invertible bloom filters, no hyper-optimized-for-the Bitcoin-elliptic-curve verification code).

If the existing code can handle 20 megabyte blocks, then I'll argue we should schedule a hard fork to raise the block size. Ideally, we would never run into the maximum block size limit -- we would always have a situation like we do today, where miners decide how large or small to make their blocks, and people using the blockchain decide whether they're willing to pay miners transaction fees to include their transactions.

So I've been testing the existing code with much larger blocks -- 20MB and 200MB.

I could create artificial tests that fill up blocks with randomly generated transactions, but it would be much better to test with real-world transactions packaged into bigger blocks. I'm running a couple of full Bitcoin nodes, so I have 20 or so gigabytes of real-world transactions.

I just had to figure out how to repackage them into bigger blocks. After a couple of false starts over a couple of frustrating days (wouldn't it be nice if our ideas always worked perfectly the first time?), I came up with the following scheme:

First, I hacked the reference implementation and changed some constants related to the block size (MAX_BLOCK_SIZE, DEFAULT_BLOCK_MAX_SIZE, MAX_BLOCKFILE_SIZE). That part is easy.

The tricky bit is how to take transactions from the 1-MB and repackage them in bigger blocks. For most transactions, it is trivial, because ordinary transactions don't care what block they are in.

There are two exceptions:

1. Transactions that use the 'locktime' feature so they are not valid until a particular block height or date. I dealt with these by simply treating all transactions as 'final'.

2. There are special rules for 'coinbase' transactions: there can be only one coinbase transaction per block, it must be the first transaction, etc. My first couple of false starts went down the path of relaxing those rules, but I gave up that approach because I found myself changing more and more code. I think if I was 22 years old I probably would have just forged ahead, unwilling to let go of a couple of days of programming effort because "just one more change and it will probably all start to work..."

I wonder if old programmers throw away more code than young programmers. I bet we do.

Anyway, after a rethink I came up with a much cleaner solution for handling coinbase transactions from the real blockchain: I write them to a 'coinbasetx.dat' file which the hacked bitcoind reads at startup and stores in memory (there are only 73 megabytes of them). As blocks are added to the big-block chain, those coinbases are added to the "unspent transaction output set" so they are available to be spent (and I set the COINBASE_MATURITY constant to zero instead of 100, so they can be spent right away, in the same big block).

I wrote a tool ("gen_megablocks") that creates the coinbasetx.dat file and blk*.dat files that are valid-but-oversized, -regtest-mode blocks.

Those big blocks are then loaded into the bigger-block-capable bitcoind using the -loadblock command-line option, where they go through full transaction and block validation and indexing.

The end result is a fully-indexed blockchain with real transactions arranged into much larger blocks. I can import some of my main-chain wallet private keys and create transactions that spend them; those transactions are valid on both the hacked big-block chain and the main Bitcoin 1-MB blockchain (because ordinary transactions are completely independent of blocks).

So far, the results are very good-- the time to process a block scales up linearly with the size of the block, there are no hidden O(n^2) gotchas waiting to bite us. My desktop machine (a quad-core, 16GB-memory, "Late 2012" iMac) can easily keep up with a 20-MB-per-block blockchain, and could even keep up with a 200-MB-per-block chain if run with bigger -maxsigcachesize and -dbcache settings.

Today I'm generating alternate chains (I added an option to gen_megablocks to skip one or more transactions and their descendants); I'll be testing long and short blockchain re-orgs, measuring CPU time and memory usage. Once that is done I'll try syncing these 'megablock' chains across the internet and will measure real-world network bandwidth usage and time. And along the way I'll import a few keys and make sure the Bitcoin-Qt wallet works properly with big blocks (it should, it doesn't contain any code that cares about the block size).

Once all that is done, I will write up the results, write up a detailed specification for increasing the block size, and will start working on a patch (and lots more tests) to make it happen.

And I'm bracing myself for  endless re-hashing of arguments about why a maximum block size tied to transaction volume or difficulty or exchange rate or total transaction fees or the phase of the moon would be better than some dumb, predictable formula....