Last night BitPay's servers started huffing and puffing. At first we thought we might have had a database corruption issue, but on further inspection we realized that one of our Bitcoin nodes seemed to be out of sync with the rest of the network.

It had gone into what is called "safe mode" and was refusing to provide details about transactions. This was affecting real time notifications of transactions as purchases were made.

We started looking around on IRC and forums and discovered that others had noticed problems as well. This led us to conclude that it was not likely an issue specific to us (likely not data corruption). Several of the developers in the #bitcoin-dev chatroom soon realized that the issue was due to pre 0.8 versions of Bitcoin-QT not recognizing recently mined blocks (due to a bug in the pre 0.8 releases). This meant that the block chain had forked. Bitcoin-QT 0.8 nodes and pre 0.8 nodes we're on a different version of the block chain.

Although it does make the network more vulnerable to a double spend attack, this actually isn't all that bad to the average user because their transactions would most likely be confirmed in both forks. However, it's very bad news if you're a miner. Ultimately one of the forks would survive and the other one would die out. If you're a miner on the fork that dies out, it means your reward for finding the blocks will be worthless. Miners needed to quickly decide which fork to support and then all get on the same page. They decided to direct all their mining efforts on the pre 0.8 fork.

This decision was made because the alternative would have required everyone to upgrade to 0.8 or for someone to develop a patch that fixed the pre 0.8 software. Mining on 0.8 would be more risky by the fact that many people might not choose to upgrade, creating a more long lasting fork where the ultimate winner is not as clear.

Once we ascertained that the pre 0.8 fork would survive, we took the affected BitPay node out of safe mode, and full service was restored.

  • Customers that placed orders during the time of the network issues may have noticed that their transactions were not immediately recognized. These transactions will eventually get credited to those invoices and confirmed by the network.
  • If you're a merchant, you may have received warnings about unconfirmed invoices, those invoices will eventually be confirmed and credited to your account.
  • We've checked all payments into BitPay and at this point there are no instances where a transaction has failed to confirm due to this incident.

If you would like more technical details on the issue, please see 11/12 March 2013 Chain Fork Information at bitcoin.org.

Kudos to the dev team, miners and pools operators for their quick diagnosis and correction of the problem. It is actually a great demonstration of how Bitcoin is more than just the software and the protocol it implements. It's also the network of users, miners, developers and service providers that can quickly react to situations such as this to preserve the integrity of the system.