DRBD node failure recovery.

Step 1 - Set all drbd volumes as secondary to avoid incidents + invalidate all data
\\ @BROKEN BOX
<code>
drbdadm secondary all
drbdadm invalidate all
</code>
(Instead of 'all', you can of course use drbd0, drbd1 etc.)

Step 2 - Overwrite data of the other peer ("bad box")
\\ @GOOD BOX:
<code>
drbdadm -- --overwrite-data-of-peer primary all
</code>

Step 3 - Connect the volumes to start the transfer
\\ @BOTH BOXES:
<code>
drbdadm connect all
</code>

Step 3 - Verify that your boxes are syncing, and wait for them to be done
\\ @GOOD/BAD, doesn't matter:
<code>
while true; do cat /proc/drbd; echo "------------------"; sleep 1; done
</code>
You can expect something like this:
<code>
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:220496 dw:220108 dr:200 al:0 bm:23 lo:97 pe:15483 ua:97 ap:0 ep:1 wo:b oos:5546848
        [>....................] sync'ed:  3.9% (5416/5628)M
        finish: 0:04:03 speed: 22,672 (15,720) K/sec
</code>

Step 4 - Set all drbd volumes as primary again ONCE THE SYNC IS COMPLETE!
\\ @(FORMER) BROKEN BOX:
<code>
drbdadm primary all
</code>

Done.