Discovering faulty NEO consensus nodes by using MinerTransactions
NEO’s consensus mechanism, dBFT, requires consensus nodes to come to an agreement about a potential block before it can be committed to the chain. This removes the possibility of blockchain forks and provides finality (irreversibility) for transactions as soon as they show up on the blockchain, which is great for the user experience, but it can result in delays as a trade-off.
Consensus nodes on NEO take it in turns to act as speaker by proposing a block. The other nodes check the validity of the block based on a few criteria, and if the block is valid, they sign their agreement. When 66% of the total number of nodes have agreed to a block, the network state is updated with the new block added to the chain.
If a speaker does not submit a valid block (whether it fails to submit a block at all or the block is invalid) within the designated time frame (currently 15 seconds), a view change occurs. Essentially this just means a new node will propose a block, in the hopes that consensus will be achieved. Unfortunately, that means another 15-second wait, so the next block will take roughly 30 seconds. These delays caused by view changes are undesirable, but the alternative would mean the loss of finality.
At the time of writing, NEO has 7 consensus nodes. By checking block explorers such as NeoScan, we can see that every 7 blocks, the block time is roughly doubled. Due to the deterministic nature of nodes being selected as speaker, this implies that a single node is failing to propose a valid block, causing a view change. So how do we figure out which node that is?
My solution here is not particularly elegant, nor is it likely to work forever; especially in the event that transaction fees are distributed to NEO holders instead of the consensus nodes. Ideally, this information would be retrieved and analyzed in a much more direct manner. But for now, this method works.
To start, I should note that every block on NEO contains what is called a MinerTransaction. An example can be seen here. If a transaction contained in a block had a priority fee attached, the MinerTransaction will send that fee to the consensus node that proposed the block (the speaker for that block).
This provides us with a cheeky way to see which consensus node proposed a specific block. To monitor the blockchain with this, we need to do two things. Firstly, we need to send priority fee transactions across a sufficiently large number of blocks. Secondly, we need to figure out which public address (from the MinerTransaction) corresponds to which consensus node. Let’s start with the latter.
On neo.org/consensus, we can see the current consensus nodes and their operators. It also includes their 33-byte public keys, which we can use to calculate the public address for each node. At the time of writing, these are the public keys for NEO consensus nodes:
- COZ: 025bdf3f181f53e9696227843950deb72dcd374ded17c057159513c3d0abe20b64
- KPN: 035e819642a8915a2572f972ddbdbe3042ae6437349295edce9bdc3b8884bbf9a3
- NF1: 02df48f60e8f3e01c48ff40b9b7f1310d7a8b2a193188befe1c2e3df740e895093
- NF2: 024c7b7fb6c310fccf1ba33b082519d82964ea93868d676662d4a59ad548df0e7d
- NF3: 02ca0e27697b9c248f6f16e085fd0061e26f44da85b58ee835c110caa5ec3ba554
- NF4: 03b209fd4f53a7170ea4444e0cb0a6bb6a53c2bd016926989cf85f9b0fba17a70c
- NF5: 03b8d9d5771d8f513aa0869b9cc8d50986403b78c6da36890638c3d46a5adce04a
Now we need to turn these keys into public addresses. I like to use NeoResearch’s NeoCompiler Eco for this purpose.
Each public address on NEO is actually a kind of smart contract. It is created by combining two specific opcodes with the public key. We’ll need opcode 21, which pushes the next 33 bytes to memory, and opcode ac, which checks the signature. Let’s use the COZ pubkey as an example.
To create our AVM hex, we simply add 21 + public key + ac. Plug it in, and out pops the public address for the City of Zion consensus node, AedGJPWWALPZgtSXZKEVZs5vYEdxffX7xs.
Repeating the same step for each consensus node public key provides us with all of the public addresses for the current nodes.
- COZ: AedGJPWWALPZgtSXZKEVZs5vYEdxffX7xs
- KPN: ATwwcpZmPFWfMaZVMsvQwFcRnbWqhZx3ZF
- NF1: AKNLArGjLisJB9mXvtSxASxHd9jaCBFs1B
- NF2: ARtmDzcTZxHCYydqFxFw31d21CpSArZwi4
- NF3: AJeAEsmeD6t279Dx4n2HWdUvUmmXQ4iJvP
- NF4: AWHX6wX5mEJ4Vwg7uBcqESeq3NggtNFhzD
- NF5: AXvbEb4TEy86LSaeMTZ57esTtgeqWeCF6q
That’s the hard part over with, now all we need to do is send a bunch of transactions with priority fees. To account for bad timing, we want to send one every 10–15 seconds for a few minutes. I sent approximately 30 transactions manually and checked the blocks for MinerTransactions. These are the addresses that showed up in the MinerTransactions for the blocks that contained my transactions:
- COZ: AedGJPWWALPZgtSXZKEVZs5vYEdxffX7xs
- KPN: ATwwcpZmPFWfMaZVMsvQwFcRnbWqhZx3ZF
- NF: AXvbEb4TEy86LSaeMTZ57esTtgeqWeCF6q
- NF: ARtmDzcTZxHCYydqFxFw31d21CpSArZwi4
- NF: AJeAEsmeD6t279Dx4n2HWdUvUmmXQ4iJvP
- NF: AKNLArGjLisJB9mXvtSxASxHd9jaCBFs1B
As suspected, only 6 nodes are receiving MinerTransactions. That means our missing node is not proposing valid blocks, ergo it is the faulty node causing delays on the network.
Referencing our list reveals one of the NEO Foundation nodes to be responsible, with the address AWHX6wX5mEJ4Vwg7uBcqESeq3NggtNFhzD. Today, all we can do is yell at NEO to sort their node out. In the future decentralized environment, we would just revoke our votes for this node and replace it with a better candidate.
By checking this address on NeoScan, we can also get an approximation of when the issue started (this definitely isn’t perfect, as not every block contains transactions with fees, which allows us to differentiate nodes via the MinerTransactions).
This was a fun little investigation that was probably unnecessarily complicated and will soon be rendered moot by changes to NEO’s design or improved consensus analytics, but hey, it works!
I hope it was interesting to some of you.