Full Report
Different nodes in blockchain need to always come to the same state for a network to work. If the network is split in some way, then the network will not be able to come to consensus, taking the entire chain down. On the Provenance testnet, they noticed that the perceived state (app has) was different. So, what happened? Using their block explorer, the final call was to MarkerTransferAuthorization with an authz wrapped call. So, the modules bank, authz, auth and marker were the only possible culprits here. The Cosmos SDK has a tool just for this type of issue: iaviewer for reviewing the AVL tree for state changes. From this tool, they wrote a diff for the two different chain states. This came with some results. The IAVL tree is just a rebalancing binary tree. The shape of it should change for each new node that is added. The good node wrote out an authz grant to the state store but the bad node did not. So, the bug must be in the authz grant side of it. When setting up the grant, it was using the time.Now() value in Go. Yikes! A big source of non-determinism. When performing an upgrade, nodes that came on sooner were fine on these grants. However, some nodes that came on later after the upgrade would have been this grant fail! In the end of the day, this was actually an issue with the Cosmos SDK itself. An interesting post on a debugging a Cosmos blockchain app hash issue. I bet I'll see this in the future so it was a super helpful post for me!
Analysis Summary
# Vulnerability: Cosmos SDK Non-Deterministic Authz Grant State Mismatch
## CVE Details
- **CVE ID**: Not explicitly listed in the article (Research suggests this relates to the family of issues addressed in Cosmos SDK security advisories regarding `authz` non-determinism).
- **CVSS Score**: N/A (High Severity for Blockchain Availability)
- **CWE**: CWE-361: Time-of-check Time-of-use (TOCTOU) Race Condition / CWE-440: Expected Behavior Violation (Non-determinism)
## Affected Systems
- **Products**: Cosmos SDK and blockchains built upon it (e.g., Provenance Blockchain).
- **Versions**: Found during Provenance testnet upgrades utilizing Cosmos SDK v0.45.x series.
- **Configurations**: Systems utilizing the `authz` module for granting permissions, specifically during node synchronization from genesis or snapshots across different timeframes.
## Vulnerability Description
The flaw stems from **non-deterministic behavior** in the `authz` module's grant handling. Specifically, the implementation used Go’s `time.Now()` function when creating or updating grants.
In a blockchain environment, every node must reach the exact same state (App Hash) after processing a block. Because `time.Now()` relies on the local system clock of the individual node rather than the block time (header time), different nodes recorded different expiration or execution timestamps. This led to a state divergence where some nodes committed an `authz` grant to the IAVL tree (the state store) while others failed the grant, resulting in an "App Hash Mismatch."
## Exploitation
- **Status**: Identified in a live environment (Provenance Testnet); no evidence of malicious exploitation, but caused functional network split.
- **Complexity**: High (Requires specific timing during chain synchronization/upgrades).
- **Attack Vector**: Network/Logic (Triggered by standard transactions under specific synchronization conditions).
## Impact
- **Confidentiality**: None
- **Integrity**: High (State divergence between nodes; breaks consensus).
- **Availability**: High (Causes nodes to halt or crash with hash mismatches, preventing the network from reaching consensus).
## Remediation
### Patches
- The Cosmos SDK team released updates to ensure the `authz` module uses `ctx.BlockTime()` (deterministic time from the block header) instead of `time.Now()`.
- Users of the SDK should upgrade to versions **v0.45.4** or higher where these non-determinism fixes were prioritized.
### Workarounds
- Forcing all nodes to sync from a common, "healthy" data snapshot rather than replaying from genesis can bypass the faulty block height, though this undermines trustless verification.
## Detection
- **Indicators of Compromise**: Nodes failing with an `AppHash` mismatch at a specific block height (e.g., block 3639413 on the Provenance testnet).
- **Detection methods and tools**:
- **iaviewer**: Use this Cosmos SDK tool to inspect the IAVL tree and perform a diff between a "good" node (synced) and a "bad" node (halted).
- **Block Explorer**: Check for `MarkerTransferAuthorization` or `MsgGrant` type messages in the failing block.
## References
- **Vendor Advisory**: [https://github.com/cosmos/cosmos-sdk/security/advisories] (Defanged)
- **Provenance Blockchain Source**: [https://medium.com/provenanceblockchain/discovering-non-deterministic-behavior-in-provenance-blockchain-and-cosmos-sdk-3b81b87b8698] (Defanged)
- **IAVL Tree Documentation**: [https://github.com/cosmos/iavl] (Defanged)