Conduit releases Degen Chain outage incident report: Improved alerts and monitoring for Orbit chain
ChainCatcher message, Conduit released a post-mortem analysis report regarding the previous downtime incident of Degen Chain. On May 10, Conduit increased the batch size for Degen and Proof of Play Apex to 10MB to reduce costs, which delayed the time for data to be batch published from these networks to their parent chain. On May 12, this configuration was reverted to fix the batch publishing. This led to reorganizations on both networks, as the batch was published after a 24-hour forced inclusion window. Arbitrum Nitro will insert any inbox messages before any transactions in the batch and replay those transactions using the new timestamps.
After the reorganization, nodes returned with corrupted databases due to their depth not being well handled by geth. This required resynchronizing the data directory from genesis. The synchronization time for each network exceeded 40 hours, with a replay rate of about 100M gas/s. Once the nodes were resynchronized, Conduit attempted various transaction replay strategies, although not all transactions could be recovered, as some relied on precise timestamps.
After negotiating with each rollup team, Conduit discussed and concurrently tried various strategies to bring the network online and restore the state prior to the reorganization. The Degen Chain network was restored online 54 hours after the outage. The Apex chain of Proof of Play was restored around the same time but became available to the public only after implementing another recovery solution.
Conduit stated that it has improved alerts and monitoring for the Orbit chain to cover such situations and is committed to working with Offchain Labs to enhance the observability of all Orbit chain operators. The team will continue to invest in and research mechanisms to better simulate mainnet conditions and transaction payloads in a testing environment. The Degen Chain Explorer is now displaying the latest status of Degen Chain normally.
