Titan Relay Incident Report: Withdrawals Root Validation Issue

We want to provide an update on the recent issue with our Titan Relay service that affected block proposers. Here's a breakdown of what happened, how we fixed it, and what we're doing moving forward.

The Issue:

On July 23, we were alerted to inconsistencies with some Titan relay blocks. Specifically, node operators were seeing this log in their consensus client:

Withdrawals root from the local payload (0x..12) was different from the proposed payload (0x..ab) Falling back to locally produced execution payload and blobs bundle

We discovered our relay was improperly handling Optimistic V2 bids, specifically related to withdrawals roots. This led to validators being forced to use the locally produced block after the consensus client rejected their invalid header for a small subset of blocks. This issue was able to go unnoticed because it only triggered local block building rather than causing the entire slot to be missed. From the relay's perspective, it's indistinguishable why local building occurred, as there are several valid reasons for this, such as the min-bid flag.

Technical Details:

1. Our relay was receiving Optimistic V2 bids with execution layer (EL) withdrawals roots instead of consensus layer (CL) withdrawals roots.

2. These bids weren't being properly validated, allowing invalid headers to pass through.

3. When these invalid bids won, validators' consensus clients rejected them, falling back to locally built blocks instead of MEV-boost blocks.

The problem stemmed from a mix-up between two types of withdrawals roots:

• CL ExecutionPayloadHeader (SSZ hash tree root)

• EL block header (RLP based)

Our relay wasn't correctly distinguishing between these for Optimistic V2 submissions.

The Impact:

The first known occurrence happened for slot 9317845 (Jun-17-2024 13:29:23 UTC) and the last incident before the issue was resolved was slot 9587911(Jul-25-2024 03:42:35 UTC). In total, we have identified an upper bound of 570 blocks over the period. We found Kiln to be the most impacted pool with 322 potential occurrences, followed by stakefish with 77 occurrences.

The Fix:

We deployed a fix on July 25 at 10:30 UTC. Further technical details can be found here: https://github.com/gattaca-com/helix/pull/29

The fix ensures proper validation of withdrawals roots for Optimistic V2 submissions.

Current Status:

We've been monitoring the situation closely with Kiln's assistance, and have not detected any further withdrawals root mismatches since the fix was implemented. In addition, we’re implementing checks to improve our detection and prevention mechanisms.

We want to express our thanks to Kiln for their support in resolving this issue.

If you have any questions or need more information, please don't hesitate to reach out.