Another report dropped - 8/4 PeerDAS Devnet report by Sunnsyide Labs is here! Curious about how nodes behave in production-style conditions with PeerDAS? 🧵

This time, Sunnyside Labs ran 2 kinds of devnets based on @ethPandaOps's fusaka-devnet-2 image, to check the network stability at more realistic environment: 1. Interop devnet (mixed CL/EL clients with 50% supernode) 2. Perfect PeerDAS devnet (each node exclusively custody a unique data column)

For each devnet configuration, we ran 3 new test scenarios on it: 1. Blobs with Large Txs This is to investigate how regular transactions affect the block propagation’s competition with the data column propagation across the nodes. We've maintained each blocks at the test to reach 22 Mgas (2MB block size). 2. Bandwidth Limit + Large Txs This test is to investigate the behavior of the network in the conditions of bandwidth-limited nodes such as home stakers. The bandwidth is set as 100/50Mbps for supernodes and 50/25Mbps for fullnodes, as per recommendations from EIP-7870. 3. Genesis Sync Test Genesis sync is tested over the devnets to find out any blockers or bugs in the sync process. Here's the insights we got from these tests:

1 - Large transactions have low impact on the network, under unconstrained network In both Interop Devnet and Perfect PeerDAS Devnet, the network withstood 60 blobs without much problems, even with additional transactions other than blobs. This result is the same as the baseline scenario with blobs only. This indicates ordinary transactions other than blobs have minimal impact on the network stability under no resource constraints.

2 - Bandwidth limit affects the network in various paths With uplinks throttled to 100/50Mbps for supernodes and 50/25Mbps for fullnodes, blob throughput collapsed to 20 blobs/block on the Interop devnet and 30 blobs/block on the Perfect PeerDAS devnet. Under these limits, a sharp rise in IWANT messages on the beacon_block topic showed that beacon blocks were struggling to propagate. Also, once nodes hit their egress ceilings, EL mempools exposed far fewer blobs, evidenced by a steep drop in getBlobsV2 success rates. Internal discussions point to the 50 Mbps upload cap on supernodes as the prime culprit: that ceiling at EIP-7870 is tailored for local block builders, not for supernodes serving the wider network. Thus, these tests were harsher than real-world conditions. Sunnyside Labs will run a follow-up test this week with higher supernode bandwidth to gauge the exact impact of this constraint.

3 - Genesis sync works for Interop devnet, but failed on Perfect PeerDAS devnet Genesis sync completed flawlessly on the Interop devnet, yet stalled just after the Fulu upgrade epoch on Perfect PeerDAS. If a node can’t obtain all columns during genesis sync, it should ideally query additional peers to fill the gaps, which is not yet implemented in any clients. Whether this cross-peer column fetching becomes a required feature before the Fusaka is still open; the decision is expected at next Thursday’s ACD.

4 - Validator custody backfill successful at @Teku_Consensys Validator custody backfill lets a client detect missing historical columns and automatically retrieve them from peers. Our dedicated test confirmed that Teku’s implementation works flawlessly - it identified gaps and fully backfilled without hiccups in regular conditions. A small bug at an edge case where there was big gap in the chain without any blobs was identified at the tests, which was quickly fixed by the team. Other clients are now working on this feature, and it will be tested in the future Sunnyside devnets as the implementation is completed.

Sunnyside Labs will continue to run devnets this week, focusing on the following topics: - Devnet Interop with higher supernode bandwidth limits and longer periods of times - More analysis into genesis sync tests on Perfect PeerDAS devnet - More devnets focusing on max block size (in snappy-compressed format), nodes corrupted with blobber, and finality on split network