TL;DR - CLRFund Round 8 was an absolute roller coaster. We smashed records, continually hit rate limits on our RPC nodes, burned out some CPUs, made some incredible efficiency gains, failed to prove on-chain, created some awesome NFTs, got $16k stuck in a contract, recovered the rest, learned a ton, and ultimately allocated a bunch of funding to some of Ethereum’s best public goods projects.

Round Review

Calling this round a rollercoaster is almost an understatement. So much went down this round that we hardly know where to start.

Here goes...

In the lead up to Round 8, we set a new record for the number of participants in a trusted setup ceremony at 1,183. Narrowly edging out Tornado Cash’s previous record of 1,114.

Within a few hours of launching round 8, the clr.fund app was unusable because hundreds of you crazy kids contributing caused us to almost immediately hit the rate limits on our RPC node providers. Luckily, the folks from pocket.network and the xDai team hooked us up with access to some RPC nodes with much higher capacity, so the app sailed smoothly from there.

We smashed our previous record for the number of contributors in the first day, ultimately finishing the round with 6,137 contributors, casting ~50k votes, contributing 33,562.00 wxDai, to allocate a matching pool of 33,594.45 wxDai. New records on literally every metric.

However, after the round concluded, we ran into a comedy of issues.

Generating the Proofs

We started running our first attempts to generate proofs on a modest consumer laptop, desktop, and bare metal server in parallel (yay for redundancy). For some reason the server kept giving us a core dump error. After numerous fresh distro install and several days of collectively banging our heads on the walls, we realized that Iden3’s ZKP libraries make use some intel assembly language, and while the server uses intel chips, it is a few generations behind and evidently does not have the full instruction set that the libraries use.

Generating proofs of message processing...
Progress: 1 / 6216; batch index: 49720
Illegal instruction (core dumped)

Nevertheless, it was running fine on the laptop and desktop, with one little caveat. After letting it run for a few hours, we estimated it was going to take more than 110 days to generate the proofs.

The first bottleneck we identified was that the MACI scripts make use of a single processor thread. We couldn’t change this for this round, but we could find the absolute fastest single-thread VPS we could get our hands on (an Amazon Z1d, just in case you were wondering). This gave us a modest improvement, but we were still over 100 days.

Generating proofs of message processing...
[2021-11-01 16:43:58] 
[2021-11-01 16:43:58] Progress: 1 / 6216; batch index: 49720
[2021-11-01 17:08:54] Up to loadJson 0.01805599999999999913
[2021-11-01 17:08:54] Items : 21
[2021-11-01 17:08:54] Total 0.40599899999999999878
[2021-11-01 17:08:56] Loading circuit from /mnt/nvme1n1p1/maci/circuits/params/batchUst32.r1cs...
[2021-11-01 17:08:58] Proving...
[2021-11-01 17:09:16] Saved /mnt/nvme1n1p1/maci/circuits/params/1635786534498.proof.json and /mnt/nvme1n1p1/maci/circuits/params/1635786534498.publicSignals.json
[2021-11-01 17:09:17] Proof is correct
[2021-11-01 17:09:17] 
[2021-11-01 17:09:17] Progress: 2 / 6216; batch index: 49712

~25 minutes per batch
~57.6 batches per day
~107.92 days to generate proofs for this round

Then Wei Jie came to our rescue with a series of on-the-fly incremental improvements to MACI’s proving scripts. First reducing the proving time to about thirty days; still long, but much more manageable. And then later improving again down to about three days.

Three days later, proofs in-hand (and backed up multiple times) we were ready to prove on chain. But the rollercoaster doesn’t stop there.

(Not) Proving On-Chain

After generating the proofs, the next step is to prove them on-chain. The CLRFund smart contracts then use those proofs to validate how much each recipient should receive from the round.

With the proofs finally generated, we were expecting smooth sailing from here on out. Then this happened.

Submitting proofs of message processing...
Progress: 1/6216
(node:13838) UnhandledPromiseRejectionWarning: Error: transaction failed

You can see the transaction returns ERROR_INVALID_BATCH_UST_PROOF = "E02"

After another few days of collectively banging our heads against the wall, and then a really productive debugging session, we realized that there was an error in the deployment of our MACI instance due to a change in the tooling we used to deploy the round.

The MACI repo contains Poseidon.sol, with two libraries, PoseidonT3 and PoseidonT6, both without any implementation. The Poseidon libraries are part of the Circom library, a suite of tooling for zero knowledge proofs.

When MACI is compiled using their compileSol script, these two empty libraries are cleverly replaced with their real counterparts.

Unintentionally, the tooling we used to deploy this round, a frontend we’ve been working on to make it easier for anyone to spin up their own instance of CLRFund, deployed the empty implementations of the Poseidon libraries.

Normally, each time a contributor casts their votes, a new leaf is added to a merkle tree and the message root is updated.

But in our case, because of our empty Poseidon libraries, the function to update the state root failed silently and the state root remained unchanged for the entire round.

The mismatch between this empty state root and the roots expected in our proofs meant that we were unable to prove the results on chain.

So what does this mean?

Round 8 Cancelled

After exhausting all other options, we have ultimately had to cancel round 8.

All contributions to recipients (33,562.00 wxDai) can be claimed by the contributors (claim yours here).

Unfortunately, roughly half of the 33,594.45 wxDai matching pool is stuck in the funding round factory contract forever. We’ll cover that out of our reserves, and the total that should have been allocated to recipients from the matching pool will be manually distributed to recipient according to the generated results of the round, even though we were unable to prove the results on-chain.

While this is not the outcome we were hoping for in round 8, it definitely validates our choice to scale CLRFund slowly, since we’re dealing with a combination of several brand new technologies. We’ve learned a lot this round, made improvements to our node infrastructure, the MACI contracts, the proving scripts, our deployment processes, the CLRFund contracts, and as a result feel much more comfortable scaling up future rounds.

Thanks to some incredible code contributions from the Ethereum.org team, our next rounds will also feature a totally revamped application and will be deployed to Arbitrum. We plan to run one last round this year and then follow up with a much larger round early next year.

Pool Party NFTs

In the meantime, as with past rounds, everyone who contributed to a project in Round 8 round will receive a POAP. Claim yours now!

We've also been working on a super secret mission with the awesome crew over at Props Supply to create Public Goods Pool Party themed POAPs and NFTs for our matching pool contributors.

Any address that contributed at least 1 wxDai to the matching pool will receive a Pool Noodle POAP. You can claim yours now and you can see all of the eligible addresses with this Dune query.

The coveted Golden Pool Noodle NFT will be given to one lucky matching pool contributor via a raffle. Each 1 wxDai contributed to the matching pool earned the contributor one ticket. You can see the list of tickets on this Dune query or this CSV on IPFS.

To select the winner, we used the first (leftmost) two bytes of the random number generated by drand.love round 1,444,000 (announced in advance on Twitter) as input to the following JS function.

Math.floor((drand_randomness / 0xffff) * 8491) + 1

The first two bytes of randomness for round 1,444,000 were 0x8251.

So, The golden pool noodle goes to... drum roll... ticket number 4323. Congratulations bykur.eth.

Lastly, the top 4 contributors to the matching pool have each received a unique inflatable pool floaty NFT, inspired by some of Ethereum’s spirit animals.

The Golden Unicorn - 0xigor.eth - 10k xwDai
The Flipping Dolphin - bitpush.eth - 200 wxDai
Doge - 0xd63d28282eeeace4d4d2c67ebb798f3a2ca782b1 - 200 wxDai
Moloch - 0xa6a0111f3401b9ede35aca7734436e26549a9a55 - 166 wxDai

A huge thank you to everyone who participated in and contributed to this round.

Can’t wait to dive into round 9 in a few weeks.

Join Us!

Want to help build or contribute to future CLRFund rounds? You can find us on Twitter, Discord, GitHub, our forum, and Telegram.