Cloudflare ERR_ECH_FALLBACK_CERTIFICATE_INVALID

Users on HIVE recently started reporting errors when trying to download files on HIVE. The site runs on Cloudflare, but the sub domain for downloading and uploading upload.hiveworkshop.com is not proxied through Cloudflare.

The error my users are seeing is this:

I tried various things to prevent this error, but would always get new reports of users experiencing this bug. Finally I reported it on the Cloudflare community forums. After having posted I followed up multiple times with more information. Turns out the problem has to do with Encrypted Client Hello (ECH). I don't know a whole lot about it, but it supposedly patches the last known hole in the HTTPS handshake.

I am happy that Cloudflare takes security so seriously and applying things like ECH on my site. But the problem arises from CF incorrectly providing the information (via DNS) that ECH is enabled on the subdomain upload.hiveworkshop.com which is not proxied by CF and does not support ECH.

A commenter on one of the tickets I found said the following:

Most likely they do it on @ if it is set to proxy mode, and then it applies to subdomains? Because on a zone of mine I do indeed not see it yet. Tho rte.ie might be on a non-free CF plan and they’re delaying the rollout for those. idk.

Which was on point. When I use dig I got the following result:

# dig +short https hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=104.21.24.123,172.67.218.199 ech=AEX+DQBB1QAgACCCe94ieZ7pFhg+8SQKfPYVTiwfqsDTLoUpA9DxUQ+rUgAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3035::6815:187b,2606:4700:3037::ac43:dac7

# dig +short https www.hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=104.21.24.123,172.67.218.199 ech=AEX+DQBBggAgACBXw8l9AXjPoyTZnJZRkVPwX0X1tUfzxFsxtfs2P/iEegAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700:3035::6815:187b,2606:4700:3037::ac43:dac7

# dig +short https upload.hiveworkshop.com
1 . alpn="h3,h2" ipv4hint=95.217.146.41 ech=AEX+DQBBggAgACBXw8l9AXjPoyTZnJZRkVPwX0X1tUfzxFsxtfs2P/iEegAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=

It shows ech=... on the upload.hiveworkshop.com domain. It shouldn't.

After disabling proxy on the hiveworkshop.com domain and keeping it on for www.hiveworkshop.com, it stopped resolving upload.hiveworkshop.com as supporting ECH. So that fixed the DNS symptom I was seeing. When disabling on the root domain I got the following when using dig:

# dig +short https upload.hiveworkshop.com

Soon after, a CF team member responded to the ticket saying:

Thanks @user3138 - We are going to rollback ECH. I think you have identified a corner case.

The HTTPS record SHOULD NOT be returned for the upload domain.

I assume you are gray clouding upload. to an Orange Clouded root domain hiveworkshop . com My quick hunch is CNAME flattening is causing this issue. We will investigate with the DNS team to fix. The ECH record should only be returned for proxied records.

The TLS 1.3 disable button would fix this as well, but we will investigate the root cause so will back this out.

Thanks for reporting and apologies.

Matt

Pretty happy with that. I came with an issue, thinking it was on me, but turns out the Cloudflare team made a mistake and they're rolling back a whole release to make sure things work.