Installing Every* Firefox Extension

Summary

*All but 8 we didn’t scrape (or got deleted between me checking the website and me scraping) and 42 missing from extensions.json.1 Technically we only installed 99.94% of the extensions. It turns out there’s only 84 thousand Firefox extensions. That sounds feasibly small. That even sounds like it’s less than 50 gigabytes. Let’s install them all! Scraping every Firefox extension There’s a public API for the add-ons store. No authentication required, and seemingly no rate limits. This should be easy. The search endpoint can take an empty query. Let’s read every page: "https://addons.mozilla.org/api/v5/addons/search/?page_size=50&type=extension&app=firefox&appversion=150.0" let res = await fetch(url) let data = await res.json() console.log(`PAGE ${page++}: ${data.results.length} EXTENSIONS`) extensions.push(...data.results)Bun.write("extensions-default.json", JSON.stringify(extensions)) The search API only gives me 600 pages, meaning I can only see 30 thousand extensions, less than half of them. A solution I found is to use different sorts. The default sort is sort=recommended,users: first recommended extensions, then sorted by users, descending. Changing to just sort=created gave me some of the long tail: "https://addons.mozilla.org/api/v5/addons/search/?page_size=50&type=extension&app=firefox&appversion=150.0" "https://addons.mozilla.org/api/v5/addons/search/?page_size=50&type=extension&app=firefox&appversion=150.0&sort=created" Bun.write("extensions-default.json", JSON.stringify(extensions))Bun.write("extensions-newest.json", JSON.stringify(extensions)) import extensions_default from "../extensions-default.json"import extensions_newest from "../extensions-newest.json"// Yes, somehow I got the same slug twicefor (const ext of extensions_default) { extensions[ext.slug] = extfor (const ext of extensions_newest) { extensions[ext.slug] = extconsole.log(`TOTAL UNIQUE EXTENSIONS: ${Object.keys(extensions).length}`) ~/Developer/every-addon> bun countTOTAL UNIQUE EXTENSIONS:...