I've been interested in the Commoncrawl dataset for a few years now. The interest might have been sparked around the time I began using a lot of OSINT tools/techniques as part of my job as an AML investigator at a regional bank back in '18/'19.
Back then I didn't quite have the same technical skills relevant for working with the data, as I do now (I had made some small C++ CLI apps - games/quizzes that were basically large if/else statements). I downloaded some of the WET files and did CTRL+F out of curiosity. It's fun to see what a relatively unfiltered internet looks like.
Now my setup is a daily job that checks for new crawls, and begins a parsing process of the parts I'm interested in. Previously I was very interested in the WET files. Still am to some extend. But mainly I'm using it to build/maintain an index of active domains.
I've used it to create my own, very basic, search engine. Additional enrichments have allowed me to crudely geolocate the majority of danish websites. They're plottet on a map here: https://map.askhviid.com These enrichments and base domains are searchable here https://cc-explore.askhviid.com
The services and tools I've spun up for this project, are usefull for other data-heavy projects as well - I've begun ingesting a lot of public register data as well (companies -> https://cvrdb.dk). And at this point I've built a primitive dataplatform as a hobby project.
Anyways. I can parse all wet files with a single small 4 CPU Hetzner VPS in a couple of days (search for- and store relevant information). The domain extraction for the URL index is faster.