Unranked, systemd, crawls

Summary

There’s been some changes to Marginalia Search:Migrated to systemd from dockerNew unranked query endpointWide domains got their own index for faster crawlsIn brief, this has removed a lot of operational headaches, reduced expensive queries leaving more computational power for the rest, and cut crawl times in half.Let’s tackle them in some order.We have docker at homeThe system has been migrated off docker compose, and onto bare systemd in production. This has been successful, and solved a number of issues.There are a few constraints that demand a non-standard deployment for Marginalia.NUMA (non-uniform memory architecture).The production server has two CPUs, each with their own RAM bank, and while they can reach over into each other’s memory, this comes at a cost.In many scenarios this isn’t a big deal, but for indexes and databases, generally bottlenecked by RAM bandwidth, this is far from ideal.Process lifecycles.A search engine is anything but stateless, some processes are fairly heavy and run for weeks, some services are very slow to start, some services need to restart relatively often.This all but demands that the search engine is cut up into different pieces.Not necessarily microservices, but at least services.IP addresses.The crawler (and crawler-adjacent processes) run off about a dozen public IP addresses, from the same host. This is done via ipvlan using network namespaces.Linux has many advanced capabilities for giving processes their own virtual network stack, which can be wired together with virtual switches and virtual patch cables. There are to my knowledge no tools for managing this I would consider good, it’s all either too low level, or abstracting away key functionality, or suffering from multiple sources of truths leading to jank when they inevitably differ.Docker can deal with all of these constraints, but at the expense of considerable jank.Docker’s networking model in particular, while it supports ipvlan, doesn’t map onto it particularly well...