Shameless plug: I am the author.
So yes, “XDG” stands for “Cross-Desktop Group” - but I don’t agree that using the spec assumes a windowing system. The base directory spec involves checking for certain environment variables for guidance on where to put files, and falling back to certain defaults if those variables are not set. It works fine on headless systems, and on systems that are not XDG-aware (I suppose that means systems that don’t set the relevant env vars).
OTOH as another commenter pointed out the base directory spec can make software work when it otherwise wouldn’t on a system that doesn’t have a typical home directory layout or permissions.