Commit Graph

5 Commits

Author SHA1 Message Date
7a65f92f53 robots.txt: Remove Bingbot exception for *.md files
Bing wins this one. It seems like their crawler is too stupid. I might
just remove the raw markdown output anyways as it doesn't really serve
much purpose. You can go to GitHub and see the raw files if you really
want.

I believe I originally added it because I thought it was somewhat hard
to get from the "View history" commit list to the actual page but I feel
like people who care enough to go looking for the source code will be
willing to use GitHub to find the actual source file.
2025-10-09 01:24:05 -05:00
f091b366be Attempt to fix robots.txt blocking /blog/*.md for all bots
According to Semrush, 741b3bc01b added a
block for /blog/*.md for User-Agent: *, even though everything I can
read says that it shouldn't apply since the blocks are separated.

I tried reordering the rules and explicitly adding an `Allow: /` rule,
both to match what's shown on Google Search documentation[1].

[1]: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt,
and
2025-09-08 19:27:12 -05:00
741b3bc01b robots.txt: Stop Bing from complaining about md output not having title 2025-08-20 19:00:42 -05:00
c7cfb9c844 Support multilingual mode (experimental)
Please report any bugs!
2021-10-11 18:24:14 -05:00
03eb50ccca Add basic theme files 2021-09-25 20:52:01 -05:00