From 7a65f92f53c3f40c33148bb88b4e1821edebee0f Mon Sep 17 00:00:00 2001 From: BBaoVanC Date: Thu, 9 Oct 2025 01:24:05 -0500 Subject: [PATCH] robots.txt: Remove Bingbot exception for *.md files Bing wins this one. It seems like their crawler is too stupid. I might just remove the raw markdown output anyways as it doesn't really serve much purpose. You can go to GitHub and see the raw files if you really want. I believe I originally added it because I thought it was somewhat hard to get from the "View history" commit list to the actual page but I feel like people who care enough to go looking for the source code will be willing to use GitHub to find the actual source file. --- layouts/robots.txt | 4 ---- 1 file changed, 4 deletions(-) diff --git a/layouts/robots.txt b/layouts/robots.txt index 7c24138..720aebc 100644 --- a/layouts/robots.txt +++ b/layouts/robots.txt @@ -1,7 +1,3 @@ -{{/* It complains about the raw markdown output having no title tag */ -}} -User-Agent: Bingbot -Disallow: /blog/*.md$ - User-Agent: * Allow: /