bbaovanc/bobatheme
1
0
Fork 0
mirror of https://github.com/BBaoVanC/bobatheme.git synced 2025-10-14 11:05:16 -05:00
Code Issues Projects Releases Wiki Activity

robots.txt: Remove Bingbot exception for *.md files

Browse Source 
Bing wins this one. It seems like their crawler is too stupid. I might
just remove the raw markdown output anyways as it doesn't really serve
much purpose. You can go to GitHub and see the raw files if you really
want.

I believe I originally added it because I thought it was somewhat hard
to get from the "View history" commit list to the actual page but I feel
like people who care enough to go looking for the source code will be
willing to use GitHub to find the actual source file.
This commit is contained in:
BBaoVanC
2025-10-09 01:24:05 -05:00
parent bb6346bbad
commit 7a65f92f53
1 changed files with 0 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
layouts/robots.txt
View File

@@ -1,7 +1,3 @@
{{/* It complains about the raw markdown output having no title tag */ -}}
User-Agent: Bingbot
Disallow: /blog/*.md$
User-Agent: *
Allow: /