bbaovanc/bobatheme
1
0
Fork 0
mirror of https://github.com/BBaoVanC/bobatheme.git synced 2025-09-12 05:57:35 -05:00
Code Issues Projects Releases Wiki Activity

Attempt to fix robots.txt blocking /blog/*.md for all bots

Browse Source 
According to Semrush, 741b3bc01b added a
block for /blog/*.md for User-Agent: *, even though everything I can
read says that it shouldn't apply since the blocks are separated.

I tried reordering the rules and explicitly adding an `Allow: /` rule,
both to match what's shown on Google Search documentation[1].

[1]: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt,
and
This commit is contained in:
BBaoVanC
2025-09-08 19:27:12 -05:00
parent 94bfef19a1
commit f091b366be
1 changed files with 5 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
layouts/robots.txt
View File

@@ -1,6 +1,8 @@
User-Agent: *
Sitemap: {{ "sitemap.xml" | absLangURL }}
{{/* It complains about the raw markdown output having no title tag */ -}} {{/* It complains about the raw markdown output having no title tag */ -}}
User-Agent: Bingbot User-Agent: Bingbot
Disallow: /blog/*.md$ Disallow: /blog/*.md$
User-Agent: *
Allow: /
Sitemap: {{ "sitemap.xml" | absLangURL }}