mirror of
https://github.com/BBaoVanC/bobatheme.git
synced 2025-09-12 05:57:35 -05:00
BBaoVanC
f091b366be
According to Semrush, 741b3bc01b added a
block for /blog/*.md for User-Agent: *, even though everything I can
read says that it shouldn't apply since the blocks are separated.
I tried reordering the rules and explicitly adding an `Allow: /` rule,
both to match what's shown on Google Search documentation[1].
[1]: https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt,
and
9 lines
184 B
Plaintext
9 lines
184 B
Plaintext
{{/* It complains about the raw markdown output having no title tag */ -}}
|
|
User-Agent: Bingbot
|
|
Disallow: /blog/*.md$
|
|
|
|
User-Agent: *
|
|
Allow: /
|
|
|
|
Sitemap: {{ "sitemap.xml" | absLangURL }}
|