BephicBot reads your robots.txt following the standard search engines use (RFC 9309), as far as needed to ask for a single page.
Which block it uses
- It looks for a block whose
User-agentis BephicBot (upper or lower case does not matter). - If there is one, it uses only that block.
- If there is not, it uses the general block,
User-agent: *. - If there is neither, it may come in.
How it decides
- It looks at the
AllowandDisallowrules of the block it uses. - The longest rule that matches the page wins (for BephicBot,
/). - If two rules tie in length,
Allowwins. - An empty
Disallow:forbids nothing. - It understands
*(anything) and$(end of the address). - It ignores what are not page rules, like
SitemaporCrawl-delay. Since it asks for one page per visit, it does not need them.
If your robots.txt does not exist or fails
What happens when asking for robots.txt | What BephicBot does |
|---|---|
| It exists and lets it in | It asks for your home page |
| It exists and does not let it in | It leaves without asking for anything else |
| It does not exist (your server says it is not there) | It comes in, as search engines do |
| Your server gives an error or does not answer | It does not come in this time |
Examples
Block BephicBot:
User-agent: BephicBot
Disallow: /Block everyone but BephicBot:
User-agent: *
Disallow: /
User-agent: BephicBot
Allow: /To test yours, use the tester on the Bephic Bot page. It uses the same rules as BephicBot.