BephicHelp Center

How does BephicBot read my robots.txt?

· Bephic team, Neuraphic Inc.


BephicBot reads your robots.txt following the standard search engines use (RFC 9309), as far as needed to ask for a single page.

Which block it uses

  1. It looks for a block whose User-agent is BephicBot (upper or lower case does not matter).
  2. If there is one, it uses only that block.
  3. If there is not, it uses the general block, User-agent: *.
  4. If there is neither, it may come in.

How it decides

  • It looks at the Allow and Disallow rules of the block it uses.
  • The longest rule that matches the page wins (for BephicBot, /).
  • If two rules tie in length, Allow wins.
  • An empty Disallow: forbids nothing.
  • It understands * (anything) and $ (end of the address).
  • It ignores what are not page rules, like Sitemap or Crawl-delay. Since it asks for one page per visit, it does not need them.

If your robots.txt does not exist or fails

What happens when asking for robots.txtWhat BephicBot does
It exists and lets it inIt asks for your home page
It exists and does not let it inIt leaves without asking for anything else
It does not exist (your server says it is not there)It comes in, as search engines do
Your server gives an error or does not answerIt does not come in this time

Examples

Block BephicBot:

User-agent: BephicBot
Disallow: /

Block everyone but BephicBot:

User-agent: *
Disallow: /

User-agent: BephicBot
Allow: /

To test yours, use the tester on the Bephic Bot page. It uses the same rules as BephicBot.

Was this helpful?

Need more help?

Ask Bephic or write to the team. We answer within 7 to 15 business days, usually much sooner.

Contact support