The Bephic Bot page has a tester: you paste your robots.txt and it tells you at once whether BephicBot can come in. It uses the same rules as the real robot.
Step by step
- Open your
robots.txtin the browser, for examplehttps://yoursite.com/robots.txt, and copy all its text. - Open the Bephic Bot page and scroll to Test your robots.txt.
- Paste the text into the box in that section, under "Paste your file and see at once whether BephicBot can reach your home page."
- Read the result. For example: BephicBot may visit your site. and, below it, No rule keeps it out.
The tester checks the home page (/), which is the only one BephicBot asks for.
What it tells you
- Whether it may visit your site or will not visit your site (BephicBot will not visit your site.).
- Which rule decides it, when there is one.
- Whether it uses the block with its name or the general one (
User-agent: *).
If the result is not what you expected
- Check whether you have a block for BephicBot: when there is one, only its rules count.
- Check the lines are spelled right (
User-agent:,Allow:,Disallow:). - Remember: the longest rule wins, and on a tie
Allowwins.
See How does BephicBot read my robots.txt?.
What the tester does not do
- It does not visit your site: it only reads the text you paste.
- It does not keep what you paste.
- It is not for BephicSEO, which does not read
robots.txt.