# How does BephicBot read my robots.txt?

Source: https://bephic.com/docs/bephic-bot/robots-rules/
Updated: 2026-09-30

BephicBot reads your `robots.txt` following the standard search engines use (RFC 9309), as far as needed to ask for a single page.

## Which block it uses

1. It looks for a block whose `User-agent` is **BephicBot** (upper or lower case does not matter).
2. If there is one, **it uses only that block**.
3. If there is not, it uses the general block, `User-agent: *`.
4. If there is neither, it may come in.

## How it decides

- It looks at the `Allow` and `Disallow` rules of the block it uses.
- **The longest rule** that matches the page wins (for BephicBot, `/`).
- If two rules tie in length, **`Allow` wins**.
- An empty `Disallow:` forbids nothing.
- It understands `*` (anything) and `$` (end of the address).
- It ignores what are not page rules, like `Sitemap` or `Crawl-delay`. Since it asks for one page per visit, it does not need them.

## If your robots.txt does not exist or fails

| What happens when asking for `robots.txt` | What BephicBot does |
|---|---|
| It exists and lets it in | It asks for your home page |
| It exists and does not let it in | It leaves without asking for anything else |
| It does not exist (your server says it is not there) | It comes in, as search engines do |
| Your server gives an error or does not answer | It does not come in this time |

## Examples

Block BephicBot:

```
User-agent: BephicBot
Disallow: /
```

Block everyone but BephicBot:

```
User-agent: *
Disallow: /

User-agent: BephicBot
Allow: /
```

To test yours, use the tester on the [Bephic Bot](https://bephic.com/bot/) page. It uses the same rules as BephicBot.

## Related articles

- [How do I block BephicBot in robots.txt?](https://bephic.com/docs/bephic-bot/block-bephicbot/)
- [How do I let BephicBot in if I block other robots?](https://bephic.com/docs/bephic-bot/allow-bephicbot/)
- [How do I test whether BephicBot can come into my site?](https://bephic.com/docs/bephic-bot/robots-tester/)
