Why Vyera respects robots.txt
Vyera only reads what a site's own robots.txt permits automated readers to see, as a deliberate boundary.
Vyera only reads what a site's own robots.txt file permits automated readers to see.
Why this is a deliberate boundary, not a limitation
Respecting robots.txt is a considered choice, not an accident of what is technically possible. It reflects the same principle behind everything Vyera collects: read what is meant to be publicly and automatically readable, and nothing beyond that.
This occasionally means a data point exists publicly to a person visiting the page directly, but is not collected, because the site has asked automated readers not to read that part of it. That trade-off is intentional.
What it does not do
Vyera does not attempt to work around a robots.txt restriction under any circumstances, and this is not something support can override for a specific competitor. If a page is genuinely excluded this way, it stays excluded.
Was this useful?
Related articles
- What Vyera does not collect
Vyera only collects public data from the sources you add, never private, login-gated or unpublished information.
- Source coverage: what Vyera can and cannot see
Coverage varies by source type, region and industry, and this is an honest inventory of where the gaps sit.
- How Vyera decides what counts as a change
Vyera filters out trivial edits like typos and formatting, so what reaches you is a change worth knowing about.
Still stuck?
Send this to the team and we will pick it up.