OpenAI Crawler Version Changes: Make Robots Rules Resilient for Hermes Agent Review URLs
SEO Signal — July 15, 2026. OpenAI’s OAI-SearchBot user-agent version may change, and robots.txt requests may include an added marker. Exact-string crawler rules can become fragile when versions or request details evolve.
Signal
Search Engine Roundtable, citing OpenAI crawler documentation, reports that the OAI-SearchBot version identifier can update and that robots.txt fetches may carry an added robots.txt marker. The practical maintenance lesson is narrow but important: crawler access policies should identify durable bot tokens or documented patterns, while server logs should be reviewed for unexpected changes. A rule tied to one exact version string can accidentally block a desired crawler path after a routine update.
Actions for Hermes Agent Review Pages
- Review
robots.txt, CDN, and WAF rules for exact OAI-SearchBot version matching; replace brittle matches with documented token/pattern handling where appropriate. - Verify that priority review and comparison URLs are crawlable under the site’s intended policy, including canonical pages and structured-data assets.
- Log crawler user-agent strings and robots fetch behavior with timestamps so policy changes can be audited after a version update.
- Do not grant broader access than intended: validate the genuine crawler identity using the platform’s current published verification guidance before changing allow rules.
- Recheck accessibility after major navigation, canonical, or deployment changes.
Measurement Discipline
Separate visibility, qualified engagement, and business outcome metrics. Treat platform pilots and reporting changes as signals to instrument and validate, not as evidence that any one page or agent has achieved a result. Record market, date range, query or product scope, attribution window, and the exact page or feed revision under review.
Source and Scope
This page summarizes a reported search or advertising-platform update and translates it into implementation questions for a technical comparison site. Availability can vary by market, account, product, or rollout stage. Consult the cited report and first-party documentation before making operational decisions.