SquishPixel Blog · AI search · SEO · WordPress

Three experts say to check AI crawler access before rewriting your site for AI search

According to DEV Community, in a post originally published by 2theTop Web Design & Marketing, you do not need a new website to appear in AI assistants such as ChatGPT. The post cites Google's guidance on AI features: there are no special requirements for AI Overviews or AI Mode. A page needs to be indexed, readable as text, and have structured data that matches what is on the page.

The post explains that an assistant runs background searches, reads the pages it finds and writes an answer. Google calls this a "query fan-out". A vague page gives it nothing to quote, while a plain page listing prices, process and service area is more likely to be used.

The article lists five fixes. Name the service and place in the H1 heading. Give each service its own page. Put key facts in text rather than PDFs, images or sliders. Use structured data, which is machine-readable markup, that matches the visible page. Show proof of real work, such as project photos and detailed reviews.

Three experts were asked what they fix first. Tory Gray of The Gray Dot Company says to check the raw HTML before any JavaScript runs, because most AI crawlers do not render JavaScript. She also says to make sure security plugins, CDN settings or a robots.txt rule are not blocking AI bots. Warren Laine-Naida, a digital skills coach, also starts with access: the WordPress "Discourage search engines" box unticked, robots.txt allowing Googlebot and OAI-SearchBot, and no host-level blocking.

Alison Ver Halen of AV Writing Services focuses on content. She says pages should be easy to scan and answer the questions prospects ask, including FAQs and objections. In her view this matters most for people comparing options in the later stages of buying.

The post ends with a repair-or-redesign test. If staff can edit page text, services have their own pages, the site loads quickly on phones and leads still arrive, it suggests repairing the site rather than rebuilding.

Why it matters

Many owners assume that missing from AI answers means the site needs replacing. This article argues the cause is often simpler: crawlers blocked by a plugin, host or robots.txt file, or key facts hidden in images, sliders or script-loaded content. These are cheaper to fix than a rebuild.

Note that this is a marketing agency's post, and some points, such as the claim about AI crawlers and JavaScript, come from individual experts rather than Google. The Google-sourced points are the practical baseline: indexed, readable text, and matching structured data. Clear, specific pages also help in ordinary search.

Source: DEV Community. This brief summarises their reporting; read the original for the full detail.