According to DEV Community, in an article originally published on kubryx.sk, people see a finished page in a browser, but search and AI crawlers work differently. They download the page code and try to work out the subject from it. If key text only appears after scripts run in the browser, a crawler may not see it.
The article explains that JavaScript is code that runs in the browser and can build or fill in a page. Crawlers handle it in different ways: some run it in a limited way and some not at all. A site drawn entirely in the browser may look almost empty to them. The piece says this can apply to sites made with AI tools. It describes server-side rendering and static generation as safer, because the server sends finished HTML with the text on first load.
It offers a five-minute check. View the page source and look for your headings and paragraphs. Turn off JavaScript and see what remains. Confirm each page has one clear main heading, a meaningful title and description, alt text on images, and ordinary links rather than script-only buttons. Open your robots.txt file and make sure it does not block pages you want public.
The article also covers structured data, a short block of code, commonly in JSON-LD, that tells machines what a page holds, such as a business, address, opening hours, product or article. It must match what is visible on the page and should be tested with the schema.org validator or Google's testing tools. It does not guarantee display or citation. Finally, it describes llms.txt as a proposed convention, not an official standard: a Markdown file at the site root that summarises the business and links key pages. The article says it is unclear which services read it, so it is cheap to add but should not be relied on.
Why it matters
If your services, prices or contact details only appear after scripts run, some crawlers may see a nearly blank page. That can limit how well search engines and AI tools understand your site. The risk is higher for sites built entirely in the browser, including some made with AI tools.
The checks the article describes are simple and need no special software. Readable source text, accurate structured data and a correct robots.txt are basics that help crawlers, while optional extras like llms.txt come second.
Source: DEV Community. This brief summarises their reporting; read the original for the full detail.