Before a website goes live, run through at least 25 checks across six categories: indexing, metadata, structured data, AI-crawler readability, performance, and measurement. Skip any one category and your site risks incomplete indexing by Google or AI engines after launch. Each item below comes with why it matters and how to verify it — several of them are lessons we learned the hard way on our own site, including a Next.js metadata merge trap, an hreflang code mismatch, and a gap between our sitemap's URL count and what Google actually recognized.
Indexing
- Sitemap covers every real route, with hreflang alternates. Only URLs listed in the sitemap reliably get crawled and indexed, and dynamically generated pages (articles, case studies) are the easiest to miss. Verify: open
/sitemap.xmldirectly, confirm the URL count matches your actual page count, and spot-check a few entries for0tags. - robots.txt allows legitimate crawlers rule by rule, and explicitly excludes admin/API paths. A blanket
Disallow: /blocks SEO and AI crawlers alike; only exclude paths that genuinely shouldn't be public. Verify: runcurl https://yourdomain/robots.txtand confirm there's no site-wide block. - Every page has a unique, self-referencing canonical tag. A wrong canonical makes Google treat multiple pages as duplicates of one another, diluting ranking signals. Verify: view source for
0, or use Google Search Console's URL Inspection tool to compare the "user-declared" canonical against the "Google-selected" one. - Track the gap between "indexed" and "excluded" page counts after submitting the sitemap. Listing URLs in a sitemap doesn't mean Google indexes all of them — in our own launch period (GSC data range 2026-09-14 to 09-19), our sitemap listed 114 URLs but Google recognized only 51 pages, with 30 indexed and 21 not indexed. Verify: check the Coverage/Pages report in Search Console for the breakdown by exclusion reason.
- Don't let locale auto-detection 307-redirect away a non-default-locale canonical page. Frameworks like next-intl default to redirecting based on the
Accept-Languageheader. Googlebot usually doesn't send it, but ChatGPT-User, some browser-based crawlers, and Lighthouse/PageSpeed Insights do — which can make your zh-TW canonical page effectively invisible to them. Verify: runcurl -H "Accept-Language: en-US" -I 0 and check whether it 307-redirects and whether the response is missing aVary: Accept-Language` header.
Metadata
- Every page has its own title and description — no shared template. Duplicate title/description tags get auto-rewritten by Google in search results and dilute the keyword targeting of each page. Verify: view source on 5-10 different pages and confirm the
0and1differ and match each page's topic. - A child page's openGraph/twitter fields shouldn't wipe out the parent's. Next.js metadata is shallow-merged, not deep-merged — if a child page returns only
openGraph: { url, images }, it silently drops the inheritedog:site_name,og:locale, andog:typefrom the parent layout, and the same happens to thetwitterobject. We hit exactly this bug and fixed it by routing every page through a singlebuildPageMetadata()helper. Verify: view source on a child page and confirm0and1are still present, not just the page's ownog:url/og:image. - hreflang codes are consistent site-wide — don't mix
zh-TWandzh-Hant. Our own 2026-09-16 audit found the in-page0tag usingzh-Hantwhile the sitemap and HTTP Link header usedzh-TW— three places sending contradictory signals; we later unified it. Verify: compare the in-page1, the sitemap, and the HTTP response'sLinkheader — all three should agree. - Dynamically generated 404 pages still carry a correct
0. Some multilingual framework versions produce a raw HTML response for dynamicnotFound()that's missing thelangattribute (a known upstream limitation we chose to accept, since 404 pages are noindex anyway). Verify:curla deliberately broken URL and check whether the response's1tag has alangattribute.
Structured Data
- Organization JSON-LD includes full legal entity info and
sameAs. Fields likelegalName, registration number, address, andsameAslinks give search engines and AI a clear record of which company this is. Verify: paste your homepage into Google's Rich Results Test and confirm Organization parses cleanly with no missing-field warnings. - FAQPage/HowTo/Article schema matches the actual content type of each page. The wrong schema type, or an incomplete one, means the markup does nothing — Google doesn't reward you for merely having JSON-LD, only for valid JSON-LD. Verify: run Rich Results Test page by page and confirm no "missing required field" or "invalid item" warnings.
- Multi-level pages use BreadcrumbList. It lets search results display a page's path within your site and reinforces the semantic hierarchy between pages. Verify: Rich Results Test, or view source and confirm the
BreadcrumbListJSON-LD'sitemListElementorder matches the actual page path. - Judge structured data by completeness, not just presence. Our own site has JSON-LD across Organization, WebSite, Service, Article, HowTo, FAQPage, Person, and DefinedTermSet types, yet the structured-data category still scored only 50/100 at our 2026-09-16 audit — broad type coverage doesn't guarantee a high score. Verify: beyond Rich Results Test's pass/warning/error status, cross-check against Google's structured data documentation for required vs. recommended fields.
AI Crawlers
- robots.txt explicitly allows the major AI crawler user-agents. Not blocking them is what gives GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended a chance to read and eventually cite your content. Our own robots.txt has 16 rules that allow these crawlers individually, disallowing only
/adminand/api/. Verify:curl https://yourdomain/robots.txtand check each AI crawler UA you want to allow. - Publish both a short
/llms.txtand a fuller/llms-full.txt. Per its spec site, llms.txt is a proposal — not a W3C or IETF standard — for a Markdown file that gives language models background and links; thousands of sites publish one (checked September 2026). We publish/llms.txtplus a full-text/llms-full.txt. Verify: open/llms.txtdirectly and rungrep -c "^##"to confirm it has real sections rather than an empty template. - Support Markdown content negotiation so AI agents can fetch content more cheaply. When an AI agent requests a page with
Accept: text/markdown, our own article pages rewrite that request via middleware to a dedicated/md/[locale]/[slug]route that returns clean Markdown instead of full HTML. Verify:curl -H "Accept: text/markdown" https://yourdomain/insights/some-slugand confirm you get plain Markdown, not HTML. - Don't assume a special AI schema is what gets you cited. Google states directly: "You don't need to create new machine readable files, AI text files, or markup to appear in these features," and "There are no additional technical requirements" beyond standard indexing eligibility — the work that matters is still standard SEO fundamentals. Verify: cross-check Google's official documentation (verified September 2026) and confirm your general indexing health is solid, not just whether you happen to have an llms.txt file.
Performance
- Run a mobile-simulated performance test and archive it. Load speed directly shapes user experience; if you never measured pre-launch, you can't tell whether a later redesign improved or regressed things. Verify: run PageSpeed Insights or Lighthouse (mobile simulation) once at launch and archive LCP, TBT and CLS.
- Find the real cause of a slow LCP — don't assume it's images. At our own 2026-09-16 audit (Lighthouse 13.4.1, mobile simulation), homepage LCP was 5.9s and article-page LCP was 5.3s. The culprit wasn't images: a background animation ate 650-1100ms of main-thread time per page mount, and an article fade-in kept server-rendered text at
opacity:0until hydration. Deferring the animation viarequestIdleCallbackand excluding the fade-in from above-the-fold text dropped live LCP to 2.5s (homepage) and 2.3s (articles). Verify: check the Diagnostics/elementRenderDelaybreakdown in Lighthouse, or record a load in Chrome DevTools' Performance panel. - Respect users' reduced-motion preference. Decorative animation that ignores
prefers-reduced-motionisn't just a UX miss — it's wasted performance budget. Our own background animation skips mounting entirely when that preference is detected. Verify: in Chrome DevTools' Rendering panel, enable "Emulate CSS media feature prefers-reduced-motion: reduce," reload, and confirm the background animation doesn't appear. - Keep tracking real-user performance after launch, not just one-off audits. Lighthouse runs in a simulated environment; real visitors' devices and network conditions vary widely. We connected Vercel Speed Insights to our own site after launch. Verify: periodically check a real-user monitoring tool (Vercel Speed Insights, or Search Console's Core Web Vitals report) instead of relying on a single simulated score.
Measurement
- Set up Google Search Console and Bing Webmaster Tools and submit your sitemap before launch. Without domain verification and a submitted sitemap, you have zero visibility into clicks or impressions. Verify: after verifying both, check the Sitemaps tab for a "Success" status, not "Couldn't fetch."
- Track "total impressions/clicks" and "coverage" separately — don't rely on one number. In our own early launch window (data range 2026-09-14 to 09-19), we had 9 total clicks, 68 impressions, a 13.2% average CTR, and an average position of 9.1, with the homepage performing best (8 clicks / 38 impressions / 21.1% CTR / average position 3.7). Over the same period, our sitemap listed 114 URLs while Google recognized only 51 pages. Good click/impression numbers don't mean coverage is healthy. Verify: check Search Console's "Performance" and "Pages" (indexing) reports independently, not just the homepage's rolled-up totals.
- Use IndexNow to speed up indexing on non-Google engines. IndexNow is supported by engines such as Bing and Yandex, but not by Google (our submission script notes this explicitly). Our own batch-submission script has pushed 406 URLs cumulatively (292 on September 19, 2026 and 114 on September 20), and Bing Webmaster Tools confirmed our sitemap's 112 URLs with 0 errors and 0 warnings, crawled successfully the day after submission. Verify: check your IndexNow submission log and response codes, and confirm crawl status in Bing Webmaster Tools' Sitemaps tab.
- Measure off-site brand signal, not just on-site technical scores. In our own 2026-09-16 GEO audit, technical foundation scored 86, but brand authority signal (a 20%-weighted category) scored only 5 — LinkedIn, Wikidata, GitHub, and third-party review platforms showed zero mentions of our company at the time. Getting the technical work right doesn't mean AI engines trust or cite you. Verify: search your company name on LinkedIn, Wikidata, Google Business Profile, and GitHub, and inventory presence/absence rather than relying solely on an audit score.
Next Steps
You don't need to complete all 25 items before going live, but run through the list at least once before you actively promote the site, and archive each verification result so you have something to compare against after a future redesign. If you're planning a new site or want to retrofit SEO/GEO fundamentals onto an existing one, see our website services, or reach out via contact to talk through where you're stuck.
References
- Google: Appearing in AI features like AI Overviews and AI Mode — Google Search Central, verified September 2026
- Google: Intro to structured data — Google Search Central, verified September 2026