Answer Engine Optimization stops being a buzzword the moment you implement its three concrete artifacts: a machine-readable content inventory (llms.txt), extraction-ready FAQ structured data, and an explicit policy for AI crawlers. Here’s how each was built into Project Spark, and the privacy and quality traps we hit.
Key takeaway: AEO implementation is three files’ worth of work: llms.txt tells AI systems what you have, FAQPage schema gives them liftable answers, and robots.txt tells them what they may do with it. Most sites have none of the three.
Why answer engines needed their own layer
Classic SEO assumes a specific consumer: a crawler that indexes pages so a human can choose among ranked links. Answer engines, ChatGPT, Perplexity, Google’s AI Overviews, and the assistants built on them, consume differently. They read content to compose an answer, sometimes with a citation, sometimes without, and the human may never visit the page at all.
That difference changes the engineering brief. For a ranked link, you optimize the page. For an extracted answer, you optimize the units inside the page: the self-contained paragraph, the labelled question-and-answer pair, the machine-readable inventory that tells a crawler which pages matter. Project Spark treats these as first-class outputs of the same pipeline that produces titles and descriptions, one content workflow, several consumers (Part 4 covers the strategy; this post covers the implementation).
A note on timing: when this layer was built, none of it was standard practice. That was the point. The artifacts are cheap, they compound, and being early costs little, being late means the answer engines learned your market from your competitors.
llms.txt: a sitemap for language models
The plugin serves a Markdown inventory at the site root: site name, description, and a link-plus-summary line for every published post and page. AI crawlers get a curated map instead of scraping navigation menus.
The concept is deliberately modest, a plain-text file, human-readable, trivially parseable. But four implementation details turned out to carry almost all of the engineering weight:
- Respect noindex, everywhere. Posts marked noindex are excluded. This sounds obvious, but it’s worse to get wrong here than in a sitemap: leaking a hidden page to a search engine gets it indexed, which is reversible; leaking it to an AI training crawler teaches models your private content permanently. There is no “remove from training data” request form. Our exclusion logic is pinned by automated tests on both the sitemap and llms.txt, so a future refactor cannot silently widen the inventory.
- Exclude media attachments. WordPress treats every upload as a “page” with its own URL. Without an explicit exclusion, your llms.txt lists every PDF, screenshot, and internal diagram as a “key resource”, including documents nobody intended to publish prominently. The exclusion is enforced in code and verified by tests.
- Summaries with a fallback chain. Each entry uses the editor’s hand-written excerpt when present, otherwise a trimmed extract of the content, so AI engines always get context, and human-curated context wins over machine-trimmed context wherever a human bothered to write one.
- Decode entities, don’t HTML-escape. The file is Markdown, not HTML. Escaping would ship
Don'tto every AI reader, literally teaching language models a corrupted version of your copy. The correct transform is strip-tags plus entity-decoding, the opposite of what web developers reach for by reflex, and a small example of why AEO artifacts deserve their own code review rather than borrowed assumptions.
Two operational details complete the picture: the file carries a noindex header of its own (it is for AI crawlers, not for search results pages), and robots.txt announces it, closing the discovery loop, a crawler that reads robots.txt finds the sitemap and the llms.txt in the same place.
FAQ schema: answers ready for extraction
Every important page carries FAQPage structured data with human-approved Q&A pairs, drafted by AI from the page content (Part 5). Voice assistants and AI Overviews extract these directly. For a Hebrew site the pairs are in Hebrew, matching how customers actually ask.
The production workflow matters more than the markup. The AI proposes question-and-answer candidates from the page’s actual content; an editor reviews each pair, rewrites phrasing into the register customers use, and approves or discards individually. Nothing is published unreviewed. Two quality rules emerged from usage:
- Questions must sound asked, not written. “What are the opening hours of the branch?” outperforms “Branch operating schedule” in every extraction context, because answer engines match against real query phrasing.
- Answers must survive being read aloud. Around thirty words is the practical ceiling (Part 12 explains why). Longer answers still work for rich results but stop being voice-eligible.
There is also a data-integrity story here that connects to the rest of the series: the FAQ storage path was one of the places where a wrong storage key had silently broken schema generation for a long period, a bug found and permanently pinned during the testing campaigns described in Part 14. Structured answers are only as good as the pipeline that stores and emits them.
AI crawler policy: an explicit decision, not a default
The plugin maintains a registry of 14 known AI bots, classified by purpose: training crawlers (like GPTBot and Google-Extended), answer crawlers, and AI search crawlers. The dashboard shows each bot’s current robots.txt status, allowed, blocked, or unset, and warns when most policies are unset, because “no decision” is still a decision: it means “allow everything by default.”
The classification is the useful part. The three bot categories have entirely different business meanings:
- Training crawlers collect content to train future models. Allowing them contributes your expertise to systems that will answer without citing you. Blocking them protects content but removes you from a channel that may matter later.
- Answer crawlers fetch content at question-time to compose cited answers. Allowing these is closest to classic SEO: visibility with attribution.
- AI search crawlers power AI-branded search products; their behaviour sits between the two.
A single “block AI” switch would erase these distinctions, and a silent default erases the decision itself. The dashboard instead forces a per-category conversation with the site owner: do you want your content in training data? In cited answers? The plugin doesn’t decide, it makes the owner decide, visibly, and keeps the decision auditable in a health check that flags when most policies remain unset.
That conversation is the real feature. In our client’s case it surfaced a genuine business position (“cited answers yes, anonymous training no”) that no default would have expressed.
Three ecosystems, three doors
The crawler policy work sharpens further once you stop saying “AI bots” and start naming families. The three ecosystems that matter commercially, GPT (OpenAI), Claude (Anthropic), and Gemini (Google), each knock on your site through their own doors, and each door can be answered differently:
- OpenAI operates separate crawlers for model training and for answer-time search and citation. That separation is a gift: a business can decline the training bot while welcoming the search bot, exactly the “cite me, do not train on me” position from the previous section, expressed per ecosystem.
- Anthropic’s retrieval crawler serves Claude’s answers; Claude’s professional-heavy user base makes it a disproportionately valuable citation source for business services.
- Google splits its classic crawler (which also feeds AI Overviews through the search index) from its dedicated AI policy token, so blocking Gemini’s generative features without harming classic search requires knowing which token controls what. The registry documents the distinction so the owner decides with the mechanics in view, not by folklore.
Measurement then closes per family: ask ChatGPT, Claude, and Gemini the same commercially important questions at review intervals, in Hebrew and English, and score mentions, citations, and factual accuracy per assistant. One foundation, three doors, three scoreboards; a business cited correctly by two assistants and hallucinated by the third has a specific, fixable data problem. The full strategy of treating the model families as both audience and toolkit is developed in the hybrid AI deep dive.
Does AEO work?
Measured honestly: AI referral traffic is still small for most businesses, but the artifacts are cheap, they compound, and they’re the same structured-data investments that classic SEO rewards today. Being early to AEO costs little; being late means your competitors’ answers get cited instead.
There is also a defensive argument that gets less attention. Answer engines will describe your business whether or not you participate, from third-party directories, reviews, stale caches, and competitors’ content. The llms.txt inventory, the FAQ pairs, and consistent structured data are how you put your own version of the facts in front of the machines doing the describing. AEO is partly optimization and partly reputation management for an audience of models.
How to see whether it is working
AEO measurement is younger than AEO itself, but “hard to measure” does not mean “unmeasurable.” Four observation points cover most of it:
- Referral traffic from AI surfaces. Assistant products increasingly send clickable citations; their referrers show up in analytics like any other source. The volumes are small today, track the trend, not the total.
- Direct interrogation. Ask the major assistants the questions your customers ask, in the language they ask them, monthly. Record whether your business is cited, described accurately, or absent. This is the AEO equivalent of checking your rankings, and almost nobody does it.
- Crawler logs. Your server logs show which AI bots visit, how often, and what they fetch. A bot that reads your llms.txt and then fetches the pages it lists is behaving exactly as designed, visible proof the inventory is being consumed.
- Accuracy of the answers themselves. When an assistant describes your business wrongly, old hours, a stale address, a discontinued service, treat it as a data defect and trace which source taught it. Usually the fix is on a surface you control: the listing, the structured data, the inventory.
The honest framing for clients: AEO today is a low-cost position, not a traffic engine. You are buying accuracy in how machines describe you, and an early seat in a channel whose growth curve everyone can see. Both are cheap now and expensive later.
An implementation checklist
For readers applying this to their own sites, the order that worked for us:
- Audit robots.txt for the 14 known AI crawlers. Record each bot’s current status. “Unset” is the answer for most sites, that is your baseline.
- Hold the business conversation. Training versus answer crawlers, per category, with whoever owns the content. Write the decision down.
- Publish llms.txt. Curated inventory, noindex exclusions, attachment exclusions, human excerpts preferred, entities decoded.
- Add FAQ structured data to your highest-intent pages first. Service pages, pricing pages, location pages, the places where a customer’s question has commercial weight. Human review on every pair.
- Pin all of it with tests if you operate the code, or with a recurring manual audit if you do not. AEO artifacts rot quietly: a redesign, a new plugin, or a template change can silently drop them.
Next: Part 10, Local SEO for Multi-Location Businesses
FAQ
What is an llms.txt file?
A Markdown file at your site root listing your key content with links and short summaries, designed so AI crawlers can understand your site’s inventory efficiently, a sitemap for language models.
Should I block AI crawlers from my website?
It depends on your goals: blocking training crawlers protects content from model training but may reduce AI visibility; allowing answer crawlers can win citations. The mistake is not deciding, audit your robots.txt and make each policy explicit.
Does FAQ schema still matter?
Yes, beyond rich results, FAQPage structured data is a primary extraction source for voice assistants and AI answer engines. Keep answers concise and human-reviewed.
Should noindex pages appear in llms.txt?
Never. Content excluded from search must also be excluded from AI inventories, leaking it to training crawlers is more permanent than a search index leak.
How do I create an llms.txt file for my website?
Publish a Markdown file at your site root listing your site name, a short description, and a link with a one-line summary for each important page. Exclude noindex content and media attachments.
What is the difference between a training crawler and an answer crawler?
Training crawlers collect content to train future AI models, usually without attribution. Answer crawlers fetch content when a user asks a question, to compose an answer that can cite you.
