Seriously, why help an industry that we all know doesn't care and will still scrap your site regardless? The least they can do it put in some minimal effort without expecting everyone to bend over for them.
No one does SEO because they're trying to help Google.
You do it because you're trying to help the people using google. (Edit: or trying to make money by driving traffic for ads)
Whether or not companies spend time on AEO is directly tied to whether LLM/agents/AI/etc end up becoming a lead channel that buyers use to research products to buy.
I'd rather have a site showing how well my site is protected from being accessed by AI agents would be preferable, and advises how I can lock it down further. Basically, the exact opposite of this.
Maybe we can start a new protocol where the html is encrypted, and the viewer must try 2^10 to 2^20 hashes before the decryption key is discovered. Same formula that BTC mining uses. It would be negligible cost for any single user but terribly expensive for crawling en-masse.
Anything that increases the entry time by a second or more is a pretty good way to make me (and probably others) just not bother with opening the website.
One could consider that the LLM paradox: If you don't want an LLM talking about how to make a nuclear weapon, you first need to explain to them how to make a nuclear weapon, which increases the likelyhood, despite your admonition, that they would talk about it.
So perhaps you can point your LLM at this and ask it to inverse the rules and make sure user design remains consistent.
The absurd process of SEO hucksters trying to pivot their obsolete services into "GEO" as most ecommerce websites realize their entire value was a list of part numbers and prices.
I don't want my site to be agent ready. I'd prefer people visit my site so that I can make revenue than have an AI scrape my content and answer the question for someone else.
I've redesigned my site to have enough content so that AI knows what I have but they have to send the user to my site to use an interactive JavaScript widget to get the final answer they need. So far so good, but not sure how long that will work for.
"Agent-ready" for me would mean they are all being locked out, given the boot, shown the middle finger, and ideally sent into an endless fractal maze never to return.
No metric for performance, obviously. That would ruin the entire narrative.
How much CPU time an average request takes is probably the most important factor in the real world. No one running a frontier AI lab is going to honor any of the metadata described here.
Ironically, this feels exactly like the various "semantic web" initiatives, only this time coming directly from the tech megacorps and not the starry-eyed "free web"/"open data" idealists.
It will hit exactly the same walls too, namely that the technical details are completely irrelevant - if adopting a standard is actually a negative for websites, because it will separate the site from its users, sites will obviously not do it.
You can lead the horse to water but you cannot make it drink, especially if the water is obvious poison.
I feel pretty uncomfortable by this being a Cloudflare product. Cloudflare is the one that I'm expecting to keep bots out of my site with their AI bot blocking feature. Feels like I'm letting the fox guard my henhouse.
The TDMRep protocol [1] is supposed to tell scrappers used for text and data mining whether a ressource can be mined or not. Naively, I would say that a website which explicitly express not wanting to be included in training data would also be considered not wanting to be pulled by agents. I know it's not the same thing, but it still itches me a bit.
We run agents on web tasks and the difference between
clean semantic HTML and a site drowning in React state is massive.
Agents can technically navigate both, but one costs 10x more in
compute and fails 5x more often.
Whether site owners have any incentive to care is a different
question. Agent traffic doesn't pay the bills yet. Something like
x402 could change that, but the payment rails aren't there.
Around 2010 I met a friend at a bar in San Francisco and within 10 minutes we were approached by someone with a chocolate bar startup. It may have been vaguely associated with developers or maybe I'm misremembering. We got a free sample and I explained I didn't live in the US and I also wasn't an investor. They left and moved on to the next group of people at the bar.
This has always stuck to me as an example of the pinnacle of collective investment delusion that seems to exist in certain circles. They idea that you can shape the world to your product instead of improving the world with your product. You just have to try hard enough.
I think this is worth typing a random website into or your website to see it’s analysis.
I’m not really interested in my website being ai ready, but it’s particularly fascinating to me that they are suggesting and interface for ai agents to make payments to secure access to an api.
Generally, when I want to pay for an api, it would be really wonderful to be able to just direct an ai to setup the account and get me some credentials.
Or it's a psyop to see which IP owns which website. Datamining this at scale, you come across isitagentready.com, chances are, you're going to plug in your own website(s) into it, so now cloudflare has a mapping of IP to website owner. If you used your home wifi, glue that info to your google/meta ad profile, and then Cloudflare also knows what's up.
It's a shame that Cloudflare rolled out a bunch of neat product announcements under the confusing, noisy umbrella of "Agent Week". Off the top of my head, Artifacts, Email, Mesh (tailscale competitor), all buried.
It would be helpful if somebody could post what it looks for so I can add it to fail2ban. I tried opening up my website temporarily but it will cancel out if it doesn't find something at /. When I retry sometimes it also says it is blocked when clearly there is not anything in my logs so it is not retrying.
I have reduced my online presence to much less than it once was partly because I don't want to feed this machine training data that I've worked hard to make for a human audience.
My traffic is down 60% year on year because of AI overviews and LLMs. They took everything without consent, used it without credit, and pushed my retirement back a few years. Now I should make their job easier?
This seems like nonsense at any angle? Like, if the agent hype comes true, then agents will be just as good at using any website as humans are, and there's no need to make any changes to your site. And if the hype doesn't come true, then who cares if your site is agent ready.
Unless of course you want to expose some functionality only to AIs, not humans. Then sure. But why would you want to do that?
so use this and then do the opposite of what it suggests if you want to have a cheap, low-effort way to prevent AI from being able to use your content effectively
I get a few points for having a robots.txt with rules specific to AI-crawlers, even though those rules are complete bans. Shame, I was hoping to get a 0.
I think this is meant for "web apps", not "websites" ("sites"). I tried emsh.cat (a blog) and got 25, it complains about missing an "API catalogue", OAuth/OIDC and a bunch of more completely irrelevant stuff. Also tried HN which is very easy for any agent worth their salt to both parse and browse, can hardly get better for an agent, and it gets a score of 17.
Seems like this belongs squarely in the fun and ever-growing collection of "Cloudflare throws vibe-slop into the world and see what sticks".
178 comments
Also AI industry: "Please make sure your website is adapted so that AI agents are able to use it."
No one does SEO because they're trying to help Google.
You do it because you're trying to help the people using google. (Edit: or trying to make money by driving traffic for ads)
Whether or not companies spend time on AEO is directly tied to whether LLM/agents/AI/etc end up becoming a lead channel that buyers use to research products to buy.
> You do it because you're trying to help the people using google.
Who are all _super_ interested in "Top 10 Ways to make a summer Mojito."
>You do it because you're trying to help the people using google.
Haha, no, people do it to try and get ranked higher and thus make more money. They're not trying to help anyone.
Thank you
The latency while browsing the web these days is brutal as a result; between Anubis and Cloudflare and the like.
Our prize for it will be the impending super intelligence our benevolent future overlords allow us to exploit, I suppose. /s
So perhaps you can point your LLM at this and ask it to inverse the rules and make sure user design remains consistent.
403 Forbidden
error code: 1106
The site is blocking our scanner. This may be due to WAF rules, bot detection, or IP-based restrictions.
Perfect :)
I've redesigned my site to have enough content so that AI knows what I have but they have to send the user to my site to use an interactive JavaScript widget to get the final answer they need. So far so good, but not sure how long that will work for.
"Now, make sure your websites are rigorously structured in such a way that allows the technology to work..."
We couldn't scan this site isitagentready.com returned 522
The site appears to be experiencing server errors. This is not an agent-readiness issue. Try scanning again later.
Oops.
(Hint: no)
We couldn't scan this site isitagentready.com returned 522
The site appears to be experiencing server errors. This is not an agent-readiness issue. Try scanning again later.
How much CPU time an average request takes is probably the most important factor in the real world. No one running a frontier AI lab is going to honor any of the metadata described here.
It will hit exactly the same walls too, namely that the technical details are completely irrelevant - if adopting a standard is actually a negative for websites, because it will separate the site from its users, sites will obviously not do it.
You can lead the horse to water but you cannot make it drink, especially if the water is obvious poison.
>
We couldn't scan this site> isitagentready.com returned 522
Ironic perfection.
[1] https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-...
Whether site owners have any incentive to care is a different question. Agent traffic doesn't pay the bills yet. Something like x402 could change that, but the payment rails aren't there.
Good.
This has always stuck to me as an example of the pinnacle of collective investment delusion that seems to exist in certain circles. They idea that you can shape the world to your product instead of improving the world with your product. You just have to try hard enough.
I’m not really interested in my website being ai ready, but it’s particularly fascinating to me that they are suggesting and interface for ai agents to make payments to secure access to an api.
Generally, when I want to pay for an api, it would be really wonderful to be able to just direct an ai to setup the account and get me some credentials.
I have reduced my online presence to much less than it once was partly because I don't want to feed this machine training data that I've worked hard to make for a human audience.
A lot of the misses are for stuff a blog doesn’t need like mcp or api catalogs. It’s a damn blog, I have no api. Unless rss feed counts.
> Goal: Return HTML responses as markdown when agents request it
Does anything legitimate use this?
If I see a request for my page as markdown, does that mean an AI scraper is poking at it? Sounds like a good time to return a zipbomb.
Unless of course you want to expose some functionality only to AIs, not humans. Then sure. But why would you want to do that?
Fix: Implement the WebMCP API by calling navigator.modelContext.provideContext()
but I already do that. the extension detects them https://chromewebstore.google.com/detail/webmcp-model-contex...
[1]: https://motherfuckingwebsite.com/
Seems like this belongs squarely in the fun and ever-growing collection of "Cloudflare throws vibe-slop into the world and see what sticks".