It is a bit weird to see LLMs suddenly being presented as the reason to follow what are basically long standing best practices.
'You must write docs. Docs must be in your repo. You must write tests. You must document your architecture. Etc. Etc.'
These were all best practices before LLMs existed and they remain so even now. I have been writing extensive documentation for all my software for something like twenty years now, whether it was for software I wrote for myself, for my tiny open source projects or for businesses. I will obviously continue to do so and it has nothing to do with:
> AI changes the game
The reason is simply that tests and documentation are useful to humans working on the codebase. They help people understand the system and maintain it over time. If these practices also benefit LLMs then that is certainly a bonus, but these practices were valuable long before LLMs existed and they remain valuable even now regardless of how AI may have changed the game.
It is also a bit funny that these considerations did not seem very common when the beneficiaries were fellow human collaborators, but are now being portrayed as very important once LLMs are involved. I'd argue that fellow humans and your future self deserved these considerations even more in the first place. Still, if LLMs are what finally motivate people to write good documentation and good tests, I suppose that is a good outcome since humans will end up benefiting from it too.
> It is a bit weird to see LLMs suddenly being presented as the reason to follow what are basically long standing best practices.
Maybe it's the speed of LLM iteration that makes the benefit more immediately obvious, vs seeing it unfold with a team of people over a longer time? It's almost like running a study?
I have a similar reaction to strong static types being advocated to help LLMs understanding/debugging code, catching bugs, refactoring... when it's obvious to me this helps humans as well.
Curious how "this practice helps LLMs be more productive" relates to studies that try to show this with human programmers, where running convincing human studies is really difficult. Besides problems with context sizes, are there best practices that help LLMs a lot but not humans?
Agreed. LLM usage just makes the impact more visible.
I bet one of the next "revelations" is going to be: Avoid sprinkling side effects throughout your code. Prefer functional code, push side effects to the boundary (functional core, imperative shell).
> It is a bit weird to see LLMs suddenly being presented as the reason to follow what are basically long standing best practices.
About 95% of the work needed to make LLMs happy is just general purpose better engineering. Units tests? Integration tests? CI? API documentation? Good example? All great for humans too!
I consider this largely a good thing. It would be much worse if the changes needed for Happy LLMs were completely different than what you want for Happy Humans! Even worse would be if they were mutually exclusive.
Reminds me of the "semantic web". Making content machine-readable has positive side effects. (clear structure, reduced ambiguity, separation of data and presentation)
It's another example where the reason for better engineering is to make machines (search engines) happy.
AI means that you cannot defer software design until you've written half code; you cannot defer documentation to random notes at the end.
It has the effect of finally forcing people to think about the software they're making, assuming they care about quality. If they didn't, then it's not practically different from an insecure low-code app or something copy-pasted from 15 year old StackOverflow answers.
Single iteration waterfall is a broken process. You really need those late stage usage feedback signals unless your requirements were somehow captured by God.
> Single iteration waterfall is a broken process. You really need those late stage usage feedback signals unless your requirements were somehow captured by God.
Waterfall, as prescribed and as practiced, had feedback signals.
> It is also a bit funny that these considerations did not seem very common when the beneficiaries were fellow human collaborators, but are now being portrayed as very important once LLMs are involved. I'd argue that fellow humans and your future self deserved these considerations even more in the first place.
The reason might have been a cultural regression. At least with documentation, it seems to have been much better and a bigger priority a couple decades ago.
At the start of my career, teams produced documents as a part of their work, and there were even technical writers on staff. Then agile hit, the writers were laid off, much of what little documentation that was created was kept in various work-tracking systems and wikis that were periodically replaced, often with little to no migration.
Well, it's timely because there's a docs platform that has surged in popularity, and it really is not a good idea for most of those who need technical docs to be using a SaaS that approximates Squarespace.
As a tech writer, it's not surprising. All LLMs did was get PMs on the same page as TWs, devs, and support toward prioritizing it, because now it benefits feature development in the short term instead of the long term. They can put it on a quarterly review slide and in an OKR, and their bosses will jump up and down and squeal and give them a raise instead of shooting them in the face.
I see this as not just internal API/architecture/code documentation, but product documentation too. We maintain internal docs about how our product is used for our support, implementation, and sales teams to reference.
Right now it's hosted externally (in our "blessed" knowledge base) but if it could be pulled into the repo, and we set an AI reviewer on every pull request to sanity check that if the changes we're doing have a material impact on the feature as described in those docs that it should be flagged (or changes proposed) that'd be a nice win for keeping them up-to-date, and it's easy enough to publish markdown as html or even script an update to the canonical site when we merge to main.
I agree and would go one step further. The way people are now talking to LLMs to write code is the way we need them to plan and discuss in meetings with humans.
Everything regarding AI-assisted development is basically training wheels for the young people coming into the workplace.
There's a pattern where people create AI-specific infrastructure for coding agents which is essentially instantly obsolete because it's pointless. Stuff like most MCPs (instead of just using a CLI), agent-specific files (CLAUDE.md, AGENTS.MD, github-instructions.NET etc.) etc.
> You should have a good, concise introduction to the codebase that allows anyone to write and test a simple patch in under 15 minutes.
Lately I have seen a lot of things coming full circle like this in a way that always seems positive for humans as well.
Many doomers are running around saying the future is grim because everything will be made for AI agents to use rather than humans. But so far everything done to push that agenda has looked more like a big de-enshittification.
Another one is Model Context Protocol, which brings forth the cutting edge (for 1970) idea of using a standard text based interface so that separate programs can interoperate through it.
If the cost of having non-user-hostile software is to let AI bros run around thinking they invented things like stdin and documentation, I'm all for it at this point.
If any AI bros are reading this here's another idea. Web pages that use a mostly static layout and a simple structure would probably be a lot easier for AI to parse. And google, it would be really beneficial to AI agents if their web searches weren't being interfered with by clickjacking sites such as Pinterest.
there's an implicit ownership change, from having technical writers own the documentation, to including it as part of the commit.
when things are tiny or resource constrained, the same people are doing each task regardless, but "technical writer" is a real job around documentation and manual writing, so there's at least sometimes some real decoupling between the code and outwards facing documents.
that also covers for cases where people can write code well, but who's english(or whatever the target documentation language is) is shaky at best.
The biggest win for me with docs-in-repo isn't the AI angle, it's that pull requests can't land without updating the relevant docs. When your support pages, privacy policy, and README all live in the same repo, they naturally stay in sync with the code.
GitHub Pages serving directly from a /docs folder makes it even simpler, no separate deploy, no separate CMS, no drift. The less infrastructure between writing and publishing, the more likely docs actually get maintained.
We just did this the other week and it's such a great setup using AI. Monorepos in general are better for coding agents since it's a single location to search. But now we have the ability to say "Add xyz optional param to our API" and claude adds the code + updates the documentation. I was also able to quickly ask "look at our API and our docs, find anything out of date".
Our set up is:
packages/
↳ server
↳ app
↳ docs
Using mintlify for the docs, just points to the markdown files in the docs folder. And then a line in the claude.md to always check /docs for updates after adding new code.
We have been on this path at work. But I challenge everyone to consider what you lose with MD vs Confluence (et al). It is NOT easier to author, comment on, label, view history of, move without breaking links, etc. markdown docs vs Confluence. If I am the sole author plus my AI and the scope is narrow (a library), I go for MD. But for a big org, process docs, fast iteration… I’m not convinced, until someone builds equally powerful editing UI on top of MD files.
Out-of-band docs have always been a constant source of frustration and discrepancies. It's really difficult to keep readme.com docs updated with actual code releases because there's no hard constraint preventing one from updating without the other. It just relies on "convention".
There’s an irresistible, almost demoralizing irony in the fact that developers are discovering docs and accessibility only now due to AI. They needed docs and didn’t know it until they had at their disposal an ersatz user in the form of an LLM that asked for context.
There's a lot of things that we mean when we say 'docs'.
The great talk "No Vibes Allowed" put me to the far end of the other extreme - persistent long term state on disk is bad. Always force agents to rebuild, aggressively sub agent or use tools to compress context. The code should be self documenting as much as possible and structured in a way such that it's easy to grep through it. No inline docs trying to describe the structure of the tree (okay, maybe like, 3 at most).
I don't have the time to build such an elaborate testing harness as they do though. So instead I check in a markdown jungle in ROOT/docs/* . And garbage collect them aggressively. Most of these are not "look for where the code is", they are plans of varying length, ADRs, bug reports, etc. and they all can and *will" get GC'ed.
I still use persistent docs but they're very spare and often completely contractual. "Yes, I can enumerate the exact 97 cases I need to support, and we are tracking each of these in a markdown doc". That is fine IMO. Not "here let me explain what this code does". Or even ADRs - I love ADRs, but at least for my use case, I've thrown out the project and rewritten from scratch when too many of them got cluttered up... Lol.
I'm also re-implementing an open source project (with the intent of genuinely making it better as a daily user, licensed under the same license, and not just clean rooming it), which makes markdown spam less appealing to me. I kind of wish there was yet another git wrapper like jujutsu which easily layered and kept commits unified on the same branch but had multi-level purposes like this. Persistent History for some things is not needed, but git as a wrapper for everything is so convenient. Maybe I just submodule the notes....
Note: my approach isn't the best, heck, 1 month ago OpenAI wrote an article on harness engineering where they had many parallel agents working, including some which aggressively garbage collected. They garbage collected in the sense that yes, prolific docs point agents to places XYZ, but if something goes out of date, sync the docs. Again, That works if you have a huge compute basin. But for my use cases, my approach is how I combatted markdown spam.
When I start a new project with a team I start off with asking 'how we will work' and part of that is 'how we will communicate'. Less is more in that world. Jira, confluence, github, slack, email, standup, ad-hock meetings, bongo drums, etc etc. The more places you communicate the harder it is to keep everyone on the same page. I have always been a fan of putting docs next to code for this exact reason and, as far as I can tell, it has been the right decisions every time.
With AI code assistants I personally spend 90% of time/tokens on design and understanding and that means creating docs that represent the feature and the changes needed to implement it so I can really see the value growing over time to this approach. Software engineering is evolving to be less about writing the code and more about designing the system and this is supporting that trend.
In the end I don't think AI hasn't fundamentally changed the benefit/detractor equation, it is just emphasizing that docs are part of the code and making it more obvious that putting them in the code is generally pretty beneficial.
Bit of a plug I suppose, but this was what motivated me to set up AS Notes, my VS code extension which makes VS Code a personal knowledge management system, with linking and markdown tooling. I've built an html converter so they can be published to github pages from the repo. It's here if it's of interest to anyone https://www.appsoftware.com/blog/as-notes-turn-vs-code-into-... ... I'm so much more motivated to write docs when a) its easy to keep them up to date using an agent, and b) someone (agents) will actually read them!
For open source, this has been the practice for many projects. The docs is often in README.md or in a separate folder "docs". For larger projects, there could be a separate repo from where a docs site is built.
However, in corporations, docs are often in Confluence or MS Sharepoint, separate from the code. Tech specs often require comments, discussion, or estimate/budget approvals from non-tech staff. Hence, some corporate AI coding tools can refer to docs in such corporate sites. That doesn't work too well yet, IMHO ... time will show.
Interesting idea overall, and I would support doing this if we can.
Some constraints are:
- Non-programmers are not used to working with Git.
- In practice, they (usually PMs or feature designers) need to write their documents somewhere else.
Possible solutions are:
- Make non-programmers use Git as a documentation tool (upgrade your tooling or GTFO).
- Build a two-way sync tool so that programmers and non-programmers can work from the same source.
- However, in practice, an SSOT (single source of truth) architecture is usually much simpler. Two-way sync tends to be quite difficult, especially across different platforms.
> Non-engineers usually don't have repo access. [Answer:] (1) You can deploy your docs on an internal-only website. (2) There is clear trend with non-engineer code access (which poses some interesting security challenges).
Regarding (2): If, on the other hand, you have your agent use MCP to query, e.g., Confluence, anyone with access to Confluence could in theory do a prompt injection and possibly get access to your repo.
At least doc changes in the repo will undergo code review.
Agree with the post. Working on https://spectagon.md which aims to improve the workflow around reviewing docs in the repo.
Reviewing docs in Github isn't great - as the post mentions, Google Docs or similar is typically where review happens. Spectagon aims to change that so that you can submit PRs for docs and you get a great review experience as a layer on top of Github.
What about a OneDrive folder shared with all developers, mounted in a place the AI can access? Putting docs in git makes it slow to iterate and share. That's my hesitancy with committing them.
Not sure I agree with this. MD files need to be constantly synced to code state- why not just grep the code files? This is just more unstructured indexing
Sounds like they are saying use a repo like git for your documents to help AI read/"understand" your docs. Is that correct ?
I am all for using a source control system for your documents, I usually use RCS. But give AI access to your docs, no thanks. If I upload any of my docs to a public server (very rarely happens), they are compressed and encrypted to make sure only I and a few people can view them.
83 comments
'You must write docs. Docs must be in your repo. You must write tests. You must document your architecture. Etc. Etc.'
These were all best practices before LLMs existed and they remain so even now. I have been writing extensive documentation for all my software for something like twenty years now, whether it was for software I wrote for myself, for my tiny open source projects or for businesses. I will obviously continue to do so and it has nothing to do with:
> AI changes the game
The reason is simply that tests and documentation are useful to humans working on the codebase. They help people understand the system and maintain it over time. If these practices also benefit LLMs then that is certainly a bonus, but these practices were valuable long before LLMs existed and they remain valuable even now regardless of how AI may have changed the game.
It is also a bit funny that these considerations did not seem very common when the beneficiaries were fellow human collaborators, but are now being portrayed as very important once LLMs are involved. I'd argue that fellow humans and your future self deserved these considerations even more in the first place. Still, if LLMs are what finally motivate people to write good documentation and good tests, I suppose that is a good outcome since humans will end up benefiting from it too.
> It is a bit weird to see LLMs suddenly being presented as the reason to follow what are basically long standing best practices.
Maybe it's the speed of LLM iteration that makes the benefit more immediately obvious, vs seeing it unfold with a team of people over a longer time? It's almost like running a study?
I have a similar reaction to strong static types being advocated to help LLMs understanding/debugging code, catching bugs, refactoring... when it's obvious to me this helps humans as well.
Curious how "this practice helps LLMs be more productive" relates to studies that try to show this with human programmers, where running convincing human studies is really difficult. Besides problems with context sizes, are there best practices that help LLMs a lot but not humans?
I bet one of the next "revelations" is going to be: Avoid sprinkling side effects throughout your code. Prefer functional code, push side effects to the boundary (functional core, imperative shell).
> It is a bit weird to see LLMs suddenly being presented as the reason to follow what are basically long standing best practices.
About 95% of the work needed to make LLMs happy is just general purpose better engineering. Units tests? Integration tests? CI? API documentation? Good example? All great for humans too!
I consider this largely a good thing. It would be much worse if the changes needed for Happy LLMs were completely different than what you want for Happy Humans! Even worse would be if they were mutually exclusive.
It's a win. I'll take it.
It's another example where the reason for better engineering is to make machines (search engines) happy.
It has the effect of finally forcing people to think about the software they're making, assuming they care about quality. If they didn't, then it's not practically different from an insecure low-code app or something copy-pasted from 15 year old StackOverflow answers.
> AI means that you cannot defer software design until you've written half code; you cannot defer documentation to random notes at the end.
> It has the effect of finally forcing people to think about the software they're making,
Ah, and all this time I was reliably assured that waterfall, design-upfront, was a broken process...
> Single iteration waterfall is a broken process. You really need those late stage usage feedback signals unless your requirements were somehow captured by God.
Waterfall, as prescribed and as practiced, had feedback signals.
> It is also a bit funny that these considerations did not seem very common when the beneficiaries were fellow human collaborators, but are now being portrayed as very important once LLMs are involved. I'd argue that fellow humans and your future self deserved these considerations even more in the first place.
The reason might have been a cultural regression. At least with documentation, it seems to have been much better and a bigger priority a couple decades ago.
At the start of my career, teams produced documents as a part of their work, and there were even technical writers on staff. Then agile hit, the writers were laid off, much of what little documentation that was created was kept in various work-tracking systems and wikis that were periodically replaced, often with little to no migration.
> The reason is simply that tests and documentation are useful to other humans working on the codebase.
Including future you
Right now it's hosted externally (in our "blessed" knowledge base) but if it could be pulled into the repo, and we set an AI reviewer on every pull request to sanity check that if the changes we're doing have a material impact on the feature as described in those docs that it should be flagged (or changes proposed) that'd be a nice win for keeping them up-to-date, and it's easy enough to publish markdown as html or even script an update to the canonical site when we merge to main.
Everything regarding AI-assisted development is basically training wheels for the young people coming into the workplace.
> You should have a good, concise introduction to the codebase that allows anyone to write and test a simple patch in under 15 minutes.
Yeah, that's the CONTRIBUTING file.
> These were all best practices before LLMs existed and they remain so even now
Okay, so what, should I be moving my docs out of the repo or something?
How should I make it as hard as possible for LLMs to make any use of or suggestions about my documentation?
Many doomers are running around saying the future is grim because everything will be made for AI agents to use rather than humans. But so far everything done to push that agenda has looked more like a big de-enshittification.
Another one is Model Context Protocol, which brings forth the cutting edge (for 1970) idea of using a standard text based interface so that separate programs can interoperate through it.
If the cost of having non-user-hostile software is to let AI bros run around thinking they invented things like stdin and documentation, I'm all for it at this point.
If any AI bros are reading this here's another idea. Web pages that use a mostly static layout and a simple structure would probably be a lot easier for AI to parse. And google, it would be really beneficial to AI agents if their web searches weren't being interfered with by clickjacking sites such as Pinterest.
when things are tiny or resource constrained, the same people are doing each task regardless, but "technical writer" is a real job around documentation and manual writing, so there's at least sometimes some real decoupling between the code and outwards facing documents.
that also covers for cases where people can write code well, but who's english(or whatever the target documentation language is) is shaky at best.
GitHub Pages serving directly from a /docs folder makes it even simpler, no separate deploy, no separate CMS, no drift. The less infrastructure between writing and publishing, the more likely docs actually get maintained.
Our set up is:
Using mintlify for the docs, just points to the markdown files in the docs folder. And then a line in the claude.md to always check /docs for updates after adding new code.GitHub Pages came out in 2008.
https://passo.uno/skills-are-docs/
The great talk "No Vibes Allowed" put me to the far end of the other extreme - persistent long term state on disk is bad. Always force agents to rebuild, aggressively sub agent or use tools to compress context. The code should be self documenting as much as possible and structured in a way such that it's easy to grep through it. No inline docs trying to describe the structure of the tree (okay, maybe like, 3 at most).
I don't have the time to build such an elaborate testing harness as they do though. So instead I check in a markdown jungle in ROOT/docs/* . And garbage collect them aggressively. Most of these are not "look for where the code is", they are plans of varying length, ADRs, bug reports, etc. and they all can and *will" get GC'ed.
I still use persistent docs but they're very spare and often completely contractual. "Yes, I can enumerate the exact 97 cases I need to support, and we are tracking each of these in a markdown doc". That is fine IMO. Not "here let me explain what this code does". Or even ADRs - I love ADRs, but at least for my use case, I've thrown out the project and rewritten from scratch when too many of them got cluttered up... Lol.
I'm also re-implementing an open source project (with the intent of genuinely making it better as a daily user, licensed under the same license, and not just clean rooming it), which makes markdown spam less appealing to me. I kind of wish there was yet another git wrapper like jujutsu which easily layered and kept commits unified on the same branch but had multi-level purposes like this. Persistent History for some things is not needed, but git as a wrapper for everything is so convenient. Maybe I just submodule the notes....
Note: my approach isn't the best, heck, 1 month ago OpenAI wrote an article on harness engineering where they had many parallel agents working, including some which aggressively garbage collected. They garbage collected in the sense that yes, prolific docs point agents to places XYZ, but if something goes out of date, sync the docs. Again, That works if you have a huge compute basin. But for my use cases, my approach is how I combatted markdown spam.
With AI code assistants I personally spend 90% of time/tokens on design and understanding and that means creating docs that represent the feature and the changes needed to implement it so I can really see the value growing over time to this approach. Software engineering is evolving to be less about writing the code and more about designing the system and this is supporting that trend.
In the end I don't think AI hasn't fundamentally changed the benefit/detractor equation, it is just emphasizing that docs are part of the code and making it more obvious that putting them in the code is generally pretty beneficial.
However, in corporations, docs are often in Confluence or MS Sharepoint, separate from the code. Tech specs often require comments, discussion, or estimate/budget approvals from non-tech staff. Hence, some corporate AI coding tools can refer to docs in such corporate sites. That doesn't work too well yet, IMHO ... time will show.
Some constraints are:
- Non-programmers are not used to working with Git.
- In practice, they (usually PMs or feature designers) need to write their documents somewhere else.
Possible solutions are:
- Make non-programmers use Git as a documentation tool (upgrade your tooling or GTFO).
- Build a two-way sync tool so that programmers and non-programmers can work from the same source.
> Answer to objections
> Non-engineers usually don't have repo access. [Answer:] (1) You can deploy your docs on an internal-only website. (2) There is clear trend with non-engineer code access (which poses some interesting security challenges).
Regarding (2): If, on the other hand, you have your agent use MCP to query, e.g., Confluence, anyone with access to Confluence could in theory do a prompt injection and possibly get access to your repo.
At least doc changes in the repo will undergo code review.
Reviewing docs in Github isn't great - as the post mentions, Google Docs or similar is typically where review happens. Spectagon aims to change that so that you can submit PRs for docs and you get a great review experience as a layer on top of Github.
I am all for using a source control system for your documents, I usually use RCS. But give AI access to your docs, no thanks. If I upload any of my docs to a public server (very rarely happens), they are compressed and encrypted to make sure only I and a few people can view them.