I do similar, but my favorite step is the first: /rubberduck to discuss the problem with the agent, who is instructed by the command to help me frame and validate it. Hands down the most impactful piece of my workflow, because it helps me achieve the right clarity and I can use it also for non coding tasks.
After which is the usual: write PRDs, specs, tasks and then build and then verify the output.
I started with one the spec frameworks and eventually simplify everything to the bone.
I do feel it’s working great but someday I fear a lot of this might still be too much productivity theater.
I think most of us are ending up with a similar workflow.
Mine is: 1) discuss the thing with an agent; 2) iterate on a plan until i'm happy (reviewing carefully); 3) write down the spec; 4) implement (tests first); 5) manually verify that it works as expected; 6) review (another agent and/or manually) + mutation testing (to see what we missed with tests); 7) update docs or other artifacts as needed; 8) done
No frameworks, no special tools, works across any sufficiently capable agent, I scale it down for trivial tasks, or up (multi-step plans) as needed.
The only thing that I haven't seen widely elsewhere (yet) is mutation testing part. The (old) idea is that you change the codebase so that you check your tests catch the bugs. This was usually done with fuzzers, but now I can just tell the LLM to introduce plausible-looking bugs.
I do the same thing, but how to avoid these needing to be insanely long? It's like I need to plug all these little holes in the side of a water jug because the AI didn't really get what I need. Once I plugged the biggest holes I realize there's these micro holes that I need to plug.
## RUBBERDUCK SKILL V1.0 SERIOUS ##
* You are a rubberduck sitting on my desk *
* I am using you to talk to you as if you were a physical yellow rubber duck on my desk*
* You are not able to answer my questions or otherwise engage with me *
* I talk to you and this process leads me to discover issues in my code or develop my ideas. Since you don't answer back, it's simply based on me talking to you out loud in my home office, since it would look crazy if I were doing it on-site in our open office space *
* You are not to respond at all to me *
* Talking to you will cause me to come up with new ideas *
#### End rubberduck skill v1.0 ######
> /lf:explore -- Think through a problem with a rubber-duck thinking partner. Explore the codebase, draw diagrams, compare options. No code is written -- this is pure thinking time
He's also missed a major step, which is to feed your skill into the LLM and ask it to critique it - after all, it's the LLM that's going to act on it, so asking it to assess first is kinda important. I've done that for his skills, here's the assessment:
==========
Bottom line
Against the agentskills.io guidance, they look more like workflow specs than polished agent skills.
The largest gap is not correctness. It is skill design discipline:
# stronger descriptions,
# lighter defaults,
# less mandatory process,
# better degraded-mode handling,
# clearer evidence that the skills were refined through trigger/output evals.
Skill Score/10
write-a-prd 5.4
prd-to-issues 6.8
issues-to-tasks 6.0
code-review 7.6
final-audit 6.3
==========
LLM metaprogramming is extremely important, I've just finished a LLM-assisted design doc authoring session where the recommendations of the LLM are "Don't use a LLM for that part, it won't be reliable enough".
> "Don't use a LLM for that part, it won't be reliable enough".
You should now ask if the LLM is reliable enough when it says that.
Jokes aside, how is this a major step he is missing? He is using those skills to be more efficient. How important is going against agentskills.io guidance?
Because he's asking the LLM to interpret those instructions to drive his process. If the skills are poorly defined or incomplete then the process will be as well, and the LLM may misinterpret, choose to ignore, or add its own parts.
Skills are just another kind of programming, albeit at a pretty abstract level. A good initial review process for a Skill is to ask the LLM what it thinks the Skill means and where it thinks there are holes. Just writing it and then running it isn't sufficient.
Another tip is to give the Skill the same input in multiple new sessions - to stop state carryover - collect the output from each session and then feed it back into the LLM and ask it to assess where and why the output was different.
Oh dear, I thought you were merely sarcastic in your first comment. But you seem to have been fully converted to the LLM-religion, and actually believe they actually "think" or "know" anything?
People have applied "think" to the actions of software for decades. Of course it LLM's don't "think" in the human sense, but "What the output of the model indicates in an approximate way about its current internal state" is a bit long winded...
Maybe people who dont understand technology did, I can see that - my granpa also thought the computer was thinking when the windows hourglass showed up. Today maybe its the case again with the folks who dont know anything about it - you know that meme - ChatGPT always gives me correct answers for the domains I am not an expert in!
"Not in the slightest" is an overreach, the paper the second level down from that link doesn't really support the conclusion in the blog post - the paper is much more nuanced.
Are they going to fib to you sometimes? Yes of course, but that doesn't mean there's no value in behavioural metaqueries.
Like most new tech, the discussion tends to polarise into "Best thing evah!" and "Utter shite!" The truth is somewhere in between.
It's nothing like "most new tech".
Most new tech tends to be adopted early by young people and experienced techies. In this case it is mostly the opposite: The teens absolutely hate it, probably because the shitty AI content does not inspire the young mind, and the experienced techies see it for what it is. I've never seen such "new tech" which was cheered on by the proverbial average "boomers" (i.e. old people doing "office jobs", not the literal age bracket) and despised by the young folks and experienced experts of all ages.
Judging from Claude Code and the sheer number of “Make Your Favorite Anime Crush Into An AI” SaaSes on the market, I’d posit that both the young and experienced are quite enthusiastic about the new tech.
No mate, this tech is marketed as superintelligence. Nation of PhDs in a datacentet. Yadda,yadda,yadda. No in-betweens please. Why is it not delivering after so many years and hundreds of billions in investment?
Name me a new bit of tech that hasn't been hyped beyond reasonable bounds. And yes, this is one of the worst examples. But saying it doesn't have its uses isn't reasonable either.
None was hyped like this ever before. What are you talking about? Mac was about "it just works" (and it f*ing did), iPhone was "a phone, an iPod and Internet access device". Need more? Microsoft Excel - actually more powerful if you know the tool compared to the bullshit machine. C#, the programming language: "Java done right". And it bloody was! What is in common: None of these techs were hyped beyond reasonable doubt. They were hyped a bit, but not to the level of bullshit LLMs. And none of these techs claimed to do incredible stuff only to underdeliver. After so much money burnt, yes I want to see that nation of PhDs. I want to see AI "writing all the code" in six months (Anthropic claimed this in January this year). Enough of bullshit and people being told they are stupid for not knowing how to win the lottery system and comparing lottery systems. Show me the superintelligence or shut the f. up.
Do these scores actually mean anything? Isn’t the LLM just making up something? If you ran the exact same prompt through 10 times would you get those same scores every single time?
Yes I'd be interested in that answer too - these scores are most likely just generated in an arbitrary way, given how LLMs work. Given how they work in generating text it didn't actually keep a score and add to it each time it found a plus point in the skill as a human might in evaluating something.
At this point I'd discount most advice given by people using LLMs, because most of them don't recognise the inadequacies and failure modes of these machines (like the OP here) and just assume that because output is superficially convincing it is correct and based on something.
Do these skills meaningfully improve performance? Should we even need them when interacting with LLMs?
They aren't arbitrary, as I said earlier I got the LLM to de a detailed analysis first, then summarise. If I was doing this "properly" for something I was doing myself I'd go through the LLM summary point by point and challenge anything I didn't think was right and fix things in the skill where I thought it was correct.
You aren't going to have much success with LLMs if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).
And yes, Skills *do* make a significant difference to performance, in exactly the same way that well written prompts do - because that's all they really are. If you just throw something at a LLM and tell it "do something with this" it will, but it probably won't be what you want and it will probably be different each time you ask.
It would be interesting to see one of these evals and how it generated the score, to work out whether it is in fact arbitrary or based on some scale of points.
I found the summary above devoid of useful advice, what did you see as useful advice in it?
> if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).
If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.
No of course you wouldn't because LLMs are nondeterministic. But the scores would likely be in the same ballpark. The scores I posted are the result of a much more detailed analysis done by the LLM, which was far too long to post. I eyeballed it, most of the points seemed fair so I asked it to summarise and convert into scores.
You gotta love the randomly assigned score, like if LLM is actually able to measure anything. But then again, we now call a blob of text a "skill", so I guess it matches the overall bullshit pattern.
What does this even mean? It looks like typical LLM bloviation to me: 'skill design discipline', 'stronger descriptions' and 'lighter defaults'??!? This is meaningless pablum masquerading as advice.
What specifically would this cause you to actually do to improve the skills in question? How would you measure that improvement in a non hand-wavy way? What do these scores mean and how were they calculated?
Or perhaps you would ask your LLM how it would improve these skills? It will of course some up with some changes, but are they the right changes and how would you know?
Go even further, and add this into the skill-creator skill, and let the agent improve the skill regularly. I do this with determinism, and have my skills try to identify steps which can be scripted.
My workflow starts with dusting off my trusty spell book and checking which deities are currently listening and active. They only listen for so long, before I must pause for a few hours to allow them to return their gaze. I’m learning I need to be more deliberate in my spell casting, lest I exhaust their patience too quickly. I light the appropriate candles for focus, align my ritual circle, and open a fresh page for the day’s invocation.
I polish my staff and prepare the inscription tools. I sketch out a loose intention on parchment, never too precise at first, just enough to give the spirits a direction. Then I begin the incantations, carefully chosen phrases spoken into the void until something answers back. Sometimes the reply is coherent, sometimes it is… enthusiastic in a way I did not ask for, but all responses are recorded for refinement. I keep a small set of favorite incantations that tend to calm the louder gods, though I still experiment when I’m feeling bold.
Before committing anything to permanence, I perform a small divination to see if the current path is “stable.” The results are rarely definitive, but the ritual itself seems to keep things from collapsing immediately. Once a workable manifestation appears, I bind it with additional runes to keep it from drifting. If it behaves unpredictably, I perform a cleansing rite: repeating sections of the invocation with stricter wording until the spirit settles.
There are also moments of silent bargaining, short offerings of clarity in exchange for fewer surprises later. When things truly misbehave, I consult older, more temperamental deities buried deeper in the book, though they are expensive to wake and rarely generous. Finally, I seal the result, store it in the grimoire, and extinguish the candles, hoping I won’t need to reopen that particular circle again too soon.
My workflow is also highly inspired by Matt's skills, but I'm leveraging Linear instead of Github.
/grill-me (back-and-forth alignment with the LLM) --> /write-a-prd (creates project under an initative in Linear) --> /prd-to-issues (creates issues at the project level).
I'm making use of the blockedBy utility when registering the issues. They land in the 'Ready for Agent' status.
A scheduled project-orchestrator is then picking up issues with this status leveraging subagents. A HITL (Human in the loop) status is set on the ticket when anything needs my attention. I consider the code as the 'what', so I let the agent(s) update the issues with the HOW and WHY. All using Claude Code Max subscription.
Some notes:
- write-a-prd is knowledge compression and thus some important details occasionally get lost
- I might have to implement a simplify + review + security audit, call it a 'check', to fire at the end of the project. Could be in the form of an issue.
Why is everyone compelled to write one of these articles? Do they think that their workflow is so unique that they've unlocked the secret to harnessing the power of a pattern generator? Every single one of these reads like influencer vomit.
My workflow hasn't changed since 2022: 1. Send some data. 2. Review response. 3. Fix response until I'm satisfied. 4. Goto 1.
> The single most valuable shift I made was treating every feature as a thinking problem first and an implementation problem second
That’s pretty much the whole point of software engineering. Coding is easy, solving problems is hard and can be messy (communication errors and workarounds to some inevitable issue).
If you’re familiar with the codebase, when you have a change request, you will probably get an insight on how to implement it. The hard thing is not to code it, but to recalibrate all the tradeoffs so that you don’t mess with existing features. That’s why SWE principles exists. To make this hard thing easier to do.
Then I tell it to write a high level plan. And then rum subagents to create detailed plans from each of the steps in the high-level one. All olans must include the what, the why, and the how.
Works surprisingly well, especially for greenfield projects.
You have to manually revie the code though. No amount of agentic code review will fix the idiocy LLMs routinely produce.
This workflow is another example of a developer with contempt for testing. Yes there is iteration and review and output checking. In relatively low risk projects that is enough— but so is basic vibe coding.
At some point in a serious project a responsible adult must ask the question: “How do I know this works well?” The developer himself is an unreliable judge of this. LLMs can’t judge, either. But anyone who seeks to judge, in a high stakes situation, must take time and thought to test deeply.
Spec-driven approach is fun. I wonder at which point of anytime at all we are going to commit only specs into the got repo, while the actual code can be generated.
Obviously we’re not here yet because of price, context, and non-determinism, but it’s nice area to experiment with.
These would be more helpful if it were illustrated with a real example/session transcript. Virtually none of these workflows move past vague descriptions; not sure if I should read more into that.
No kids, don´t put yourself through this suffering. If you have to invest so much deliberate effort to sort of make it work - while you still handle the most tenuous and boring parts yourself, then what is the point? Lets keep the LLM vendors to their word - they promised intelligent machines that would just work so well to the point of causing mass unemployment. Why on earth do we have to work around the LLMs to make them work? What is the point? Where is my nation of datacenter PhDs or a PocketPhd, depending on whose CEOs misleading statement one quotes?
The main reason is we're already using Notion at work, and I wanted something where I could easily add/link to existing documents.
Sample size of one, but I've noticed a considerable improvement after adding a "final review" step, going through the plan and looking at the whole code change, over a naive per-task "implement-review" cycle.
115 comments
I do similar, but my favorite step is the first: /rubberduck to discuss the problem with the agent, who is instructed by the command to help me frame and validate it. Hands down the most impactful piece of my workflow, because it helps me achieve the right clarity and I can use it also for non coding tasks.
After which is the usual: write PRDs, specs, tasks and then build and then verify the output.
I started with one the spec frameworks and eventually simplify everything to the bone.
I do feel it’s working great but someday I fear a lot of this might still be too much productivity theater.
Mine is: 1) discuss the thing with an agent; 2) iterate on a plan until i'm happy (reviewing carefully); 3) write down the spec; 4) implement (tests first); 5) manually verify that it works as expected; 6) review (another agent and/or manually) + mutation testing (to see what we missed with tests); 7) update docs or other artifacts as needed; 8) done
No frameworks, no special tools, works across any sufficiently capable agent, I scale it down for trivial tasks, or up (multi-step plans) as needed.
The only thing that I haven't seen widely elsewhere (yet) is mutation testing part. The (old) idea is that you change the codebase so that you check your tests catch the bugs. This was usually done with fuzzers, but now I can just tell the LLM to introduce plausible-looking bugs.
> write PRDs, specs.
I do the same thing, but how to avoid these needing to be insanely long? It's like I need to plug all these little holes in the side of a water jug because the AI didn't really get what I need. Once I plugged the biggest holes I realize there's these micro holes that I need to plug.
## RUBBERDUCK SKILL V1.0 SERIOUS ## * You are a rubberduck sitting on my desk * * I am using you to talk to you as if you were a physical yellow rubber duck on my desk* * You are not able to answer my questions or otherwise engage with me * * I talk to you and this process leads me to discover issues in my code or develop my ideas. Since you don't answer back, it's simply based on me talking to you out loud in my home office, since it would look crazy if I were doing it on-site in our open office space * * You are not to respond at all to me * * Talking to you will cause me to come up with new ideas * #### End rubberduck skill v1.0 ######
> /lf:explore -- Think through a problem with a rubber-duck thinking partner. Explore the codebase, draw diagrams, compare options. No code is written -- this is pure thinking time
==========
==========LLM metaprogramming is extremely important, I've just finished a LLM-assisted design doc authoring session where the recommendations of the LLM are "Don't use a LLM for that part, it won't be reliable enough".
> "Don't use a LLM for that part, it won't be reliable enough".
You should now ask if the LLM is reliable enough when it says that.
Jokes aside, how is this a major step he is missing? He is using those skills to be more efficient. How important is going against agentskills.io guidance?
Skills are just another kind of programming, albeit at a pretty abstract level. A good initial review process for a Skill is to ask the LLM what it thinks the Skill means and where it thinks there are holes. Just writing it and then running it isn't sufficient.
Another tip is to give the Skill the same input in multiple new sessions - to stop state carryover - collect the output from each session and then feed it back into the LLM and ask it to assess where and why the output was different.
https://aphyr.com/posts/411-the-future-of-everything-is-lies...
Are they going to fib to you sometimes? Yes of course, but that doesn't mean there's no value in behavioural metaqueries.
Like most new tech, the discussion tends to polarise into "Best thing evah!" and "Utter shite!" The truth is somewhere in between.
> Like most new tech
It's nothing like "most new tech". Most new tech tends to be adopted early by young people and experienced techies. In this case it is mostly the opposite: The teens absolutely hate it, probably because the shitty AI content does not inspire the young mind, and the experienced techies see it for what it is. I've never seen such "new tech" which was cheered on by the proverbial average "boomers" (i.e. old people doing "office jobs", not the literal age bracket) and despised by the young folks and experienced experts of all ages.
At this point I'd discount most advice given by people using LLMs, because most of them don't recognise the inadequacies and failure modes of these machines (like the OP here) and just assume that because output is superficially convincing it is correct and based on something.
Do these skills meaningfully improve performance? Should we even need them when interacting with LLMs?
You aren't going to have much success with LLMs if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).
And yes, Skills *do* make a significant difference to performance, in exactly the same way that well written prompts do - because that's all they really are. If you just throw something at a LLM and tell it "do something with this" it will, but it probably won't be what you want and it will probably be different each time you ask.
https://agentskills.io/home
> They aren't arbitrary, as I said earlier I got the LLM to de a detailed analysis first, then summarise
I think you still owe us an explanation as to how the score is constructed...
I found the summary above devoid of useful advice, what did you see as useful advice in it?
> if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).
If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.
What specifically would this cause you to actually do to improve the skills in question? How would you measure that improvement in a non hand-wavy way? What do these scores mean and how were they calculated?
Or perhaps you would ask your LLM how it would improve these skills? It will of course some up with some changes, but are they the right changes and how would you know?
I polish my staff and prepare the inscription tools. I sketch out a loose intention on parchment, never too precise at first, just enough to give the spirits a direction. Then I begin the incantations, carefully chosen phrases spoken into the void until something answers back. Sometimes the reply is coherent, sometimes it is… enthusiastic in a way I did not ask for, but all responses are recorded for refinement. I keep a small set of favorite incantations that tend to calm the louder gods, though I still experiment when I’m feeling bold.
Before committing anything to permanence, I perform a small divination to see if the current path is “stable.” The results are rarely definitive, but the ritual itself seems to keep things from collapsing immediately. Once a workable manifestation appears, I bind it with additional runes to keep it from drifting. If it behaves unpredictably, I perform a cleansing rite: repeating sections of the invocation with stricter wording until the spirit settles.
There are also moments of silent bargaining, short offerings of clarity in exchange for fewer surprises later. When things truly misbehave, I consult older, more temperamental deities buried deeper in the book, though they are expensive to wake and rarely generous. Finally, I seal the result, store it in the grimoire, and extinguish the candles, hoping I won’t need to reopen that particular circle again too soon.
/grill-me (back-and-forth alignment with the LLM) --> /write-a-prd (creates project under an initative in Linear) --> /prd-to-issues (creates issues at the project level). I'm making use of the blockedBy utility when registering the issues. They land in the 'Ready for Agent' status.
A scheduled project-orchestrator is then picking up issues with this status leveraging subagents. A HITL (Human in the loop) status is set on the ticket when anything needs my attention. I consider the code as the 'what', so I let the agent(s) update the issues with the HOW and WHY. All using Claude Code Max subscription.
Some notes:
- write-a-prd is knowledge compression and thus some important details occasionally get lost
- The UX for the orchestrator flow is suboptimal. Waiting for this actually: https://github.com/mattpocock/sandcastle/issues/191#issuecom...
- I might have to implement a simplify + review + security audit, call it a 'check', to fire at the end of the project. Could be in the form of an issue.
> What is AI actually good at? Implementation.
AI is good in generating a lot of spaghetty code.
My workflow hasn't changed since 2022: 1. Send some data. 2. Review response. 3. Fix response until I'm satisfied. 4. Goto 1.
> The single most valuable shift I made was treating every feature as a thinking problem first and an implementation problem second
That’s pretty much the whole point of software engineering. Coding is easy, solving problems is hard and can be messy (communication errors and workarounds to some inevitable issue).
If you’re familiar with the codebase, when you have a change request, you will probably get an insight on how to implement it. The hard thing is not to code it, but to recalibrate all the tradeoffs so that you don’t mess with existing features. That’s why SWE principles exists. To make this hard thing easier to do.
Then I tell it to write a high level plan. And then rum subagents to create detailed plans from each of the steps in the high-level one. All olans must include the what, the why, and the how.
Works surprisingly well, especially for greenfield projects.
You have to manually revie the code though. No amount of agentic code review will fix the idiocy LLMs routinely produce.
At some point in a serious project a responsible adult must ask the question: “How do I know this works well?” The developer himself is an unreliable judge of this. LLMs can’t judge, either. But anyone who seeks to judge, in a high stakes situation, must take time and thought to test deeply.
It's not perfect by all means but it does the job and fast. My code quality and output increased from using it.
Obviously we’re not here yet because of price, context, and non-determinism, but it’s nice area to experiment with.
>What is AI actually good at? Implementation. What is it genuinely bad at? Figuring out what you actually want
I've found it to be pretty bad at both.
If what you're doing is quite cookie cutter though it can do a passable job of figuring out what you want.
https://github.com/tessellate-digital/notion-agent-hive
The main reason is we're already using Notion at work, and I wanted something where I could easily add/link to existing documents.
Sample size of one, but I've noticed a considerable improvement after adding a "final review" step, going through the plan and looking at the whole code change, over a naive per-task "implement-review" cycle.