I don't have any benchmarks avalable right now, and honestly I found pretty hard to make them considering that the workflow I have set up is not fully automated, but there is a lot of human intervention in the pre-coding phases.
I feel the problem of token wasting a lot, and actually that was the first reason I had to introduce a hierarchy for instructions, and the artfact indexes: avoid wasting. Then I realized that this approaches helped to keep a lean context that can help the AI agent to deliver better results.
Consider that in the initial phase the token consumption is very limited: is in the implementation phase that the tokens are consumed fast and that the project can proceed with minimal human intevenction. You can try just the fist requirement collection phase to try out the approach, the implementation phase is something pretty boring and not innovative.
I am playing around with building my own similar and am faced with the question you pose.
How can you tell if your prompt process works? I feel like the outputs from SDLC process are so much more high level than could be done with evals, but I am no eval expert.
For sure the proposed approach is more token-consuming than just ask high level the final outcome of the project and make an AI agent to decide everything and deliver the code. This can be acceptable for small personal projects, but if you want to deliver production ready code, you need to be able to control all the intermediate decisions, or at least you want to save and store them. They are needed because otherwise any high level change that you will require will not be able to make focused and coherent enough code changes, with previous forgotten decision that are modified and the code change that will produce lots of side-effects.
How does your framework compare to spec-driven development e.g. https://github.com/github/spec-kit? In my experience, spec-kit produces a lot of markdown files and little source code.
Thoughts on publishing an example output perhaps in another repo? Perhaps just the first two phases? Would be interesting to see what the output looks like practically speaking (before committing to using it for a project).
We have built something similar for our SDLC, but it is based on Claude Code slash commands:
- /tasks:capture — Quick capture idea/bug/task to tasks/ideas/
- /tasks:groom — Expand with detailed requirements → tasks/backlog/
- /tasks:plan — Create implementation plan → tasks/planned/
- /tasks:implement — Execute plan, run tests → tasks/done/
- /tasks:review-plan — Format plan for team review (optionally Slack)
- /tasks:send — Send to autonomous dev pipeline via GitHub issue
- /tasks:fast-track — Capture → groom → plan → review in one pass
- /tasks:status — Kanban-style overview of all tasks
Workflow: capture → groom → plan → implement → done (with optional review-plan before implement, or send for autonomous execution).
12 comments
I feel the problem of token wasting a lot, and actually that was the first reason I had to introduce a hierarchy for instructions, and the artfact indexes: avoid wasting. Then I realized that this approaches helped to keep a lean context that can help the AI agent to deliver better results.
Consider that in the initial phase the token consumption is very limited: is in the implementation phase that the tokens are consumed fast and that the project can proceed with minimal human intevenction. You can try just the fist requirement collection phase to try out the approach, the implementation phase is something pretty boring and not innovative.
How can you tell if your prompt process works? I feel like the outputs from SDLC process are so much more high level than could be done with evals, but I am no eval expert.
How would you benchmark this?