I have detailed knowledge of dynamodb. This code does not properly mimic the service exceptions, input validations, eventual consistency, or edge cases. I would not feel comfortable developing against ministack or running tests against it. A lot of these services also have free tiers, so the use cases are quite minimal.
Hello! The target use case is integration testing, verifying that your application calls the right operations with the right data, not production fidelity. For that, MiniStack works well and is free. For anything requiring exact DynamoDB behavior (capacity throttling, consistency window, stream processing), you're right that it's not a substitute. Contributions to improve error fidelity are very welcome (we have pending ProvisionedThroughputExceededException, TransactionConflictException, ItemCollectionSizeLimitExceededException)
I ran into the same thing with Kinesis. The Kinesis API has strict limits on batch size, message size, and rate limiting. Any client writing to Kinesis must deal with all of these constraints and handle errors properly.
I briefly looked at localstack to see how they implemented the Kenesis api and other than a KINESIS_ERROR_PROBABILITY option to simulate rate limiting, they did not implement any of the constraints.
Thanks for pointing this, I am adding a few extra validations in v 18 (to be released in a few hours) feel free to raise an issue, this is an open source project but I do review each PR and I keep the code as clean as I can... And yes we do use AI but we know what we are doing :)
I really don't see how any clone is going to manage to do what localstack couldn't - maintain compatibility with tons of AWS services while not getting paid for it. If this were viable, why would it not have worked before?
The only things I can think of are that perhaps LocalStack was just a mess of a codebase that couldn't maintain velocity or attract contributors, or it just failed to steward new contributors or some such thing.
Hello! We won't have the broad coverage that Localstack has... we're not aiming to be the "next Localstack"... just want to keep the core services that were available for free in the LS community up to date. If you’re looking for larger services like MWAA, sorry, but we won't be supporting them.... Most core AWS services don't receive many updates anyway (their APIs don’t change drastically or frequently)
If you’re looking for a Localstack alternative, it’s often because you were using the community tier... In that case, Ministack can serve as a drop-in replacement... otherwise, the price reduction in LS might already address your needs
Personally, I would get value out of really solid compatibility of the base features of a few core services (sqs, s3, kms, and maybe dynamo are the main ones that come to mind) with a light weight gui interface and persistence.
If I’m getting into esoteric features or some “big” features that don’t make sense locally, then I just spin up a real dev account of aws, so I know I’m getting the real experience.
Interesting — those are the ones that depending on the case, approximating them with local redis/postgres/vms without the AWS specific APIs on top is often good enough — because my app is just talking to them over native protocols anyway.
Or I am doing something so specific, that a local emulation of the aws api isn’t ever going to be good enough, so there’s not a lot of point in trying. For example, writing code that handles automatically spinning up RDS instances from an RDS snapshot — a local emulation of that process is going to be so far off from what would actually happen no matter what they do.
I'd put ALL the ones listed above: SQS, S3, KMS, DynamoDB, EC2, RDS, Redis in the required "core" services column, and also throw in IAM, SNS, and SecretsManager as well. Those are all table stakes imho.
I'm using all of the above in LocalStack today. Frankly, I don't believe this is as "impossible" a task as several in this thread are insinuating. It's the type of rote work you can delegate to AI to build these days, as observed in this OP.
Building a test suite to validate the state-transient mocks in question against the real deal is not difficult. Only annoyingly expensive (in time and money) if run often, which is exactly the problem they're solving.
This is exactly the approach fakecloud takes — depth over breadth. 13 services right now, but each one is conformance tested against the AWS Smithy API models (34,856 test variants). The bet is that fewer services with correct behavior is more useful for integration testing than broad coverage that surprises you in production.
If this were viable, why would it not have worked before?
Because something wasn't in place to make it work. There are millions of examples where the second mover won, often because the first mover was too early, or the tech wasn't there to make it work, or the market didn't understand the value, etc.
In this case I imagine there are three massive benefits that you'd have over being the first mover:
- AWS is more mature and therefore moving slower, so it's easier to keep up.
- AI is useful for building 80% working code, so there's a lot less to do to keep up.
- There's a lot of devs looking for ways to move off Localstack due to the price change, which gives you (potentially) a pool of willing volunteers to contribute to an OSS alternative.
You can also learn from Localstack's open source version about what's needed, what works, what doesn't work, etc.
"LocalStack was just a mess of a codebase" - very true.
I do think there's potential to semi-automatically create a compatible suite of services, but it'll require some very talented use of LLMs and some novel testing approaches. Not something I want to sign up for.
I evaluated Floci, but that has the typical issues you'd expected with freshly minted vibe code.
What are the typical problems you mentioned with Floci? I just went into their repo, and the community is very active; I see a lot of contributions. I'm not sure what you mean by typical problems.
I wouldn’t hold my breath. They do have a dynamo db fake, but they don’t even release the code for it! Perhaps they’re concerned about making it really easy to clone their stuff.
But we can still do that ourselves, dynamically interrogating the real thing and comparing it to the fake.
I suppose (among many other things) LLMs are changing this. We no longer need that many contributors when we can use AWS docs, intercept AWS API calls and give it to AI agent to mimic. Of course, contributors are still needed for maintaining tests and validations.
That's true if you assume making these things is manual work. But of course with well documented API service and AI coding tools, making a functional local equivalent of any given service is not all that hard.
I am currently implementing S3, with correct error handling for edge cases. Its been about a month and I have almost the whole API surface done, with some gaps (you actually need a broader surface eg some IAM, KMS encrytion needs a persistence layer so only done sse-c so far). Close to the point where you could drop it in as a local emulation tool, its a sub 9MB binary (Rust).
Literally as I write this the AI code review said "suspended-bucket current-version selection is still wrong when the numbered version and the newer null version land in the same millisecond." - thats the level of detail you have to deal with.
Actually no, because they actually have to worry about their millions of customer deployments not randomly breaking every other week. So, they are generally very good about not breaking compatibility unless they have a really good reason to. And when they do, they typically give people years of time to deal with any backwards compatibility breaking changes. Because otherwise they are dealing with lots of support overhead and angry customers. That and the multiple independently developed s3 compatible clients or alternatives make this probably the easiest thing to simulate in the AWS universe as it is probably the most widely used thing in AWS.
Not really, the main thing they could change is the order in which errors are checked when there are multiple errors, but even a lot of that is not likely to change, eg html based stuff comes first. There are actual reasons for a lot of the subtleties, plus they have to have a stable contract for all the success cases as people rely on it. And its not hard to update, I have 800 or so tests against AWS that run in a few minutes.
The real issue with LocalStack was always drift. Tests pass locally, then something breaks in staging because the S3 response format is slightly different or DynamoDB throttling doesn't match. After getting burned enough times we just switched to short-lived real AWS environments for integration tests. More expensive, but way fewer surprises in prod.
i would imagine localstack is a sniff-test - aka, if it doesn't pass localstack, you don't run the more expensive real AWS integration test. In other words, like food, you don't eat the cost unless it passes the sniff test.
Ugh, I hadn't heard the news about the LocalStack licensing changes. I had some great results building AWS services for local dev as well as CI/CD and testing in GH actions with LocalStack in previous jobs.
I secretly always hoped Amazon would buy out LocalStack and make it the official free local development environment for AWS work, but I guess it probably would reduce revenues spent on AWS based dev and test environments. The compatibility with the AWS CLI was mostly excellent in my experiences.
Unclear what LocalStack's end game is tbh. My company has an active enterprise license atm so their recent changes won't affect me in the short term at least. As of writing I'm still a happy user of LocalStack. Disappointed with their overall direction, but I hold no ill will and I'm sure they had their reasons. I wish them luck.
Hopefully this change was not just an short term attempt to lock in current enterprise customers by shoring up existing income streams. That'll only work in the VERY short term.
It's not difficult to forsee the inevitable customer drain to free competitors or private one-shots easily produced by genai from publicly available AWS SDK code. Maybe they're already feeling that pressure and that's all this change is. I hope not.
AFAICT, they have no appreciable moat to retain customers long term. For example, I have absolutely no interest in their "Pods" or even their console UI, so thosr aren't keeping me around forever. For their sakes I hope they're still shopping themselves around and didn't take some VC poison pill with preconditions for killing the community edition. Really It's anyone's guess though.
> The compatibility with the AWS CLI was mostly excellent in my experiences.
Interesting, I've had the opposite experience. Every single AWS service I've ever tried to build tests around with LocalStack has run into compatibility issues. Usually something works in LocalStack but fails when it hits the real endpoint.
I guess the CLI itself has mostly worked, its more the LocalStack service not behaving the way the real service is documented/works.
Got any concrete examples? I've been happily using LocalStack for roughly a decade now and haven't run into a single compatibility issue, aside from the obvious missing of net new services for the first N months after AWS product launches. Things like AppConfig, etc, but those gaps got filled in time. They clearly prioritized the 95pct/most used features of each service first though. There's a long tail in some AWS services, as one might expect. I've never used any of the more esoteric AWS feature sets of any of their services. Those are the things that tend to end up deprecated. So requiring those long tail featur sets may be the simple answer to having very different experiences.
Any project with Claude’s signature misaligned ASCII diagram just screams DON’T USE to me. If you can’t even bother to read your damn generated README, how do I know you have QC’ed anything else? At the very least supported features claimed in README / on website could be totally fake, which is extremely common among all the vibed Show HNs.
I love how a comment pointing out glaring lack of human review of this project (with clear evidence) is flagged to death. With zero counter-argument no less. Can’t wait for the bright future of vibed crap with zero human quality control getting hacked every day.
I think it's still very early for this to be adapted in the SDLC, but looks very promising. I am frustrated with Localstack at this point, and this may just do the job.
67 comments
But, for anybody else who needs a quick fix - you can pin to the
community-archivetag and it should unblock you quickly:https://hub.docker.com/layers/localstack/localstack/communit...
Same reason I wince when people say things like "we just use SQLite in tests, but our real db is mysql/psql/etc"
The test results have no merit if they are testing against a glorified mock.
I briefly looked at localstack to see how they implemented the Kenesis api and other than a
KINESIS_ERROR_PROBABILITYoption to simulate rate limiting, they did not implement any of the constraints.The only things I can think of are that perhaps LocalStack was just a mess of a codebase that couldn't maintain velocity or attract contributors, or it just failed to steward new contributors or some such thing.
> We won't have the broad coverage that Localstack has... we're not aiming to be the "next Localstack"
you should tell that to the LLM that writes your website:
> MiniStack is your free, MIT-licensed drop-in replacement.
is it a drop-in replacement, or not?
If I’m getting into esoteric features or some “big” features that don’t make sense locally, then I just spin up a real dev account of aws, so I know I’m getting the real experience.
> getting into esoteric features
The problem is that everybody needs different "core" features
> > compatibility of the base features of a few core services (sqs, s3, kms, and maybe dynamo are the main ones that come to mind)
For instance, I don't care about any of those features at all. But I would care a lot about EC2, RDS, and ElastiCache Redis
Or I am doing something so specific, that a local emulation of the aws api isn’t ever going to be good enough, so there’s not a lot of point in trying. For example, writing code that handles automatically spinning up RDS instances from an RDS snapshot — a local emulation of that process is going to be so far off from what would actually happen no matter what they do.
I'm using all of the above in LocalStack today. Frankly, I don't believe this is as "impossible" a task as several in this thread are insinuating. It's the type of rote work you can delegate to AI to build these days, as observed in this OP.
Building a test suite to validate the state-transient mocks in question against the real deal is not difficult. Only annoyingly expensive (in time and money) if run often, which is exactly the problem they're solving.
Because something wasn't in place to make it work. There are millions of examples where the second mover won, often because the first mover was too early, or the tech wasn't there to make it work, or the market didn't understand the value, etc.
In this case I imagine there are three massive benefits that you'd have over being the first mover:
- AWS is more mature and therefore moving slower, so it's easier to keep up.
- AI is useful for building 80% working code, so there's a lot less to do to keep up.
- There's a lot of devs looking for ways to move off Localstack due to the price change, which gives you (potentially) a pool of willing volunteers to contribute to an OSS alternative.
You can also learn from Localstack's open source version about what's needed, what works, what doesn't work, etc.
I do think there's potential to semi-automatically create a compatible suite of services, but it'll require some very talented use of LLMs and some novel testing approaches. Not something I want to sign up for.
I evaluated Floci, but that has the typical issues you'd expected with freshly minted vibe code.
I mean, if we can use "virtual-AWS", it would dramatically lower entry-barrier for devs/companies who are scared of "tales of huge aws bills" and such
But we can still do that ourselves, dynamically interrogating the real thing and comparing it to the fake.
Literally as I write this the AI code review said "suspended-bucket current-version selection is still wrong when the numbered version and the newer null version land in the same millisecond." - thats the level of detail you have to deal with.
I secretly always hoped Amazon would buy out LocalStack and make it the official free local development environment for AWS work, but I guess it probably would reduce revenues spent on AWS based dev and test environments. The compatibility with the AWS CLI was mostly excellent in my experiences.
Hopefully this change was not just an short term attempt to lock in current enterprise customers by shoring up existing income streams. That'll only work in the VERY short term.
It's not difficult to forsee the inevitable customer drain to free competitors or private one-shots easily produced by genai from publicly available AWS SDK code. Maybe they're already feeling that pressure and that's all this change is. I hope not.
AFAICT, they have no appreciable moat to retain customers long term. For example, I have absolutely no interest in their "Pods" or even their console UI, so thosr aren't keeping me around forever. For their sakes I hope they're still shopping themselves around and didn't take some VC poison pill with preconditions for killing the community edition. Really It's anyone's guess though.
> The compatibility with the AWS CLI was mostly excellent in my experiences.
Interesting, I've had the opposite experience. Every single AWS service I've ever tried to build tests around with LocalStack has run into compatibility issues. Usually something works in LocalStack but fails when it hits the real endpoint.
I guess the CLI itself has mostly worked, its more the LocalStack service not behaving the way the real service is documented/works.
https://github.com/Nahuel990/ministack/blob/a1b1d20a27d2238d...
Something more realistic would be:
+-----+
| foo |
+-----+
> Copyright (c) 2024 MiniStack Contributors
Already lying or totally unreviewed AI slop ?
If you go on Reddit, there are at least a dozen AI "vibe coded" shitty local stack alternatives being posted daily. Whats special about this??