I did something similar to a local company here in Spain. Not medical, but a small insurance company. Believe it or not, yes, they vibecoded their CRM.
I sent them an email and they threatened to sue me. I was a bit in shock from such dumb response, but I guess some people only learn the hard way, so I filed a report to the AEPD (Data protection agency in Spain) for starters, known to be brutal.
I've also sent them a burofax demanding the removal of my data on their systems just last friday.
A similar thing happened to me back in the day when Wi-Fi was still new.
I joined an open network and it turned out to be a law firm. All their computers were on a Samba network with full C: drives shared.
I wrote README.txt files on their drives telling them about the issue, but after some time it was still the same.
Then I went directly to the place to talk to them and also with the idea I could land my first job fixing that mess. But... They got incredibly angry with me, since they claimed they had some very good and expensive contractors taking care of their computers and network, and that I had basically broken in.
At one point I worked as a customer support agent outsourced to Apple via the company. Apple forced us to us some very outdated browser UIs, basically for filling in forms, across maybe 4-5 different services in some cases. The machines we were given by this outsourcing company of course where Apple computers, fairly locked down.
But one thing they hadn't locked doll wn, was installing extensions in Safari, and given I had some development chops from coding a bunch in my freetime, I saw the opportunity to write a tiny extension that saved me a ton of time by merely copy-pasting stuff into the right forms and so on. Basically making the whole thing more efficient for me.
Everything was great, until the person next to me saw I had something different. Cautiously eager, I let them try the extension too, they loved it, and without thinking about it, spread it to other people in our team. Eventually, the manager and the IT team picked up what was going on, said they'investigate if I could maybe start doing those kind of things full-time instead of being a support agent, and just focus on tooling.
Fast forward two weeks, I get called into a meeting, apparently someone in the company had been "stealing" CC numbers from the customers on the calls, and since they don't think they've found the right person who did it (or something like that), the person who was known for "doing stuff to the computers" was the next possible suspect, and they fired me right there.
Eventually this firing let me find my first actual programming job, so I'm not too mad about it, but it really shows how out of touch lots of companies and people are when it comes to how computers actually work.
Nice. I wish more countries had something like that. Many of these organizations are lethargic and have to be forced into action by civilian efforts or the press.
> [burofax is] a service that allows you to send a document with certified proof of delivery and confirmation of the date of receipt, and this confirmation has legal validity
I'm also curious how much effort it would be to setup some OWASP tools with an agent and crawl for company tools. I'm sure I'm not the first one to think of this, but for local businesses it would give a solid rep, I suppose.
I have a feeling that next year's theme will be security. People have turned off their brain when it comes to tech.
People building these apps often have no idea about various data privacy rules.
I am part of a forum with many small business owners. One particular owner has been gung-ho about how he built his entire business app using vibe coding. And my first reaction was - All the power to him. It’s his business and he is free to do so.
But then came the question of data privacy rules and he had no clue. This was concerning because the impact went beyond his business. His response when the oversight was pointed out to him was that being ignorant of the law was enough to save him. Still he went to one of the vibe coding Reddit subs to get help. Then came back fuming because devs on Reddit asked him to hire real developers. He believes that these developers are delusional and a dying breed and AI is so ahead that developers are going to be dead in a years time.
Meanwhile on Linkedin… Every sales bozo with zero technical understanding is screaming top of their virtual lungs that evrything must be done with AI and it is solution to every layoff, economic problem, everything.
It is just a matter of time when something really really bad happens.
I think vibe-coding is cool, but it runs into limits pretty fast (at least right now).
It kinda falls apart once you get past a few thousand lines of code... and real systems aren't just big, they're actually messy...shit loads of components, services, edge cases, things breaking in weird ways. Getting all of that to work together reliably is a different game altogether.
And you still need solid software engineering fundamentals. Without understanding architecture, debugging, tradeoffs, and failure modes, it's hard to guide or even evaluate what's being generated.
Vibe-coding feels great for prototypes, hobby projects, or just messing around, or even some internal tools in a handful of cases. But for actual production systems, you still need real engineering behind it.
As of now, I'm 100% hesitant to pay for, or put my data on systems that are vibe-coded without the knowledge of what's been built and how it's been built.
I saw something very similar a few months ago. It was a web app vibe coded by a surgeon. It worked, but they did not have an index .html file in the root web directory and they would routinely zip up all of the source code which contained all the database connection strings, API credentials, AWS credentials, etc.) and place the backup in the root web directory. They would also dump the database to that folder (for backup). So web browsers that went to https://example.com/ could see and download all the backups.
The quick fix was a simple, empty index.html file (or setting the -Indexes option in the apache config). The surgeon had no idea what this meant or why it was important. And the AI bots didn't either.
The odd part of this to me was that the AI had made good choices (strong password hashes, reasonable DB schema, etc.) and the app itself worked well. Honestly, it was impressive. But at the same time, they made some very basic deployment/security mistakes that were trivial. They just needed a bit of guidance from an experienced devops security guy to make it Internet worthy, but no one bothered to do that.
Edit: I do not recommend backing up web apps on the web server itself. That's another basic mistake. But they (or the AI) decided to do that and no one with experience was consulted.
All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one command away from anyone who looked
This is the top!
This is a typical example of someone using Coding Agents without being a developer: AI that isn't used knowingly can be a huge risk if you don't know what you're doing.
AI used for professional purposes (not experiments) should NOT be used haphazardly.
And this also opens up a serious liability issue: the developer has the perception of being exempt from responsibility and this also leads to enormous risks for the business.
Software engineering is looking more and more like it needs a professional body in each country, and accreditation and standards. Ie it needs to grow up and become like every other strand of engineering.
Gone should be the days of “I taught myself so now I can [design software in a professional setting / design a bridge in a professional setting].” I’m not advocating gatekeeping - if you want to build a small bridge at the end of your garden for personal use, go for it. If you want to build a bridge in your local town over a river, you’re gonna need professional accreditation. Same should be true for software engineering now.
I know, through personal acquaintance, of at least one boutique accounting firm that is currently vibe-building their own CRM with Lovable. They have no technical staff. I can't begin to comprehend the disasters that are in store.
Every other field that's figured out high stakes failure models eventually landed on the same solution - make sure two people that understand the details are looking at it - pilots have copilots surgeons with checklists and nuclear plants have independent verification. Software was always the exception, cause when it broke it mostly just broke for you, vibe coding is not going to change the equation, it barely removes one check that existed before is that the people who wrote the code understood what was going on, but now that's gone too
What would a responsible on-boarding flow for all of these tools look like?
> Welcome to VibeToolX.
> By pressing Confirm you accept all responsibility for user data stewardship as regulated in every country where your users reside.
Would that be scary enough to nudge some risk analysis on the user's part? I am sure that would drop adoption by a lot, so I don't see it happening voluntarily.
What if that AI was trained to hide implementation as much as possible? E.g. by making the client as thin as possible, using OAuth for authentication, following some robust template, would it be better?
Avoiding such low-hanging-fruit disasters is doable and major players have incentives to fix it.
This however wouldn't change the goal - replace all software engineers with DIY nephews with LLM in their hands.
Even worse - people start thinking it's infeasible to learn hard things, because it prevents you from moving fast, from one domain to another.
My area of expertise is cryptography-adjacent. Non-trivial fraction of students think that cryptography and security is a dead-end skill as all companies in this area are being replaced with AI. I asked them to implement web-bot-auth as simple as possible, because I know how AI can read specification and follow it.
Vibe coding is fun, but I can't trust it to make any serious decisions. Like, it knows what's the best way to do a thing, but when encounters challenges, it started to make all kinds of excuses to cut corners, just like humans. "but honestly, it's cluster internal traffic so unencrypted traffic is fine". "Given the urgency and tight timeline, your best option is bypassing the pipeline and deploying it manually". "Per my research, XXX also did this so you are fine".
If I don't have disciplines or principles, or if I am just technically incompetent, its suggestions would sound so reasonable.
I think the issue here is less about AI misbehaving and more about people doing things they should not be doing without thinking too hard about the consequences.
There are going to be a lot of accidents like this because it's just really easy to do. And some people are inevitably going to do silly things.
But it's not that different from people doing stupid things with Visual Basic back in the day. Or responding to friendly worded emails with the subject "I love you". Putting CDs/USB drives in work PCs with viruses, worms, etc.
That's what people do when you give the useful tools with sharp edges.
Is there anybody making some framework where you declare the security intentions as code (for each CRUD action) and which agents can correctly do and unit test? I have seen a Lovable competitor's system prompt have 24 lines of "please consider security when generating select statements, please consider security when generating update statements..." since it expects to dump queries here and there.
It's reminiscent of the 90s, where every middle manager had dragged and dropped some boxes on some forms, and could get a salesman to sell it, without a care in the world for what was going on behind the scenes.
Until something crashed and recovery was needed, of course.
Kinda crazy but hopefully the future holds a Clippy-esque thing for people who don’t know to set up CI, checkpoints, reviews, environments, etc. that just takes care of all that.
It sorta should do this anyway given that the user intent probably wasn’t to dump everyone’s data into Firebase or whatever.
I personally would like this as well since it gets tiring specifying all the guardrails and double-checking myself. Using this stuff feels too much like developing a skill I shouldn’t need while not focusing on real user problems.
I believe there are various dimensions to vibe coding. If you work with an existing codebase, it is a tool to increase productivity. If you have domain specific knowledge, in this case - patient management system, you can build better systems.
Otherwise, you endup simulating the production. Lot of the non technical folks building products with AI Vibe coding are basically building Product Simulations. It looks like a product, functions like a product but behind the scene, you can poke holes.
The takeaway is to vet new companies one is dealing with - even just calling them up and asking if they've AI generated any system which deals with customer/patient data.
This is going to get more common (state sponsored hackers are going to have a field day)
I interviewed some years ago for an AI related startup.
After looking at the live product, first thing I see is their prod dB credentials and openAI api key publicly send in some requests...
Bad actors will be having a lot of fun these days
I have my doubts on the story. I consulted on a medtech project in the recent past in similar space, and at various points different individuals vibe-coded[0] not one but three distinct, independent prototypes of a system like the article describes, and neither of them was anywhere near that bad. On the frontend, you'd have to work pretty hard to force SOTA LLMs to give you what is being reported here. Backend-side, there's plenty of proper turn-key systems to get you started, including OSS servers you can just run locally, and even a year ago, SOTA LLMs knew about them and could find them (and would suggest some of them).
I might be biased by my experience, because we actually cared about GDPR and AI act and proper medical data processing, and I've spent my fair share of time investigating the options that exist. Still, I'm struggling to imagine how one could possibly screw it up anywhere near as what the article described. Like, I can't think of a way to do it, to the point I might need to ask an LLM to explain it to me.
--
[0] - Not as a means of developing an actual product, but solely to see if we can, plus it was easier to discuss product ideas while having some prototypes to click around.
To me it just sounds like eventually someone will figure out how to make vibecoding more reasonably secure (with prompts to have apps be looked at for security practices?)
unless cybersecurity is such a dynamic practice that we can't create automated processes that are secured
Essentially a question of what can be done to make vibecoding "secure enough"
The only thing what helps is deleting the database. Every day. Until the thing goes down because the 'developer' thinks he has a bug that he can't find.
"That sparked an idea: why use an industry-proven solution when you could just build your own patient management system?" lol
I wonder if it is an actual dunning kruger effect at play here (confidence issue) or the lack of critical thinking. The first one could be excused somewhat, but I am afraid it is more of the second one
211 comments
I sent them an email and they threatened to sue me. I was a bit in shock from such dumb response, but I guess some people only learn the hard way, so I filed a report to the AEPD (Data protection agency in Spain) for starters, known to be brutal.
I've also sent them a burofax demanding the removal of my data on their systems just last friday.
I joined an open network and it turned out to be a law firm. All their computers were on a Samba network with full C: drives shared. I wrote README.txt files on their drives telling them about the issue, but after some time it was still the same.
Then I went directly to the place to talk to them and also with the idea I could land my first job fixing that mess. But... They got incredibly angry with me, since they claimed they had some very good and expensive contractors taking care of their computers and network, and that I had basically broken in.
I left the place quickly...
But one thing they hadn't locked doll wn, was installing extensions in Safari, and given I had some development chops from coding a bunch in my freetime, I saw the opportunity to write a tiny extension that saved me a ton of time by merely copy-pasting stuff into the right forms and so on. Basically making the whole thing more efficient for me.
Everything was great, until the person next to me saw I had something different. Cautiously eager, I let them try the extension too, they loved it, and without thinking about it, spread it to other people in our team. Eventually, the manager and the IT team picked up what was going on, said they'investigate if I could maybe start doing those kind of things full-time instead of being a support agent, and just focus on tooling.
Fast forward two weeks, I get called into a meeting, apparently someone in the company had been "stealing" CC numbers from the customers on the calls, and since they don't think they've found the right person who did it (or something like that), the person who was known for "doing stuff to the computers" was the next possible suspect, and they fired me right there.
Eventually this firing let me find my first actual programming job, so I'm not too mad about it, but it really shows how out of touch lots of companies and people are when it comes to how computers actually work.
> AEPD […] known to be brutal.
Nice. I wish more countries had something like that. Many of these organizations are lethargic and have to be forced into action by civilian efforts or the press.
> [burofax is] a service that allows you to send a document with certified proof of delivery and confirmation of the date of receipt, and this confirmation has legal validity
I have a feeling that next year's theme will be security. People have turned off their brain when it comes to tech.
I am part of a forum with many small business owners. One particular owner has been gung-ho about how he built his entire business app using vibe coding. And my first reaction was - All the power to him. It’s his business and he is free to do so.
But then came the question of data privacy rules and he had no clue. This was concerning because the impact went beyond his business. His response when the oversight was pointed out to him was that being ignorant of the law was enough to save him. Still he went to one of the vibe coding Reddit subs to get help. Then came back fuming because devs on Reddit asked him to hire real developers. He believes that these developers are delusional and a dying breed and AI is so ahead that developers are going to be dead in a years time.
I think that having paper documentation will be safer very soon :)
It is just a matter of time when something really really bad happens.
It kinda falls apart once you get past a few thousand lines of code... and real systems aren't just big, they're actually messy...shit loads of components, services, edge cases, things breaking in weird ways. Getting all of that to work together reliably is a different game altogether.
And you still need solid software engineering fundamentals. Without understanding architecture, debugging, tradeoffs, and failure modes, it's hard to guide or even evaluate what's being generated.
Vibe-coding feels great for prototypes, hobby projects, or just messing around, or even some internal tools in a handful of cases. But for actual production systems, you still need real engineering behind it.
As of now, I'm 100% hesitant to pay for, or put my data on systems that are vibe-coded without the knowledge of what's been built and how it's been built.
The quick fix was a simple, empty index.html file (or setting the -Indexes option in the apache config). The surgeon had no idea what this meant or why it was important. And the AI bots didn't either.
The odd part of this to me was that the AI had made good choices (strong password hashes, reasonable DB schema, etc.) and the app itself worked well. Honestly, it was impressive. But at the same time, they made some very basic deployment/security mistakes that were trivial. They just needed a bit of guidance from an experienced devops security guy to make it Internet worthy, but no one bothered to do that.
Edit: I do not recommend backing up web apps on the web server itself. That's another basic mistake. But they (or the AI) decided to do that and no one with experience was consulted.
>
All "access control" logic lived in the JavaScript on the client side, meaning the data was literally one command away from anyone who lookedThis is the top!
This is a typical example of someone using Coding Agents without being a developer: AI that isn't used knowingly can be a huge risk if you don't know what you're doing.
AI used for professional purposes (not experiments) should NOT be used haphazardly.
And this also opens up a serious liability issue: the developer has the perception of being exempt from responsibility and this also leads to enormous risks for the business.
Gone should be the days of “I taught myself so now I can [design software in a professional setting / design a bridge in a professional setting].” I’m not advocating gatekeeping - if you want to build a small bridge at the end of your garden for personal use, go for it. If you want to build a bridge in your local town over a river, you’re gonna need professional accreditation. Same should be true for software engineering now.
> Welcome to VibeToolX.
> By pressing Confirm you accept all responsibility for user data stewardship as regulated in every country where your users reside.
Would that be scary enough to nudge some risk analysis on the user's part? I am sure that would drop adoption by a lot, so I don't see it happening voluntarily.
Avoiding such low-hanging-fruit disasters is doable and major players have incentives to fix it.
This however wouldn't change the goal - replace all software engineers with DIY nephews with LLM in their hands.
Even worse - people start thinking it's infeasible to learn hard things, because it prevents you from moving fast, from one domain to another.
My area of expertise is cryptography-adjacent. Non-trivial fraction of students think that cryptography and security is a dead-end skill as all companies in this area are being replaced with AI. I asked them to implement web-bot-auth as simple as possible, because I know how AI can read specification and follow it.
If I don't have disciplines or principles, or if I am just technically incompetent, its suggestions would sound so reasonable.
There are going to be a lot of accidents like this because it's just really easy to do. And some people are inevitably going to do silly things.
But it's not that different from people doing stupid things with Visual Basic back in the day. Or responding to friendly worded emails with the subject "I love you". Putting CDs/USB drives in work PCs with viruses, worms, etc.
That's what people do when you give the useful tools with sharp edges.
https://archive.ph/GsLvt
https://web.archive.org/web/20260331184500/https://www.tobru...
It's reminiscent of the 90s, where every middle manager had dragged and dropped some boxes on some forms, and could get a salesman to sell it, without a care in the world for what was going on behind the scenes.
Until something crashed and recovery was needed, of course.
The piper always needs to be paid.
It sorta should do this anyway given that the user intent probably wasn’t to dump everyone’s data into Firebase or whatever.
I personally would like this as well since it gets tiring specifying all the guardrails and double-checking myself. Using this stuff feels too much like developing a skill I shouldn’t need while not focusing on real user problems.
Otherwise, you endup simulating the production. Lot of the non technical folks building products with AI Vibe coding are basically building Product Simulations. It looks like a product, functions like a product but behind the scene, you can poke holes.
Does the company which willingly sells the polymorphic virus editor bear any responsibility, or should the unaware vibe coder be incumbent ?
This is going to get more common (state sponsored hackers are going to have a field day)
(In not a lawyer and so I might be mistaken about this; Especially the level of intentionality might be a factor)
Interesting how unquestioning the responses are that this isn’t engagement bait
I might be biased by my experience, because we actually cared about GDPR and AI act and proper medical data processing, and I've spent my fair share of time investigating the options that exist. Still, I'm struggling to imagine how one could possibly screw it up anywhere near as what the article described. Like, I can't think of a way to do it, to the point I might need to ask an LLM to explain it to me.
--
[0] - Not as a means of developing an actual product, but solely to see if we can, plus it was easier to discuss product ideas while having some prototypes to click around.
unless cybersecurity is such a dynamic practice that we can't create automated processes that are secured
Essentially a question of what can be done to make vibecoding "secure enough"
https://news.ycombinator.com/item?id=47707681
I wonder if it is an actual dunning kruger effect at play here (confidence issue) or the lack of critical thinking. The first one could be excused somewhat, but I am afraid it is more of the second one