This agent stuff is really making me lose respect for our industry
All the years of discussing programming/security best practices
Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter, determinism is going out the window, and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
We didn't collectively decided, we've got this forced down our throats to apply a novel tool to any imaginable situation because the execs got antsy about being left behind.
A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.
That's it.
The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".
I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.
But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...
We had it forced down our throats by CEOs and CTOs who thought that it would improve our productivity. Nobody forced it down their throats, though. Instead, they were seduced. They went willingly.
In one gig I was on, a consultant showed up and started saying that the platform was not good because it didn't have any machine learning(this is pre-AI buzz words). So the executives asked me when can I fix the platform to have machine learning in it. They didn't have an answer when I simply asked "machine learning to do what?" and my explanation of what machine learning is or can be used for went to deaf ears. So yeah, definitely agree on seduced and then went willingly and blindly.
no. openclaw wasnt forced by ceo's. it was forced by the same people who though there was money to be made in crypto then ICO then NFT. a bunch of scammers that bring negative value to the world
And they make money. A scammer is the President of the United States.
At a certain point why blame people for trying to keep up? Why are scammers so successful? It seems to me we have a systemic failure at a societal level. Until we are honest about that it will only get worse. Until then maybe some rouge LLM botching some critical system will be the wake up call we need.
I am not sure what to make of critiques that seem to rest on notions of a small population of scammers preying upon the doe-eyed public. I think the situation is a bit closer to Carlin: garbage in, garbage out. A critique that holds up quite excellently in this AI age.
western society is a shelve of its former glory. it did not last long but there was an age were man was capable of greatness. the early internet kinda was the last stretch of this short run then money corrupted it. the underlying issue stems from abandoning cultural education as a Western value. Instead, we've opted to dispense raw ideology devoid of any thinking mechanism that we now seek so dearly to integrate to LLMs so that they can be more like us. This sloppening manifested in our lives through every medium.
We witnessed it when animation shifted to 3D, providing slop and poorly designed characters and stories. We witnessed it when video games all adopted the same game engines, look and feel and lack of narrative stakes, slopping ideology down players’ throats- no nuance, no wit, just mind-numbing dogma that punishes anyone who dares to criticize.Perhaps most damaging was Netflix's infiltration of our households that has accelerated our collective intellectual atrophy through relentless ideologically charged content parroting as entertainment. Meanwhile, our children's minds are being shaped not by family or tradition but by the algorithms of TikTok and Snapchat.The past decade and a half hasn't just prepared LLMs to replicate human abilities it has systematically stripped away human complexity, reshaping us into predictable patterns, not to raise LLMs to our level, but to reduce us to theirs, until the distinction no longer matters.
The whole industry is like a fashion show and has been for a long time. This is just exceptionally stupid compared to moderately stupid things before. I see it ore that everyone's wearing pink feathered chicken suits because it's in fashion. If you don't wear a pink feathered chicken suit then you're a luddite scumbag who doesn't deserve the respect of your peers.
However some of us still have enough self-respect not to be seen dead in a pink feathered chicken suit. I mean I'm still pissed off at half the other stuff we do in the industry. I haven't even really looked at the chicken suits yet.
If you work in a tech company with >5k employees it's extremely likely it's been forced down on you to wear the pink feathered chicken suit, and told to not complain about the pink feathered chicken suit because it is the inevitable future, and no one will be wearing anything that doesn't look like it ever again. Also, we are watching every straggler not in a pink feathered chicken suit, put yours on or leave the building.
my assessment of the situation: "we've spent so much money on AI's promise to give us 5x, 10x returns, that now we have to earn it back by foisting the burden on developers to make up the gains by working harder, at least enough to recoup the exec's decision to pour money into the boondoggle".
"Hey developers, we spent $x million on Claude, who promised 7x returns, so YOU better make it 7x more efficient so we don't look bad".
Force is seeping in. Managements are expecting that LLM-driven prouctivity-enhancers will be deployed and give broad-based boosts. More are each week. Supposedly cheaper than people. Those that aren't yet might be soon.
When your performance review includes facility with and productivity with LLM tools, you are being forced.
This is a "monopolized sector." They absolutely forced it on you. In most cases, sure, not directly, but their influence is the only driving force. Absent this no one would have jumped on this flimsy bandwagon.
Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.
Agents are providing to employees the long overdue benefits limited liability companies long enjoyed: Gambling with upside for themselves and other peoples downsides.
The media isn’t helping. This wasn’t a “rogue AI”. It was a system that was given permission by a human operator.
We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.
The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.
I’ve never had respect for the industry as a whole, only individuals within. There has a been a serious lack of rigor and professionalism in software engineering for as long as I’ve been a part of it
I think it might be because we (or at least I) used to associate insecure actions with people, not computers. Computers should know better, right? Recently, I spotted that Opus 4.6 found config files for one of its tools and gave itself access to my whole filesystem. Similarly, Gemini CLI will rewrite itself if you let it.
There's nothing "collectively" about it. I don't know what industry you work in, but in mine it's a top down mandate to use AI everywhere, tracked with KPIs, from the CEO down, and supported and pressured by companies like Amazon and MS.
We're the dummies that have to run around picking up dookies like a new puppy in the house.
People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.
My thinking is, this will increase the demand for backup and other resilience solutions.
> cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter
I saw the sea change in 2008 when quality process got replaced with velocity and testing tasks. I've watched everything from Experian and health record data leaks to Windows 11 since that change. Software quality hasn't mattered for a long time.
The frustrating part is watching all the careful thinking about reliability and failure modes get thrown out the window the second something new gets hyped. It's not even that people disagree with the principles, they just stop applying them.
I think it's batshit crazy. That's why I wrote yoloAI, so I could sandbox it up properly and control EXACTLY what comes out of that sandbox, diff style.
Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.
Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.
To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.
This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).
AI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.
“Meta spokesperson Tracy Clayton said in a statement to The Verge that ‘no user data was mishandled’ during the incident.”
Wow, no mishandled user data? A striking change of standard operating procedure from Meta here.
Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it.
Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case.
The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.
> "Had the engineer that acted on that known better, or did other checks, this would have been avoided."
takes long drag tweet[1] here>
I personally find "LLMs can do $THING poorly" and "LLMs can do $THING well" articles kinda boring at this point. But! I'm hopeful that stories like this will shift the industry's focus towards robustness instead of just short-term efficiency. I suspect many decision making and change management processes accidentally benefited from just being a bit slow.
The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
142 comments
All the years of discussing programming/security best practices
Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter, determinism is going out the window, and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.
That's it.
The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".
I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.
But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...
At a certain point why blame people for trying to keep up? Why are scammers so successful? It seems to me we have a systemic failure at a societal level. Until we are honest about that it will only get worse. Until then maybe some rouge LLM botching some critical system will be the wake up call we need.
I am not sure what to make of critiques that seem to rest on notions of a small population of scammers preying upon the doe-eyed public. I think the situation is a bit closer to Carlin: garbage in, garbage out. A critique that holds up quite excellently in this AI age.
> At a certain point why blame people for trying to keep up?
No.
The whole industry is like a fashion show and has been for a long time. This is just exceptionally stupid compared to moderately stupid things before. I see it ore that everyone's wearing pink feathered chicken suits because it's in fashion. If you don't wear a pink feathered chicken suit then you're a luddite scumbag who doesn't deserve the respect of your peers.
However some of us still have enough self-respect not to be seen dead in a pink feathered chicken suit. I mean I'm still pissed off at half the other stuff we do in the industry. I haven't even really looked at the chicken suits yet.
"Hey developers, we spent $x million on Claude, who promised 7x returns, so YOU better make it 7x more efficient so we don't look bad".
This is a "monopolized sector." They absolutely forced it on you. In most cases, sure, not directly, but their influence is the only driving force. Absent this no one would have jumped on this flimsy bandwagon.
We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.
The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.
> Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter
Is this new to people? I figured this out when I first entered the industry. The messages have never been particularly subtle.
We’ve covered so many issues already on our blog (grith.ai)
We're the dummies that have to run around picking up dookies like a new puppy in the house.
My thinking is, this will increase the demand for backup and other resilience solutions.
> cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter
I saw the sea change in 2008 when quality process got replaced with velocity and testing tasks. I've watched everything from Experian and health record data leaks to Windows 11 since that change. Software quality hasn't mattered for a long time.
https://github.com/kstenerud/yoloai
I can't go back anymore. Going back to a non-sandboxed Claude feels like going back to a non-adblocked browser.
It makes it sound like a rogue AI hacked Meta.
Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.
Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.
To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.
This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).
Because a human would have been fired for posting something that incorrect and dangerous
And there was no test environment to validate the change before it was made.
Multiple process & mechanism failures, regardless of where the bad advice came from.
The language of this article is a great example, "... thanks to an AI agent that gave an employee inaccurate technical advice ...".
It should more-correctly read, " ... thanks to the people who made it possible for an AI agent to give an employee inaccurate technical advice ... ".
It is at our peril that we deem it acceptable to blame a black box for an error, especially at scale.
Wow, no mishandled user data? A striking change of standard operating procedure from Meta here.
Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it.
Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case.
The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.
> "Had the engineer that acted on that known better, or did other checks, this would have been avoided."
I personally find "LLMs can do $THING poorly" and "LLMs can do $THING well" articles kinda boring at this point. But! I'm hopeful that stories like this will shift the industry's focus towards robustness instead of just short-term efficiency. I suspect many decision making and change management processes accidentally benefited from just being a bit slow.
[1] https://waffles.fun/amy.png
> A rogue AI led to a serious security incident at Meta
The AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failure
AI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans