In what world would I ever expect a commercial (or governmental) entity to have precise alignment with me personally, or even with my own business? I argue those relationships are necessarily adversarial, and trusting anyone else to align their "AI" tool to my goals, needs, and/or desires is a recipe for having my livelihood completely reassigned into someone else's wallet.
Why would relationships with a commercial entity be "necessarily adversarial"? A commercial relationship depends on the product providing more utility than the cost (for the consumer) and providing more revenue than cost (for the commercial entity). This means that while some components of the relationship may be adversarial in some areas, it cannot really be entirely adversarial.
Interesting you single out commercial and government entities but not people. What defines the difference? Bureaucracy? Concentration of resources? Legal theory?
I guess I'm trying to wonder why this line of thinking (in theory) doesn't turn to paranoia about everybody. I don't know much ethics or political theory or anything.
It does. People drive these entities. People hide behind the liability shields and authority of these entities. Also notice that I generalized with the phrase “…and trusting anyone…”
I'm not an expert in political theory or ethics either, but in my worldview, power relationships matter in these discussions. I believe power and responsibility should go hand in hand, and I hold entities to a standard that is proportional to their power to influence others lives.
If an entity's power is decentralized, for example when it is democratically organized to some degree, then that disperses both power and responsibility.
You can tell that broad alignment between people is natural just by looking at the effort that corporations and governments make to undermine it. Alignment between people is perhaps not a state of nature, but it really is a pretty normal consequence of a fairly small amount of education and of middle-class existence that is left to itself (i.e. without brain-washing and deliberately working to create out-groups). If you're eating enough and have a few brain cells to rub together, then you definitely want that for your neighbors too because it promotes stability.
Uh, what? People have been killing each other over values misalignments since there have been people. We invented civilization in part to protect our farms and granaries from people who disagreed with us on whose grain was in said granaries.
Fair enough. We are a social species. But those alignments occur in small groups. You don’t need effort by “corporations and governments” for nations of millions of people to schism. If anything, those large institutions drive broad-based alignment.
Methinks you've been sitting in your armchair too long.
Broad-based alignment doesn't come from nothing, but it is surprisingly easy to achieve when a population recognizes a shared stake. A synthesis between selfishness and altruism emerges when you consider who you can call a "neighbor".
it is surprisingly easy to achieve when a population recognizes a shared stake
Sure. But it takes work for anything larger than a small, close-knit community. I’m pushing back on the notion that this comes naturally and is a default state. It’s not, at least not relative to people naturally forming in and out groups.
The armchair commenters are probably folks who have never organized a group of people before outside a commercial context.
You might be treating "neighbor" too literally. People understand the global nature of the limits on resources and by extension the world economy better every year. The boundary of who shares 'stake' grows likewise.
But that shared stakeholding doesn’t naturally drive alignment. You need journalists, fiction writers, organizers and delegates. Travel and curiosity. These each take effort, resources and organization. It’s something we do well. But it isn’t spontaneous in the way small-group kinship is—it literally emerges if you put people in proximity.
And if my grandmother had wheels she’d be a bicycle. The process of creating an in group naturally creates out groups. The “brainwashing” OP describes is just as natural as social alignment through an innate drive for conformity.
Sure. Push and pull. The point is that needs effort to work at larger scales. We don’t “naturally” organize into nations of three hundred million or a billion. To the extent we do, we also “naturally” go to war.
There is a pretty interesting study of a large group of chimps. I dont remember where exactly but they have been civil warring the last 15 years or so. Point is, it seems that there is some kind of innate group formation process.
> You can tell that broad alignment between people is natural
It really isn't. The whole point of the market system is to collectively align people's actions towards a shared target of "Pareto-optimized total welfare". And even then the alignment is approximate and heavily constrained due to a combination of transaction costs (which also account for e.g. externalities) and information asymmetries. But transaction costs and information asymmetries apply to any system of alignment, including non-market ones. The market (augmented with some pre-determined legal assignment of property rights, potentially including quite complex bundles of rules and regulations) is still your best bet.
AIUI David Graeber famously pointed out that people in small groups can form the equivalent of a "market" simply by exchanging favours ("I'll scratch your back if you scratch mine") in an informal gift economy, without any money-like token or external unit of account. That's quite in line with what I said.
You understanding is mistaken. Graeber's "everyday communism" is not a market, and his whole larger point is that contorting everything to the lens of markets is simply ahistorical and unempirical.
I'd strongly suggest reading his books. They profoundly changed my understanding of how human institutions and society form.
Unless it's some sort of complete post-scarcity, it has to be understandable in market terms. What happens if people try to free-ride on the whole "communist" system? If they get excluded from its benefits, that's equivalent to enforcing some bundle of property rights.
> Unless it's some sort of complete post-scarcity, it has to be understandable in market terms.
No, it does not, and that's Graeber's whole point.
"Markets" are not some sort of physical law of the universe.
A simple example of this is it's the norm in hunter gatherer societies to take care of people who never will make an equal contribution back in the transactional sense.
Because the social ties in those societies are not simply transactions.
If your model fails to accurately describe empirical reality, time to improve/expand the model.
These social ties are real (they are a kind of wealth, or social capital, for the persons involved) but they're also limited to very small social groups, the equivalent of a modern small village neighborhood or HOA. The point of the market is that it scales well beyond those.
I like economics and math too, but the whole discussion of markets is a terrible starting place for deriving results in ethics/psychology. If you insist though, notice that unions will happen unless some other organization is working to prevent them. What do you suppose this means? People are aligned with each other exactly because they've noticed their coworkers are not corporations or governments.
Although the two are entangled, politics is a more relevant framing than economics here. If people weren't broadly aligned on basic stuff, then autocrats, theocrats, kleptocrats and so on would simply not be interested in dismantling democracies. They make that effort because they must.
> the whole discussion of markets is a terrible starting place for deriving results in ethics/psychology.
Historically, we did essentially the opposite. We figured out many aspects of human ethics and psychology first, and deduced from them how and why markets work as they do.
> ... If people weren't broadly aligned on basic stuff, then autocrats, theocrats, kleptocrats and so on would simply not be interested in dismantling democracies. They make that effort because they must.
This implies that people are only weakly aligned in the first place, otherwise no such attempt at dismantling could ever succeed. That's not a very interesting claim; it does not refute the usefulness of some external mechanism to more directly foster aligned action. Markets do this with a maximum of decentralized power and a minimum of institutional mechanism.
The market aligned us with children working in sweat shops after we outlawed it by convincing us it was OK if it was foreign kids and we got to share in pocketing the savings not just the evil factory owner.
Yes I'm well aware. Of course that's not how things are advertised to people, and they absolutely hate it when this is pointed out to them. This tells me that deep down they don't actually agree with how the system operates.
> Interesting you single out commercial and government entities but not people. What defines the difference? Bureaucracy? Concentration of resources? Legal theory?
Not OP, but for me, kind family and friends, and various feel-good pieces of fiction and other writing, at least let me envision the possibility of a perfectly kind/dedicated/innocent/naieve individual who is truly on my side 100%. But even that is mostly imagination and fiction... although convincing others of that isn't necessairly an argument worth making.
Commercial entities have a fundamental purpouse of profit. While profit doesn't have to be a zero-sum game - ideally, everyone benefits in a somewhat balanced way - there's some fundamental tension, in that each party's profit is necessairly limited by the other party's.
Government entities have a fundamental purpouse of executing the will of the state, which is rather explicitly not the same thing as the will of you as an individual.
Both commercial and government entities also tend to involve multiple people, which gets statistics working against you - you really gathered that many people who would put your needs above their own, with exactly zero "imposters" - which in this context just means people with a bit of rational self interest?
> I guess I'm trying to wonder why this line of thinking (in theory) doesn't turn to paranoia about everybody. I don't know much ethics or political theory or anything.
Just because you're paranoid, doesn't mean they aren't out to get you. Trust, but verify.
You might not be able to put absolute blind trust in anybody. I certainly can't. However, one can hedge one's bets, and diversify trust. Build social circles of people with good character, good judgement, and calm temperments - and statistics will start working for you. It's unlikely they'll all conspire to betray you simultaniously, especially if you've ensured betrayal costs much and gains little. While petty and jealous people can indeed be irrational enough to betray under such circumstances, it'll be harder for them to create the kind of conspiracy necessary for mass betrayal that might cause significant enough damage to warrant proper paranoia. You might still have to watch out for gaslighters stealing credit (document your work!) and framing people (document your character!) and other such dishonest and manipulative behavior... but if everyone's looking out for the same thing, well, that's just everyone looking out for everyone else! That's a community looking out for each other, and holding everyone honest and accountable. Most find comfort in that, rather than the stress paranoia implies.
Put yourself in a room full of manipulators and schemers, on the other hand, and "parnoia about everyone" might be the only reasonable or rational response!
> But even that is mostly imagination and fiction... although convincing others of that isn't necessairly an argument worth making.
There was a Japanese visual novel in the 2000s about a girl who was your personal maid, and was so devoted would always take your side in any conflict, accept and support you just the way you are, even if you were a horrid person to your friends. It turns out she was a ghost, or a kind of yokai, or something. Anyhoo, back on 2ch she attracted a fandom, and there was a second group of people on 2ch who labelled her a "useless person manufacturer" because if you actually had a person who always accepted you just the way you are and never pushed back, that can be actually a trap that prevents you from developing.
It's a theme that's relevant today when people have AI servitors that always glaze them. It puts even certain utopian AI fiction, like Richard Stallman's story "Made for You", into a whole new light.
My family accepts me just the way I am a bit too much. I can't bring myself to blame them, when past "reformist" pressures have been misguided/misapplied and backfired, but I recognize the trap. It'd also be hypocritical to blame them, when I also accept me just the way I am a bit too much! I'd like to think I'm decent enough to people, but I'm certainly more useless than I'd like to be. (Un?)fortunately, I'm not in a position to suffer, and I'm at least aware of the problem!
One of the ideas I've toyed with, even before all the AI hype, is a dumb, semi-adversarial servitor. Something to nag or taunt me about chores not done, to interrupt me when I'm doomscrolling, to use as a vessel for precommitment, to challenge me in various ways. I've been too lazy to build it thus far. Many tools overlap the problem space, so I shouldn't be using that as an excuse - perhaps I should give StayFocusd another shot.
Conflict and other stressors - in moderation, within the limits of one's ability to handle - are important for growth and health. A tree shielded from wind is weakened as it fails to develop stress wood and structural strength. A good debate can sharpen my thoughts and mind, walking to lunch keeps my cardiovascular system healthy, rising to life's various challenges gives me the security of knowing I can rise to the occasion and gives me more skills.
> Profit is obtained by maximizing traded benefits and minimizing costs.
Gain is obtained by the easiest means available. Your narrow definition of profit is seldom the easiest, cheating is far "superior" especially when it's legal for some.
> None of this requires taking anything away from any other party.
"required" and "preferred" (e.g. because it's far easier) are different like night and day.
If you've built an agent that can act even vaguely close to a paperclip maximizer, you've already solved 99.999% or more of the alignment problem. The hard part of alignment so far is getting the AI to do something useful in pursuit of the right goal, and not just waste energy. We still have no idea how to do this with any effectiveness: even modern "RL from verified feedback" systems are effectively toys, the equivalent of playing video games, not really of doing something useful in the real world.
Huh? Modern RLVR systems are toys that can’t do anything useful in the real world?
We must be living in completely different worlds. Claude and other agents have completely upended work for me and every single other software engineer I know.
181 comments
1. Introduction: <https://news.ycombinator.com/item?id=47689648> (619 comments)
2. Dynamics: <https://news.ycombinator.com/item?id=47693678> (0 comments)
3. Culture: <https://news.ycombinator.com/item?id=47703528>
4. Information Ecology: <https://news.ycombinator.com/item?id=47718502> (106 comments)
5. Annoyances: <https://news.ycombinator.com/item?id=47730981> (171 comments)
6. Psychological Hazards: <https://news.ycombinator.com/item?id=47747936> (0 comments)
And this submission makes:
7. Safety: <https://news.ycombinator.com/item?id=47754379> (89 comments, presently).
There's also a comprehensive PDF version for those who prefer that kind of thing: <https://aphyr.com/data/posts/411/the-future-of-everything-is...> (PDF) 26 pp.
(Derived from aphyr's comment: <https://news.ycombinator.com/item?id=47754834>.)
In what world would I ever expect a commercial (or governmental) entity to have precise alignment with me personally, or even with my own business? I argue those relationships are necessarily adversarial, and trusting anyone else to align their "AI" tool to my goals, needs, and/or desires is a recipe for having my livelihood completely reassigned into someone else's wallet.
> Why would relationships with a commercial entity be "necessarily adversarial"?
Because they want to separate me from as much of my money as they can, and I want to keep as much of my money as I can.
I guess I'm trying to wonder why this line of thinking (in theory) doesn't turn to paranoia about everybody. I don't know much ethics or political theory or anything.
> … paranoia about everybody
It does. People drive these entities. People hide behind the liability shields and authority of these entities. Also notice that I generalized with the phrase “…and trusting anyone…”
I'm not an expert in political theory or ethics either, but in my worldview, power relationships matter in these discussions. I believe power and responsibility should go hand in hand, and I hold entities to a standard that is proportional to their power to influence others lives.
If an entity's power is decentralized, for example when it is democratically organized to some degree, then that disperses both power and responsibility.
>
broad alignment between people is naturalUh, what? People have been killing each other over values misalignments since there have been people. We invented civilization in part to protect our farms and granaries from people who disagreed with us on whose grain was in said granaries.
Broad-based alignment doesn't come from nothing, but it is surprisingly easy to achieve when a population recognizes a shared stake. A synthesis between selfishness and altruism emerges when you consider who you can call a "neighbor".
>
it is surprisingly easy to achieve when a population recognizes a shared stakeSure. But it takes work for anything larger than a small, close-knit community. I’m pushing back on the notion that this comes naturally and is a default state. It’s not, at least not relative to people naturally forming in and out groups.
The armchair commenters are probably folks who have never organized a group of people before outside a commercial context.
>
boundary of who shares 'stake' grows likewiseBut that shared stakeholding doesn’t naturally drive alignment. You need journalists, fiction writers, organizers and delegates. Travel and curiosity. These each take effort, resources and organization. It’s something we do well. But it isn’t spontaneous in the way small-group kinship is—it literally emerges if you put people in proximity.
>
Couldn't read the next sentence before wading in, huh?Whatever the difference between naturalness and a state of nature, it has nothing to do with education or middle-class existence.
> i.e. without brain-washing and deliberately working to create out-groups
> You can tell that broad alignment between people is natural
It really isn't. The whole point of the market system is to collectively align people's actions towards a shared target of "Pareto-optimized total welfare". And even then the alignment is approximate and heavily constrained due to a combination of transaction costs (which also account for e.g. externalities) and information asymmetries. But transaction costs and information asymmetries apply to any system of alignment, including non-market ones. The market (augmented with some pre-determined legal assignment of property rights, potentially including quite complex bundles of rules and regulations) is still your best bet.
What you describe is factually not how human society formed.
I'd strongly suggest reading his books. They profoundly changed my understanding of how human institutions and society form.
> Unless it's some sort of complete post-scarcity, it has to be understandable in market terms.
No, it does not, and that's Graeber's whole point.
"Markets" are not some sort of physical law of the universe.
A simple example of this is it's the norm in hunter gatherer societies to take care of people who never will make an equal contribution back in the transactional sense.
Because the social ties in those societies are not simply transactions.
If your model fails to accurately describe empirical reality, time to improve/expand the model.
> it has to be understandable in market terms
I like economics and math too, but the whole discussion of markets is a terrible starting place for deriving results in ethics/psychology. If you insist though, notice that unions will happen unless some other organization is working to prevent them. What do you suppose this means? People are aligned with each other exactly because they've noticed their coworkers are not corporations or governments.
Although the two are entangled, politics is a more relevant framing than economics here. If people weren't broadly aligned on basic stuff, then autocrats, theocrats, kleptocrats and so on would simply not be interested in dismantling democracies. They make that effort because they must.
> the whole discussion of markets is a terrible starting place for deriving results in ethics/psychology.
Historically, we did essentially the opposite. We figured out many aspects of human ethics and psychology first, and deduced from them how and why markets work as they do.
> ... If people weren't broadly aligned on basic stuff, then autocrats, theocrats, kleptocrats and so on would simply not be interested in dismantling democracies. They make that effort because they must.
This implies that people are only weakly aligned in the first place, otherwise no such attempt at dismantling could ever succeed. That's not a very interesting claim; it does not refute the usefulness of some external mechanism to more directly foster aligned action. Markets do this with a maximum of decentralized power and a minimum of institutional mechanism.
"Understandable in market terms" doesn't mean the thing is actually understood, and in fact may be dangerously misunderstood.
> Interesting you single out commercial and government entities but not people. What defines the difference? Bureaucracy? Concentration of resources? Legal theory?
Not OP, but for me, kind family and friends, and various feel-good pieces of fiction and other writing, at least let me envision the possibility of a perfectly kind/dedicated/innocent/naieve individual who is truly on my side 100%. But even that is mostly imagination and fiction... although convincing others of that isn't necessairly an argument worth making.
Commercial entities have a fundamental purpouse of profit. While profit doesn't have to be a zero-sum game - ideally, everyone benefits in a somewhat balanced way - there's some fundamental tension, in that each party's profit is necessairly limited by the other party's.
Government entities have a fundamental purpouse of executing the will of the state, which is rather explicitly not the same thing as the will of you as an individual.
Both commercial and government entities also tend to involve multiple people, which gets statistics working against you - you really gathered that many people who would put your needs above their own, with exactly zero "imposters" - which in this context just means people with a bit of rational self interest?
> I guess I'm trying to wonder why this line of thinking (in theory) doesn't turn to paranoia about everybody. I don't know much ethics or political theory or anything.
Just because you're paranoid, doesn't mean they aren't out to get you. Trust, but verify.
You might not be able to put absolute blind trust in anybody. I certainly can't. However, one can hedge one's bets, and diversify trust. Build social circles of people with good character, good judgement, and calm temperments - and statistics will start working for you. It's unlikely they'll all conspire to betray you simultaniously, especially if you've ensured betrayal costs much and gains little. While petty and jealous people can indeed be irrational enough to betray under such circumstances, it'll be harder for them to create the kind of conspiracy necessary for mass betrayal that might cause significant enough damage to warrant proper paranoia. You might still have to watch out for gaslighters stealing credit (document your work!) and framing people (document your character!) and other such dishonest and manipulative behavior... but if everyone's looking out for the same thing, well, that's just everyone looking out for everyone else! That's a community looking out for each other, and holding everyone honest and accountable. Most find comfort in that, rather than the stress paranoia implies.
Put yourself in a room full of manipulators and schemers, on the other hand, and "parnoia about everyone" might be the only reasonable or rational response!
> But even that is mostly imagination and fiction... although convincing others of that isn't necessairly an argument worth making.
There was a Japanese visual novel in the 2000s about a girl who was your personal maid, and was so devoted would always take your side in any conflict, accept and support you just the way you are, even if you were a horrid person to your friends. It turns out she was a ghost, or a kind of yokai, or something. Anyhoo, back on 2ch she attracted a fandom, and there was a second group of people on 2ch who labelled her a "useless person manufacturer" because if you actually had a person who always accepted you just the way you are and never pushed back, that can be actually a trap that prevents you from developing.
It's a theme that's relevant today when people have AI servitors that always glaze them. It puts even certain utopian AI fiction, like Richard Stallman's story "Made for You", into a whole new light.
One of the ideas I've toyed with, even before all the AI hype, is a dumb, semi-adversarial servitor. Something to nag or taunt me about chores not done, to interrupt me when I'm doomscrolling, to use as a vessel for precommitment, to challenge me in various ways. I've been too lazy to build it thus far. Many tools overlap the problem space, so I shouldn't be using that as an excuse - perhaps I should give StayFocusd another shot.
Conflict and other stressors - in moderation, within the limits of one's ability to handle - are important for growth and health. A tree shielded from wind is weakened as it fails to develop stress wood and structural strength. A good debate can sharpen my thoughts and mind, walking to lunch keeps my cardiovascular system healthy, rising to life's various challenges gives me the security of knowing I can rise to the occasion and gives me more skills.
> each party's profit is necessairly limited by the other party's
Profit is obtained by maximizing traded benefits and minimizing costs. None of this requires taking anything away from any other party.
> Profit is obtained by maximizing traded benefits and minimizing costs.
Gain is obtained by the easiest means available. Your narrow definition of profit is seldom the easiest, cheating is far "superior" especially when it's legal for some.
> None of this requires taking anything away from any other party.
"required" and "preferred" (e.g. because it's far easier) are different like night and day.
> precise alignment with me personally, or even with my own business
Seems like a strawman, I don't think anyone means this when talking about alignment.
More general goals, like avoiding paperclip maximization, are broadly applicable to humanity.
We must be living in completely different worlds. Claude and other agents have completely upended work for me and every single other software engineer I know.