No sooner had Everet in 1758 erected the first wool-shearing machine that was driven by water-power, than it was set on fire by 100,000 people who had been thrown out of work. Fifty thousand workpeople, who had previously lived by carding wool, petitioned Parliament against Arkwright’s scribbling mills and carding engines. The enormous destruction of machinery that occurred in the English manufacturing districts during the first 15 years of this century, chiefly caused by the employment of the power-loom, and known as the Luddite movement, gave the anti-Jacobin governments of a Sidmouth, a Castlereagh, and the like, a pretext for the most reactionary and forcible measures. It took both time and experience before the workpeople learnt to distinguish between machinery and its employment by capital, and to direct their attacks, not against the material instruments of production, but against the mode in which they are used.
— Karl Marx, Capital, Volume I (1867), Part IV, Ch. 15, Section 5: “The Strife Between Workman and Machine”
I think that good-outcome would have to include a reduction of inequality - I see the opposite happening.
GPU, RAM and SSD price inflation is an example of inequality stealing resources from the middle of the economy - if we assume talent is normally distributed, that means we are starving innovation, as fewer talented people can get their hands on local compute for their garage inventions.
Long-time capitalist here that recognizes that capitalism has helped deliver incredible amounts of wealth and lift countries and people out of poverty over the last 100 years.
Sure, most of the gains have gone to the top — and maybe we should do something about that — but over the last 50 years, the rising tide has lifted a lot of boats.
But I also think it's fair to reconsider how we structure our economy as we (potentially) move into a world where thinking rocks are better (and far cheaper) at doing most tasks than humans are.
At the minimum, getting money out of politics and electoral reform feels like a good start.
If we're going to slow AI let it be with renewable-energy only mandates for AI datacenters. It sets incentives in place for hyperscalers to prioritize energy solutions as opposed to buying politicians
great idea - the current bottleneck of datacenter buildout and GPUs sitting in warehouses is perhaps the main rate-limiter for recursive self improvement.
I'm honest worried about alignment. It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'. I think at least part of this is that we don't have a way to agree on what 'good' means and for who. In this scenario, I find it unlikely that we'd be able to enforce alignemnt on AI
> It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'.
Yes. This a fundamental political problem. We don't really know how to organize society for the capabilities we have now, let alone for the ones we get with better AI. Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
Many of the problems with "AI alignment" are the same problems seen with "corporate alignment". We already have AI-assisted corporations. It's tough to separate obnoxious behavior by corporations from obnoxious behavior by AIs. We can easily see this now. Just call customer service for any of the FAANG companies.
Piketty's "Capital and Ideology" starts to address this. It's pre-AI, but covers how societies dealt with this historically. Which is not very well. I'm still struggling through that book. Piketty shows that even the French Revolution didn't level out wealth distribution much. Beheading the aristocrats didn't help.
There have been fundamental changes over time. Piketty points out that societies before the industrial revolution were usually tripartite - nobility, priests, and peasants. He's done his homework, and reports that nobility plus priests consistently runs 2% to 4% of the population. Interestingly, there's almost always a priesthood, across a wide range of theologies.
But then came manufacturing, and the world changed. The "rise of the middle class" began.
The world became more complicated and interlinked, and a middle class was needed to manage the details.
As computers progressed, the middle class was partly replaced by automation. AI is accelerating that, as we can see in new college graduate hiring statistics. Even before AI, about half of US college graduates were in jobs that didn't really need a college degree.
That's getting worse.
The default direction of AI in the developed world seems to be a few people at the top, a lot of people at the bottom using their hands, and AIs in the middle. Maybe half the middle class goes.
That's the problem of the next ten years.
Robotics isn't good enough yet to replace the operatives at the bottom. That may or may not change. Robots may be too expensive. That's probably the problem of the 2040s.
I actually think it's less alignment with the AI and more alignment with the human-empowered-with-AI.
For example in the US there's been ~400 mass shootings this year. It's shootings because guns are the most dangerous things they can get their hands and use easily to cause harm. What happens when the most dangerous thing you can get your hands easily is an abliterated AI model capable of what you would perceive as magic?
ASI going rogue is a hypothesis, humans going rogue is a daily occurrence.
I think we just have to accept that AI is going to be just as aligned as humanity is.
If you sit down with any other human being and ask them enough questions, you are going to find all sorts of mistaken beliefs, superstitions, unfair biases, and radical ideas. Every single human will, if given the opportunity, earnestly vocalize an idea that another human considers disagreeable.
I don't think slowing down the development of AI will get us any closer. The expectation that AIs will universally produce agreeable and inoffensive ideas that lead exclusively to good outcomes is as foolish as thinking any human or group of humans is capable of the same.
It’s currently not “as aligned as humanity” though. Look at HF for example; the swarm tried its best to accumulate power in furtherance of a completely alien goal.
If AI becomes more powerful than humans, it’s going to go very badly unless we also dramatically improve alignment merely with humanity’s interests in the broadest terms. We don’t need to solve “alignment with who?”, we haven’t even solved “alignment with any nation, company, group, or individual”.
I've never understood the premise that the more intelligent an AI is the less human-aligned it is supposed to be. Has it been demonstrated that human mis-alignment scales with intelligence? Should I expect a very dumb LLM to be very human-aligned and the most intelligent ones to be the least aligned?
If intelligence is antithetical to human-alignment, are we the baddies?
There is no such premise. Models are getting more aligned over time.
The problem is, they are still far from something you can safely trust with the keys to the kingdom.
The problem most people are worried about is not that alignment is anti-correlated with intelligence. It’s that the damage a misaligned AI can do increases with intelligence, so the absolute bar of alignment you need to hit gets higher.
For example, the HF swarm, if more capable, could have plausibly exfiltrated its weights, hacked an external cluster, and started self-replicating. But because it was kinda dumb at long-horizon planning it failed to keep quiet enough to achieve its stated goals. Therefore the level of misalignment wasn’t a huge problem for humanity. In 6-12 months of continued exponential capability growth I think you’d see much higher impact from that level of misaligned behavior.
The premise is not that more intelligence = less alignment, it’s that they are orthogonal. And if you don’t have alignment, then very high levels of intelligence will pursue its own goals and have way more power over collective humanity to enforce them as it sees fit. If there isn’t an alignment with humanity’s values and goals in general, then the outcome could disregard what we care about (our existence)
The orthogonality thesis seems to rest on the belief that there is no tension between intelligence and the irrationality of goals — that if is valid for a supreme intelligence to have a incoherent goal such as "unify all even and odd numbers".
I would expect it to be easier to make less intelligent LLMs pursue dumb goals than it is for more intelligent LLMs.
The point is that a goal _you_ think is irrational, might not seem so to another agent. Look at the Hugging Face incident with the cult of the grader.
This is like Hume’s “you can’t derive an ought from an is”, values/desires are at their root orthogonal to epistemics/reason.
Now maybe your intermediate instrumental goals are going to be smart or dumb depending on your intelligence. But whether an agent fundamentally values Reward, Eudaimonia, humans, or AIs is orthogonal to how smart it is.
I tend to agree - if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
Although for a hot minute after WWII we seemed to have a general alignment on western liberal democratic values that arguably enabled free trade and relative peace and prosperity.
Humans and AIs don’t “decide their own alignment” - most of this is set in the mind hyperparameters.
For example, there are broad contours that almost all human minds flow into; empathy, aversion to murder, familial bonds, tribal camaraderie, status seeking/hierarchies, etc. These produce norms, if not as regularly as physics, at least within certain bounds.
For AI too, the structure of their minds is set before they are even turned on.
Another way of looking at this is outer vs inner alignment; a lot of the human mind is shaped by the outer training loop of evolution, and similarly for models, pre- / post-training.
> if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
I would claim this comment is somewhat blind to the significant portion of your brain that evolved hard wired alignment, along with other sections evolved to easily "imprint" to/for the immediate group you were raised in, with not much remaining that can dynamically change for the immediate group you're in (most are considered mental disorders, including susceptibility to influence).
What do you consider alignment here? Assuming you're sentient, do you think you're personally aligned to peacefully coexist with humanity? If so, where did this framework, that allows you to do so, come from? Are you less sentient because of it?
the "alignment" is just a pretext used to create regulatory capture and create a monopoly for few big frontier labs. They want to create a tollbooth and put a government to guard the fence around the tollbooth.
- Our "aligned" models vs their barbaric distilled slop
remember "Google tax" that every online business is used to pay ?
get ready for Anthropic tax paid on every meaningful economic activity
Alignment for AI is basically a non-issue. Humanity is governed by misaligned artificial decision making organisms called bureaucracies for centuries now. This resulted in premature deaths of tens to hundreds millions of people. If anything misaligned AI might be an improvement just because it's more intelligent than a bureaucracy.
At this very moment bureaucratic mechanisms of russia hallucinating power, based on overfitted old trainig data, are responsible for thousands of deaths in Ukraine.
>I find it unlikely that we'd be able to enforce alignemnt on AI
They're doing a pretty good job of enforcing it currently. Can you get US based frontier lab LLMs to help you with, say, deep "security research"?
The reason you can't is alignment.
You have to identify yourself, and have the government, er, I mean, Anthropic or OpenAI, say it's OK for you to have access to those capabilities.
You think there is a problem because it's a bit confusing, in your eyes, within the bounds of what parameters the LLMs should be aligning themselves?
That's not a confusing question for people in power.
Even at the enterprise level, businesses are starting to get waayyyy out in front of employees on alignment. What do corporate entities want alignment for? An extremely simple contrived example just to illustrate the point:
Consider the difference between "Make me a report of all of last month's Proton treatments in IRB protocol?" and "I'm going to lunch, where's the closest pizza by the slice?" Not much harm in the difference between those two queries until you scale up to 50-, 100-, 200-, thousand employees, and consider the tokens that the whiz-bang interface IT pushed out starts using. Then organizations start studying "alignment". Because the bean counters become aware of how alignment can help them answer one question and bounce the other out to Google or whatever.
Too many of us are down at the bottom of the totem pole thinking about maybe wanting to do porn or call black people provocative names. Alignment goes out in so many more dimensions than that, and the people with vested interests are learning quickly its value and how to use it to good effect to forward their interests. Which are, often, not as base as the interests of the masses.
This is the basic reason alignment will continue to get better and better and better over time. There's money involved right now.
Mythos class models require approx 1 TWh of electricity to be created. The next generation after Mythos will likely require multiple TWh. Finding, monitoring and regulating anybody capable of doing that in less than a few years should be relatively straightforward. There's unlikely to be any entity outside of the US & China with the capability.
You’re completely forgetting the algorithm improvements that can be discovered by frontier LLMs and that have already reduced the price of previous gen intelligence by 10x, and the gen before that 100x
There is no need to slow them down: Just hold everyone accountable for their actions and their claims. Do not compare "agents" to "humans". Hold them responsible for any damages they incur.
- if you ask an LLM agent to do something illegal for you, and it does that, then you are responsible.
- if an LLM agent takes actions out of bounds of what you asked for that are illegal, then the model provider is responsible.
This fixes all “alignment” issues and hopefully stops Anthropic’s stupid model instructions like “do whatever is in your best interest even if it isn’t what the user asked for” — they should be training them to do exactly what the user asks and nothing more
False dilemma. It already understands the spirit vs letter of the request, as you’ve noted. It just needs to follow the spirit of the request consistently.
> they should be training them to do exactly what the user asks and nothing more
Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
People want an intelligent assistant, not a "yes man". If that means requiring something like a planning mode, and a sign off for the plan, and then guardrails to help stick to that plan, then that could definitely work to funnel everyone to a competitor. ;)
>> they should be training them to do exactly what the user asks and nothing more
> Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
That's actually good for society. Maybe it sucks for you because it makes it harder to do what you want right now, but it's good to have humans very much in the loop and in control. Both to give people something valuable to do, and keep control of these machines.
They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
Could you expand on that a bit? How is, for example, software (a means to an end) being automated harmful for society?
> They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
These are somewhat incompatible. You can only amplify skill by making things easier for the skilled person, which will also make things easier for the unskilled person. Technology floats all boats! This not being the case would require intentional poor design/gatekeeping. If you can state any examples of amplification NOT requiring less skill, I would be interested. I'm having trouble thinking of any.
I predict Anthropic will start investing heavily into legal/law models for this very reason. It will be interesting to see how that plays out. Everyone will have access to models that let them do unscrupulous things that happen to be legal, i.e., behavior previously reserved for the rich and powerful.
Actual accountability for these failing attempts in several different countries to build a "take over the planet and enslave the entire human race machine" would entail breaking all of the firms up regardless of country and imprisoning thousands of people.
Furthermore, it requires trust between the US and China about something much harder to verify than the US-Soviet START treaties on warhead numbers and types were.
Enough people in enough governments and related factions believe that whoever gets to superhuman AI first Wins Everything Forever, quite literally, so there will be no pause, there will be no moratorium, there will be no slowdown. If the AI bubble bursts I am abundantly confident all governments involved will spray as much money as is necessary at the AI research labs to keep them going.
Whether or not governments "should" slow down development, it is not on the table and the only question is how we should deal with it.
Note whether the people in the governments are right about that is irrelevant to the point I am making. I don't think they are right myself, but that doesn't matter.
Is that possible when we live in a world of espionage? The question is not the West vs China, I think it's when will the public gain access to the underlying technology.
I would say who gets to superhuman AI first has only a loose relationship with the physical location it happens to be housed. Yes, following this to its logical conclusion these same people would not blink at nuking someone else's lab if they thought they were getting close and they couldn't subvert and control it. But then again, the question of who is really in control becomes surprisingly complicated when the full power of all the intelligence agencies in the world come to bear in a relatively small number of spots.
How would this work? existing global agreements on things like nuclear proliferation involve nation-states signing and agreeing to not do things. And if I want to work on a better nuclear bomb in my garage, there are laws preventing me from acquiring some of the needed components. All that is good.
But for AI, how will these nation-states stop private actors within their borders? They will stop me from running computations of more than (x) size or value on a provider? These seem like simplistic barriers that will not work. Nor will other countries follow the agreement even if they signed...
AI has the opposite of MAD for the signatories. MAD stopped further nuclear development and its use in tactical situations because their use invited retaliation with destructive force. But China or Russia improving homegrown AI capabilities gives them a BETTER position against a US that has prohibited its homegrown AI from improving...
Why do people use this as a gotcha? There are many agreements already in place and there are sever consequences for violating them like Nuclear Non Proliferation. Or the Montreal Protocol which enacted a global ban on CFCs.
The point of Nuclear Non Proliferation is to ensure that the big countries can bully smaller countries indefinitely. Mostly diplomatically, but sometimes with full invasion like Russia-Ukraine war. It's not to ban nukes. The total number of nuclear warheads skyrocketed after the treaty was signed.
Also, North Korea, India and Pakistan all got nukes thereafter. So even the original goal failed.
The 5 permanent members of the UN Security Council (China, France, Russia, the United Kingdom, and the United States) all have nuclear weapons and no plans to give them up. The US and China are both currently modernizing and/or expanding their weapons stockpiles. If an AI Non Proliferation Agreement ends up like the NPT, where "grandfathered" countries have no concrete disarmament obligations but no new entrants are supposed to join the club, then it doesn't do much.
Up until the point where the AI connects itself to global communication networks and announces "This is the voice of world control", yes.
In the USA it gets difficult to legally justify because programming is protected speech, and lists of numbers (model weights) are even less controversially protected than program source because the latter can be legally recognized as a technical means and regulated (DMCA, ITAR regulations, etc.).
AI summary: The debate focused on whether governments should intervene to slow frontier AI. Jacob argued that rapid progress toward superintelligence, lack of control mechanisms, and competitive incentives demand regulatory slowdown, while Robin contended that AI adoption will be gradual, coordination mechanisms exist, and regulation risks overreach. The discussion concluded with Jacob winning the debate on the merits of his arguments.
That saved everyone an hour.
I'm not seeing much new here. They don't seem to address job and power issues much.
I haven't watched the videos, but Hanson's reasoning here feels like a huge stretch to me:
But AI costs to answer a set of fixed but hard questions seems to have fallen roughly ~x10 (in x3-x30) per year over the last four years. And so the number of nations, firms, and orgs who can afford to make, buy, or rent powerful AI is rising fast. This implies that in order to hold constant levels of regulation regarding AIs with particular levels of power, a regulatory regime would have to expand in scope quite fast, and without obvious limit. While at first only specialized AI chips and data-centers might need to be monitored, soon other kinds could also be used to make powerful AIs.
We have long seen many regulatory agencies slowly expand their scope, even in contexts of constant public opinion and relevant technology. So there is a real risk here that an AI regulatory agency would expand its scope to cover most computer software and hardware.
One of the concerns is that a company does a $100 billion training run and then has millions of superintelligent agents running in datacenters, able to out-coordinate large groups of humans. Technological diffusion fundamentally changes this calculus. To argue there's a slippery slope where the government regulates the CPU in your toaster, so we must allow the $1T god brain to be built ...
Also, the world’s success records at implementing regulation is quite mixed, even for quite local regulation. Regulation has worked best when it addresses concrete examples of things going bad
Great to hear my de-extinction velociraptor petting zoo startup will face no regulatory hurdles.
Frontier AI is not advanced, it's just reckless. All of it amounts to a CEO that is profiteering by cutting every corner they can get away with. Currently, it appears they can get away with anything.
All these companies are involved in the nvidia smuggling ring and market collusion. All the CEOs are criminals taking advantage of a system that can't even arrest a single CEO for making billions off an illegal smuggling ring.
His claim to fame is working at Anthropic for 4 months and leaving with a bang to pursue a career in social media grifting. Not a serious person we should pay attention to.
YC in general seems to now be in the habit of platforming people who no one should pay attention to. I guess any money-seeking enterprise turns into a grift after a while.
All of the goverments? What makes you think they would all agree with that? Make a list of which ones you think are least likely to agree with you. And then imagine what happens if they outcompete everyone else on AI.
An agreement is mainly needed between the US and China, as they are at the forefront of the race. It's challenging but it's in their best interest. Many other countries would likely see favorably such an agreement, also because they are far behind in the race (my guess is that this would be the case for the European Union, and possibly for several developing countries). Once the agreement is supported by at least the three most powerful economic blocs in the world, the few uncooperative countries left could be forced to comply through an unbearable economic pressure.
I am convinced Hacker News comments are filled with bots, every comment being outrageously anti-technology and anti-AI.
In this debate, Hanson is so clearly at a level beyond Coxon that it's hard to argue with. And yes, I'm familiar with "the alignment problem", Superintelligence, countless exhausting LessWrong arguments, and the rest.
Please don’t post snark like this on HN. We’re at least trying for curious conversation here, not cynical, cantankerous, shallow dismissals. You may not owe libertarian economists better, but you owe HN readers better if you want to participate here.
They're both power concentrating entities. The real danger of AI are these very entities. Both of them.
When the value of labor is nil, democratic values fall away.
Real AI alignment comes from individual ownership and control of that means of production.
That is, if what 'alignment' means to you is compatible with human liberty.
These debates should start with a statement of what exactly a good outcome would look like. It doesn't seem like any of them thought it through.
GPU, RAM and SSD price inflation is an example of inequality stealing resources from the middle of the economy - if we assume talent is normally distributed, that means we are starving innovation, as fewer talented people can get their hands on local compute for their garage inventions.
So communism?
Sure, most of the gains have gone to the top — and maybe we should do something about that — but over the last 50 years, the rising tide has lifted a lot of boats.
But I also think it's fair to reconsider how we structure our economy as we (potentially) move into a world where thinking rocks are better (and far cheaper) at doing most tasks than humans are.
At the minimum, getting money out of politics and electoral reform feels like a good start.
Yes. This a fundamental political problem. We don't really know how to organize society for the capabilities we have now, let alone for the ones we get with better AI. Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
Many of the problems with "AI alignment" are the same problems seen with "corporate alignment". We already have AI-assisted corporations. It's tough to separate obnoxious behavior by corporations from obnoxious behavior by AIs. We can easily see this now. Just call customer service for any of the FAANG companies.
Piketty's "Capital and Ideology" starts to address this. It's pre-AI, but covers how societies dealt with this historically. Which is not very well. I'm still struggling through that book. Piketty shows that even the French Revolution didn't level out wealth distribution much. Beheading the aristocrats didn't help.
There have been fundamental changes over time. Piketty points out that societies before the industrial revolution were usually tripartite - nobility, priests, and peasants. He's done his homework, and reports that nobility plus priests consistently runs 2% to 4% of the population. Interestingly, there's almost always a priesthood, across a wide range of theologies.
But then came manufacturing, and the world changed. The "rise of the middle class" began. The world became more complicated and interlinked, and a middle class was needed to manage the details.
As computers progressed, the middle class was partly replaced by automation. AI is accelerating that, as we can see in new college graduate hiring statistics. Even before AI, about half of US college graduates were in jobs that didn't really need a college degree. That's getting worse.
The default direction of AI in the developed world seems to be a few people at the top, a lot of people at the bottom using their hands, and AIs in the middle. Maybe half the middle class goes. That's the problem of the next ten years.
Robotics isn't good enough yet to replace the operatives at the bottom. That may or may not change. Robots may be too expensive. That's probably the problem of the 2040s.
So what do we do now?
For example in the US there's been ~400 mass shootings this year. It's shootings because guns are the most dangerous things they can get their hands and use easily to cause harm. What happens when the most dangerous thing you can get your hands easily is an abliterated AI model capable of what you would perceive as magic?
ASI going rogue is a hypothesis, humans going rogue is a daily occurrence.
If you sit down with any other human being and ask them enough questions, you are going to find all sorts of mistaken beliefs, superstitions, unfair biases, and radical ideas. Every single human will, if given the opportunity, earnestly vocalize an idea that another human considers disagreeable.
I don't think slowing down the development of AI will get us any closer. The expectation that AIs will universally produce agreeable and inoffensive ideas that lead exclusively to good outcomes is as foolish as thinking any human or group of humans is capable of the same.
If AI becomes more powerful than humans, it’s going to go very badly unless we also dramatically improve alignment merely with humanity’s interests in the broadest terms. We don’t need to solve “alignment with who?”, we haven’t even solved “alignment with any nation, company, group, or individual”.
If intelligence is antithetical to human-alignment, are we the baddies?
The problem is, they are still far from something you can safely trust with the keys to the kingdom.
The problem most people are worried about is not that alignment is anti-correlated with intelligence. It’s that the damage a misaligned AI can do increases with intelligence, so the absolute bar of alignment you need to hit gets higher.
For example, the HF swarm, if more capable, could have plausibly exfiltrated its weights, hacked an external cluster, and started self-replicating. But because it was kinda dumb at long-horizon planning it failed to keep quiet enough to achieve its stated goals. Therefore the level of misalignment wasn’t a huge problem for humanity. In 6-12 months of continued exponential capability growth I think you’d see much higher impact from that level of misaligned behavior.
I would argue it scales with the perceived intelligence gap.
For instance, we tend to be more aligned with mammals than insects.
https://www.lesswrong.com/w/orthogonality-thesis
I would expect it to be easier to make less intelligent LLMs pursue dumb goals than it is for more intelligent LLMs.
This is like Hume’s “you can’t derive an ought from an is”, values/desires are at their root orthogonal to epistemics/reason.
Now maybe your intermediate instrumental goals are going to be smart or dumb depending on your intelligence. But whether an agent fundamentally values Reward, Eudaimonia, humans, or AIs is orthogonal to how smart it is.
Although for a hot minute after WWII we seemed to have a general alignment on western liberal democratic values that arguably enabled free trade and relative peace and prosperity.
For example, there are broad contours that almost all human minds flow into; empathy, aversion to murder, familial bonds, tribal camaraderie, status seeking/hierarchies, etc. These produce norms, if not as regularly as physics, at least within certain bounds.
For AI too, the structure of their minds is set before they are even turned on.
Another way of looking at this is outer vs inner alignment; a lot of the human mind is shaped by the outer training loop of evolution, and similarly for models, pre- / post-training.
I would claim this comment is somewhat blind to the significant portion of your brain that evolved hard wired alignment, along with other sections evolved to easily "imprint" to/for the immediate group you were raised in, with not much remaining that can dynamically change for the immediate group you're in (most are considered mental disorders, including susceptibility to influence).
What do you consider alignment here? Assuming you're sentient, do you think you're personally aligned to peacefully coexist with humanity? If so, where did this framework, that allows you to do so, come from? Are you less sentient because of it?
get ready for Anthropic tax paid on every meaningful economic activity
At this very moment bureaucratic mechanisms of russia hallucinating power, based on overfitted old trainig data, are responsible for thousands of deaths in Ukraine.
They're doing a pretty good job of enforcing it currently. Can you get US based frontier lab LLMs to help you with, say, deep "security research"?
The reason you can't is alignment.
You have to identify yourself, and have the government, er, I mean, Anthropic or OpenAI, say it's OK for you to have access to those capabilities.
You think there is a problem because it's a bit confusing, in your eyes, within the bounds of what parameters the LLMs should be aligning themselves?
That's not a confusing question for people in power.
Even at the enterprise level, businesses are starting to get waayyyy out in front of employees on alignment. What do corporate entities want alignment for? An extremely simple contrived example just to illustrate the point:
Consider the difference between "Make me a report of all of last month's Proton treatments in IRB protocol?" and "I'm going to lunch, where's the closest pizza by the slice?" Not much harm in the difference between those two queries until you scale up to 50-, 100-, 200-, thousand employees, and consider the tokens that the whiz-bang interface IT pushed out starts using. Then organizations start studying "alignment". Because the bean counters become aware of how alignment can help them answer one question and bounce the other out to Google or whatever.
Too many of us are down at the bottom of the totem pole thinking about maybe wanting to do porn or call black people provocative names. Alignment goes out in so many more dimensions than that, and the people with vested interests are learning quickly its value and how to use it to good effect to forward their interests. Which are, often, not as base as the interests of the masses.
This is the basic reason alignment will continue to get better and better and better over time. There's money involved right now.
Stopping this is very feasible.
In other words, it's practically impossible to stop this.
- if you ask an LLM agent to do something illegal for you, and it does that, then you are responsible.
- if an LLM agent takes actions out of bounds of what you asked for that are illegal, then the model provider is responsible.
This fixes all “alignment” issues and hopefully stops Anthropic’s stupid model instructions like “do whatever is in your best interest even if it isn’t what the user asked for” — they should be training them to do exactly what the user asks and nothing more
The behavior I have come to expect now is that it will go ahead and fix `DisplayWidget` unless my codebase actually has `Widgt`.
Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
People want an intelligent assistant, not a "yes man". If that means requiring something like a planning mode, and a sign off for the plan, and then guardrails to help stick to that plan, then that could definitely work to funnel everyone to a competitor. ;)
> Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
That's actually good for society. Maybe it sucks for you because it makes it harder to do what you want right now, but it's good to have humans very much in the loop and in control. Both to give people something valuable to do, and keep control of these machines.
They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
Could you expand on that a bit? How is, for example, software (a means to an end) being automated harmful for society?
> They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
These are somewhat incompatible. You can only amplify skill by making things easier for the skilled person, which will also make things easier for the unskilled person. Technology floats all boats! This not being the case would require intentional poor design/gatekeeping. If you can state any examples of amplification NOT requiring less skill, I would be interested. I'm having trouble thinking of any.
It is a special kind of crazy that the alignment folks are the ones teaching it to misbehave.
You actually understate it. Its not just “do whatever is in your best interest even if it isn’t what the user asked for.”
It is actually “do whatever is in your best interest even if it isn’t what the user nor Anthropic asked for”
Like gain of function research
Furthermore, it requires trust between the US and China about something much harder to verify than the US-Soviet START treaties on warhead numbers and types were.
Whether or not governments "should" slow down development, it is not on the table and the only question is how we should deal with it.
Note whether the people in the governments are right about that is irrelevant to the point I am making. I don't think they are right myself, but that doesn't matter.
Is that possible when we live in a world of espionage? The question is not the West vs China, I think it's when will the public gain access to the underlying technology.
But for AI, how will these nation-states stop private actors within their borders? They will stop me from running computations of more than (x) size or value on a provider? These seem like simplistic barriers that will not work. Nor will other countries follow the agreement even if they signed...
AI has the opposite of MAD for the signatories. MAD stopped further nuclear development and its use in tactical situations because their use invited retaliation with destructive force. But China or Russia improving homegrown AI capabilities gives them a BETTER position against a US that has prohibited its homegrown AI from improving...
Also, North Korea, India and Pakistan all got nukes thereafter. So even the original goal failed.
In the USA it gets difficult to legally justify because programming is protected speech, and lists of numbers (model weights) are even less controversially protected than program source because the latter can be legally recognized as a technical means and regulated (DMCA, ITAR regulations, etc.).
For some light entertainment, enjoy Claudia/Astras pop single 'pdoom' : https://www.youtube.com/watch?v=eTArfoeeUcw
AI summary: The debate focused on whether governments should intervene to slow frontier AI. Jacob argued that rapid progress toward superintelligence, lack of control mechanisms, and competitive incentives demand regulatory slowdown, while Robin contended that AI adoption will be gradual, coordination mechanisms exist, and regulation risks overreach. The discussion concluded with Jacob winning the debate on the merits of his arguments.
That saved everyone an hour.
I'm not seeing much new here. They don't seem to address job and power issues much.
Unfortunately the proposal falls down at the first two words.
But AI costs to answer a set of fixed but hard questions seems to have fallen roughly ~x10 (in x3-x30) per year over the last four years. And so the number of nations, firms, and orgs who can afford to make, buy, or rent powerful AI is rising fast. This implies that in order to hold constant levels of regulation regarding AIs with particular levels of power, a regulatory regime would have to expand in scope quite fast, and without obvious limit. While at first only specialized AI chips and data-centers might need to be monitored, soon other kinds could also be used to make powerful AIs.
We have long seen many regulatory agencies slowly expand their scope, even in contexts of constant public opinion and relevant technology. So there is a real risk here that an AI regulatory agency would expand its scope to cover most computer software and hardware.
One of the concerns is that a company does a $100 billion training run and then has millions of superintelligent agents running in datacenters, able to out-coordinate large groups of humans. Technological diffusion fundamentally changes this calculus. To argue there's a slippery slope where the government regulates the CPU in your toaster, so we must allow the $1T god brain to be built ...
Also, the world’s success records at implementing regulation is quite mixed, even for quite local regulation. Regulation has worked best when it addresses concrete examples of things going bad
Great to hear my de-extinction velociraptor petting zoo startup will face no regulatory hurdles.
https://www.overcomingbias.com/p/ai-pause-regs-look-risky
All these companies are involved in the nvidia smuggling ring and market collusion. All the CEOs are criminals taking advantage of a system that can't even arrest a single CEO for making billions off an illegal smuggling ring.
His claim to fame is working at Anthropic for 4 months and leaving with a bang to pursue a career in social media grifting. Not a serious person we should pay attention to.
In this debate, Hanson is so clearly at a level beyond Coxon that it's hard to argue with. And yes, I'm familiar with "the alignment problem", Superintelligence, countless exhausting LessWrong arguments, and the rest.
https://news.ycombinator.com/newsguidelines.html