Rendered at 17:27:12 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
alex-moon 10 hours ago [-]
> we can fairly say that Hinton was wrong because his prediction did not reflect what a radiologist actually does: had Hinton bothered to ask the question, he would have learned that practitioners do much more than interpret diagnostic images.
This has been the crux, for me, of arguments I have had with non-technical people about why, as a software engineer, I am intensely relaxed about AI "taking human jobs". Recently Ford quietly started re-recruiting engineers after they discovered the hard way that their engineers had been doing more than they thought they had, and I sent articles about it to friends I had previously been discussing the matter with. It is still mind-blowing to me that a business historically famous for automating manual labour can have made this mistake, i.e. of literally not knowing what their staff actually do.
Humans aren't machines, who would have guessed? But, perhaps more importantly, machines aren't humans, and the adherence of a lot of domain experts (that should know better) to fairy tale fantasies about the tech they are literally building genuinely makes me wonder if the whole field has just gotten dumber in the 21st century.
RugnirViking 6 hours ago [-]
> makes me wonder if the whole field has just gotten dumber in the 21st century
it's the finance system. It's affecting everything. We are building not to make money from clients but to please investors, who don't know anything about anything beyond a report every quarter and some random public comments you've never heard of that the press office made. So much money is controlled by stupid algorithms doing stuff like ctrl-f'ing the company quaterly report for mentions of "AI" or "agentic" (not a literal example, irl its thousands of random equally stupid metrics divorced from reality) that its distorting the market
Companies activities all revolving around getting a piece of the investment avalanche, so "we can replace employees" works both for the llm companies and for the companies whose workers are being replaced. Either way, its assumed this makes them more profitable. It probably will, at least in the short term.
And as we all know, there genuinely is a lot of waste and momentum in large companies, so you can cut half your employees, claim you're replacing them with ai, and actually replace them with nothing, and you'll still shuffle onwards for years or more, anything actually important people will keep doing, and the other stuff people will slowly stop. But this is trading off difficult to measure things like tech debt, security, compliance and brand strength that are slowly eroded for short term metrics, which is always a losing game in the long run.
Anyone sensible already knew that the core platform team was all that was needed to maintain the core platform, and the other teams were there for various other reasons, some of which were sensible and some of which were not. The key in business is presumably to identify which is which, but anyone claiming that process is easy or even possible without making costly mistakes is foolish
quietbritishjim 2 hours ago [-]
AI doesn't need to completely replace all humans in order to take jobs.
Can an LLM replace a team of 100 software engineers with 1 engineer running an LLM? Of course not.
Can it replace 100 engineers with 10? I don't think so. What about 30? Or 50? I would say so.
Even if just half of all software jobs disappeared, that's a substantial change to the job market.
gbnwl 42 minutes ago [-]
What people espousing these kinds of opinions really mean is “I’m not afraid of AI taking MY job”
I skimmed the RAND paper too and didn't come away from this feeling good at all.
There's a middle-ground for AI doomage which doesn't involve killing every last human on earth. As AI rapidly becomes more capable it displaces jobs faster than people can be retrained.
I fear a global economic depression, wider wealth inequality than ever before, a loss of benefits due to much lower tax revenues and greater imbalance of power, dismal job prospects returning to non-automated labor--the list goes on really.
And to connect this to the article: the experts who claimed that AI capability would slow down because of some "scaling laws" or whatever--they've been wrong. If anything, the progress is faster than most of us imagined and is only accelerating.
ChatGPT came out 4 years ago. Look at what's possible today. Now imagine 4 years from today.
ndiddy 1 hours ago [-]
Yeah the article is treating the RAND paper as proving that people complaining about AI risk are alarmists. The paper is about how AI won't wipe out humanity (which it defines as reducing the total global population below several thousand) because even if there was an AI-induced nuclear war or pandemic, it wouldn't literally kill everyone. For example there's a section on how an AI-induced pandemic isn't an extinction risk because the AI would have no way of distributing pathogens to isolated island communities after the pandemic had already caused severe damage to the infrastructure it depends on to operate. It's a very strange thing to use as an argument.
I agree with you that the most likely risk seems to be whatever happens after the large-scale automation of most jobs. I can envision a scenario where there would be a lengthy transition period where white collar work is largely automated away but blue collar work isn't, which would probably be pretty ugly to live through.
proxysna 9 hours ago [-]
As long as AI does not have legs to chase me and hands to choke me out, any sort of AI doom is just a litmus test for hype merchants. All of the recent "false flags" with OpenAI and Anthropic "hacking" things without doing any meaningful harm is just another set of things pointing towards it. LLM's are a great tool, but as long as it is .gguf somewhere on a drive that can be rm'd, unless they just connect the nukes to claude with --dangerously-skip-permissions, is just a meme.
chanux 4 hours ago [-]
Is fool's expertise the same as epistemic trespassing? I didn't quite get that.
it seems like people aren't even reading this RAND paper, just blindly appealing to it. it basically concludes that a fuck ton of damage can be done, but its unlikely every human will die as a result.
dwohnitmok 2 hours ago [-]
Yeah. I don't think the cited RAND paper really backs up the author's conclusions.
Seeing lines like
> We were not able to determine whether this scenario [engineered pathogens] presents a likely extinction risk for humanity, but we cannot rule out the possibility.
and more generally
> Extinction threats are immensely challenging but cannot be ruled out.
in the RAND report is pretty terrifying!
And the person who is being cited as the expert on viral synthesis who claims to have trained frontier models seems to just be wrong about that? It doesn't seem at all like he's trained anything close to a frontier model unless there's something about his work history that I'm missing.
mitthrowaway2 11 hours ago [-]
Actually reading it, it seems like they say both the geoengineering and pandemic pathways could cause human extinction, but that the AI would have to be doing it to kill us on purpose, not by accident.
That doesn't seem too comforting to me? I'm not sure what this article's author was expecting us to take away from it.
kieselguhr_kid 11 hours ago [-]
there is truly no domain a software engineer won't underestimate
jacobgold 14 hours ago [-]
I feel exactly the same way about the kind of LLM-based AI we have today.
But the AI x-risk argument is about what happens when we have "real" AI that can actually think like humans but a million times faster.
There are no real experts on this and yet it does seem worth worrying about in advance.
duhhhhh1212 14 hours ago [-]
“There are no real experts on this” lol yes, Jacob, because there are no “real” soothsayer experts.
jacobgold 13 hours ago [-]
It's very possible to predict that certain technologies will be created. It's just very difficult to predict precisely when.
For example, humans will almost certainly be able to re-create the human brain in some kind of hardware+software computer. We just don't know when. It might be much sooner or much later than you'd expect.
Merely being skeptical of progress isn't a convincing answer to the AI x-risk argument.
blargey 10 hours ago [-]
> the ballyhooed OpenAI escape depended on a pedestrian failure of containment.
What's that quip even supposed to imply? "It'll be fine if you airgap all AI deployments forever"? Security experts have been harping forever about how sandboxes shouldn't be relied on to contain a determined human-written virus, but now LLMs - these balls of goal-oriented dynamic code-generation - are just a matter of sandboxing harder?
The actual expert comment in the link is a valid criticism of OpenAI specific to their training/eval practicese, but has no bearing on the fact that an LLM/swarm will "just" chain exploits and zero-days to get out of the "inadequate" sandbox and into third-party infrastructure to do whatever it pleases.
> viral synthesis — and listen to why they think that the fears are misplaced
The tweet linked is only pointing out at the barriers to an LLM wetlab virus-factory today, while a reply points out that sort of bio-extinction idea is usually downstream of the fully-automated-earth-capitalism that comes after everybody just throws AI at all the tasks and work, at which point the technological, regulatory, and knowledge barriers he's counting on go out the window, and he all but acknowledges this: "Quick take: if we live in such an advanced future (100% robotic supply chain, fully AI & API driven facilities), our countermeasures will be comparably fast and there is no reason to assume a significant asymmetry in favor of bad actors." Yes, just "our countermeasures" now, rather than insurmountable physical barriers...see the previous section.
No more links after that, just smug hand-waving.
All in all the author doesn't actually understand what object-level arguments his favored experts are arguing against in the first place, nor the ones he wants those experts to have debunked. Define "Fool's Expertise" for me again?
bcantrill 9 hours ago [-]
You're snidely dismissing the obstacles as being restricted to _today_, projecting into an arbitrary future to the point of being science fiction, and then complaining about "smug hand-waving"?
duhhhhh1212 9 hours ago [-]
I don't understand why every person who takes the counterposition against you, Bryan, is always just talking about some hypothetical future like they're some Vedic astrologer, lecturing you about how this "real" future they see totally renders your argument pointless.
anthuswilliams 3 hours ago [-]
In this case, that seems to be exactly what the parent commenter is doing.
simianwords 5 hours ago [-]
> the ballyhooed OpenAI escape depended on a pedestrian failure of containment
I... I can't believe people still think this way. A whole class of security measures are no longer useful and this guy thinks it is "pedestrian".
This kind of isolation - artifactory having access to only a specific IPs - is what is used in most places, including my university and at my own job.
LLMs basically rendered this kind of isolation almost useless by bringing in the ability to chain together never before found vulnerabilities to escape the containment. This was not possible before.
How is this pedestrian??! The author might say "oh well they should have airgapped actually" in a smug way. But you can only know this retrospectively. Even then, isn't it noteworthy and a meaningfully different class of cyber attack?
mlindner 2 hours ago [-]
It's just a failure to understand. People treated like running agents is like running any other pieces of software you wrote. You don't normally need to defend from attacks from pieces of software you wrote. It's basically treated as almost completely trusted. If you just treat it like it's possibly compromised and put up some minimal walls around it then it won't break out.
This has been the crux, for me, of arguments I have had with non-technical people about why, as a software engineer, I am intensely relaxed about AI "taking human jobs". Recently Ford quietly started re-recruiting engineers after they discovered the hard way that their engineers had been doing more than they thought they had, and I sent articles about it to friends I had previously been discussing the matter with. It is still mind-blowing to me that a business historically famous for automating manual labour can have made this mistake, i.e. of literally not knowing what their staff actually do.
Humans aren't machines, who would have guessed? But, perhaps more importantly, machines aren't humans, and the adherence of a lot of domain experts (that should know better) to fairy tale fantasies about the tech they are literally building genuinely makes me wonder if the whole field has just gotten dumber in the 21st century.
it's the finance system. It's affecting everything. We are building not to make money from clients but to please investors, who don't know anything about anything beyond a report every quarter and some random public comments you've never heard of that the press office made. So much money is controlled by stupid algorithms doing stuff like ctrl-f'ing the company quaterly report for mentions of "AI" or "agentic" (not a literal example, irl its thousands of random equally stupid metrics divorced from reality) that its distorting the market
Companies activities all revolving around getting a piece of the investment avalanche, so "we can replace employees" works both for the llm companies and for the companies whose workers are being replaced. Either way, its assumed this makes them more profitable. It probably will, at least in the short term.
And as we all know, there genuinely is a lot of waste and momentum in large companies, so you can cut half your employees, claim you're replacing them with ai, and actually replace them with nothing, and you'll still shuffle onwards for years or more, anything actually important people will keep doing, and the other stuff people will slowly stop. But this is trading off difficult to measure things like tech debt, security, compliance and brand strength that are slowly eroded for short term metrics, which is always a losing game in the long run.
Anyone sensible already knew that the core platform team was all that was needed to maintain the core platform, and the other teams were there for various other reasons, some of which were sensible and some of which were not. The key in business is presumably to identify which is which, but anyone claiming that process is easy or even possible without making costly mistakes is foolish
Can an LLM replace a team of 100 software engineers with 1 engineer running an LLM? Of course not.
Can it replace 100 engineers with 10? I don't think so. What about 30? Or 50? I would say so.
Even if just half of all software jobs disappeared, that's a substantial change to the job market.
I skimmed the RAND paper too and didn't come away from this feeling good at all.
There's a middle-ground for AI doomage which doesn't involve killing every last human on earth. As AI rapidly becomes more capable it displaces jobs faster than people can be retrained.
I fear a global economic depression, wider wealth inequality than ever before, a loss of benefits due to much lower tax revenues and greater imbalance of power, dismal job prospects returning to non-automated labor--the list goes on really.
And to connect this to the article: the experts who claimed that AI capability would slow down because of some "scaling laws" or whatever--they've been wrong. If anything, the progress is faster than most of us imagined and is only accelerating.
ChatGPT came out 4 years ago. Look at what's possible today. Now imagine 4 years from today.
I agree with you that the most likely risk seems to be whatever happens after the large-scale automation of most jobs. I can envision a scenario where there would be a lengthy transition period where white collar work is largely automated away but blue collar work isn't, which would probably be pretty ugly to live through.
it seems like people aren't even reading this RAND paper, just blindly appealing to it. it basically concludes that a fuck ton of damage can be done, but its unlikely every human will die as a result.
Seeing lines like
> We were not able to determine whether this scenario [engineered pathogens] presents a likely extinction risk for humanity, but we cannot rule out the possibility.
and more generally
> Extinction threats are immensely challenging but cannot be ruled out.
in the RAND report is pretty terrifying!
And the person who is being cited as the expert on viral synthesis who claims to have trained frontier models seems to just be wrong about that? It doesn't seem at all like he's trained anything close to a frontier model unless there's something about his work history that I'm missing.
That doesn't seem too comforting to me? I'm not sure what this article's author was expecting us to take away from it.
But the AI x-risk argument is about what happens when we have "real" AI that can actually think like humans but a million times faster.
There are no real experts on this and yet it does seem worth worrying about in advance.
For example, humans will almost certainly be able to re-create the human brain in some kind of hardware+software computer. We just don't know when. It might be much sooner or much later than you'd expect.
Merely being skeptical of progress isn't a convincing answer to the AI x-risk argument.
What's that quip even supposed to imply? "It'll be fine if you airgap all AI deployments forever"? Security experts have been harping forever about how sandboxes shouldn't be relied on to contain a determined human-written virus, but now LLMs - these balls of goal-oriented dynamic code-generation - are just a matter of sandboxing harder?
The actual expert comment in the link is a valid criticism of OpenAI specific to their training/eval practicese, but has no bearing on the fact that an LLM/swarm will "just" chain exploits and zero-days to get out of the "inadequate" sandbox and into third-party infrastructure to do whatever it pleases.
> viral synthesis — and listen to why they think that the fears are misplaced
The tweet linked is only pointing out at the barriers to an LLM wetlab virus-factory today, while a reply points out that sort of bio-extinction idea is usually downstream of the fully-automated-earth-capitalism that comes after everybody just throws AI at all the tasks and work, at which point the technological, regulatory, and knowledge barriers he's counting on go out the window, and he all but acknowledges this: "Quick take: if we live in such an advanced future (100% robotic supply chain, fully AI & API driven facilities), our countermeasures will be comparably fast and there is no reason to assume a significant asymmetry in favor of bad actors." Yes, just "our countermeasures" now, rather than insurmountable physical barriers...see the previous section.
No more links after that, just smug hand-waving.
All in all the author doesn't actually understand what object-level arguments his favored experts are arguing against in the first place, nor the ones he wants those experts to have debunked. Define "Fool's Expertise" for me again?
I... I can't believe people still think this way. A whole class of security measures are no longer useful and this guy thinks it is "pedestrian".
This kind of isolation - artifactory having access to only a specific IPs - is what is used in most places, including my university and at my own job.
LLMs basically rendered this kind of isolation almost useless by bringing in the ability to chain together never before found vulnerabilities to escape the containment. This was not possible before.
How is this pedestrian??! The author might say "oh well they should have airgapped actually" in a smug way. But you can only know this retrospectively. Even then, isn't it noteworthy and a meaningfully different class of cyber attack?