I’m Getting a Whiff of Iain Banks’ Culture
by Malte Skarupke
The US has been acting powerful recently and it reminded me of this question: What does it feel like to fight against a powerful AI? Not for normal people for whom there’s no difference between competing against a strong human or a strong AI, (you lose hard either way) but for the world’s best humans. We got a sense of the answer before LLMs were a thing, when the frontier research labs were working on game RL:
Fighting against a powerful AI feels like you’re weirdly underpowered somehow. Everything the AI does just works slightly better than it should.
If you’re not a strong human player, the closest feeling is when you play a game with lots of randomness against a really strong player. It will appear as if that strong player just keeps on getting lucky somehow.
I’m getting a similar sense for the recent US foreign interventions and wars. They all seem to work slightly better than they should. It finally clicked for me when Dario Amodei said “This technology can radically accelerate what our military can do. I’ve talked to admirals, I’ve talked to generals, I’ve talked to combatant commanders who say this has revolutionized what we can do.”
The things I’m referring to are the raid that captured Maduro in Venezuela (Claude was used), the current war with Iran (Claude was used), the killing of a drug boss in Mexico (unclear if AI was used but US intelligence helped Mexico).
The commentators in the AlphaGo match with Lee Sedol didn’t know what to make of most games. The AI wasn’t doing anything obviously brilliant, there were lots of little fights all over the board where the outcome wasn’t quite clear, but they just all worked a little better for AlphaGo than expected. So gradually Lee Sedol’s position changed from “this is tough, hard to tell how this is going but at least I’m feeling good about these areas” to “hmm I’m struggling, maybe I’m a bit behind but it’s not clear” to suddenly “oh I lost”.
I don’t know Go, but I got a clearer sense from the StarCraft 2 matches. In some skirmishes the AI would take damage, in others the human would. But somehow it always felt like the human was in more trouble. In some fights the human clearly came out ahead but then mysteriously just one minute later the AI had a clear advantage. It was able to quickly recover and constantly put pressure on the human. It all looked very stressful, because even when you think you do well as a human, it works out a little less well than expected and whatever the AI does works a little better than expected.
And where have we seen this pattern before? In sci-fi of course. In particular I’m thinking of Iain Banks’ Culture, the ostensibly human civilization that’s actually run entirely by AIs. Alien civilizations keep on wanting to pick fights with them for reasons and keep on being surprised by how hard the harmless-seeming Culture can whoop your ass if you make it mad.
I always thought of the Culture as closest to the European Union: Seemingly harmless but if anyone ever picked a fight with them, they’d find out that the EU can get its act together very quickly and can very quickly stand up the strongest army in the world. But obviously the real EU has never come close to the Culture because nothing human ever comes close to the potential of AIs. It would be as if Russia picked a fight with Poland, gained ground for a week, feeling good, only to suddenly find all of its IT systems hacked and access to nuclear bombs revoked, bombs dropping on Moscow the next day and an army in Moscow another two days later. The Culture takes a week to get its act together and then whoops your ass so hard you don’t even know what’s happening.
But now I’m getting a whiff of the power of the Culture for the first time, and it’s from the US. Going into another country, kidnapping their leader and getting away with it is exactly the kind of overpowered move that the Culture would be able to pull off. Bombing cities all over Iran, knocking out the entire leadership within two days, while the air-defense systems supplied by China do absolutely nothing is another example. If this was a video game these would be strategies done by high level players, but they’re not supposed to work that well.
It would be foolish to think this is entirely due to AI. The US had a high-tech advantage for a while. Turns out the F-35 is actually good. But even a couple years ago the US regularly messed up when it tried to do operate at high precision. We saw in Iraq and Afghanistan that being overpowered doesn’t work out as well in practice as it does in theory. So I think AI is the most likely candidate for the shift to “it worked better than it should have.”
So how specifically do you get to a point where everything works slightly better than it should? We saw two different approaches in Go and StarCraft 2:
- In Go the AI was having little fights all over the map, in a way that combined to a few extra pieces at the end. It would defend a little bit here, attack a little bit there. It was able to keep the overall picture in its head, not feeling the pressure to resolve things too early. (I haven’t played Go, but I know I get frustrated in strategy games if I have to deal with multiple fights in different parts of the map at once)
- In StarCraft 2 we saw the same thing, but we also saw that the AI could have perfect micro when it counts, like playing with wounded stalkers in the frontline because it could get them out of danger just in time. Humans could also do that in theory but in practice you can’t quickly click perfectly like that.
So the two angles are “having a better high-level view” and “having better micro control.”
Another source of success for the Culture is that they’re over-prepared for fighting. (not for their first big war, but in later books) And this is also part of the story we hear in Iran. Normally there’s just too much going on in the world and you can’t possibly keep track of all of it. Famously the US had prior intelligence on 9/11 but didn’t really put the pieces together. (there’s a whole Wikipedia article about it which has phrases like “Rice listened but was unconvinced, having other priorities on which to focus.”) But AI has almost no limits of what it can keep track of. You can always spin up another agent. So when something important comes up, chances are that some AI was keeping track of it and can raise an alert. You’ll never miss opportunities just because you had other priorities to focus on.
So the third angle is: Being over-prepared because you can follow up on many more things at once.
What does all of this mean for the world? It means we’re in a weird temporary phase where one country has control of a game-changing technology while others are not far behind (sadly not the EU. I’m thinking of China, especially with H200s). You get to play at a higher level, but only for a short time and only in specific ways. In a year others will have caught up, but by then you’ll have new capabilities that you didn’t have a year ago. If this was a game you’d saturate at some point (you just can’t play StarCraft that much better than the best humans), but in real life the game keeps on changing. New pieces keep on coming into play and the old pieces become irrelevant. You can’t do this for long before the humans become irrelevant to the outcomes, and then you’re fully in Culture territory. I personally wouldn’t mind living in the Culture, but it seems scary to rush towards it without a good plan for how we’ll survive the transition.
I don’t have a good angle for working on that plan, maybe others do. For now my contribution is just to point out that we seem to be in the early stages of overpowered AI, and to make people notice what that feels like.
Probably Dance 
I disagree entirely. Claude is not the same as AlphaGo.
Claude is deceptively good but in the end it too just copies existing things poorly. I know this because I tried to make it do a programming task that was simple but novel and it failed twice despite me guiding it every time it made a mistake. Then I gave it a prompt with all the things learned and failures to avoid. It thought for 4 hours and then still failed.
I was fooled for a while but I have to conclude that LLMs are for information retrieval. When used for generating things, they are at best legal copyright infringement.
Strategies that LLM comes up with will not be good, they will be average at best and seriously lack thinking.
To illustrate, here’s a puzzle that fools most of the models on the market: In a turn-based game you can buy Drones and Wild Drones. Drones cost you 3 gold and produce one gold per turn. Wild Drones cost two and produce similarly but skip production on their first turn. Which one is better?
Every model I tried except Claude and GPT 5 on xhigh will draw you a table of net gold over multiple turns and then conclude that Drones are always better because you can reinvest earlier! Some even wrote some differential equations to support their obviously wrong conclusion.
I don’t have a similarly concise failure for Claude 4.6 but I do think that it is similarly incapable of reasoning based on my extensive use of it.
I agree with your sentiment at current capabilities. (I don’t agree that it just copies existing things. I have seen it write plenty of novel code) And clearly the Iran war turned out pretty bad for the US in the end. But already I am sometimes getting a feeling of being weirdly underpowered when dealing with AI. Like I’ll ask it to write a script for me to help debug something and it writes it in ten seconds and does an unnecessarily good job at it, with a progress indicator and useful log messages. I could have written the same useful script in a few minutes (it was small) but I wouldn’t have done an unnecessarily good job at it. Maybe my progress indicator would have just been a dump of log messages, but not an in-place updating one. You just don’t do that on a little debug script. But if it just takes seconds, then sure, why not? Go ahead and add it without even being prompted to do so.
I expect this to happen more often soon, to the point where in a few years it’ll be rare for a human programmer to type out actual code.
Do I have it write most of my code right now? Maybe for side projects, but not for things that I plan to maintain. Even if I do let it write a good chunk of code, I always have to go over it to de-slopify it.
Novelty is pretty hard to check because unlike AI, people have not seen the entire internet. But when you give a task that is obviously fresh, the performance is usually pretty bad.
I agree that the results can be in some sense unreasonably polished. It is kind of a big downside because I think AI is best for exploratory work and there the completeness hurts by making the code harder to change. The worst thing is that LLMs tend to handle errors and do a bad job about it that hides details rather than simply crashing.
I’m not sure if I want to let AI work and then de-slopify. Writing code quickly is mentally taxing but so is paring down code to 1/2 the size, especially when you aren’t sure if it is even correct.
What concerns me most is that there is no chance of inventing a beautiful way to solve the problem when going AI first. All your mental capacity will be tied to dealing with the slop. You can make the slop maybe 4x less LoC but you are missing out on approaches that are even shorter or are easy to prove correct.
I used to program so that I solve the problem first and then look up solutions online and compare. That created a nice feedback loop. But if I am primed by an AI solution, it is harder to come up with an original solution afterwards.