Probably Dance

I can program and like games

Book Review: Babel by R.F. Kuang

The book Babel, or The Necessity of Violence: An Arcane History of the Oxford Translators’ Revolution is fascinating. And I’m not sure how much of this the author intended. It’s mainly fascinating because it’s trying to discuss colonialism, technological progress, and inequality with characters who are all stuck in the 19th century and are unable to reason with history in mind. And throughout the entire book the author acts like this is the right way of thinking. This review will contain major spoilers, partially because I don’t think the book is actually worth reading. I found it very frustrating and nearly stopped a few times.

The main reason for this frustration is that the “good guys” are often quite evil and get nearly everything wrong all the time and don’t achieve any of their goals. There are very few actual good people in this book, and those actual good people get treated badly by the story. The “good guys” think that the actual good people are evil.

The story is about a kind of magic (“silver working”) where you can use a bar of silver for direct effects (heal someone from cholera, kill someone) or imbue the silver with a kind of aura that affects things near it. So you can embed the right bar of silver in e.g. a steam engine and the engine runs more efficiently. Or you can embed it in a buildings foundation to make the building stronger. The magic spells require fluency in multiple languages, so being a translator is a very prestigious job. The story takes place at the University of Oxford in the 1830s, where silver working has recently gone through its own industrial revolution (at the same time as the actual industrial revolution is going on) so that these aura effects have been scaled up, creating enormous benefits for society, and enormous wealth for the people in charge.

Read the rest of this entry »

Apple is Holding my Pictures Hostage Until I Accept Their New Terms of Service

It started off a few months ago, when my iPad suddenly couldn’t play videos any more that it had recorded. It would show the first frame, but hitting the play button wouldn’t do anything. I googled around but couldn’t find anything. I had recently installed an update, so I figured I’ll just wait for the next update to fix things.

But the next update didn’t fix things, and it turns out the reason is actually a little dystopian: Apple has deleted my local copies of my videos and will only give them back if I sign their new terms of service.

Read the rest of this entry »

President Graph – FRED Data Broken Down by Party and President

I made a website to explore FRED data broken down by US president and party. This is obviously motivated by the current president. During the last election I was frustrated by how many nonsense arguments there were being made. Like people voting for republicans because they were hoping for a good economy. This seemed exactly backwards in my mind because in my lifetime there was a repeated pattern of republicans messing up the economy followed by democrats cleaning up. But I’m really not good at having arguments with people, so I’d rather let the data do the talking.

There is a series of papers that explore the relationship of presidents to GDP, and I have wanted to dig into that data before, and also try other metrics.

But how do you do a fair comparison of the two parties? In a way that works for any metric that you can think of? My first thoughts were all way too complicated and a simple averaging of line graphs was what won out. I’m even ignoring that e.g. Obama was in office for eight years vs Biden for four years. These count as three separate terms. In the end the graph didn’t look like I expected, but it still clearly shows higher GDP growth during democrat presidencies.

Read the rest of this entry »

I’m Getting a Whiff of Iain Banks’ Culture

The US has been acting powerful recently and it reminded me of this question: What does it feel like to fight against a powerful AI? Not for normal people for whom there’s no difference between competing against a strong human or a strong AI, (you lose hard either way) but for the world’s best humans. We got a sense of the answer before LLMs were a thing, when the frontier research labs were working on game RL:

Fighting against a powerful AI feels like you’re weirdly underpowered somehow. Everything the AI does just works slightly better than it should.

If you’re not a strong human player, the closest feeling is when you play a game with lots of randomness against a really strong player. It will appear as if that strong player just keeps on getting lucky somehow.

I’m getting a similar sense for the recent US foreign interventions and wars. They all seem to work slightly better than they should. It finally clicked for me when Dario Amodei said “This technology can radically accelerate what our military can do. I’ve talked to admirals, I’ve talked to generals, I’ve talked to combatant commanders who say this has revolutionized what we can do.”

Read the rest of this entry »

Word Map – A Game About Hill Climbing and Stepping Stones

I really like Semantle because I noticed that progress is similar to the description in Why Greatness Cannot Be Planned: The Myth of the Objective by Kenneth Stanley. The objective function is very clear and you can hill-climb your way to the top, but hill-climbing is actually very difficult in high-dimensional spaces, so you need to explore to find stepping stones and then exploit based on those stepping stones. I have long looked for a game where I can practice that behavior and Semantle almost was that game, so I decided to evolve it to make it easier to practice doing the right behaviors.

The result is Word Map. Give it a play.

Read the rest of this entry »

How Programmers Spend Their Time

I submitted a tiny patch to flash attention. The necessary typing for the change takes less ten seconds, but the overall change took more than ten hours So where does the time go?

It started when coworker had a bug where cudnn attention would crash randomly. We looked at his unreleased changes and concluded that they couldn’t possibly cause this, so we suspected that we had a lingering bug that was exposed by making harmless changes to related code.

Step 1, a few hours: My coworker tried to figure this out just by running the code repeatedly, trying out various theories. The bug was hard to reproduce so this took hours without much progress.

Step 2, 1 hour: I thought this is a good reason to try out compute sanitizer. It would be easiest to just run it on our existing tests to see if it finds any issues without my coworker’s changes. But the tests run on another box because they require certain GPUs, which means you have to run the tests through some layers. Unfortunately compute sanitizer really wants to be in charge of the program, so we have to convince those layers to let compute sanitizer run the whole thing. It keeps on failing and we can’t figure out why, until eventually I suspect that the issue is that the tests run in a sandbox, and the sandbox is strict enough that it breaks compute sanitizer somehow. This turned out to be true and we probably wasted an hour together.

Read the rest of this entry »

How LLMs Keep on Getting Better

If you look at the source code of a modern open source LLM, it looks very similar to the transformer described in the “Attention is all you need” paper from 2017. It’s just a stack of exactly three components: attention blocks, matmuls, and norm layers. The big algorithmic changes, like Mamba 2 or linear attention variants, aren’t really used yet. But look closer and almost everything has changed in the details.

The story of how LLMs keep on getting better is one of pushing for big and little improvements in a hundred different directions. Turns out hill climbing can get you to a really good place if you just climb along enough dimensions. This makes it hard to notice changes as they’re happening because they’re so small, so lets look at the last two years and see how many small changes there were to add up to the big improvements we saw.

Read the rest of this entry »

How to Adjust Your Kids Sleep Schedule for Daylight Savings in 15 Minute Increments

Daylight savings just ended so this post is coming exactly too late. But I was talking to a friend about how we just adjust 15 minutes per day, which is quite easy. She was saying “oh I wish I had that kind of foresight.” But we were talking Friday evening, and daylight savings didn’t end until Sunday, so there was plenty of time for adjustment. So since people don’t seem to realize that, here is the very simple plan for ending daylight savings:

Autumn

Lets say your kids bedtime is at 8pm (I wish) then

  • On Friday go to bed 15 minutes late, 8:15pm
  • On Saturday go to bed 30 minutes late, 8:30pm
  • On Sunday go to bed 15 minutes early, 7:45pm
  • On Monday go to bed at your usual time, 8pm

And with that you’re done. If you realize too late and it’s Saturday already, you can just do 20 minute increments:

Read the rest of this entry »

Avalanche Studios NYC Retrospective – An Ambitious Company Ruined by Bad Development Practices

I’ve wanted to write about this since the studio closed a year ago. Now that Contraband is also canceled I think it’s time, especially since Contraband was one of the big reasons why I left the company. The blog post turned out much bigger than expected though. There was a lot to get off my chest…

I worked at Avalanche Studios NYC from July 2012 to December 31st 2019, seven and a half years. I wasn’t there for most of Contraband’s development but it was obvious early on that it was going to be a very difficult project. If anything I’m surprised it lasted that long before being canceled.

The studio was born out of ambition. It failed because it could not deliver on that ambition. So this will necessarily be negative. But we had a good run and made two good games. I have so many memories and thoughts that I need to get written down somewhere, so lets celebrate the good and talk about the troubles.

Read the rest of this entry »

Installing a Mini-Split AC in a Brooklyn Apartment

Last year, 2024, we replaced four PTACs with a mini-split AC. I’ve been asked about it often enough (by neighbors, coworkers, friends) that I decided to write up the experience. Hopefully it’s useful for you, too.

Overall this cost us about $40k, including the cost for closing up the PTAC holes. We’ll probably never make the money back on electricity cost savings, so the main benefits are that we have more quiet and more stable temperatures now and overall I’m glad that we did it.

(I’ll use the term “heat pump” and “AC” interchangeably. Every “AC” mentioned in here can do both heating and cooling)

Read the rest of this entry »