If AI Writes All the Code, What Do the Programmers Do?

Eight months ago I was producing roughly 90% human written code and 10% AI written code. This has switched surprisingly rapidly and now my code is probably 90% AI. So what do I do all day?

I’ll go through a change that I made to a matrix-multiply kernel, for which I’ll sadly have to be a bit vague. Since GPUs are now giant matrix-multiply chips, where north of 96% of the flops are in the tensor cores (latest Nvidia GPUs have 2250 tflops in bfloat16 matmuls, compared to 75 tflops for everything that’s not a matmul), you’d think that they’d make it easy to use all those flops. But no, matrix multiply kernels are giant crazy beasts that are incredibly tricky to get right. Some quick googling finds this explanation on Nvidia hardware and this one on AMD hardware. Just open those two and scroll down both to get a feeling for how much work is involved. (and yes, these implement the simple O(n^3) matmul loop where you iterate along repeatedly and multiply and add all the numbers)

The particular kernel I was optimizing was using Cluster Launch Control (CLC), a complexity that’s covered in neither of the blog posts linked above, and it had some bad interactions with some particular inputs. The resulting change was 95% written by AI. So I’ll just go through what I did. AI asks are bolded:

Read the rest of this entry »