Every new model launch brings the same panic. I have watched it four times now. The panic is "this one will replace me". I have not shared it once. Let me tell you why.
What the panic gets wrong
The panic assumes the job is the same as the model. A new model comes out that is better at the model part. Therefore the job is gone.
But the job is not the model. The job is the loop. The loop is: get a request → figure out what is actually being asked → set up the context → pick a model → ask → check the answer → ship the result. The model is one of six steps. A better model makes one step cheaper. It does not replace the loop.
If you are selling the loop, the better model is good news. If you are selling the model — "I am the person who runs GPT-N" — then yes, you are in trouble. But that was always a strange job.
What I am actually worried about
What I am actually worried about is not a smarter model. It is the opposite. It is that I get attached to the model I already know, and stop being curious about the new one. It is that my "default" calcifies, and the gap between my default and the best available tool widens month by month.
The way I deal with this is the same as the way I deal with any competence rot. Once a quarter, I run a small experiment. I take a task I do every day, and I do it on a model I do not usually use. I score the result. Sometimes the new model wins. Sometimes the old one is still better for the task. The point is that I know which, and the answer is not "the one I started with".
What to do when a new model comes out
Here is the protocol. It takes about 2 hours. It is the same protocol as my cheap-AI essay.
- Take 5 tasks you do every week. Make sure they are real tasks, not tests.
- Run each on the new model and on your current model. Same prompt, same input.
- Score both on a 5-point rubric you wrote before you saw the answers.
- Note the cost. The new model might be cheaper, or more expensive, or have a quota.
- Decide, for each task, whether to switch.
If the new model wins on 4 of 5 tasks, switch. If on 1, keep the old one. If on 3, switch for those 3 and keep the old one for the others. This is a per-task decision, not a per-model one.
The thing I keep coming back to
The job of an operator is to make decisions about which tool to use, when, and how. That job does not disappear when a new model comes out. It gets more important. The set of available decisions grows. The value of being good at the decision grows with it.
This is why I do not panic. Panic is what happens when someone is selling a model, not a decision. I am selling a decision. The market for decisions is fine.