From a startup CTO: "Our evals test suite costs us about $50 to run. We run it to catch regressions, but of course not part of CI becaue of this." What a big shift: I used to think about CI/CD cost in time to run, and perhaps machine costs. But now we can add token costs!
52,73K