Do Agent Skills Matter?

There’s an ongoing discussion about what to do with Agent skills. Last year, there was a flurry of development around skills. People shared them widely on social media. In the blockchain security world, auditors would share massive dumps of skill files, claiming that they contained all of the secret knowledge distilled from hundreds of public audit reports.

At the same time, I don’t think anybody has benchmarked these things. Developing good benchmarking software for machine learning evaluations is really hard generally. Testing against something like Claude Code is even harder, as the agent is trained to learn about you in particular and on your data generally. There are few constants in the equation, and everything drifts.

I see a lot less emphasis on skills now. Where have all the prompt engineers gone? If anything, I see more calls to abandon skills. Why? Because the cloud models are really good at a lot of things and they don’t need skills to perform tasks. If anything, skills are likely to overprescribe methodologies that may actually not be optimal. Compared to a pure run, the skills may end up polluting the context. A tension can arise between the model’s “desires”: to perform the task well vs. doing it according to how the task was described. Suboptimal descriptions lead to suboptimal results, or else an avoidable round of back-and-forth where the model suggests a better way to the operator. Daniel Miessler has good articulations of this as Bitter Lesson Engineering.

So, do agent skills matter? I believe they do, at least in a few cases.

Case 1: Genuine Alpha

One way that may clarify thought is the recognition that LLMs are index funds. An agent operator is in the role of an investor in this metaphor. Your skills are bets that you can outperform the index. Do you feel lucky?

The traditional investing wisdom is that you should avoid such bets. “You” here means “the ordinary person” who does not have special expertise or hidden alpha that would allow you to perform better than the average (the index fund).

Now, that’s not to say that outperforming the index via custom agent skills is impossible, or even rare. It’s that you’ll be unlikely to do it for a long time, in a sustained way, even if you are an expert.

That leaves open the possibility that some experts will absolutely be able to outperform the index – for a while, within their niche. That might be all you care about! If you’re a bug bounty hunter, the only secret sauce you may need pertains to finding security issues. You may have a skill here that’s a great bet, and allows you to eat well off of it.

However, if you’re overall bullish on the abilities of LLMs, that also means you should be bearish on skills. It’s also very easy to trick yourself that you have some special talent in placing great bets if everything goes up at once.

But if skills are alpha, why might a company release it widely? The moment they do this, the alpha ceases to be alpha by definition. In most cases, I believe that skill releases are mostly just marketing material. A company that makes wide, frequent releases of its private methods would not be acting rationally. For this reason, skill releases should be met with a lot of skepticism: what they are releasing to the world could be considered exactly what not to adopt in your workflow. You’d be better off riding the index rather than making yesterday’s bet.

Case 2: Glue & Taste

If you need a highly specific output format, skills will absolutely help you out. LLMs often run wild with formatting outputs. They love rendering markdown tables in the CLI, for example. This suits the immediate chat interface but it’s not composable with other text processing tools or even a simple copy-paste.

Skills help here. In this form, they are simple little recipes cooked just the way you like. There’s no claim that the output is superior here, but simply that it’s well-suited for the situation at hand.

Similarly, skills can help those of us who are tinkerers who love building utilities and networking them together. If you spent much of your programming life customizing neovim, shell scripts, or ricing a Linux distro, you’re likely to have an illegible wilderness of personal tools that cloud models know nothing about. Skills can help an agent drop into your mad little garden and not make a mess of things.

Case 3: Local Models

Local models are behind the cloud models. Not by much, but it’s still a meaningful gap. Skills here could be genuine augmentation of the models’ abilities. You’d only want a skill that clearly demonstrates better outcomes. As compared to the cloud model case, you at least have an oracle when testing the local model. You run the task locally with no skills, then in the cloud with no skills, and then again locally with the skills attached. This last case should outperform the pure local case – otherwise, the skill is garbage.

For people taking a hybrid approach, this raises the question of how to manage your skills and your harnesses. Your skills help the local model but likely hurt the cloud model. This invites experimentation: perhaps subagents are configured with or without skills depending on where you run them; or, you could consider tracking everything in one repo and using separate branches or worktrees depending on whether you’re working locally or not.

Skill Issues

When performance is your top concern, then skills are bets, not infrastructure. Fund them like bets: verify against baseline, cut losers. Articulate the ideal state and let the model work – reach for a skill only when you can name (and ideally measure) the alpha.

But let’s be real. Nobody really tests these things (Halvar Flake). And most people don’t need the absolute optimal performance out of cloud models. Outside of research or vicious PvP environments, it’s not worth worrying all that much about.

When you’re working locally, doing cozy computing for its own sake, skills are just the thing to make working with agents more personal and fun.