Hello friends. I was dearly hoping to be
able to come to the AI engineer World's
Fair, but family matters have intruded
and I'm not able to make it. However, I
will not be leaving you empty-handed.
I'm going to give you the talk that I
would have given in San Francisco. This
talk is called the missing manual. How
to write great skills. And I think that
the ability to distinguish good skills
from bad skills is only getting more
important. As developers, we seem to be
pretty talented at finding different
forms of hell for us to go to. In like a
few years ago, we had tutorial hell,
which is where you would go into a bunch
of tutorials trying to learn something,
not be able to piece it together and
sort of just get into this cycle you
couldn't get out of. We had framework
hell where every other 10 minutes there
was a JavaScript framework being
announced and you know, you had to learn
the hot new thing all the time. And now
I think we have another version of hell
which is skill hell. Skill hell is where
you have all of these skills available
freely available that you can download,
contribute to, you can figure out on
your own, but you don't really know how
the pieces all work together. You can't
tell a good skill from a bad skill. And
this means that people are trying to
piece together these frameworks trying
to try everything that's out there all
at once. And they sort of can't, or
rather, they don't get the results that
the skills themselves promise. This is
true at an individual level, but it's
also true at an organization level, too.
Organizations have no way or no
understanding on how to build good
skills. How to take their operating
procedures and turn them into things
that an agent can do. And if you don't
do that, then it's hard to get the
bounty that skills can offer. Just one
more skill, bro. That's kind of seems
like what we're saying. And I feel a bit
of guilt here, too, because we have Matt
PCO skills, which is my skills repo,
which is one of the most popular
engineering skill sets out there. And so
I feel like I want to help the people
who use my skills get out of skill hell.
So how do we do it? How do we get out of
this? Well, what is actually missing
here? Well, in my opinion, the thing
that we're missing is we don't know what
makes a skill great. We can't yet look
at a skill and go, okay, this skill is
doing these good things and these bad
things. There's no shared rubric, no
framework for looking at a skill and
making it better. And so that's what I'm
going to give you in this talk. I'm
going to give you a skill checklist. A
checklist of things you can look at
inside the skill to make sure that it's
doing what it says it's doing and ways
you can improve it, way you can write
skills. This checklist looks like this.
We start with the trigger of the skill.
How the skill is invoked and the
decisions that you need to design there.
Then the internal structure of the
skill, how the skill is actually
composed and laid out internally. Then
number three is how do you actually
steer using the skill? How do you get
the skill to tell the agent what to do?
And then four, how do you make the skill
as small as possible? Because once we've
got a working skill, we then need to
basically maximize it. Prune out all of
the irrelevant stuff, prune out all of
the noops. There's one handy advantage
of me not being in the room with you,
which is you can immediately go and try
this out because I've encoded all of
this into a new skill in my repo called
writing great skills. So, if you've got
an immediate use case for this, then
just go to this my skills repo. you
know, just close this browser, get out
of here, and go and use this skill to
either improve your skills or write
great new ones. But let's start going
through the checklist. Then we have
number one, the trigger, the way the
skill is invoked. And in order to talk
about this, I'm actually going to do a
bit of comparison here, which is that my
skills are often compared to another set
of extremely popular engineering skills
called superpowers. And I'm really often
asked the question, how do your skills
compare to superpowers? What's the
difference between them? To understand
that we need to understand the
difference between user invoked and
model invoked skills. Anytime you have a
skill you can always invoke it manually.
So the skill sits on your file system.
The agent will just be able to pull up
the skill and understand what's in
there. And you can always do that by
communicating that to the agent. Doesn't
always look like this forward slash
depending on the harness but that you
can always user invoke your skills.
Another way that skills can be invoked
is by the agent itself. off. These are
called model invocable skills or model
invoked skills. You can take a
description. So the description of the
skill always ends up in the agent's
context and the agent can look in that
and go okay based on that description
I'm going to invoke the skill and I end
up reading the skill.md file which is
where the meat of the skill is into my
context window. That's how you invoke a
skill. That's what happens when a skill
is invoked. So this description serves
as a kind of context pointer. It sits in
the agents context pointing to another
file where the agent can go if it wants
more context for that context pointer.
You don't need to put it into the
agent's context. It can just be
invisible from the agent. And that is
what we call a user invocable skill. So
some skills can only be invoked by the
user because they don't have this
context pointer. It's optional. For
instance, we can see in my codebased
design here, this is a model invocable
skill. It has a description that ends up
in the agents context window. But if we
look at my grill me skill instead, we
can see it has disable model invocation
true. This means that this little
description here will only show to the
user. It won't be visible to the agent.
So this then is tip number one. Decide
if your skill is user invoked or model
invoked. Now you might think that model
invoked skills are better, right?
Because either the model can invoke it
itself or the user can invoke it. It's
more flexible. But every time you add a
model invoked skill into your agent's
environment, it increases what I'm going
to call the context load on that agent,
it adds a new description, which is
costing you tokens on every request, but
also adding a different thing for the
agent to think about. So if you have a
100 model invoked skills, that's going
to be 100 descriptions inside the
context for your agent. So it seems to
make sense then to either tamp down the
number of model invoked skills or to
just use all user invoked skills. But
user invoked skills have a different
load which is the more user invoked
skills you have the higher cognitive
load on the user. In other words, the
more things the user needs to keep in
their head, the more um skill you
require from the pilot. And so if we
compare Matt Pot skills to superpowers,
superpowers is primarily model invoked
skills. gives the agent superpowers.
Whereas my skills, I much prefer to be
in full control. That means I get to
keep the context load on the agent as
small as possible, but it does impose
more of a cognitive load on me. So I
need to understand the skills really
deeply in order to get the most use out
of. So why have I done this? Why did I
prefer user invoked skills? Well, every
time you have a model invoked skill, it
basically you get a cost in
unpredictability because every time you
have a context pointer pointing from one
resource to another, the model may just
choose not to follow it. You know, even
if it's absolutely perfect for the task,
it may just choose not to invoke the
skill. I much prefer removing that level
of unpredictability, imposing a bit more
cognitive load on the user. And what you
get is just you you're removing a class
of problem from even being a problem
because this unpredictability leaves
people to need to eval their skills to
make sure they're being called at the
right time which is uh really nasty and
it's a problem that I prefer to avoid.
But what I'm hoping to show you here is
that model invoke skills and user invoke
skills both have their same costs. So
it's not an easy decision which one you
choose. So that then is the trigger how
the skill gets invoked. Now let's talk
about the structure. the internal layout
of the skill. I think of there as being
two main units that you need to put into
most skills. These two units are the
steps and the reference. The steps are
the step-by-step procedure that the
skill is going to walk through. And the
reference is any supporting information
that helps it walk through those steps.
You can have skills that have no steps
and are only reference. And you can have
skills that are no reference and only a
set of simple steps to walk through. But
if you start thinking of skills as
composed of these two units, it really
helps just break them down a lot more.
If we look at an example, one of my
skills called two PRD creates a product
requirements document out of the current
context window. It's got three steps in
it. So it finds the relevant context. It
confirms the test seams with the user.
So there's like a little human in the
loop checkpoint there just to make sure
we're not doing anything weird with the
testing, which I find really important.
And then we write the product
requirements document to handle those
three steps. We've got two bits of
reference material. We've got a little
bit of reference on what is a test seam.
And then we've got a product
requirements document template. So just
a literal markdown template which is
used to write the PRD. So this is a
great way to write a skill from scratch.
You work out if you need some steps.
Then you write those steps and you work
out what reference material those steps
need and you put it in a separate little
spot in the skill which is for reference
material. However, there's a really
important constraint that we need to
think about which is tip number three.
We want to make the main skill.md file
as small as possible. Every skill is
composed of its description and then a
skill.md file and then any reference
material that branches off that. And
this skill.md file, if we make it small,
then we're saving in a bunch of
different ways. Smaller skills are just
easier to maintain, easier to audit,
fewer words to think about. And every
time you shave off a word, that is a
token shaved. That multiple token shave
from your skills cost. So I do believe
that small skills are really important
both for maintainers and for users. One
really useful way you can make your
skill smaller is by thinking about the
different branches of the skill, the
different ways the skill can be used.
Because if you have reference material
that's only used in one branch, then
that's a candidate for being removed
from the main skill.md. For instance, if
we look at my two PRD here, we have two
pieces of reference material. What is a
test seam and the PRD template? Well, we
need the PRD template every single time
because we are always creating a PRD.
And we probably also need the what is a
test seam information every time because
we're always asking about the test
seams. So 2P there's only one branch and
all the reference material belongs on
that branch. So it probably also belongs
in the skill.md file. However, if we
look at a different skill of mine which
is domain modeling, domain modeling does
two things. It updates a local glossery
called context.md and then it also
creates architectural decision records.
In other words, it's doing two different
things. Or it might actually choose to
do neither of these in which case it
doesn't need the template and it doesn't
need the ADR template either. So in
other words, domain modeling has two or
maybe three branches. And this means
that we don't need to include the ADR
template or the context.md template into
the main skill. They can be moved into
separate zones. The way you do that is
you have the skill.md file. Then you put
it behind a context pointer and you
point the context template to a separate
markdown file inside the skills folder.
That context pointer literally just says
if you need the template or if you need
to update the context.md file, go to
this file. And I call that an external
reference. It's a reference that's
external to the skill.md that you can
just easily reference. The agent can
pull in very easily because it's bundled
along with the skill. So this is a
technique you can use for making the
skill.md as small as possible, which is
has so many benefits. Hide branching
reference material behind context
pointers. In other words, if you feel
like your skill is going to be used in
lots of different ways, then take the
reference material that's relevant for
those branches and hide them behind
context pointers. So that is structure.
We need to think about making the
skill.md super duper small. We need to
think about the branches in our skill
moving material out behind context
pointers. And we need to think about
steps and reference, which are the two
main units inside a skill. Let's go next
to steering, the actual ways we get the
agent to do what we want it to do. And
for me, steering comes down to one
really cool technique, which is the kind
of main thing I want you to get from
this talk. This technique fixes this
issue, which is the agent doesn't do
what I want. In other words, I specify
something in the skill. I think that
I've been clear and then it just doesn't
do the thing. Now, I think the main
reason this happens is because you're
not using a technique called leading
words. The idea of leading words or
lightvert if you like literary theory I
suppose is that there are certain words
that pack in a bunch of meaning into a
very small space. These leading words
are really powerful with agents because
you put the leading word in the skill
itself in the text and then the agent
will repeat the leading word back to
itself as part of its operations as part
of its thinking tokens and as part of
its output to you. And then because it's
re-emphasizing that word and that word
hopefully describes what you want from
the agent that then goes and changes its
behavior. Let's make this more concrete
with an example. So let's imagine that
we have a problem which is a classic
problem with agents which is that they
code layer by layer. In other words, if
you give them a big tranch of work to
do, they will generally code up all of
the database layer, then all of the
schemas, then all of the API endpoints,
then all of the front end. they don't do
the sort of typical um human thing which
is to seek feedback early on, get
something small working and then expand
out from there. Now, we can try to
encourage the agent to do that by just
saying, you know, don't code layer by
layer. Um make sure that you create a
small slice first and then go from
there. But what if instead we used a
leading word? We said vertical slice is
our leading word. We want to slice up
the work instead of horizontal slices
into vertical slices. A vertical slice
is a pretty well-known terminology in
development. And so this will hopefully
trigger the agent's prior, and it will
understand what we mean. We don't just
have to like have a two-word skill where
it just says vertical slice. What we're
doing is we're packing lots of meaning
into a relatively short phrase that we
then repeat throughout the skill. The
cool thing about this technique is you
can know if it's worked because you say
vertical slice in your skill and then
you'll notice in the reasoning traces
that it's saying, "Okay, we're going to
do this as a thin vertical slice." then
you should get better implementation
plans. Everyone I've explained this
technique to sort of feels like, "Oh
yeah, I've been doing that for a while.
I've been using these little phrases to
try to encourage the agent to do what I
want." All I'm asking you now is to use
those consistently within your skills
and watch in the thinking traces as the
agent adopts your way of doing it. So
often if the agent isn't doing what you
want, you need to make your leading
words more consistent, more powerful,
and look for others because you know,
English is a pretty wide API in terms of
different functions you can call,
different things you can experiment
with, and there are many leading word
candidates out there, and agents are
actually pretty good at helping you
think of them. Another little lever you
can use with agents is sometimes the
agent just doesn't do enough leg work.
What I mean by this is that okay, we're
on a step, let's say, and maybe the step
is to ask clarifying questions or to
explore the codebase and the agent just
doesn't do enough of it. It doesn't put
enough effort into that particular step.
A real classic case of this and
something that I have found almost
everywhere it exists is plan mode.
Because in plan mode, we have two steps.
We have ask clarifying questions and
then create a plan. And what I have
found in every single implementation of
plan mode I've tried is that ask
clarifying questions just, you know,
doesn't ever do enough leg work. It sees
that its ultimate goal is to create a
plan. And so it just does a small amount
of leg work with ask clarifying
questions, ask you a couple of things,
and then eagerly creates the plan. So
what was my solution here? Instead of
doing plan mode, I instead have a skill
called grill with docs, which is kind of
my ask clarifying questions phase. And
then I split that up into a separate
skill. So I split the planning into its
own skill. So grill with docs now is its
own skill where the agent only sees that
part of the process. And then after
grill with docs completes, we then go
and do two PRD. In other words, we have
step one and step two, but the agent
only sees one step at a time. So, this
is a really cool technique for
increasing leg work on the step that
you're on by hiding the future goal,
hiding the future steps. It's not always
necessary to split skills into
individual steps, but in particular
cases where you really want an extra
chunk of leg work. It really there's no
technique like it. It works very, very
well. So, that is steering. using
leading words to capture what you want
in small reusable tokens and then making
sure that it's doing the right amount of
leg work per step. So let's head now
into pruning. Now pruning really is just
a quickfire set of failure modes,
different things that you can get wrong.
And the first is fairly obvious is we do
not want massive skills. Massive skills
are usually a kind of symptom of
something else going wrong. So a symptom
of one of these other failure modes. And
the first one is pretty simple. Don't
repeat yourself. You need to make sure
you're watching out for duplication. And
in general, I like to have every part of
the skill to have a single source of
truth. In other words, if you have a
piece of reference material like the PRD
template, let's say, or something even
smaller like what is a test seam, you
make sure that you don't repeat that in
several places or like cover multiple
steps in multiple places. Just make sure
each part has a single source of truth
and you're not repeating yourself even
across reference material too. The next
way that skills get big is via sediment.
And sediment is just a classic thing
when people are working on the same set
of docs really which is that everyone
starts contributing to a shared markdown
file. People add their own stuff. They
don't feel brave enough to delete and
modify anyone else's. And so you just
end up with this huge amount of sediment
with often irrelevant material for the
skill, especially stuff that hasn't been
laid out properly. With a skill with a
lot of sediment, you really need to look
at structure. That's the first thing you
need to do. You need to make sure that
the stuff that's been added is relevant
for all branches. If it's not, then move
it into the correct branches. Or if it's
just totally irrelevant, maybe just
remove it or kill it. Or maybe there's
stuff in there that's totally stale, in
which case you just need to kill it
dead. The next failure mode is really
common when an agent writes your skills,
which are no ops. So things inside the
skill that appear to do something, but
don't actually influence the agent's
behavior inside the context of the
skill. Let's imagine we have an
implement skill and we have an entire
paragraph of the skill that tells the
agent to write a long detailed commit
message. What would happen if you just
deleted that paragraph? Well, the agent
would probably still write a decent like
long commit message. People ask me a lot
how I get my skills so small and it's
just using these techniques, using
deletion tests, using um making sure
that I compact things into leading
words, I don't have anything irrelevant
in there, and I don't have any sediment.
And that finally brings us to the full
sweep of things. Number one, we check
the trigger. We make sure that it's
firing at the right times. We check
whether we're imposing context load or
cognitive load. With structure, we think
about branches. We think about um
structuring things into steps and
reference and we make sure that um
material that's only relevant for one
branch is outside of the main skill.mmd.
For steering, we're thinking about
condensing text down into leading words
and watching those leading words appear
in the reasoning traces. And we're also
thinking about leg work. Should we break
this skill down further to increase its
focus on the current phase by hiding the
future phrase phase from it? And with
pruning, we're doing a final pruning
pass over the entire skill, watching out
for sediments, watching out for crud,
and watching out especially for no ops.
Now, all of this stuff, the best way to
get started with this framework is
inside this skill, inside the writing
great skills skill. You can check it out
from map. Got skills, download it, uh,
use it to improve your own skills and
maybe even use it to run over some
community uh, authored skills so that
you can check that the skills that
you're actually pulling in are any good.
If you want to follow along with my
stuff, then I have a newsletter up on
aihero.dev. And my plans for the next
few months are to release an AI coding
crash course, which is an intro to a lot
of the stuff I've been talking about and
how you get off the ground working with
engineering and AI. I hope that what
I've given you is enough to help you
escape from skill hell or at least try
to make the bitter journey out of there.
I'm so sorry not to be able to attend in
person, but thanks for watching. I'll
see you very