Herrington Darkholme (ast-grep) on Teaching AI to Write Rules - Episode 8

[00:00:00] Introduction

[00:00:00] Jakob Heuser: What's up, everybody? Welcome to The Task at Hand. I'm Jakob Heuser, founder and CEO at Taskless. Today, I have Herrington Darkholme on the show, creator of ast-grep, contributor to tree-sitter.

if you've ever wanted to find every place that a code pattern shows up, no matter how it's formatted, rewrite patterns across a whole code base in, one go, and you wanna do it quickly, boy, do I have a tool for you and the person behind it. it's also the engine Taskless runs its rules on, so I've been living inside his work for about eight and a half, nine months now.

And so I'm personally very excited to be doing this interview. Herrington's been teaching AI to write ast-grep rules, and he wrote up what worked and what didn't. We'll be talking about that later on in the show. we've been effectively fighting the fight from the other side here at Taskless. Both of us care about how to make an agent behave correctly, and we both think that static analysis and determinism is part of the answer.

So I'm looking forward to comparing notes today. Herrington, welcome to The Task at Hand

[00:01:00] Herrington: Hey, Jakob. so glad, Taskless Podcast can have me here. I'm Herrington, Herrington Darkholme. you can find me on Twitter, with the handle name hd_nvn. I created ast-grep. I love, I, I love, programming with Rust, so I wrote this program. It's, it's a fun project to me, and I'm ver- very, very honored that a lot of people are using this technology, including Taskless

[00:01:27] Jakob Heuser: Yeah, let's,

[00:01:28] Origin of ast-grep

[00:01:28] Jakob Heuser: let's actually start with sort of the, the g- the genesis, the beginning.

[00:01:33] Herrington: Yeah

[00:01:33] Jakob Heuser: why did you create ast-grep? Was there a specific code base, a bad afternoon? Did ESLint do you dirty? Like, when d- when did you go, "The tool ch- the tools we have right now aren't enough"?

[00:01:47] Herrington: Oh, yeah. So I created ast-grep before the, the AI era, so it's around twenty-twenty-one or twenty-twenty-two. so back then I was working in, big tech and, almost every, week or, every month, I have a lot of, say, technical migration project because our infrastructure is ever, ever, ever grow- growing. So the infrastruct- structural code is always changing, and we have to, as application developer, have to, to change our code. And, this migration is boring and tedious, very error-prone. So I always, always want a tool that can do some, say, pattern match on, on code and migrate some existing usage to another one. So I find... I, I just started my, exploration of the tool. So there are quite a lot existing tools there say ESLint, jscodeshift, Code Rewrite or something like that. But I find that they are quite hard to use. Like you have to write some imperative code, like, some, kind of a AST node and write some- something you, you need to, take care of this, the tree, the, the tree visiting.

You also need to think about, hey, hey, there is some state I need to s- to store, I need to remember. And also, AST is a tree structure, right? So it is, naturally a, a deep structure, and you need to, pattern match deeply. So it isn't fun to write some very deep code just to find a piece of, say, a console.log. And, in, if you, you visit ast-grep's doc, the, the motivation is there. if you want to write a console.log in y- in Babel or ES, Esprima, you will have to something like the, the, the, the, a call, construct a function call, and you need to match the ca- caller and the callee and an argument. And when these trees match, you can match your, console.log. But in ast-grep, I want something, yeah, I want something, something that's easy, that you can write something, say, looks very like your, your ordinary code, like console.log, with dollar variable. So it looks like what you will write in plain JavaScript, right? and with some, tricks like a, a meta variable, a placeholder, then you can match a, a certain kind of structure in your code. So that is very, handful, and it can help me.

[00:04:25] Jakob Heuser: that's very

[00:04:26] How ast-grep works

[00:04:30] Jakob Heuser: cool. 'Cause like I think as someone that came from, ast-grep and has worked with like Biome and GritQL and a bunch of other static analysis tools, I think part of the magic is with ast-grep you don't have to write custom code. You don't have to understand the syntax tree part of ast-grep to make ast-grep work.

ast-grep handles, oh, you've written the code this way. I know that's a declaration. I know that's a function. I know how to do the matching for that. You don't have to do the tree walking to build that hierarchy or mental model yourself. And then like you said, there's some really, there's some really nice sort of polish that you add where like a double dollar sign helps you define like this is k- this is keywords, this is meta variables, these are constructs that aren't used in other programming languages so that you can call out the placeholders.

And then you can just write statements about the placeholders. hey, the placeholder has to look like this, and that makes it very, very accessible, and it makes it very portable. ast-grep, ast-grep supports what? 30 languages I think out of the gate

[00:05:38] Herrington: Oh, yes, like tw- 20 plus or 25 something

[00:05:41] Jakob Heuser: Yeah. It's, it's a lot, and it, it makes it very great for monorepos because now you've got one rule engine that works for Python, it works for ESLint. And something that we've heard from folks is that they actually run tree-sitter and ast-grep alongside existing code analysis tools. So if there's already existing battle-tested libraries, they'll use those, but then they'll bring in ast-grep because it's really easy to write, what they call shop style rules.

"Hey, we call things this way." And instead of going through the hassle of building a really complex, syntax tree rule for it and having to write it in, write it, go write it in Python for Pylint and then go write another version for ESLint, they just write it once in, in ast-grep, and they're able to use that rule in two different contexts and just basically simplify, simplify the work they have to do I know ast-grep is built on tree-sitter.

You're a big contributor to tree-sitter. why tree-sitter? Was it just a sh- shopping around looking at what was out there and something clicked? Or was there something specific that stood out to you as "This is the technology ast-grep needs to be built on"?

[00:06:52] Herrington: Well, I, I

[00:06:52] Why tree-sitter

[00:06:59] Herrington: think it's the later. That tree-sitter is definitely what ast-grep is looking for. So first, I designed ast-grep to be language general. So just as you, you have said, ast-grep is general enough to fit different language, even if, they have different, say, syntax rules or, or semantic rules. so I need something that's general enough care of all the language. So tree-sitter is almost the, the only, solution that is adopt wide enough used as a language general parser. we do have some, say alternate native solutions like, writing everything in, like a Bison or, or some, pa- you're using some parser generator, but we will have to maintain the language grammar ourselves, right? But with tree-sitter, we have a wide community that already built this beautiful language, language parser, and we don't need to maintain the, the grammar. So that's the one big, big, say, the adoption part of why ast-grep, choosing tree-sitter. The other part is technical, because we need to handle so many language, right?

So we cannot have a, a typed interface for our, our AST. by typed say, say, in JavaScript you have a statement, right? And for statement, say, say you have a function d-declaration, and for function declaration, you have its name, its arguments, its return type. that depends on if you are, are using, TypeScript or not. So, typed AST means every AST node will have a, a, a named field for these, for these children. tree-sitter is not. tree-sitter has a uniform interface. Like you have a tree and, you have a node, and then you have, its children. you can iterate, over its children to access its, its content. additionally, it also has something called field. So, you cannot know, say beforehand, what kind of, ch- children you have, but you can visit the field, and the field has name on to it. I have a, the, the, the, say I have a, I have, five children for my function declaration, and the second ch-child is function name with field name. So this uniform API interface is also pretty important, for language general, general work

[00:09:25] Jakob Heuser: So the,

[00:09:26] Semantic vs syntactic

[00:09:29] Jakob Heuser: the example I usually give people when they ask about what's the difference between AST grip and, the TSC compiler or something like that. And what I've usually done now, and you can always correct me if this doesn't match your mental model, is that a tool like TypeScript can tell you the types and whether you're using the wrong type in the wrong place.

But if you wanted to answer a question of there can't... I don't wanna have more than three infer statements in my generic. That's not a TypeScript problem, that's a tree problem. And if you take a TypeScript generic and you expand out the structure behind that code, you have something you can build assertions against.

It'll still be valid TypeScript, but the difference is there's something about the semantic correctness you want to enforce, not the syntactical

[00:10:29] Herrington: it, it's, maybe it, it's, something, catchier like, so first, TypeScript type, it's, it's somehow gener- semantic. you have this type, or, or, or, an identifier, and that identifier have some types, and that type, we usually call it semantic, information. but on the other hand, if you have, an identifier and it appears on the return position, then we call it, it- so we can just look at the tree without resolving the, the types. it is more like a, a syntactic co- co- problem. back to, to, to your, your, your scenario, like, you have a valid, type check to the TypeScript, right? And it's a generic, you have it, for generic parameter, and you don't want the, the generic parameter to exceed the safe rule. And this i- this can be semantic syntactical, but it's, for this case, it's, it's definitely can be solved at a syntax tree level. of course, it, you can enforce, the, the program writer to not exceed f- four parameters. And this is something, I would say it's linked. It's linting. It's a linting rule. It's not a hard error in terms of the language, but it's a bad practice, right? So we definitely don't want it, especially in AI coding. AI can do pretty many na-- r- r-- like technical correct code, but practically, wrong

[00:11:59] AI's technically correct but wrong code

[00:11:59] Herrington: code

[00:12:00] Jakob Heuser: No, that, that's actually a big thing. I think the way you describe it, technically correct but practically useless, is a very good way to describe, what I've seen in Claude and Sol and a couple of frontier models when they start trying to get very clever. we have a rule in-house that says there's no premature optimization, and it's a really simple AST grep rule.

It just looks at the number of, the number of statements within a function, and if that number of statements is less than five, it throws a warning.

[00:12:38] Herrington: Oh, yeah

[00:12:39] Jakob Heuser: And it just... It's because we found that Claude would prematurely optimize and be like, "Oh, I need to pad this string left. Let me create my own string pad function that has three lines inside of it, and let me then also create, seven lines of a test to go with it that already assert the thing we already know."

And it was like you could just use the built-in padding functions. And

[00:13:07] Herrington: problem so, so many times. in my project, the, the cloud or, or, or other models will do something like a small f-file just for, say, this kind of specific, problem, but it can be merging to other files, like because it's a bigger, say, a, a domain, a bigger idea.

But you cannot, say, without looking at the code, you cannot not know that, right? But this rule is pretty, smart. Like you can just look at, at the, the, the, the line number and to guess, hey, this probably doesn't worth a, a standard standalone function or standalone file

[00:13:44] Jakob Heuser: So one of, one of our big aha moments at Taskless, and a big reason why we build what we build now, was we had that rule, and all it did was emit a warning. And the instant Claude checked itself, the warning was written like an instruction for the agent.

[00:14:04] Herrington: Yeah

[00:14:06] Jakob Heuser: I, I actually have it here. The, the w- the warning says, "Functions under five lines are a sign of premature optimization.

Before agreeing to use this line, double-check that this function has novel functionality that is used in multiple places that would justify extraction." And I kid you not, it gets that error on standard error, and it immediately pauses and said... And I have the transcript log right here, my JSONL, the Claude response, "Let me go look at it.

Let me go look at the call sites to see if this is worth inc- worth persisting." And it just started cleaning itself up because it was just never told that it wasn't allowed to do that, and you n- you were never gonna go put that in your system prompt anyway

[00:14:49] Herrington: yeah,

[00:14:50] Just-in-time prompting

[00:14:50] Herrington: yeah, yeah, yeah. Yeah, yeah, yeah. I really like the, the idea of using AI script like that. I also have a, a, a a, a, I always given it a name, like just-in-time prompt. Like you will prompt your AI only when it does something, right? So it, the, the prompt is injected into the, AI at just at that time. So you, you will not have something like the, your prompt is too long, so your, your AI cannot follow that. you have something like, oh, the, your prompt, is too long, so, AI just selectively fi-find something and then forget other things. So just-in-time is really powerful

[00:15:31] Jakob Heuser: It's, it's actually a big reason I think Rust with AI has been so successful is because Rust errors are so good about telling you when you break a convention. They tell you why it did this, what you need to do instead, and that's, as you said, just-in-time prompting. The agent gets that output and immediately knows what its next step should be.

so Rust has just been taking over kinda everywhere. I suppose it's a good time to talk about some of the fun stuff that you've been doing in ast-grep. first off, 0.5.0 is out. It has support for Zig. I'm gonna ask you about what's driving the language work in a moment, but I also wanna take a moment and just kinda talk about sort of the Rust work inside of ast-grep, 'cause that was some non-trivial work.

There was that, there was tree-sitter. You've been on a Rust kick lately. So talk to me a little bit about what you've been doing with Rust in the, in the syntax tree system, and how we've all benefited from this work, 'cause I think the stuff you're doing's super cool

[00:16:29] Herrington: Oh, yeah.

[00:16:30] Porting tree-sitter to Rust

[00:16:31] Herrington: So, first asgrep was written in Rust. I didn't pick up other language because, I really cannot write correct C++. I'm sorry. And, also, sim-similarly, I cannot pick Zig because I'm not confident in myself, like I can manage the memory well. So Rust is somehow, an external tool that help me to write a, a correct or memory safe code. that's, big, big perk to me because, I write this, project for fun. I don't want to spend time to fight this memory, safety nightmare. And, yes, I also did something, say, asgrep, I also tried to dig deeper into the syntax, syntax tree world. So I tr- one thing that I really cannot optimize away, from t- asgrep is tree-sitter.

So I have to spend like, at least one third of time or, or half of time to just parse the syntax tree, and that's the bottleneck that isn't easy to, to solve. So I tried something to look at tree-sitter code itself. so it's written in C. that's great, because it's portable. But, for asgrep, I would want something, more tailored, to the Rust ecosystem, also have some opportunity to, to optimize the, the memory allocation in the original tree-sitter, program and do, do something better.

Like I can apply some data-oriented design. so I just tried to rewrite tree-sitter in Rust and, it's an interesting journey. Like at first, it run fast. Yeah, it run fast. But later you will bump into some memory issues like the, the, the segfault or something, and I don't know how to fix that, to be honest.

what I ga-gave my, my ChatGPT, I rewrote, tree-sitter with ChatGPT, and Chat... I just ta-told ChatGPT, " Hey, please, optimize it, it and make it fast enough, like ten, 10% or 20% faster." Yeah, it did

[00:18:42] Jakob Heuser: And it, I don't think that would've been possible, honestly, without the, the tree-sitter team's C core having just such an incredible test suite too.

that set of expectations gave you... I think without that, a port would've been incredibly difficult because that contained every error that tree-sitter had ever run into and how it was solved.

[00:19:02] Herrington: Yeah

[00:19:03] Jakob Heuser: what you ported to had to, had to basically pass every test that was created

[00:19:06] Herrington: Yes, yes. So I basically think if without the tree-sitter test case that the port won't be possible at all. I can copy some, some behavior, using the, the existing thing repo, but I will never know or understand the, the nitty-gritty details of tree-sitter, right? So, so, so to make it truly compatible and, can work with the ecosystem, you-- I have to run some, some test cases, and these test suites are very, very helpful.

[00:19:40] Jakob Heuser: And then you made some design decisions too. incremental parsing went out the window. and then the Wasm grammar loading also went away. for those that don't know, Wasm, WebAssembly, it is a portable partial binary format. You can always decompile it into WAT files. it is a pretty good answer for a lot of cross, cross-architecture modules that you need.

[00:20:02] Herrington: Yes

[00:20:03] Jakob Heuser: you decided both of those needed to get swapped out with something else, and I wanna hear a little bit about sort of what that decision process was like and how you selected the candidates that went in, because there's a lot of language work showing up, so clearly you've, you've created a gateway to make it easier to create languages that run on AST Grow.

[00:20:24] Herrington: yeah. okay.

[00:20:25] Dropping incremental parsing

[00:20:27] Herrington: So first, why, so there is something I dropped during the report. So first is incremental parsing. So one thing I did that is increment pa-parsing is mainly for editor usage. to be honest, at least me, I, I no longer write code by hand, that much. Like, I occasionally do that, for fun or for, for some, some, for some training.

Like, yeah, you need mental training. Like you do some work, workout in gym for your, for your body. Then, then programming by hand is somehow workou-workout for your mental, right? So but besides th-this part, most of code is generated by AI both in my, in my, in my work, in my day job or in my, my, hobby project, I use AI to generate code a lot. it no longer makes sense, like, or it no longer makes this im- that important to, to have in incremental parsing. And incremental parsing im- add some complexity and overhead to parsing. So but since AI usually generate code in, in batch, like, in, in one edit they will change the whole file, or in one batch they will change multiple files across the system. So you will need some, thing like faster, say like a, like a faster fu-full parsing. that's what you, you will need in an AG- AG- agentic coding world.

[00:22:02] Jakob Heuser: One of the advantages to that too that I think, came out of this that a lot of people don't appreciate and understand is that by dropping incremental parsing, ast-grep can choose to discard part of the tree. So if you have a syntax error or a just structural error in, say, an if branch, that doesn't interfere with the parsing of the rest of the file.

And this is something that's very different than running like Bison or a PEG parser or any s- other sort of like grammar structure, is that you get that partial, you get that partial tree, and that allows you to then basically save the run. And that means you can still get value out of a full file parse versus having to stop at the first error

[00:22:44] Herrington: Oh, yeah. but, but, a lit- a little correction. So because you have, incremental parsing, it is somehow orthogonal to error recovery. So, like, you have some typos or you have some, syntax errors in, in your code. but tree-sitter somehow can, can figure that out. Like, it, it still give you, a valid-- it's not valid tree.

it still give you some useful tree out of invalid programs. So this is error tolerance. And on the other hand, you can, type, some new, say, say, type some, some new text i-in your, in your code, and, say the parenthesis is not closed. And these things, you can also error tolerant. But, add new fi-- new things to the old tree and reuse the old tree to create a new tree is something called incremental parsing. they are related, but not necessarily the same thing

[00:23:36] Jakob Heuser: Okay. And then the Wasm stuff, like you actually made a choice to swap that out. where was WebAssembly not going to help, help you get where you needed to go? What, what did you swap it out for, and sort of what, what was the end result of that?

[00:23:51] Herrington: to be honest, that

[00:23:52] Dropping Wasm support

[00:23:54] Herrington: decision is more, like just out of, practical reasons. Like I don't have enough coca-- I'm sorry. So I don't, don't have the budget to, to, to migrate to the Wasmpar. And the Wasmpar is quite heavy. Like it, it's, if I remember correct, it's four thousand or five thousand line of C code, and translating that to, to Rust will easily break like five thousand line of code or even more like si- six thousand. So I, I didn't have the budget to do that, so I decided to cut that. another thing is the Wasm ecosystem. So, tree-sitter is using Wasmtime to drive the, the Wasm parser. So Wasmtime only supports several, say several, platform like, the, the, the main Mac, Mac, m- the, Windows, Linux and, Mac, and only for thirty-two-bit or sixty-four-bit, ARM or, or, x86. But you cannot support some, lesser supported platform. for example, on Linux, Wasmtime only support libc like, GNU Lib-libc, right? But it does not support something like, like, like, musl, the

[00:25:07] Jakob Heuser: so the Wasm layer was effectively taking away some of those portability gains from the C library at that point.

[00:25:14] Herrington: Nope,

[00:25:15] Jakob Heuser: No?

[00:25:16] Herrington: WebAssembly Time does not support these, these smaller platform. So that's why. Oh, because I already built, ast-grep, pre-built ast-grep for these platforms. So, so if WebAssembly Time only support these platform, well, it does not actually add new things because for a larger support platform, WebAssembly Time does not support them as well. I decided, okay, I didn't get enough, say gain from supporting WebAssembly Time or WebAssembly, so I j- just dropped them

[00:25:48] Jakob Heuser: And it, it doesn't seem to have hurt the, number of languages that are showing up that are both in tree-sitter and ast-grep. Like, I saw ast-grep 0.5 just came out with Zig support. What's driving all the languages? Like, there's just been this flurry of new languages landing in ast-grep, and I mean, that's very exciting obviously as a downstream consumer because it means the stuff we're building works in more contexts as well.

But it definitely feels like it's either gotten easier or the tools have gotten better to where there's more languages coming up much quicker now

[00:26:23] Herrington: Oh, yeah.

[00:26:25] Language support and security

[00:26:27] Herrington: So it's, so supporting new, new language in ast-grep is very, very easy. Like we talked before, ast-grep and the tree-sitter are language general. So you just need to grab, tree-sitter grammar into ast-grep's repo, and then write some, a very few lines of, configuration code, like how your language handle, say the dollar sign, because dollar sign is special in ast-grep's pattern syntax.

So you only need to configure this, the, the, the, we call it a meta variable, the, the dollar sign, how you handle dollar sign in your language. that's it. Then write so-some, some code testing, and you are good to go. So

[00:27:05] Jakob Heuser: awesome

[00:27:05] Herrington: new language is very, very easy thanks to tree-sitter. but at the other hand, as a library maintainer, I have to if it makes sense to include some language as built-in.

So it has to have a, a big ecosystem or some interest, in ast-grep. Otherwise, I would say, you can use our custom la-language lo-loading scheme. So you, you can, you... Though you will not have, it built in, you still have something you can run

[00:27:40] Jakob Heuser: Yeah, and I think in the, most recently in the CLI too, custom languages are no longer on by default. You have to opt into them for security reasons.

[00:27:49] Herrington: yeah. Yeah, yeah, yeah,

[00:27:49] Jakob Heuser: Because again, agents have started getting clever, and that sort of just necessitates needing to, needing to improve our sandboxes

[00:27:57] Herrington: yeah. And because, yeah, custom language is, implemented using dynamic loading. So it, it's permission-- It doesn't have permission. It, it is in a sandbox. So

[00:28:08] Jakob Heuser: Yeah, it, it inherits whatever ASC grape runs in, right?

[00:28:11] Herrington: It can, can, it can do more. Like ast-grep actually just, touch your, your file system and, read some code. It's quite safe, but using Dynat library means you can run arbitrary code. you can call, you can, you can call a process, you can spawn, you can call, HTTP call. Like all these things that no one stops you to do that. So, so somehow I, I have to, to add some opt-in mechanism. So if you are, some, you are cloning or, or, or reading some, say untrusted code repo. Like y- they can just put something in their sgconfig.yml, in the, their ast-grep, and when you run that, that, the ast-grep it automatic- automatically load the, the malicious code.

So this is very dangerous

[00:29:05] Jakob Heuser: Yeah, so anyone on ast-grep need, needs to go through the upgrade 'cause it's, it's one of those things you could just run ast-grep in anyone's code base and surprise, you've just loaded arbitrary code without realizing it

[00:29:19] Herrington: Yeah, yeah, yeah

[00:29:20] Jakob Heuser: And technically other libraries have this problem. I think the approach you did with the opt-in is fantastic.

I know a lot of other, syntax tree parsing libraries that have custom modules and runtime, runtime code. They do not offer any sort of guarantees or assertions about the code. So if you run ESLint and there are custom plugins, that JavaScript code is going to run, and it might very we- a custom code very well might have just done a read file on your .ssh directory.

[00:29:51] Herrington: Oh, yeah. Yeah

[00:29:52] Jakob Heuser: And that's like a real concern 'cause like nowadays, as you said, agents do so much code, and an agent's decision to check code, it might check out a code base, it might run ast-grep, ESLint, PyLint. It might run one of those straight away, and in doing so could potentially just be running arbitrary code without realizing it.

[00:30:20] Herrington: Yes

[00:30:21] Jakob Heuser: And to the agent, even with like Claude's auto mode and like permissioning, it looks like running PyLint. Running PyLint should never have been an unsafe operation in the first place. And I think you realized that a lot faster than a lot of others, where you were like, "Wait a minute, this can just run whatever it wants.

This isn't actually safe. This needs to be, this needs to be gated in some way

[00:30:43] Herrington: Yeah.

[00:30:44] YAML rules and JSON schema

[00:30:45] Herrington: Yeah, say one thing ast-grep is good i-is that, say, the rule is written in YAML. oh, well, YAML isn't that safe to, to be honest. but at least in Rust YAML's library, you cannot run arbitrary code like, but, but that's not same for JS YAML or Ruby YAML. But at least it, it's, it's safer

[00:31:08] Jakob Heuser: Yeah, one of the things that, Taskless did was, because we really like-- We use ast-grep heavily, as one of the libraries that lets us build a rules engine for agentic coding. One of the things that we started doing was we take the YAML, the YAML rule that an agent creates, and we don't trust it.

But I can turn YAML into JSON, and I can take that JSON and run it against ast-grep's published JSON schema. And so once I have those pieces, I can now effectively assert the correctness, a very different type of correctness, on the generated YAML. And this has been an absolute, just force multiplier in getting an agent to write a really good ast-grep rule because it will sometimes wander.

It will sometimes mess up kind declarations. It will sometimes, for a while there, it would use synonym in Haskell when it was actually supposed to be synonym because there was a typo. And it did, it didn't know better, but the schema knew, and the schema could tell that it was wrong. And if you just have a good JSON schema parser, you could actually guide the agent to fixing its own YAML,

[00:32:25] Herrington: Yeah,

[00:32:25] Jakob Heuser: which then ended up being very safe YAML for Rust, and everything is a lot more secure all the way through.

[00:32:32] Herrington: Yes, yes

[00:32:33] Jakob Heuser: on that, I

[00:32:34] AI writing ast-grep rules

[00:32:54] Jakob Heuser: think it's really kind of fascinating. AI generating rules is a very sort of new idea here, that you could just ask your agent like, "Hey, I want a rule that presents, prevents four levels of infers in my TypeScript generics." And then the agent figures out what to do with that. about a year ago, rule writing with ast-grep using an AI was difficult.

The model corpus wasn't that good. There wasn't enough training data. There wasn't enough examples. I don't think that's the case anymore. I'd love to hear your thoughts. You, you have some sort of firsthand experience with people using ast-grep, and it seems like it's writing rules much better than it ever was before.

[00:33:16] Herrington: Oh, yes.

[00:33:17] Jakob Heuser: being AIs at this point

[00:33:19] Herrington: Yes, yes. Yes, I have tried AI with ast-grep for a long time. Like, my initial foray to AI writing ast-grep was-- came back to twenty twenty-four. Like tw-- yeah, early twenty twenty-four the agentic programming is nothing everything we do is prompting engineer, like write, to construct a well-crafted prompt sent to AI and pray for a, a response from, from AI.

So that's back then. AI back then wr-wrote cannot, really cannot write anything useful. Like they, they hallucinate a lot. Like, "What's ast-grep? I don't, don't even know that. Oh, it's a pattern matching language? Okay, I write some-something for you." And then everything explode, nothing works. so I tried something, called, using DSPy. So DSPy is a framework that you can, say, orchestrate AI program-- AI's response in a bigger workflow use something like, curate some data set and, work-- run the workflow again and optimize that. I tried that, but unfortunately it doesn't work that well. Like, the issue is the, the atomic, the LLM isn't that smart at that time.

Like, if I, we-- I-- what I can do is, that's my workflow. If I remember correct, I asked AI first to translate user's intent, into some code example and propose, in human language how to, how to, say, match this code example. then using that, that description to, to break down some sub-steps that can translate directly into ast-grep rule then compose them together and then run some, some test case. unfortunately, it didn't work well, because every step AI struggles to do, do that. Like translate in-intention, that's maybe the best case. its pass rate is six percent, if I, I remember correct, in my experiment. But when, we want to translate the intention into sub-step, well, that, the, the, the, the, that, that, that's, that was, AI PS code.

Like you can-- you really cannot extract anything useful. And then translate, sub-steps to the rule. Well, that also failed because AI cannot write one single, say, say, valid ast-grep pattern, so that also fail And then, then, about one year later or, or by the end of 2024, yeah, by the end of 2024, for like November 2024, Cursor went out and they, they gave us something like agent coding. Oh, that really worked. That's my first time thought, wow, AI now can write, AST grep rule. So it somehow, it still hallucinate. It still makes AST grep with, with other tools, the syntax, somehow it can now translate my intention faithfully to, to a rule. It's just atomic. it d- doesn't work that well, but it can some smaller rules. And, then it came out Opus five or 4.6, 4.8. And when the agent words comes out, changed. Everything changed. So, so AI can compose very compli- complex AST grep rule for me. And to be honest,

[00:36:53] Jakob Heuser: Y-

[00:36:54] Herrington: AI now writes better rules than me.

[00:36:59] Jakob Heuser: Oh, that's good. you also, you had a loop that you used. You talked about it where you write a s- you write a snippet, and then you write the rule, and then you test the rule against the snippet.

[00:37:09] Herrington: Yes

[00:37:10] Jakob Heuser: And I think that one of the cool things, we found this out at Taskless as well, was there's a point in time where you realize the AI has made a mistake in your code.

And at that point in time, you have the best example you could ever ask for of the wrong behavior. And if you can save that and you can work with the AI to fix it and make it right, now you've got the example of the right behavior. And that is the two hardest things that are necessary in order to set up an AI s- for success if it's gonna write your rule for you, is you now have your canonical failure case and your canonical passing case

[00:37:49] Herrington: Yes. So yeah, AC Grape also has a built-in test command for that. So I hope it will help you.

[00:37:58] Jakob Heuser: Yeah, it does. it helps a lot. And then one of the things that we started doing was we're expanding and adding full fixtures because we also do cross-file and variant checks. So alongside your rule, there's like a .test directory, and then it has multiple test fixtures that create simulated examples of this sort of code e-example should fail, this sort of code example should pass, to just go beyond the, the YAML-based pass-failure fixtures.

And then you're just running ast-grep against, literally against the local .test directory that's right alongside your rules, being like, "Are all of my, are all of my canonical examples correct or not?" And it's, it's helped a lot. and obv- and obviously, like, I think we're just kinda coming at the same problem from two different ways here.

Taskless was starting from this idea of like, you've already got a problem, and you're like, "I don't want this to ever happen again." And some of the stuff you're proposing as well is like, how do I start from a plain text, I don't want this to happen in the future, and how do I get an AI to iterate to a point where it can create a reliable rule?

And I think both are necessary. You have to be able to generate the test. You have to be able to verify. And tools like ast-grep have actually made that loop for agents very, very easy. And so we all get better rules as a result

[00:39:18] Herrington: Oh, yes. Yes

[00:39:21] Jakob Heuser: some quick

[00:39:21] The AI hype cycle

[00:39:23] Jakob Heuser: questions on the way out here. where do you feel we're at in terms of the AI hype cycle? Is it, is it all hype? Are we finally over the hype and getting into the useful bits? Was it ever, was it ever hype to begin with? Where are you at along that sort of cycle?

[00:39:37] Herrington: Oh, wow. I think that, depends on the industry. Like in IT or in software industry, AI has passed the, the hype, period. Like, I think most of, most, most developers now using AI to generate code. And, but in other industry, I don't think it's necessarily over the hype. Like, or it's, it's probably under the hype. Like in, but that, that depends. Like, if you are in, say, say, media industry, you are in filming industry, so you can use AI to create, a lot of, like this short film or something. Like are real, and they impact the industry. So, like it or not, but that's the, the s- s- that, that's the reality that you-- we have to, to embrace. But in other industry like finance, or like, like, like legal, I believe there are some effort, to push AI, but they are not wide enough

[00:40:38] Jakob Heuser: Yeah, there's,

[00:40:39] AI in creative industries

[00:40:45] Jakob Heuser: there's a lot of scenarios too where even in creative industries like film, AI can be used for the non-creative bits.

[00:40:50] Herrington: Yes,

[00:40:51] Jakob Heuser: we do with this podcast, for example, is done in this tool called Descript, and it's just JSON under the hood. And splicing together the scenes, removing transcript pauses, building the transcript itself can all be done with an LLM and save us a lot of production headache.

It's very menial work that honestly gets out of the way so we can go have a really good hour conversation

[00:41:20] Herrington: Yes. but back then if we want, want to have a interview and have a podcast, well, the editing part will be like hours and I... or days now.

[00:41:31] Jakob Heuser: Yeah, and it,

[00:41:32] Herrington: want to do that

[00:41:33] Jakob Heuser: if AI can get us to 80% so I can spend some time instead just making sure that we have taken out pauses, that we have a really tight set of show notes, that your li- socials are linked, to me, that's where all the value is. That's the good stuff

[00:41:49] Herrington: Yeah, that's the, like, the boring part AI can take care of it. But the, the interesting part, like the, the creative part, I don't think AI can do that.

[00:41:58] Jakob Heuser: Yeah, and I, I don't think it should. I think the human part is what makes this whole thing kind of fun

[00:42:04] Herrington: yes. Yes. For example, if you saw some, claims over, over the internet about, "Hey, I can use AI." Well, Opus 5.5 is, amazing that it can create these games. It can. Like, it can generate some small game demos, but they are not fun because there is no creativeness in that, right? So why we play games? Because there are human creativeness there, and we appreciate the, the fun part where we human appreciate another human's creative.

[00:42:39] Jakob Heuser: Yeah. And I

[00:42:40] Herrington: that...

[00:42:40] Jakob Heuser: I don't think AI can ever replace that part, 'cause that's... I don't know, that's actually the neat part.

[00:42:46] Herrington: Yep

[00:42:48] Jakob Heuser: editing for me though. It can, it can eliminate silences, it can normalize volumes, it can do all that for me. That's, that's fine

[00:42:54] Herrington: But it cannot replace you and me, right?

[00:42:57] Jakob Heuser: No, and that, I think that's probably a good thing, at least near term.

[00:43:00] Herrington: Yeah, I'm real.

[00:43:02] Jakob Heuser: Yeah, and,

[00:43:02] Where to find Herrington

[00:43:05] Jakob Heuser: and speaking of, where can people find you? So, I have you down as HD_nvim on X and hd-nvim on BlueSky. Is that correct?

[00:43:14] Herrington: Yes, that's my primary, appearance on, on, on the internet

[00:43:19] Jakob Heuser: Okay, and then obviously you've got a GitHub as well. We'll be sure to link ast-grep. We'll be sure to link, tree-sitter as well, for which you're a contributor, in the show notes, just in case people wanna dig into some of these technologies. 'Cause I think syntax trees are probably gonna be one of the best ways for us to manage the sort of entropy that coding agents introduce.

[00:43:40] Herrington: Yes

[00:43:41] Jakob Heuser: Everybody, I'm Jakob Heuser again, Task at Hand, having conversations today with Herrington Darkholme, creator of ast-grep, contributor to tree-sitter, talking about AI agents. I appreciate you all. Take care of yourselves and I'll see you next time on the Task at Hand

[00:43:57] Herrington: See ya