Chris Griffing (GitKraken) on Where Agents Still Need Your Taste - Episode 7

[00:00:00] Welcome to The Task at Hand

[00:00:00] Jakob Heuser: What's up, everybody? Welcome to The Task at Hand. I am Jakob Heuser, founder at Taskless.

[00:00:05] Today, I am talking with Chris Griffing, AKA cmgriffing on Twitch, also a DevRel at GitKraken, builder and maker, creator of things. It's gonna be really exciting. He's gonna be talking about a lot of stuff he's building. He's on the front lines building things with AI, and that's not how things started. So we're gonna hear a little bit about how Chris went from snowboarding to development to slinging code several hours a day in front of a live audience, and how that's changed.

[00:00:33] That's happening today on The Task at Hand. Chris, welcome to the show.

[00:00:37] Chris Griffing: Thanks for having me

[00:00:38] Jakob Heuser: Yeah, so let's just start with where you're at now. You stream on Twitch. You actually build things live. And first off, I think that's just amazing and cool. it's like doing a live interview in front of a studio audience several days a week.

[00:00:54] And let's just start there, 'cause I don't think you started as a streamer. You started as a snowboarder, right?

[00:01:01] From Snowboarding to Software

[00:01:01] Chris Griffing: Yeah, so when I graduated high school, I didn't have the best grades, so I knew I wasn't gonna get into any, good university or college, so I could do community college at any point, and I decided to be a snowboard bum. So I went to Mount Hood that summer after graduation, and, yeah, I fell in love with it and started doing that, 200 days a year.

[00:01:18] Or, let's say 150 days a year for the next 10 years

[00:01:22] Jakob Heuser: You and I were looking for sponsorships at one point. I think I remember seeing, you were, you have, footage. You have B-roll

[00:01:28] Chris Griffing: Oh, yeah, yeah. Yeah, I have a Sponsor Me video from 2006. I feel like they recompress it every year on YouTube just as, a celebration of how long they've been around, 'cause there is so much, JPEG-style artifacting in it that it's bad. But yeah, I got free gear. I never got a paycheck from a company, but free gear is, couple grand worth of gear a year, so

[00:01:46] worth it

[00:01:47] Jakob Heuser: That's, seriously pretty dope. But obviously, that doesn't pay the bills.

[00:01:50] Chris Griffing: No, definitely not

[00:01:51] Jakob Heuser: but snowboard and software, beyond them both starting with S, how did you actually get to software then? 'Cause that's... You don't go halfway down the mountain and be like, "I should be a web developer."

[00:02:01] Chris Griffing: Well, I mean, I'm old. The beanie maybe doesn't show that, but I'm like, I'm 42 now, and, like I grew up with GeoCities, which taught me HTML. I grew up with MySpace, which taught me CSS. I did some Flash stuff in high school, like so ActionScript. As a millennial, they didn't have the tech to abstract these things away from us, so we just had to figure it out and learn it.

[00:02:27] So I think I was already kinda set up for that. But also, my parents were in tech as well. They were in like telecommunications, and we moved around into many different states in the country just for them chasing like the next big contract, which was fun

[00:02:40] Jakob Heuser: Okay. So what was it like when you started in software then? So I think we started in tech close to around the same time then. and it was "something planet" everything, like it was 2000s era. It'd be right around there

[00:02:52] Chris Griffing: I started, properly in tech, I wanna say, maybe 2008 or '9. I started building, a WordPress site for my friend's snowboard shop, and then pivoted that to a Magento site for e-commerce, and then during that time was getting my two-year degree over the course of, three and a half years, 'cause snowboard bum only did it during spring and fall when I could. but really, heavy into JavaScript frameworks and things like that around, let's just call it 2011 and 2012.

[00:03:18] Jakob Heuser: Okay. So

[00:03:19] Chris Griffing: When did you start?

[00:03:20] Jakob Heuser: That was a, that was a pretty, that was a pretty big time. Right around 2011, 2012 was, React was in its infancy. that sounds like around React, like zero dot, like it was in zero dots, like super early zero dots

[00:03:33] Chris Griffing: Yeah, I don't remember React being much of a thing until really 2014.

[00:03:39] Jakob Heuser: Yeah, so this would've been

[00:03:40] Chris Griffing: did that first React Conf talk and did the DBMon like visualization and all that, right?

[00:03:45] Jakob Heuser: Yeah, so 12 would've been Backbone, I think, like Backbone Marionette type stuff

[00:03:50] The Ember Years

[00:03:50] Chris Griffing: That would be for like React devs, but I was in the Ember community.

[00:03:53] Jakob Heuser: Oh, cool

[00:03:55] Chris Griffing: So back in 2012 when I first started going to like meetups and some of these things to try and like break into the industry and get my first real tech job instead of just freelancing, which sucks, I was going to the Ember meetup and there was like 100 plus people going there.

[00:04:06] Ryan Florence was the person, or him and Michael Jackson created Ember Router, which is like the prototype for all routers at this point anyway, right? The way that like the pages and the routes are represented, all that stuff was the gold standard, right? Before that you had AngularJS 1.x and their router was not great, So, the Ember community really did push some of those boundaries. Like Ember CLI was the first like CLI focused for scaffolding and building out like front-end frameworks like that. Angular CLI, when Angular 2 came out famously showed Ember CLI in like the footnote 'cause they just forked it. 'Cause it was like it was such a good tool. but yeah, like starting out in that era and seeing just the churn from framework to framework over time was really interesting and quite fun. Like I still remember Meteor and I still have a Meteor project running right now,

[00:04:57] Jakob Heuser: Wait, wait, you still have a Meteor project running? Tell me about that

[00:05:00] Chris Griffing: Well, I think I made it in 2011, 2012 maybe. And it's just a tool for chat to, vote on things. And you see, the bar of, the vote going up and down. But I wanted it to be something that was better than just, the Twitch chat poll. I wanted to be able to have more than three items. I wanted to, see the things changing in real time, and I wanted to maybe show it on stream. And Meteor, with its just, the way that it long tails the Mongo instance, it made it to where, this, interactive live feed thing was easy without having to, do the WebSocket stuff myself.

[00:05:34] Jakob Heuser: Was that one of your first projects on a stream? 'cause that sounds...

[00:05:37] Chris Griffing: No?

[00:05:38] 2011, 20... Or not 2011, 2021, 2022. So I

[00:05:42] I had already been streaming for, three or four years at that point

[00:05:44] Jakob Heuser: Oh, dang. So yeah, so you've been, you've been in the streaming game for a while

[00:05:47] Chris Griffing: Yeah, since 2018.

[00:05:49] Jakob Heuser: did it did it coincide with full-time software or did it come before or after?

[00:05:53] First Streaming Attempt at Mixer

[00:05:54] Chris Griffing: came after full-time software. So I got my first full-time gig around 2014, 2015, something like that. but I had interviewed at Beam, I don't know... Or technically Mixer, once Microsoft bought it. So for people who don't remember Mixer, it was this thing that was called Beam that was owned by a kid who started out doing it as, a Minecraft hosting, server thing, and they wanted to stream video for it, and they created FTL, and FTL now powers Twitch, and it was, like, what Microsoft bought, Beam for and turned it into Mixer.

[00:06:24] And that video tech is still now alive in Microsoft Teams, as an example. So I was interviewing for Mixer, because I had a friend that had gone there, and during the interview, I'm sitting there chatting with the guy who was the owner, the CEO, whatever. I don't remember what his title was at Microsoft, probably VP something or other. Young kid, very cool, and he's just chatting. We're, like, talking about games we play, and he's like, "Man, it'd be cool if you streamed 'cause then you'd understand our customers better." And I was like, "Oh, yeah, I'll give that a shot." which was really fun. And so I gave it a shot. I did both Mixer and Twitch at the same time at one point, and the numbers for software dev on Mixer were really bad, right?

[00:07:05] Over the course of one stream, I might get, 30 to 100 viewers on Twitch, and over the course of a month on Mixer, I got 30 views that number was no good. So I just stopped streaming to Mixer

[00:07:14] Building a Streaming Audience

[00:07:16] Jakob Heuser: But you, but you actually did start streaming, and that's, that's not nothing because you started streaming before Twitch looked anything like it does now. and also before software looks like it does now. Really

[00:07:27] Chris Griffing: have a programming category that was underneath, I think creative at the time.

[00:07:31] Jakob Heuser: Okay

[00:07:32] Chris Griffing: some people like, his name was Adam13531, back then, but now it's like Adam Learns Live. Awesome streamer, really cool dude. He's also up in the Seattle area too. and like seeing him just working on side projects was exactly what I wanted to do too, which was, yeah, a big inspiration

[00:07:49] Messijo and Lenses

[00:07:49] Jakob Heuser: And that's, that's what your channel is now. Fast-forward to 2026 and, you've built a ridiculous amount of things on stream. So like what are the current projects? 'Cause I know you've got a couple in the hopper

[00:08:00] Chris Griffing: Well, let's see. I mean, Messijo is my current, like SaaS that I'm trying to monetize and all that, and it's just, listening for social media keywords. And then I have a thing called Lenses that analyzes, takes a prompt from the user to see if it matches a certain prompt. So if you're looking up the word elixir, you might get stuff that's related to the Elixir programming language, which you might want, but you might also get things like Dungeons & Dragons type content because elixirs exist in, the fantasy realm. So could you ask an AI, "Is this related to the NFL?" Boom. Or not the NFL, sorry. I use football as a different one. But, is this related to a new feature in the Elixir programming language? Boom. And then you can sort, my example is, three thousand six hundred events down to fourteen.

[00:08:44] Jakob Heuser: Okay.

[00:08:45] Chris Griffing: yeah.

[00:08:45] that,

[00:08:46] thing, and it's really fun to build.

[00:08:48] Jakob Heuser: and nowadays that, even that gets offloaded now that Jev's around with classifiers. Like the whole what is this questions have changed. And I think that's kinda cool because I've been modding for your channel and I've watched you go live from basically a week to solve a UI problem to bing, bang, done, six hours, whiskey in hand, yapping it up with people.

[00:09:09] Dev just doesn't look like it used to anymore

[00:09:11] Live Coding on Twitch

[00:09:13] Chris Griffing: Yeah, I mean, even before AI I would have the whiskey in hand. I've stopped drinking on stream these days at least just, keeps it a little more even-keeled. Sometimes I'd go off the rails with the whiskeys. that's, that's for off-stream content, but I don't know, even then before AI was a big thing, it was still like be working and doing stuff, and I'd have the whiskey, and we'd be chatting, and people would be surprised that something would even happen and get done during that time 'cause they just see me like BS-ing the whole time, which is funny.

[00:09:42] Like, when I interviewed at Mixer, one of the interviews with, was with, Connor Pete, and they had this thing for, It was like a toy, problem that they solved in the app of doing the emojis. And if like I have my chat overlay, I've actually done that in my own chat overlay of just replacing words or whatever with an emoji.

[00:10:00] That was the task, and I'm sitting there just BS-ing with him like, like I do on stream, and like hammering it out. I go to run the code. Oh, I forgot the zero case of an empty string. Solve that. test passes, and then boom, 100% test passes, and he just goes, "Wait, what?" Because he-- we were just BS-ing, So I've kinda built that skill since before streaming even.

[00:10:20] Jakob Heuser: It's a good skill. It's a good skill. And you get a lot done when people aren't looking. You also only do all your work on stream. That was the other thing I thought was fascinating. you don't build anything outside of the stream. You do it all live. Scuffed and all

[00:10:33] Chris Griffing: Yeah, I mean, it's, it's also like a motivational thing for me. There are some days where I just wanna work on the project and some days where I just wanna chat. But if I have that double motivation to click that go live button, it keeps me going live five days a week, because I have like, when one thing lacks the motivation, the other thing kinda lifts it up

[00:10:54] Jakob Heuser: That makes sense. And Messijo's required a lot of attention.

[00:10:57] Chris Griffing: Yes.

[00:10:58] Jakob Heuser: Like it is,

[00:10:59] Chris Griffing: yeah. It is, it is. I have several babies, right? There's, my feature flag tool. There's the one for, like monetization of, incremental billing, like usage-based billing. I have several of these, but I think Messijo is the easiest to maybe, get over the finish line to, monetize.

[00:11:15] Messijo Before and After Agents Got Good

[00:11:15] Jakob Heuser: Yeah, so let's talk about your baby because it's been going on for so long as a project.

[00:11:20] Chris Griffing: Mm-hmm

[00:11:21] Jakob Heuser: Um, it has the unique joy, if joy's the right word for it, of being on both sides of when agent output got good. You there was Messijo before Claude could find its way out of a paper bag, and there's Messijo now of throwing all of the menial tasks, and sometimes the more complex tasks, but mostly the menial tasks, at it.

[00:11:47] And I want you to share a little bit about what that moment looked like inside it because you had a software project that was in flight when there was such a disruptive technology that landed

[00:11:59] Chris Griffing: Yeah, so I think it's still let's just say a year ago, the models like turned a corner and got really good at code. I still didn't actually use any of the models for Messijo stuff until maybe three or four months ago, just because I still wasn't convinced. I was still seeing a lot of garbage output. I really liked it for greenfielding a prototype, for things like that. But when I really started trusting the models to be inside of the Messijo code base and like doing things and changing things, that's actually still somewhat recent. and I think the main reason is I still wanna be able to understand when something goes wrong, where should I start looking, like real quick.

[00:12:41] And the more you let AI take the wheel on that without paying attention to the code, the harder that task is. And for something that I'm planning on making money off of, that I'm planning on guaranteeing uptime, doing SLAs, et cetera, I need to be able to get in there ASAP and fix the thing without burning a, like a billion tokens figuring it out

[00:13:01] Jakob Heuser: Yeah. talk to me about that moment because I've seen you on stream. There's still a lot of vibe coding that goes on with things like Messijo, but then there's also a lot of getting into the nitty-gritty and going back and looking at what it wrote, and critiquing what it wrote, and fixing what it wrote.

[00:13:17] Because in your own words, even the best harnesses now still don't do it the way you would do it, and it needs to fit your mental model 'cause you're the, you're the guy on the hook for supporting this later

[00:13:29] Where Models Still Lack Taste

[00:13:29] Chris Griffing: Yeah. Yeah, so I have a couple good instances of that from Messijo. the one that I think sticks out most recently where I just had to go in and do it myself was I was implementing the MCP server, which meant implementing the OAuth flow for that, like, consent and all that stuff, and granting, which organizations you want the token to represent, just security-wise. That form, for the consent form, it did an okay job at first, but the, the line items weren't, fully clickable. they looked like a button, but you had to click the text explicitly because of how it was doing the label for the form element, right? this is one of those things where I've been a front-end engineer by my career since I started, right?

[00:14:10] 2015. I have taste and opinions about what makes a good user experience that I still think models are terrible at. They can make things look pretty, but they don't actually make things usable in a way that someone with expertise can do

[00:14:24] Jakob Heuser: I heard someone describe it as AIs are really good at the tactical, but they're really bad at, the systems level, how all the pieces fit together

[00:14:32] Chris Griffing: Yeah. I think this is, this is an analogy I like to make all the time. So there... I forget the name of the law or whatever, but it's from, like, the '90s, the '80s, whatever, where you could be reading a newspaper and, you flip to an article that's about something you know a lot about, and you start reading it, and you realize how out of touch and how incorrect the author of that article is, right?

[00:14:56] Like, all the things that you can just, nitpick and understand, well, no, this is just a reporter reporting outside of their boundaries. And then you flip the page, and you look at an article that you don't know anything about, and this amnesia kicks in where you just realize, or you don't realize.

[00:15:10] You just think, "Oh, okay, I'm gonna take what they're saying verbatim because I don't know enough to, critique it and tell it when it's wrong." That's actually exactly the same way that I think AI works. People who don't do front end a lot think front end is solved. People who don't do back end a lot think back end is solved, when really, if you have taste, you have to, really exercise that taste at every level.

[00:15:32] None of these things are fully solved

[00:15:34] Jakob Heuser: Yeah, that makes sense. And it, I think for you in particular with Messijo, you're on both sides of this. So you have had to, you've had to deal with everything from the Bluesky firehose, which is just atproto and, a nonstop deluge of information, to the database layer, to the auth layer, to the front end, which by the way, has an excellent...

[00:15:55] I think it's that bright yellow blocky style is so signature. It's so good

[00:16:01] Full Stack, and Letting AI Tune the SQL

[00:16:01] Chris Griffing: Yeah. I mean, I've been full stack my entire, dev life, but as far as what people paid me for, it was, like, leading front-end teams just because that's what they needed at the time, and we had plenty of people that could do the back end in whatever we were doing. So yeah, my career just funneled towards front end and making that stuff happen. But as far as the back end goes, I still have a, the first, RustConf T-shirt from twenty sixteen. I've been heavily interested in all this stuff. Elixir was big in the Ember community, because they stemmed from Rails, but Elixir was this performant thing that had that same beautiful syntax but a really good framework like Phoenix that replaced the need for Ruby on Rails. so I've just been fascinated with all of this stuff for ages. On stream building Messijo, I've had to really get in and optimize some of the SQL queries, and AI has been really good at that actually, right? If I get a first pass SQL query joining the data I need, making sure I get what I want, but then realize that it's slow, it can optimize that.

[00:17:02] Not-- And it's not like missing indexes. It's like, "Oh, how could we do this in a way that the query optimizer can actually, really get in and, fine-tune?"

[00:17:11] Jakob Heuser: The kinds of things you used to have to use a DBA for, like

[00:17:13] Chris Griffing: yeah.

[00:17:14] Jakob Heuser: specialized experience

[00:17:16] Chris Griffing: Yep, exactly. and, that's also like hearkening back to my parents doing dev. That was one of the things my mom did for ages.

[00:17:23] She would optimize SQL queries at Microsoft, towards the end of her career that were running for, forty-eight hours down to the matter of, six hours, which when you're doing a business query like that's like literally quantifiable, money saved,

[00:17:34] Jakob Heuser: Yeah, that makes sense

[00:17:36] Chris Griffing: but yeah. So, optimizing these database queries, I have, some taste, but I haven't been that deep like a DBA would be, like, spending their entire career doing it. AI helping me solve that was really useful, but I would do it in very small chunks 'cause I didn't want it to rewrite the entire thing 'cause we know how it loves to create needless abstractions, right?

[00:17:55] The Rule of Three

[00:17:55] Jakob Heuser: yes. I have a rule in my, in my, both in Claude and then obviously day job in Taskless that prevents any functions that are less than five lines long.

[00:18:04] Chris Griffing: Great.

[00:18:05] Jakob Heuser: It's banned. I look at the function body, and if it's got, if it's got five lines or fewer, that is just premature optimization.

[00:18:12] Stop doing that

[00:18:13] Chris Griffing: I mean, if it's being used in a bunch of different places, then maybe, sure. But it's rare. It usually does this when it's only used in one place,

[00:18:21] Jakob Heuser: Yeah, even twice. Like I have a general rule, like I gotta have three before I really wanna, I wanna invoke the DRY mallet and like refactor stuff

[00:18:30] Chris Griffing: Yeah. I like the rule of three as well because it's not a hard rule. It's, the rule of three could be the third iteration or it could be the third repetition, which would be number four,

[00:18:39] right?

[00:18:39] Jakob Heuser: I'd, I'd posit another one. I'm annoyed by it three times.

[00:18:43] Chris Griffing: Oh, there you go. Okay

[00:18:44] Jakob Heuser: 'cause if I see it a bunch and I'm like, "Eh, it's not that bad," then it's really not that bad. But if I see it and I'm like, "God, I gotta do something about this," and that's the third time I've said that, I'm like, "All right, let's go do something about that

[00:18:56] Chris Griffing: Yep, the thing on the backlog becomes a thing on the current sprint.

[00:18:59] Jakob Heuser: Now I can throw it, now I can throw it at the LLM potentially.

[00:19:02] Chris Griffing: Exactly. Yeah, for sure

[00:19:03] Jakob Heuser: Messijo is actually not only you've used LLMs in development, a part of Messijo runs on top of LLMs. there was a talk you gave called My AI Braindump, and I'd like to hear what was it about running LLMs in production that you learned that coding with them could never have taught you?

[00:19:22] Running LLMs in Production

[00:19:23] Chris Griffing: Well, yeah. So in production, you typically have these things where you just wanna do a quick request response. So you're not worried about context rot because it's-- the context is gone after a single request response cycle. So like some of the concerns you might have are less. I was using GPT-OSS 120B for the lenses, the things that would do like that filtering of the regex firehose, and the model I was using, GPT-OSS 120B, was cheap.

[00:19:54] It was pretty intelligent, but depending on the provider you were using, right? Because I'm using OpenRouter, I could route between all sorts of providers. I have some that I ignore 'cause they're always too expensive, and I have some, grudges. We don't need to get into those, but I do have grudges against some of those AI provider companies. But, for GPT-OSS 120B, it's an open model, so there's seventeen providers that you could, pick and choose from that are affordable and fast and worth using. I was running into issues where at least a couple of them, I think DeepInfra or, one of them, and then, Novita were giving me back a bunch of, garbage output.

[00:20:33] But the garbage output wasn't, an invalid response. It was just padding it with thousands of tokens of semantic-- or not semantic, of of white space. So, it would have a bunch of tabs and spaces in places where it didn't need it, and it was going from what should've been maybe two hundred output tokens to, seven thousand, which, output tokens are the most expensive.

[00:20:53] So the cost I was saving on using a cheaper pr-provider was getting thrown away, let's say five percent of the time,

[00:21:00] Jakob Heuser: Oh, wow

[00:21:01] Chris Griffing: just terrible. But Novita fixed it. I think, I could go back and look at one of the other providers that was messing it up that I still have turned off 'cause they never got back to me when I reported it. But Novita was like, "Oh, thank you." Right? I reported it to them. They got back to me via DM on Twitter and, resolved that and fixed it. And yeah, so Novita gets a big thumbs up from me

[00:21:20] Jakob Heuser: Nice. And there was another one you talked about too on that talk that, originally I I've made this mistake, so I'll be right here on the podcast saying it straight up. I prompted asking for JSON output without using tool use. I am a bad engineer that did not know better. and learning about tool use was one of those moments where it was like, oh, this changes how I interact with an LLM suddenly as a service

[00:21:49] Structured Output and Reasoning Tokens

[00:21:49] Chris Griffing: basically what, this is less tool use and more of, like, a structured output, right? So you can ask for a schema back when doing a request to the OpenAI compatible APIs, which OpenRouter supports. So when you ask for that schema back, it then does some, reasoning and stuff and outputs it directly into that schema. I was also, testing and validating whether telling it in the input prompt to do it as JSON was useful, and I started out doing it that way. Someone told me about structured output, so I did that. But then I actually, looked at the numbers that were coming out of it, and by asking it for JSON structured output in the prompt, it was more input tokens, sure, but it was somehow saving tokens on the reasoning to get it to the output, because reasoning is billed as output. So because output's more expensive, by saving on the reasoning, by giving it just a little bit more in the prompt, I was able to make a savings of, I wanna say, ten to twenty percent. I forget what I, what I had measured, but it was, like, a measurable output when, if you're doing hundreds of thousands of tokens a day, that starts to add up

[00:22:57] Jakob Heuser: Damn. Yeah, that over the lifespan of something like Messijo, which is supposed to be ingesting everything that comes through Bluesky ever in the history of Bluesky, that's a lot. Like that adds up quick

[00:23:09] Chris Griffing: Well, keep in mind, I'm only running the LLM on things that we've keyword matched with the regular expression first, because regular expressions are much cheaper, right? But then once we've matched, again, there could be an entire just a massive amount of false positives with just a keyword match.

[00:23:24] So then doing the LLM or technically Jev now, to evaluate these things is massively, cheaper and more performant. But like optimizing that, I'm still saving myself money,

[00:23:35] Jakob Heuser: Yeah, that makes sense. I promise I won't go into Messijo after this talk and add keywords like and, of, by, but, and just see what happens.

[00:23:42] Moving Lenses to Jev

[00:23:42] Chris Griffing: Oh, go for it. I do that with, ... I've done it with foo, which gets like, foot, football, fool, right? All that, just so that I could evaluate that. And when I was looking at Jev that first day it came out, I did foo, and we got 50,000 events over the course of the day before it, cut off because I have billing metrics of like when we just stop serving you that stuff. But 50,000, it was 80 cents, theoretically because it was free from Jev for that first day. and it was like $2 with GPT-OSS 120B. So a massive savings,

[00:24:16] Jakob Heuser: Oh, yeah.

[00:24:17] Chris Griffing: cheaper,

[00:24:18] Jakob Heuser: Yeah, and like getting exactly what you wanted out of it, which is a determi- not a deterministic answer, but a structured answer.

[00:24:24] Chris Griffing: Yep, exactly

[00:24:26] Jakob Heuser: I should be careful there. It's not deterministic, but it is structured

[00:24:29] Chris Griffing: Yep. Yeah, and it has this confidence value, and if the confidence value is low, I can fall back to the LLM anyway. So I get this like, speed at which these things are happening. I have this like, in my graph, like i-in my database, I was recording the timings of these things. With GPT-OSS 120B, it was anywhere from 2.5 seconds to all, to seven seconds of just like the latency for each request. Whereas it was 70 milliseconds to 150 milliseconds for Jev. What that means is the back pressure on my message queue is much lower now because these things are resolving quicker

[00:25:02] DevRel When an Agent Builds the Demo

[00:25:02] Jakob Heuser: Okay And you actually share all this as a DevRel too. that's the other half of the gig. You're not just on Twitch, you're actually in developer relations over at GitKraken. And you your day job is building demos and sample apps and showing people how tools work and what they can do.

[00:25:20] Chris Griffing: Yeah

[00:25:21] Jakob Heuser: I guess one of the questions I have, because things are moving so fast, kind of the workplace side.

[00:25:26] When an agent can get a demo app done in an afternoon, how does that change the dynamics of a DevRel? Like, what does the day job become at that point?

[00:25:35] Chris Griffing: So yeah, I mean, in the past, if I needed to create like a demo app or a demo project that had merge conflicts in it, I would have to go and like manually create these merge conflicts. Now with LLMs, I don't even have to tell it to create a merge conflict. I just do concurrent work on two different branches and then try and merge them, and it's gonna have a merge conflict 'cause it wants to rewrite the world anyway. So I don't even need to prompt it sometimes to make a demo. I just need to make two separate features in two separate branches. But in general, yeah, being able to just prompt out like an example repo is really nice. What I found is actually more important for DevRels is the ability for LLMs, especially like Opus 5.5 these days, to generate, like motion graphics and things like that you can plug in and talk over.

[00:26:26] in the past, like creating motion graphics is the heaviest part of making a video for YouTube for GitKraken, right? Like just even animating like when I'm capturing a screen and zooming in to certain parts, that takes more time than it does like recording the thing in the first place

[00:26:43] AI in the Video Edit

[00:26:43] Jakob Heuser: And now you can just offload a lot of that. I know we were talking ahead of time, we do all our recording here on Descript, which is all JSON.

[00:26:50] Chris Griffing: Yep

[00:26:52] Jakob Heuser: information, just recording that as basically JSON data on as like the data format, AIs can just stitch all that together now because again, it's a format they're just really familiar with

[00:27:03] Chris Griffing: Mm-hmm. Yeah, I mean, I still don't necessarily trust it to do the entire editing process. Uh, I haven't used Descript. I would guess that maybe that's a bit better because you have that data to work with the LLMs understand. I've been using Riverside, and Riverside does have a thing where it'll, find moments that are clippable.

[00:27:25] But the way... the in and out of when it clipped it and some of those things is actually still bad. So I really just use it as, okay, here's the timestamp that I should go in and figure out the clipping and maybe the layout of and everything myself still. Yeah, finding those moments is really powerful.

[00:27:45] Jakob Heuser: Descript's still a little bit sloppy as well, honestly.

[00:27:47] Chris Griffing: Yeah

[00:27:48] Jakob Heuser: where it's really helped us has been there's a structure for this podcast. Host left, guest right. We have a bump and a transition. We have intro music. Throw the assets in the project, and then it just JSONs away and we get to about 80%, which that represents so much cutting and slicing that, I don't have to do, and I can spend more time talking to cool people

[00:28:11] Chris Griffing: Yep. Yeah, exactly. But there's the age-old adage, first 80% of the project takes 80% of the time. The second, the last 20% of the project takes the other 80% of the time, right?

[00:28:20] A Human Wrote the README

[00:28:20] Jakob Heuser: Yeah. and you actually care a lot about the human piece of that too. I know that, even when you're going in on Riverside or Descript, you're doing, you're putting together your stuff, you put a human touch in there. so much so that your git-meta README actually says, "A human wrote the README."

[00:28:36] Um, "AIs wrote some of the implementation and spec based on that human written file." And I think that care, that matters a lot to you. you care very deeply that there is a human piece of this process

[00:28:48] Chris Griffing: Yeah, I mean, if a human's meant to consume it, a human should write it. That's the way I feel about it. So all the blog posts I write for GitKraken, they're all just me writing. Maybe they're not perfect writing, but it still feels real in a way that is hard to, to really, make an AI do. And the effort to make an AI feel human, almost like it's a lot of effort, but also it feels disingenuous.

[00:29:13] It's like you're trying to lie to people by making the AI seem like you. It isn't you. use it for research, use it for maybe critiquing your writing, but don't just blindly copy and paste stuff from it

[00:29:26] Jakob Heuser: Use it for the mechanical tasks, not for the creative ones.

[00:29:29] Chris Griffing: Yep

[00:29:31] Grading Models with GitBench and OngoBench

[00:29:31] Jakob Heuser: Um, and you've used it on a lot of mechanical tasks. Your most recent tangent, your side quest, if you will, has been benchmarking.

[00:29:37] Chris Griffing: Yep

[00:29:39] Jakob Heuser: ever a place to throw LLMs at the mechanical, build me 100 demos and put 100 demos through the demo grinder.

[00:29:46] GitBench, GK CLI Bench, OngoBench. you're a devrel that writes benchmarks now as part of your gig.

[00:29:52] Chris Griffing: Yeah

[00:29:53] Jakob Heuser: what got you from using the models to grading them? what was that sort of "I need to actually validate this stuff"? What was that moment?

[00:30:01] Chris Griffing: So, I mean, I-- every time a new model comes out, I hop over to Artificial Analysis. I keep checking DeepSWE. I wish they would update it. They haven't updated it since the twenty-second. So ugh, I wanna see some of these newer models in like the DeepSWE stuff, right? They still don't even have, DeepSeek-41-Flash, which I really wanna see how it compares.

[00:30:20] the model providers do their own DeepSWE measurement, but can we trust it? I'd rather see some third party do it. I've been looking at all these benchmarks and recognizing the value of them, what they're good at, right? SWE-bench basically looks at like pull requests that have been merged.

[00:30:38] It turns a model loose and then sees like, well, how close does it get to the code that got merged? And that's basically SWE-bench. I forget what DeepSWE is doing. It's like a multi-turn, like long horizon thing that it does. I'd have to look at the specifics. But I didn't see anything that was measuring how well these agents do at Git things, and I think we've all seen the random posts on social media where someone says "Oh, and then the model deleted my entire home directory," or whatever, right?

[00:31:06] They're prone to doing these destructive things, and Git has some very destructive operations it could do where it modifies your entire history. And the models may sometimes even decide, "I'm gonna Git force push this," Like it could make all sorts of things terrible. Now, some things, like Fable, it may do a worktree command and flip the directory you wanna put it in and the name of the worktree. Okay, that's a simple enough thing where it realizes it does it wrong, it flips the arguments and then does it right. But that's not too destructive. That's still burning expensive tokens to do so.

[00:31:42] Jakob Heuser: Yeah

[00:31:43] Chris Griffing: I think at the end of the day, there are other Git operations that are so destructive you don't want a model going oopsie and trying to fix it, right?

[00:31:51] 'Cause then you're just burning way more tokens to do the fix.

[00:31:54] Jakob Heuser: So what does that, what does that look like? Do you go the sandbox route? Do you have basically a very neutered Git CLI, like with a Git Git branch, Git push, Git pull, like all of those instrumented? how do you actually go about saying, "This thing does a bad thing three out of 400 times"?

[00:32:14] Chris Griffing: So technically I'm not having the model do the operations. All I'm doing is I I have this like playlist of the operations to set up the initial state of the repo, and then I pass that state and a prompt to the agent, via just OpenRouter, and I get a request response back. So all I'm looking at is what are the, like the commands it says it wants to run. Yeah, I don't even need to run them. I can just look at it and see what is correct, what is incorrect, because we know what we're expecting out of it

[00:32:46] Jakob Heuser: I see. And so you you are you looking at then the, raw logs like the JSONLs or are you just looking at the sort of transcript as it plays?

[00:32:54] Chris Griffing: I'm not even looking at a transcript because this is a single request response. There's no, turns. It's a single turn because

[00:33:01] Jakob Heuser: Okay

[00:33:01] Chris Griffing: here's the state, here's the prompt. What, based upon what I'm saying, what is the command to do blank? And it gives me the command, and if that command is correct or not is really easily evaluatable

[00:33:12] Jakob Heuser: And that's obviously highly important to GitKraken who wants to ideally not have teams blow up their repos using any sort of GitKraken product

[00:33:21] Picking the Right Model at GitKraken

[00:33:21] Chris Griffing: Yeah, we have some GitKraken AI tools built into GitKraken Desktop and now Kepler and all these things, like generating a commit message. Well, that's not destructive, but is it good at it? And should you be wasting Fable tokens to generate a commit message? Definitely not. So when we were rolling out some of these features, I kinda realized, well, could we figure out and make recommendations for our users about what model they could be using instead? And also for us, if we're running these things and the user just says, "I don't care what model, just do it," well, we still are paying for these things on behalf of our users. They get an allotment of credits, but we could still optimize that cost for them and for us

[00:34:04] Jakob Heuser: Yeah, 'cause that means they'll use GitKraken more because they're not out of tokens. So

[00:34:08] Chris Griffing: Exactly

[00:34:09] Jakob Heuser: you're, incentivized to make it cheaper for them too

[00:34:11] Chris Griffing: Yep, 100%.

[00:34:13] MCP Servers That Do Tasks

[00:34:13] Jakob Heuser: and speaking of I guess this is a really bad segue. We're talking about like the adoption of like benchmarks and GitKraken and Kepler.

[00:34:20] GitKraken has an MCP server. I poked around at it. it's really good, first off.

[00:34:26] Chris Griffing: No, thank

[00:34:27] Jakob Heuser: anytime you can get an MCP server and it has reasonable disclosure, like progressive disclosures, you don't have to load your whole context up with every single thing that you can do in GitKraken, it makes me very happy inside, and it makes my token usage very happy inside

[00:34:40] Chris Griffing: Mm-hmm. was actually a process too, because we started out with 40 plus tools, right? All very like fine-grained. and that's also one of the things that like the FastMCP folks, they have this like generator. Turn your OpenAPI loose like, into an MCP with one, you know, CLI command.

[00:34:57] But they even have an article saying you probably shouldn't do that, right? Because you're gonna blow up people's context, and you're gonna be like creating these like fine-grained things where like just making an API wrapper isn't as useful as you think it is, right?

[00:35:11] Jakob Heuser: Yeah, not REST APIs. It's, it's almost like a task API, which I think is something that mentally the first generation of MCP didn't do.

[00:35:18] Chris Griffing: Mm-hmm.

[00:35:19] Jakob Heuser: It was

[00:35:20] Chris Griffing: part of why MCP is, so looked down upon on social media, right? Because it's like, "It's just an API wrapper." Well, yeah, that's because they did it wrong. If you are doing it right, your MCP tool is gonna be making multiple API calls and stringing things together into a flow that the agent actually uses.

[00:35:36] So, in the article, they even say, should-- you could use this for prototyping, and then you should be measuring what calls are being grouped together by the agent anyway. And once that's happening, you can really understand the workflows that agents need to run and turn that into specific tools that do multiple things, not just, little micro transactions.

[00:35:55] Jakob Heuser: That makes a ton of sense. we went the CLI route at Taskless because at the time we were like, "What would even MCP look like for code generation?"

[00:36:02] Chris Griffing: Mm-hmm

[00:36:03] Jakob Heuser: Uh, but we had this whole progressive disclosure conversation around how do you actually... Even with a CLI, you can't just yeet a help file into the context.

[00:36:11] Chris Griffing: Right.

[00:36:11] You're gonna

[00:36:12] Jakob Heuser: it'll just... Yeah, if I throw a 700 kilobyte man page into a Can I do it? Sure. Should I do it? No.

[00:36:20] Chris Griffing: Right

[00:36:21] Jakob Heuser: And I think it's that same problem of how do you show enough to be useful?

[00:36:25] Chris Griffing: Mm-hmm.

[00:36:26] Jakob Heuser: And MCPs, I think, get double... They get the bad rap twice. Once because it's an internet connection, and then a second time because MCP in the first generation, there was a lot of good stuff there, and we all raced to re-implement like MCP in our own flavors.

[00:36:43] And now the version of MCP doesn't even match the MCP spec right now. Most people don't implement MCP 100% correct anymore.

[00:36:51] Chris Griffing: Mm-hmm.

[00:36:52] Jakob Heuser: And there's MCP2,

[00:36:53] Chris Griffing: Yep

[00:36:54] Jakob Heuser: which is only going to be relevant, I guess, if it takes off.

[00:36:59] Chris Griffing: Yeah

[00:36:59] Jakob Heuser: And where does it, where does that leave you? Obviously, Messijo has an MCP.

[00:37:03] Chris Griffing: Mm-hmm.

[00:37:04] Jakob Heuser: GitKraken has an MCP.

[00:37:05] You're stuck in this ether of do we support MCP2? Do we not? And is it worth supporting it if nobody's ready to accept it? MCP1 might be around forever

[00:37:16] Are MCP Servers Dead?

[00:37:16] Chris Griffing: Yeah, so this was actually fun. So I gave this talk about are MCP servers dead at Commit Your Code Conference 2026, and they're doing it again next year. It's a really good conference. I just gotta plug it. Danny Thompson killed it organizing it. $40 tickets, so good, Instead of like the $1,000 you see at other conferences. But when I gave this talk, they they say, "Hey, don't... we're not gonna do Q&A." Well, there was no one going after me, and I finished five minutes early because I love talking faster than I do in my practice runs. So I had five minutes in which I just answered people's questions. One of the people, pointed out they spent so much effort and time building their, V1 MCP server, and now that V2's out, are they gonna have to rewrite the whole thing? And the answer is no, because they're not gonna deprecate V1 for a long time, because that would, number one, require that all of the clients actually implement V2, which isn't gonna happen for a long time, right? So yes, like the-- what you were getting at is the clients have been the biggest bottleneck for making MCP good. Technically, in the V1 spec, that progressive disclosure of like tool discovery was built into the spec, but none of the clients like Cursor were actually doing that round trip discovery. They would just do an initial when the server launches, get the tools list, and that's it, So being able to get the tools list and update that as the agent is asking for things is really important, and it's now part of the spec for V2 to do that in an even cleaner way, but it requires that clients actually do that.

[00:38:45] Skills, MCP and OAuth

[00:38:45] Jakob Heuser: So do you think Messijo's gonna have a suite of skills first or MCP2 first? Just 'cause we've been talking about Messijo off and on throughout the interview today

[00:38:54] Chris Griffing: yeah. MCP is already built for Messijo. It's live, it's usable, it's done. and the thing I think, a list of skills will never be able to do is auth. How do you actually log in for that user and grant them access to a specific thing to, then do stuff on their behalf their, like, login credentials, right?

[00:39:12] on behalf of that user, what do they have access to? All that. OAuth is the way you do that, and how do you do that from a skill? I don't even know,

[00:39:20] Jakob Heuser: Yeah.

[00:39:21] Chris Griffing: right?

[00:39:22] Jakob Heuser: we do it with the CLI. We offload to a CLI and run device flow. But, that was because when our, with our early testing, localhost for OAuth callbacks is, the most brittle piece of things out there on the internet. It just turns out that a larger percentage than I thought don't actually resolve localhost where it's supposed to go.

[00:39:42] Chris Griffing: Right

[00:39:43] Jakob Heuser: So if your callback's localhost, congratulations. you're probably not getting your MCP callbacks where they're supposed to go

[00:39:48] Chris Griffing: Yeah. Yeah, that's tough. So that's basically the difference between, an HTTP MCP server and a CLI MCP server. And for what I was doing, I think it just made sense. I already had OpenAPI specifications. I already had, a, like a a, an, a token-based thing where, you could just turn a a tool loose by just doing, an API token instead. So I had all of that kinda built out, and doing an HTTP server just made the most sense

[00:40:15] Is AI Hype?

[00:40:15] Jakob Heuser: That totally makes sense. So where are we at now? Is AI hype? is it is it the real deal? Are we somewhere in between? Where are we at with this?

[00:40:25] Chris Griffing: So there is definitely a lot of hype around AI, but it is also very useful. The thing I say on stream all the time is that there is like a 5% of people at the anti-AI end of the things, and the 5% at the, AI's, the, the next coming of whatever deity you believe in, right? those people are equally insufferable. the world and, the most reasonable people are somewhere in that middle 90%, and they are much easier to have conversations with because they're not dogmatic about these things, right? They're, they're willing to understand that there is nuance to the world, and the people who are simply, the polar black and white, yes or no binary people, they're just... They're both missing things because they're, like, dogmatic about it

[00:41:12] What Chris Wants to Offload

[00:41:12] Jakob Heuser: Yeah. And these tools are gonna continue to improve. I mean, software does move forward in terms of capability. So one of the last questions I love to ask people before we ask about where we can find you on the internet is what's something that you're hoping AI will eventually be able to take over?

[00:41:27] Something you'll be able to offload onto AI

[00:41:30] Chris Griffing: Um I mean, if it could make video editing better, that would be great because I loathe video editing, and I don't have the viewership or profit from streaming to pay an editor. I probably should just bite the bullet and pay someone anyway because, takes money to make money or whatever, right?

[00:41:48] Like that adage. wish it was better at editing, at least to get it to a good 80% for, a general video, not necessarily like a podcast, which has a very, like, explicit format. But when I'm doing just a video about, I don't know, like one I wanna make is about, how to do custom, shaders in Ghostty, right?

[00:42:07] I've done that. I've, I have like a playbook I run to convert a Shadertoy shader into a Ghostty shader, and I need to make a video about that. But I am dreading the editing process, so I'd love it if AI could at least, get me part of the way there for that editing, and then I could fine-tune it and finish it

[00:42:23] Jakob Heuser: Like hook it up, let her rip, and have it come back with the editing points so you can at least, you can talk, you can present, and then let it at least scope down the talk so that it fits within the structure or shape you wanted. And then you can make it extra nice

[00:42:36] Chris Griffing: Exactly. And I mean, I still need to, put in the effort to understand what I want editing-wise. I think Theo even especially espouses that idea, where, you shouldn't be farming out your editing to someone else immediately. You should be putting in the effort to edit so you understand that the editors are skilled labor and they're worth paying, but also so you understand what kind of content you even wanna make.

[00:42:59] You don't know until you're spending multiple hours watching yourself and cringing at the sound of your voice by the end of it, to then understand, what makes a good video versus a bad video.

[00:43:10] Jakob Heuser: And what makes a good video for you, coming back to the taste. And an LLM can never, can never be your editor because you know what makes you entertaining

[00:43:19] Chris Griffing: Yeah. Yeah, exactly

[00:43:21] Where to Find Chris

[00:43:21] Jakob Heuser: Awesome. Chris, where can people find you on the internet? Obviously Twitch, cmgriffing. Where else?

[00:43:26] Chris Griffing: cmgriffing everywhere. So cmgriffing on Twitch, Bluesky, Hacker News, X, YouTube, just cmgriffing everywhere

[00:43:35] Jakob Heuser: Yeah, everywhere, everywhere. cool. I will get those in the show notes. Everybody that tuned in today, I appreciate you all. This is, this is Chris Griffing, DevRel at GitKraken, builder of a lot of cool stuff, creator of Messijo. Follow what's going on with your social media mentions. Actually get on top of stuff.

[00:43:51] Support indie developers and do cool shit. Chris, thank you for being on The Task at Hand today. I appreciate you

[00:43:57] Chris Griffing: This was a lot of fun. Thank you