CloudBlogDocs

Many Languages, One Ruleset

Jakob Heuser

The question we hear most from teams with good tooling already in place is some version of this: we run ESLint, Ruff and Prettier, our AGENTS.md points the agent at all of them, so what does Taskless do that those don't?

It's a good question, because every tool you add is one more thing to maintain. However, the rules worth enforcing in an agent-written codebase increasingly live between files, and between languages. Linters, type checkers and formatters are built to look inside one file of one language at a time. That focus is why they're fast and good at what they do. It's also why a whole class of mistakes walks straight past them.

This post covers where the tools you already run stop, why agents make the gap matter more than it used to, and four rules that are better as Taskless runtime rules than as anything you'd hand-roll in ESLint, Biome's GritQL or Ruff.

Your tools are built around one language and one concern

Every static analysis tool I run is excellent inside its lane, and every one of them has a lane.

ESLint lints JavaScript and TypeScript, and with its newer language plugins it can lint JSON, CSS and Markdown too. Each of those is still a separate pass over a single file. A rule gets the file it's looking at, and that's the unit of work. The cache works the same way: eslint --cache re-lints a file when that file or your config changes. A rule that secretly depends on a second file won't rerun when only the second file changes.

Ruff ships over 900 rules compiled into the binary, and it's fast because of exactly that. It doesn't support third-party plugins yet, though the project says a plugin system is in scope. If your team needs a rule Ruff doesn't have, the path is writing it in Rust and either getting it accepted upstream or maintaining a fork. For a rule like "our billing service never calls the payments client outside a transaction," that path doesn't exist, because nobody upstream should accept it.

Biome went a different way and adopted GritQL for plugins, which is a real step forward. You write a structural pattern, Biome reports a diagnostic when it matches. It covers JavaScript, TypeScript, CSS and JSON today. A newer JavaScript plugin API is in developer preview. Both kinds of plugin work on the file being analyzed.

The rest of the toolchain looks the same. tsc and ty check one language's types. Prettier formats. markdownlint checks Markdown, Vale checks prose, ShellCheck checks shell, and an i18n extractor pulls strings out of one framework's components. Each of them is single-language or single-concern, and that's the right design for each of them.

None of this means throwing them out. When there's already a battle-tested rule, use it. If you want to ban a syntax pattern in TypeScript, ESLint's no-restricted-syntax already does it, and Taskless will offer it before writing anything new, because there's no reason to grow the amount of rule code you own when a maintained rule exists.

The trouble is that a repo isn't any one of those things.

Your repo was never one language

Open any production repo and count. A TypeScript frontend. A Python worker. SQL migrations. YAML for CI and deploys. A .env.example. JSON locale files. A Dockerfile. A README.md, a docs/ folder, and now an AGENTS.md that the agent reads on every run.

That's normal, and it always has been. What's changed is how much of it an agent touches in one pass. A single task like "add a dark mode toggle" can edit a React component, a settings model in Python, a migration, a locale file and the docs, in under a minute.

Each of those files can be individually perfect and the change can still be broken. The new setting is read in Python and never declared in .env.example. The toggle's label was never extracted from your JSX and put into en.json for localization. The docs describe a flag the CLI renamed last week. Every linter in the repo passes, because each one checked its own file and found nothing wrong with it.

Even a repo that's "just TypeScript" has this problem the moment it has docs. Markdown is a language, and you expect it to agree with your code. Today that expectation lives in a senior engineer's head and in review comments.

Agents have to be non-deterministic

The tempting fix is to tell the agent. Put "always update every locale file" in AGENTS.md and move on.

Agents are non-deterministic by design, and they need to be. Sampling is how a model explores more than one path to a solution. Turn the temperature to zero and you still don't get the same output across model versions, providers or context windows, and you lose the variety that makes an agent useful on problems it hasn't seen. So the agent that followed your AGENTS.md instruction yesterday may skip it today, and nothing tells you it did.

At a dogfooding event in August, a team ran Taskless against a banned-terms list they'd already written into their AGENTS.md. It came back with 175 findings, each one a violation of a rule sitting in the file the agent loads on every run.

So the determinism has to live somewhere other than the agent. A rule that runs on every change says the same thing about the same code every time, regardless of which model wrote it or how the prompt was phrased. Markdown instructions still have a job, which is carrying intent and judgment. Anything precise enough to check should be checked.

And the rules have to keep up. An agent makes a new kind of mistake, you correct it, and that correction needs to become a rule today. A fixed list like Ruff's can't grow at that speed, and a hand-written ESLint plugin grows only as fast as someone on the team has time to write and maintain one.

What a runtime rule is

Taskless has a few kinds of rules. Most are ast-grep patterns, structural matches like a linter's, in any language ast-grep supports. Prose rules run on Vale. When a correction needs more than one file to decide, it becomes a runtime rule.

A runtime rule is a small directory: one or more ast-grep capture rules, and one check.ts. The captures are the cheap narrow. They find every spot worth a closer look, in whatever languages the rule needs. The check runs only when something matched. It gets the repo root and the matches, reads whatever else it needs from disk, and returns findings.

.taskless/rules/runtime/env-declared/
├── captures/
│   ├── env-read-ts.yml
│   └── env-read-py.yml
├── check.ts
└── tests/
    ├── pass/
    └── fail/

The contract is one function:

export default async function check(root: string, matches: Match[]): Promise<Finding[]>

That's the piece a linter's plugin API doesn't give you: one rule that sees matches from several languages at once, plus permission to go read the file that holds the evidence. The tests are whole fixture repos, so a rule proves itself against a passing tree and a failing tree before it ships.

Here are four rules where that shape is the difference. The first three are in taskless/examples with their full checks and fixtures.

1. Every environment variable you read is declared

The invariant: if code reads API_KEY, then .env.example declares API_KEY. Miss it and the read returns undefined (or None), the program carries on, and it fails somewhere far from the cause.

In a repo with a TypeScript app and a Python worker, the reads live in two languages. Here's the Python capture:

id: env-read-py
language: Python
rule:
  any:
    - pattern: os.environ[$VAR]
    - pattern: os.environ.get($VAR)
    - pattern: os.environ.get($VAR, $$$)
    - pattern: os.getenv($VAR)
    - pattern: os.getenv($VAR, $$$)
metadata:
  taskless:
    version: 1
    kind: runtime
    name: env-read-py
    check: check.ts
    match: anchor

The TypeScript one, env-read-ts, is a single pattern: process.env.$VAR. ast-grep treats .tsx as its own language, so a React app adds a third capture with language: Tsx. The check parses .env.example once and holds both languages to it:

const READS = new Set(["env-read-ts", "env-read-py"]);


export default async function check(root: string, matches: Match[]) {
  const declared = declaredKeys(join(root, ".env.example"));
  return matches
    .filter((m) => READS.has(m.rule))
    .map((m) => ({ ...m, name: m.captures.VAR.replace(/^["']|["']$/g, "") }))
    .filter((m) => !declared.has(m.name))
    .map((m) => ({
      file: m.file,
      line: m.line,
      message: `${m.name} is read here but isn't declared in .env.example`,
      severity: "error" as const,
    }));
}

To do this by hand you'd write two rules in two ecosystems. An ESLint rule that reads .env.example from disk, which works until the cache skips it because only .env.example changed. Then a Flake8 plugin for the Python side, because Ruff can't take one. Two parsers of the same dotenv file, in two languages, that have to agree forever. The runtime rule has one parser and one place to change it. Extending it to check that each key is also set in .github/workflows/deploy.yml is a few more lines in the same file.

2. The docs describe the CLI you actually ship

The invariant: every flag the CLI defines appears in docs/cli.md, and every flag the docs mention exists.

The second half matters more than it used to. People skim docs. Agents believe them. An agent asked to "run the migration in dry-run mode" reads the docs, finds --dryrun, and uses it with total confidence, even though the flag was renamed to --dry-run two releases ago.

The capture is TypeScript, matching a commander definition:

id: cli-option
language: TypeScript
rule:
  any:
    - pattern: $CMD.option($FLAG)
    - pattern: $CMD.option($FLAG, $$$)

The check pulls the long flag out of each match, reads docs/cli.md, collects every --flag mentioned there, and compares the two sets in both directions. A flag missing from the docs is reported at its definition. A stale flag in the docs is reported at the line in the Markdown file, so the fix lands where the problem is.

markdownlint can tell you the docs are well-formed Markdown. ESLint's Markdown support can lint the code blocks inside them. Neither one knows what your CLI defines, because that's a fact about a different file in a different language.

3. Every user-facing string goes through your translations

The invariant: text a user sees comes from a locale file, and every key the code uses exists in every locale.

Agents break this at the first step. Ask for a dark mode toggle and you'll get the component, the handler, and <label>Dark mode</label> written straight into the JSX. The string was never extracted, so en.json has never heard of it, and neither have de.json or ja.json. When an agent does remember to extract, it adds the English key, and whether the other locales get it depends on the run.

Part of this already has a maintained rule. eslint-plugin-i18next ships no-literal-string, which flags bare text in JSX, and if you already run it, keep it. What it can't tell you is what to write instead, because that answer lives in the locale files.

The runtime rule has three captures: t($KEY) calls in TSX, _($KEY) calls in the Python code that renders emails, and text inside JSX elements, which matches on the syntax node itself:

id: jsx-literal
language: Tsx
rule:
  kind: jsx_text
  regex: \S

The check loads every file under locales/ and flattens the nested JSON into dotted keys. Then it reports two things:

  • A hardcoded JSX string. If the same text already exists as a value in en.json, the finding names the key to use.
  • A key the code uses that's missing from a locale, once per locale it's missing from.
src/components/Settings.tsx:42  "Dark mode" is hardcoded. locales/en.json already has it as settings.darkMode.label
src/components/Settings.tsx:57  settings.darkMode.hint is missing from locales/de.json, locales/ja.json

The first finding is the one an agent gets the most out of. It names the existing key, so the fix reuses it and the agent doesn't invent a duplicate.

Doing this by hand means a JSX lint rule for the literals, an extraction tool for the JavaScript keys, a separate one for Python's gettext calls, and glue that fails one PR from all three. Here it's one rule with one notion of what a key is.

4. A test exercises something worth testing

This last one stays inside TypeScript, and it's here because it shows the cross-file cost even when there's only one language.

Agents are very good at producing tests that pass. Some of those tests assert that a getter returns the field it returns, or that a constant equals itself. The coverage number goes up and nothing is protected. We built a runtime rule for this, trivial-test-guard, that flags a test whose subject is too simple to be worth a test: fewer than three statement-bearing lines and no branching, loops, throws or awaits.

Deciding that means following the expect() argument back to the function it calls, which usually lives in another file, often re-exported through a barrel index.ts. Naming heuristics break on barrels, so the check walks the real expression with the TypeScript compiler, rooted at the matched files so it fits inside the ten-second budget a check gets by default.

You can write this as an ESLint rule. With type-aware linting turned on, typescript-eslint hands a rule a real ts.Program, which is more than most fast linters offer. Building trivial-test-guard taught me what you sign up for when you do, and the rule is the smallest part of it:

  1. The analysis itself.
  2. A safety argument for code that reads arbitrary files and starts a compiler on every push.
  3. A multi-file fixture harness, since ESLint's RuleTester takes one code string and this rule needs at least two files.
  4. Proof that a passing fixture actually exercised the rule and didn't silently do nothing.
  5. Carrying all of the above through every TypeScript and ESLint major version.

A runtime rule comes with the harness, the multi-file fixtures and the trust model, and there's no ESLint version to track. You write the analysis, or more often, your agent does from the correction you gave it.

What runtime rules cost

If a rule fits inside one file of one language, the tool you already run is faster and better integrated into your editor than a runtime rule will be. Taskless routes rules that way on purpose: when you write a correction, it checks what you already run and reaches for the lightest tool that can enforce it. Most corrections end up as plain ast-grep rules, and plenty of them are better as a setting in the linter you already have.

Runtime rules have real costs too. They run code, so they're slower than a pattern match and they only run where their captures fired. Taskless Cloud generates and validates each runtime rule in a sandbox, then signs it, and that signature is how the CLI knows a check is safe to run on your machine. That means runtime rules need an account, and offline they won't run unless you opt in with --dangerously-run-scripts. And a check is only as good as its fixtures. We've shipped captures that never fired on the pass case, which made the test prove nothing until we caught it.

Those costs are worth paying for the rules in this post, because nothing else in the repo can enforce them at all.

The rules between the files

Cross-file invariants are the part of Taskless I'm most excited to be building. They're the rules every team already has and never writes down, because no tool they own could run them. The env var that has to be declared. The doc that has to match the code. The label that has to come from a locale file. Senior engineers enforce them in review, one comment at a time, and agents break them a little differently every run.

Each one is a correction you've made before. With Taskless it becomes a rule once, and from then on it applies to every language in the repo.

npx @taskless/cli

Then ask your coding agent to onboard you onto Taskless. Docs for runtime rules are at docs.taskless.io/rules/runtime-rules.