An AI-written data pipeline isn’t production-ready until your team’s constraints can run against it as code — not as review comments, not as a README, not as the thing the senior engineer remembers to check.
The generation half has become fast and routine. dltHub — the company behind the open-source dlt ingestion library — reported that by January 2026, 91% of new dlt pipelines were agent-authored, and my own ingestion estate has been built that way since November 2025: agents read the API docs, write the rest_api config, and handle the pagination edge cases I used to budget afternoons for. This is good. I want more of it.
What didn’t change is the bill. Requests burn finite quota — some are billed per call — and a bad load can surface weeks later as a stale dashboard. Generation got cheap; mistakes stayed priced. That gap is what this post — and the package it introduces — is about.
What are agents actually good at?
Producing grounded first drafts of declarative sources, quickly. Grounded means the agent writes against current, retrievable documentation instead of remembered training data: dlt is indexed on Context7, a live documentation index coding agents can query, and dltHub publishes machine-readable llms.txt indexes of its docs. Pointed at those, an assistant writes against APIs that exist.
dlt’s declarative rest_api format narrows the task further. Its fields — base URL, authentication, pagination, endpoints — mirror the upstream API’s own documentation, which removes most of the hand-written pagination code where subtle mistakes usually hide. It doesn’t validate those choices, though: a wrong endpoint, auth method, or pagination strategy survives until something executes.
dltHub’s own AI Workbench post puts the remaining problem plainly: “the bottleneck in data engineering has moved. It’s no longer writing the code. It’s trusting the code.” That matches my experience exactly — and trusting the code is a different job than writing it.
What does production add that a playground doesn’t?
Constraints — decisions about money, credentials, time, and shape that no amount of reading the upstream API’s documentation can settle, because your answers aren’t in it.
- Money. Ingestion is the expensive kind of code to be wrong in: requests consume finite quota, some APIs bill per call, and every load spends warehouse capacity. A retry loop that works is still a bill if it re-extracts a year of history.
- Credentials. Your team has a sanctioned way to provision secrets — a secret manager, a vault, orchestrator-managed variables. Which mechanism is right isn’t the point here; the point is that the choice is a policy, and a token pasted into source code violates it even when it works.
- Time. A schedule is a contract with the orchestrator, not a comment. “Runs daily” has to be machine-readable somewhere, or it’s tribal knowledge.
- Shape. Which columns land, with which types, with what uniqueness — and what happens on the day the upstream adds a field.
None of this is AI-specific. Every pipeline I wrote by hand had to answer the same four questions. A repo-grounded agent can apply your existing policy; what it can’t do is invent it — your quota budget, your sanctioned secret backend, your cadence are not derivable from the API’s docs. What AI changed is one variable: authoring volume. In many codebases — mine included, before this tooling — those answers live in READMEs and reviewers’ heads, and that holds while sources arrive one per quarter. When they arrive in minutes, enforcement-by-memory stops scaling.
That’s the layer I extracted into dlt-ops: ingestion policy for dlt projects, encoded as rules that run. It is not an AI product — the same rules gate pipelines written by hand.
What does it mean for a constraint to be executable?
It has four properties and a tool that runs it: a scope (what it applies to), an enforcement point (when it runs), a failure behavior (what happens on violation), and an explicit exemption path (how you overrule it on purpose, in writing). Without a tool behind them, style guides, review checklists, and tribal knowledge stay advisory.
A constraint that lives in a README is a wish. A constraint that runs is a rule.
dlt-ops enforces at three distinct moments:
dlt-ops pipeline validatechecks everything statically expressible before pipeline execution: layout, naming, config, schedules, schema contracts, column models, import safety — 21 core rules, plus plugin-owned ones.- Runtime preflight re-checks the critical subset on every
runandbackfill, because a production scheduler does not run your CLI steps first. - Pre-load assertions inspect extracted data between extract and load — row-count floors and ceilings, required columns, in-batch uniqueness, custom predicates. A violation fails the run by default, or warns; row-level assertions can instead quarantine the offending rows into a
_dlt_rejectedtable. Assertions gate what lands; they can’t refund extraction cost already spent.
Scheduling shows what the framework looks like end to end. Every source declares a schedule in TOML, from a closed set of tags (@hourly through @monthly, plus @manual) that the Airflow adapter groups into DAGs:
[sources.github_issues.dlt_ops]
schedule = "@daily"Scope: every source. Enforcement point: validate fails on a missing or invalid value. Failure behavior: the finding names the source and the fix. Exemption: @manual is the declared escape hatch for sources triggered externally or run ad hoc.
The same pattern governs the other rules.
Secrets are a policy, not a value check. Whatever your team sanctions — secret manager, vault, orchestrator variables — plugs in as a secret backend, and validate proves the sanctioned backend is registered and healthy. Resolution still happens at runtime: no static check can prove the secret’s value is right, and this one doesn’t claim to.
Every resource declares its shape. columns= with a Pydantic model is mandatory (a rule), and a resource without a schema contract gets the canonical freeze contract auto-applied — new upstream columns don’t silently land. Evolving contracts are an explicit, justified opt-in.
Imports must be side-effect-free. One rule of the 21, with the most machinery behind it: validate imports each source module in a throwaway child process behind a CPython audit hook and reports network and disk activity at import time. It is a detector, not a preventer — the call really fires, once, inside the sandboxed child — and the finding reads like this:
✗ [github_issues] import_safety: Rule 15: network at import of
github_issues.py — socket.connect(('140.82.121.6', 443))That’s the class of bug that runs fine on a laptop and then fires on every scheduler heartbeat that parses the file — invisible to linters and type checkers, because it’s valid Python.
Two more properties hold package-wide. Exemptions are first-class: rules can be disabled project-wide, and most can also be exempted per source with a mandatory written reason — reviewable config, not a shrug. Import safety is the deliberate exception: it takes no per-source exemption, only the project-wide switch.
And enforcement is honest about where it can act. Five adapter-backed features — assertion quarantine, checkpoints, backfill, remote clean, drift reconcile — refuse at preflight on a destination without a DestinationAdapter, rather than degrade silently. The runs ledger is the one feature that skips with an INFO line instead, and pipeline status then reports it as unsupported. DuckDB, Postgres, and BigQuery adapters ship first-party.
Most of these rules are production scar tissue. The set accreted over nine months of running dlt in production at Earlybird, the Berlin VC fund where I build the data platform — not every rule maps to an incident, but the ones you’d trip over first do. A durable rule begins with a failure worth preventing.
What can’t rules catch?
Plenty — and an honest checker tells you which checks ran, not that you’re safe.
Two design choices follow from that. First, coverage is reported, not assumed: when a module fails the sandboxed import, every rule that inspects sources skips it, and validate says so explicitly — a validation_coverage error naming what is now unknown, rather than a shorter, greener report. Second, the rule set doesn’t pretend to enumerate unknown failure modes. There is no recall number for agent-introduced defects here, and I won’t invent one.
The sharpest example is a policy question wearing a bug costume. A source without an incremental cursor re-extracts the full window on every run — on a billed API, that’s real money. Default validation permits it, because a full refresh is sometimes exactly what you want. Whether it’s an error is itself a policy — so it ships as an opt-in rule: incremental_cursor_required (added in 0.1.0) is off by default, and turning it on makes a full refresh something you declare in writing, per source, as an exemption.
Rules are a floor you raise per incident, not a proof of safety. Import safety detects; it does not prevent. A passing validate means the configured predicates found nothing — that sentence, and no more.
Why this loop fits agents
Deterministic findings make the fix loop bounded. The working loop, adapted from the docs:
- Ground the assistant in current dlt and dlt-ops docs (Context7,
llms.txt). - Let it write the source.
dlt-ops pipeline validate— findings are structural and specific: a missing column model, a missing schedule, a socket opened at import.- Feed the findings back; ask the agent to address them.
- Rerun the same check until it passes. Same code, same config, same environment, same ruleset — same answer.
- Then
dlt-ops pipeline run -s <source> -yagainst a development destination — local DuckDB — anddlt-ops pipeline statusto read the outcome back.
A passing run proves that one execution against that upstream and that destination succeeded — not production correctness. It’s evidence from one run, recorded where the next engineer can find it — not a guarantee. And the determinism is the point: a second model reviewing the first gives you a second opinion; given the same inputs, a rule gives you the same answer twice.
Try it
pip install "dlt-ops[duckdb]"
dlt-ops init demo --example # scaffold with a runnable example source
cd demo
dlt-ops pipeline validate # pre-execution checks: layout, config, contracts, import safety
dlt-ops pipeline run -s demo_events -y
dlt-ops pipeline status # the run ledger, read back from the destinationThe quickstart runs fully offline: the example source is fixture-backed and lands six typed rows in a local DuckDB file — no credentials, no cloud.
What it’s not: dlt-ops ships no connectors — your dlt code does all the ingesting (the one place it writes rows itself is diverting assertion-rejected ones to _dlt_rejected). No model, no agent, no codegen. Not an orchestrator — it declares schedules and generates DAGs for one. And it’s a third-party project, not affiliated with dltHub.
Docs: earlybirdvc.github.io/dlt-ops · source: github.com/earlybirdvc/dlt-ops · Apache-2.0.
Next post: I built the same four-source estate twice — once in vanilla dlt, once in dlt-ops — and measured the difference. Vanilla structured fine, my strongest claim didn’t survive the measurement, and the wins were narrower and more specific than I expected.