Data pipelines that run unattended

Teams whose models, reports or products depend on data from outside: APIs, files, vendor feeds or web pages. It fits best when someone on the team spends mornings rerunning jobs by hand.

• A job fails overnight and someone reruns it by hand in the morning.
• A rerun downloads everything again from the start.
• Bad rows reach a model or a dashboard before anyone notices.
• When one source is down, the whole run stops.
• Nobody can say which data a given result was built from.

Checkpointed ingestion

A failed run resumes where it stopped.

Validation gates

Bad data is stopped and named, not passed on.

Fallback sources

If a second source exists, the run switches to it. If not, the run carries on without that source and says so.

Safe reruns

Running a step twice gives the same result as running it once.

Alerts and a run log

You know what failed, where, and in which run.

Tests and a runbook

The next engineer can change it without guessing.

All of it is code in your repository, running in your accounts.

• Access to the sources, or samples of them.
• The place the data must land: a database, a bucket or a warehouse.
• One person who knows what correct data looks like.
• A way to run code in your accounts, such as a scheduler, a server or a cloud project.

One source, end to end: fetched, checked, stored and alerted on, running on a schedule in your account. You can run it yourself every week while it is built.

The scope, the price and the length are agreed in writing before it starts. How a pilot works.

• A managed connector already does the job well. We will tell you to use it.
• The data cannot be fetched or stored under its licence or terms of use.
• The real problem is that nobody has agreed what the data means. That conversation comes first, and we will say so.