What is a data pipeline?

A data pipeline is the automated route data takes from the systems where it is created to the place where it is used, cleaned and joined on the way. IBM describes a system that "ingests raw data from multiple data sources", converts it, and then loads it "into a data store such as a data lake or data warehouse for analysis and operational use".

What it does

A pipeline runs on a schedule or continuously. It collects new records from each source, such as the ERP, the scale system, spreadsheets and documents. It checks and converts them, joins them with other records and loads the result where reports, people and AI tools can use it.

The real test

Any pipeline works on a normal day. The test is what happens when something changes. A scale system update renames a field. A supplier sends invoices in a new layout. Tickets for Friday arrive on Tuesday. A record can't be matched to a contract.

In our view, a good pipeline accounts for every record: loaded, corrected, or set aside with a reason someone can see. One that drops what it can't handle can still report success, and the gap surfaces later in a report.

Late discovery is common. In a March 2023 survey of 200 data professionals, run by Wakefield Research for Monte Carlo, which sells data monitoring software, 68% said it took four hours or more, on average, to detect a data incident.

In a timber business

Timber pipelines carry awkward data: weights and volumes in different units, late tickets, paper documents, and records from contractors' systems the business doesn't control. Each needs a rule, and each rule needs a place to put the exceptions.

What to check

Ask what happens to a record the pipeline can't process, and who sees it. Ask how late data changes figures already reported. Ask for a count, per run, of records received, loaded and set aside.

How Quarri works explains the platform as a layer over existing systems, not a migration.

Quarri is an AI-native data platform for the timber supply chain. It connects buying, production, sales and inventory for forest management, sawmill, wood products and pulp, paper and packaging operators.