Data Quality
Data Quality rules watch a dataset and record whether it passes or fails over time. A rule is a check — "this column is never null," "row count didn't collapse," "the feed is fresh" — that the platform runs on a schedule and shows as a status on the dataset's Data Quality tab and in the monitored overview.
Rules come in two forms: built-in templates (no SQL) and custom SQL rules. Both compile to the same internal contract, so they report and alert identically.
Built-in rule library
The rule library covers routine checks without writing SQL. Pick a type, fill in the parameters:
| Type | Checks that… |
|---|---|
not_null | a column has no null values. |
unique | a column (or column set) has no duplicates. |
accepted_values | every value is in an allowed set. |
range | a numeric column stays within [min, max]. |
regex | every value matches a pattern. |
freshness | the newest row is within a max age. |
row_count_min | the table has at least N rows. |
Each type stores its parameters on dq_rules.rule_type / dq_rules.params and generates the same {sql_check, sql_failing} pair a custom rule stores — one query that returns the pass/fail verdict and one that returns the offending rows. GET .../rules/types returns the catalog the builder renders.
Custom SQL rules
For anything the templates don't cover, author a rule as a SQL check that returns a failing-row count (or the failing rows). It stores the same {sql_check, sql_failing} contract, so it participates in the timeline, thresholds, and alerts exactly like a built-in rule.
Soft and hard thresholds
A rule can carry two thresholds:
- Soft threshold — crossing it marks the rule warning (amber). The dataset is still usable; someone should look.
- Hard threshold — crossing it marks the rule error (red). Treat the data as not trustworthy.
The two-tier model lets a rule tolerate a little drift before it escalates, instead of flipping straight from green to red.
Status timeline
Every run appends to a status timeline on the dataset's Data Quality tab — a per-rule history of pass / warn / fail with the measured value, so you can see when a check started degrading and correlate it with an upstream change.
Drift rules
A drift rule compares the current run against a baseline (a prior window) and flags a statistically significant shift — a distribution moving, a category appearing or disappearing, a mean sliding. Drift catches "the data changed shape" even when every row is individually valid.
Rule templates
Reusable rule shapes let you apply the same check across many datasets. Define the template once (e.g. not_null on a key column), then stamp it onto each dataset that should carry it, rather than re-authoring the rule per table.
Provenance & failed rows
- Provenance — a rule records what it ran against and when, so a failure is explainable rather than a bare red dot.
- Extract failed rows — from a failing check, pull the actual offending rows (
sql_failing) to a preview or a managed folder, so you can hand the exceptions back to whoever owns the source.
Monitored overview
The monitored overview rolls every dataset's Data Quality status into one board — which datasets are green / warning / failing right now, and which rules are the worst offenders. It is the fleet view for data trust, alongside the per-dataset timeline.
Permissions
Authoring, editing, and deleting Data Quality rules is gated on org.admin (the data_quality_rules router). Anyone who can read a dataset can see its rule status.
See also
- Datasets — the Data Quality tab lives on the dataset detail page.
- Catalog & Lineage — DQ chips surface in catalog browse and Collections.
- EDA & Statistics — profiling and statistical tests for exploring a dataset.