Tests, observability, lineage, PII and governance
Practise these as flashcards. One or two lines each; the ones you miss come back first.
Start freeThe main dimensions are accuracy, completeness, consistency, timeliness, uniqueness, and validity. Each dimension checks a different aspect of whether the data is reliable and fit for purpose.
I would run a query to count duplicate values in the key column, or use a dbt uniqueness test to automatically flag duplicates during pipeline runs.
PII stands for Personally Identifiable Information, such as names, emails, or addresses. It must be protected to comply with privacy laws like GDPR and to prevent misuse or data breaches.
A data catalog is a searchable repository of metadata about datasets, including schema, descriptions, owners, and lineage. It helps users discover and understand data assets.
dbt tests automate checks like uniqueness, null values, and referential integrity at the model level. They run during builds and can block deployments if tests fail, ensuring issues are caught early.
I would track the timestamp of the latest data load and set up alerts if data is not updated within expected windows. This ensures downstream users always get timely data and can catch delays quickly.
Schema drift is when the structure of source data changes unexpectedly, like columns being added or removed. I handle it by monitoring schema versions, alerting on changes, and coordinating with upstream teams to update transformations as needed.
I use techniques like hashing, encryption, or partial masking to hide sensitive fields. For example, I might store only the last four digits of a phone number, or encrypt emails so only authorized users can access them.
A data contract defines the schema, quality, and delivery expectations between producers and consumers. It prevents breaking changes and clarifies responsibilities, making data pipelines more reliable.
Great Expectations focuses on granular, row-level and statistical validations, while dbt tests are mostly schema-level checks. Great Expectations can run on raw data and supports custom expectations, making it suitable for more complex quality checks.
I would first alert stakeholders, triage the issue, and rollback if needed. Then, I'd investigate the root cause, add new tests or monitoring, and document the incident to prevent recurrence.
I track statistics like mean, median, min, max, and histograms for key columns. Monitoring distributions helps catch silent errors, such as data skew or outliers, that basic tests might miss.