Write a data contract before writing code
Agree, in writing, the fields, types, formats, required values and what happens when they are missing. Share it with every party that sends or receives the data. Most integration failures are contract failures.
Quarantine bad records, do not drop them
Silently discarding invalid rows hides problems until a report looks wrong. Put them aside with a reason, alert an owner, allow correction and reprocessing.
Why every run should be safe to repeat
If a run fails halfway, you should be able to run it again without creating duplicates. Design with unique keys and upserts. It makes recovery a routine action rather than a crisis.
Batch or real time
Real-time pipelines are more complex to build, test and monitor. If decisions are made daily, a daily batch is not a compromise; it is the right tool.
Schema drift is inevitable
Sources change without warning. Detect new, missing or renamed fields on arrival and alert someone before transformation, not after delivery.
Collect less personal data
If a workflow only needs a postcode district, do not ingest full addresses. Minimisation reduces risk and simplifies compliance.