Every analytics project eventually meets the system that predates it. There is no REST API, the vendor is unresponsive, and the one person who understands the schema retires next year. There are still workable paths — and one very tempting bad one.
Read replica: the first thing to ask for
A read-only replica of the database gives clean access without touching production load. It is the least invasive option and usually the fastest to arrange. Ask for this before designing anything more elaborate.
Scheduled extract: unglamorous and reliable
A nightly job writing CSV or Parquet to a landing area handles most reporting needs. It is easy to reason about, easy to re-run, and easy for a successor to understand. If the requirement tolerates a day of latency, stop here.
Change data capture: when latency matters
Reading the transaction log gives near-real-time changes without polling. It needs database-level cooperation and adds operational complexity, so it should be justified by a genuine latency requirement rather than by preference.
The tempting bad one: screen scraping the UI
It works in the demo, and it breaks on the next patch, silently, in a way nobody notices until a report is already wrong. If it is genuinely the only path, treat it as temporary, alarm it heavily, and write down when it will be replaced.
Any integration that cannot tell you it has stopped working is worse than no integration, because it fails quietly and the decisions carry on.
Expect the data to be messier than the schema suggests
- Free-text fields used as structured ones
- Codes retired years ago but still present in history
- Multiple conventions for the same entity across regions
- Deleted rows that are actually flagged, not deleted
- Dates stored in local formats that vary by module
Land raw, transform later
Keep an untouched copy of what came out. When a number is questioned six months later, being able to prove what the source actually said is worth the storage cost several times over.
Document the tribal knowledge while you can
The person who knows why column 14 is only valid after 2019 is the most important part of this project. Write down what they know as part of the work, not as a nice-to-have at the end.