Import data into new software: turn an export into reviewed records

Decide which information is genuinely needed

Begin with the task supported by the first software release. The consultancy may need only active customers with outstanding enquiries. Old drafts, unused fields and closed records might require different treatment. Describe selection through a date, state and accountable owner so two people can independently recognise and explain the same export, rather than relying on a vague request to copy everything relevant.

Record what deliberately remains outside the first import and how staff can find that information when necessary. A smaller selection is not a quality improvement if important work silently disappears. Ask the data owner whether the initial dataset supports the chosen task and which historical context remains essential to understanding the records. Selection is a business decision as well as a technical filter.

Turn field names into meaningful mapping

For each source field, state its destination and required transformation. Describe meaning too. A date might indicate receipt, an appointment or the latest change. A source state called “done” might mean an employee finished their activity while the new portal uses it for completed customer confirmation. Comparing column names alone leaves that difference invisible and can produce consistently wrong records without any loading error.

An explicit translation boundary can help where systems use different meanings. Microsoft describes that idea in its Anti-Corruption Layer pattern. A simple documented transformation may be sufficient for this import; an additional platform is not mandatory. Preserve the selected meaning and the person who confirmed it so others can later explain why a source value became a particular destination value.

Microsoft Azure Architecture: Anti-Corruption Layer pattern ↗

Preserve identity and relationships

Choose a recognisable source reference for each imported object. A name or email address is not automatically a stable key. Two contacts may share an address, and a company name can change. Record which reference identifies a customer, enquiry or document and how its relationship with the new record will be retained or recovered during review and subsequent support investigations.

Test a customer with several enquiries and an enquiry with multiple attachments. Include a document with an unclear customer relationship. The aim is not to resolve every ambiguous case automatically. Create a review list where someone can decide what should be linked, separated or excluded. An uncertain relationship should not silently become a final merge that is difficult to explain or reverse afterwards.

When an old system remains operational, migration also needs a transition period and a recovery decision: Migrating a running legacy system: ownership, cutover and fallback.

Agree file format and missing-value rules

Describe encoding, column order, delimiters, date representations and the notation for missing values. An empty cell might mean absent information, an intentional blank or not applicable. Choose explicit destination values or a review route for those cases. Import should follow agreed meaning instead of the accidental default setting of whichever program happened to produce or read the export file.

A database tool such as PostgreSQL COPY can load files into tables, but that command does not decide whether imported business records are useful. Select tools appropriate to the environment and examine error treatment. Retain a small suitable trial set containing important formatting differences. A later export or parser change can then be checked against the same cases instead of relying on the memory of the original importer.

PostgreSQL: COPY ↗

Start with a representative bounded trial

Use an isolated trial environment and a dataset covering relevant situations. Include ordinary records alongside missing references, duplicate names, altered states and an attachment with a different type. Use appropriate fictional or carefully prepared test information. The aim is evidence about the design; a random collection of real customer files is not automatically a proportionate or suitable test dataset.

After loading, inspect actual screens and relationships. A technical success log does not establish that a customer contact can find the correct enquiry. Ask a member of staff to perform the selected task using imported examples. Separate business-meaning errors from formatting failures. This reveals whether mapping needs adjustment or the original file needs clarification from its data owner before another technical attempt is made.

Reconcile counts without overestimating them

Summarise selected source records, accepted destination records, rejected input and deliberately excluded cases. Explain differences. Count relationships and attachments where they matter to the work. Matching customer totals can still coexist with incorrect addresses, missing enquiries or a state translated incorrectly for every record. A green count comparison should not conceal those different kinds of potential defect.

Add targeted review of named substantive properties. Select customers with several enquiries and trace some source references fully. The extent and selection of review depend on the information and consequences of mistakes. A simple count and a focused business trial complement each other. Neither should be represented as evidence that every conceivable error has become impossible or that no subsequent correction will ever be needed.

Make rerunning the import predictable

Trial imports often need repeating after a mapping or file correction. Agree how another execution recognises the same source records. Will trial data be deliberately replaced, existing destination records updated or new records added? The intended behaviour should be visible before rerunning so correcting a trial does not create another set of unexplained duplicates and make the evidence harder to interpret.

Retain the input version, mapping and selection used for each execution. Keep an error list with source references and outcomes. The team can then investigate differences between trials without guessing which file was loaded. This article is limited to an import dataset; continuing changes across live systems additionally require a transition design and explicit decisions about where people write information during that period.

Ask the data owner to make a substantive decision

Identify who has enough knowledge and responsibility to assess the imported information. Different types of records may have different owners. Customer administration might review identity and contact information while a case handler checks open-enquiry meaning. Avoid requesting a blanket signature on a file nobody could understand; connect sign-off with concrete findings, reviewed properties and known limits of the evidence provided.

The decision should state what is accepted, which corrections remain necessary and which unresolved issues block the next step. Preserve examples supporting that conclusion. The developer can demonstrate technical loading while the organisation explains why the resulting information is usable for the initial task. These responsibilities complement each other. Database access does not automatically make someone the substantive owner of all information it contains.

Deliver the import knowledge with the application

Retain mapping, source selection, execution references, checks and known exceptions alongside the new application. Explain how staff report a potentially misimported record and how its origin can be found. An operator should not need the personal recollection of whoever prepared the first file to investigate a customer question or explain why a particular historical value appears in a new screen.

After introduction, review actual defects and distinguish import issues from new user operations. Do not silently change historical source information to disguise a discrepancy. Record the correction and reason under the selected working method. Import then remains an explainable project step, with a bounded dataset, repeatable procedure and an accountable business decision about the usefulness of its results instead of a mysterious technical event.

Turn the explanation into a project decision

Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.

• Custom software development ↗
• Customer portal development ↗

Questions about this topic

Continue with a related question