Connected is a lousy acceptance test for an integration

Imagine your online store shows 120 paid orders. The warehouse has 117. The connector dashboard is green, and nobody can tell you which three orders are missing.

This is a hypothetical example, but it is a useful test of what you are buying. An integration should help someone find those orders, explain their state and recover them safely. A successful connection test cannot tell you whether it will do that.

If I were reviewing the proposal, I would want the recovery demonstration before signing off. That work belongs in the original scope.

This week's changes make the question concrete

On September 16, Shopify announced changes to Events payloads, subscription configuration and delivery headers. Its notice explicitly says classic Webhook subscriptions are unaffected. Events is currently in developer preview, so this is a specific change for adopters, not evidence that every Shopify integration has broken.

Separately, Atlassian's September 17 developer notice described incomplete Marketplace partner invoice reporting for September 15–17. It said the underlying issue was fixed and queued data was being backfilled, with no data lost. It advised fetching the affected reports again once recovery finished.

One example concerns a changing interface; the other concerns delayed reporting. For a buyer, both raise a practical question: how will the people using your integration know when its output is incomplete?

Give the missing order an identity

Return to the 120-order example. First, make sure both systems are answering the same question. Perhaps one includes canceled orders, or their reporting windows use different time zones. A count mismatch is a reason to investigate, not proof that a connector dropped data.

Once the definitions agree, the integration should be able to compare the actual order identifiers. Which orders were accepted by the warehouse? Which are waiting? Which failed validation? A log saying that a request was sent leaves the most useful part unanswered.

I would ask for a small operational view that a member of staff can use without opening a developer console. It should show the affected order, the last confirmed step, the time of that confirmation and the next action. If the destination has not confirmed receipt, say that. Do not turn an unknown outcome into a reassuring success label.

That view does not need to reproduce the entire order record. Keep customer details out of routine logs unless they are necessary for the job, and restrict access to the people who need it.

Ask for a demonstration with something broken

In a test environment, have the team walk through a few ordinary failures:

• The same order arrives twice. Show that the second delivery does not create another shipment or customer notification.
• The receiving application stops responding. Show where the unfinished work waits, when someone is alerted and how it resumes.
• An older update arrives after a newer one. Show that a stale message cannot quietly overwrite the current order state.
• A request may have succeeded, but its confirmation was lost. Show how the system checks the destination before repeating an action with real consequences.

These are proposed acceptance checks, not a claim that every integration needs the same architecture. An overnight reporting feed can tolerate delays that a warehouse dispatch workflow cannot.

Shopify's delivery guidance warns that a webhook can arrive more than once. Its webhook overview also says ordering and delivery are not guaranteed, and recommends periodically checking source data to restore consistency. Designing for those behaviors is part of using the platform correctly.

Recovery needs rules about what may happen again

Checking the source against the destination is usually called reconciliation. The business decision is how often that check happens, what counts as a meaningful discrepancy, and who acts on it.

For our hypothetical store, comparing identifiers and expected states could expose the three unconfirmed orders. But copying every old message into the workflow again would be a dangerous recovery plan. An order might already have shipped. A customer might have canceled it since the original event. A refund might already exist in the receiving system.

The recovery path needs to distinguish a missing record from a completed action whose acknowledgment went missing. Where the destination supports a stable operation identifier, use it to recognize repeated requests. Where it does not, define how an operator confirms the result before trying again. Some exceptions should remain manual.

Agree on acceptable delay in business terms. “We need to know about unconfirmed dispatches before the carrier collection” gives the team something useful to design and test. “Make it real time” leaves a lot of interpretation.

Put the ongoing work in the proposal

An integration depends on systems that other people can change. Someone needs to review relevant notices, maintain credentials, test updates and respond to failures. Name that owner, and make clear what the maintenance arrangement includes.

Keep the answer proportionate. A low-volume workflow might be adequately served by a daily exception report and a documented manual correction. Higher-volume or time-sensitive work may need a queue, automatic checks and alerts. The consequences of a missed or repeated action should drive that choice.

Rosecraft's integration and modernization work starts with the workflow the business needs. For an existing connector, a technical assessment can help identify which states are visible and which recovery steps are still assumptions.

Before accepting the build, ask the team to lose a test order on purpose and show you what happens next. That demonstration will tell you far more than the green checkmark.