Build APIs Your On-Call Future Self Can Live With

The incident started with a timeout. A client retried a payment request, the API created the same operation twice, and everyone spent the next hour arguing about whether the caller or the service was at fault.

The caller had retried exactly as its documentation recommended. The service had treated both requests as new work. Nobody had designed the contract for the boring reality that networks fail after the server commits but before the client receives a response.

That is the test of API design. Not whether the endpoints look tidy in Swagger. Not whether the first integration ships quickly. A good API stays understandable when requests arrive twice, dependencies stall, schemas change, and the original author is asleep.

Start with failure, not the happy path

Most API reviews begin with resources and verbs. I start with a less pleasant set of questions. What happens when the caller times out? Can it retry safely? How does it distinguish invalid input from a temporary dependency failure? If processing continues after the connection closes, how can the caller discover the result?

Any endpoint that creates work should have an explicit retry story. For synchronous operations, an idempotency key is usually the cleanest answer:

POST /v1/orders HTTP/1.1
Content-Type: application/json
Idempotency-Key: 8e65b5c4-6bc1-4d12-a3d7-04ce77ddaa12

{
 "customer_id": "cus_1842",
 "items": [{"sku": "SSD-2TB", "quantity": 2}]
}

The server stores the key alongside a hash of the request and the resulting response. Repeating the same request returns the original result. Reusing the key with different content returns a conflict. Put a retention period in the contract, because keeping keys forever is not a design.

Do not bolt this onto the controller with a cache lookup and hope for the best. The idempotency record and business operation need a transaction boundary, or another mechanism with equivalent guarantees. Otherwise, the duplicate you were trying to prevent will arrive in the gap between them.

Long-running operations need a different shape. Return 202 Accepted, provide an operation resource, and let clients poll or subscribe to events:

{
 "operation_id": "op_01J9QK7F2V",
 "status": "pending",
 "status_url": "/v1/operations/op_01J9QK7F2V"
}

This is less magical than holding an HTTP connection open for four minutes. Less magical is good at 2am.

Your database schema is not an API model

Exposing tables through HTTP feels productive until the first modernization project. Then a harmless database change becomes a breaking API change, and a temporary column name survives for six years because three internal tools depend on it.

API models should describe the domain clients need, not the storage choices the service happens to use. That separation gives the service room to split tables, move data, introduce queues, or replace a component without dragging every consumer through the migration.

It also forces better naming. A field called status_code means very little outside the database that created it. A field called fulfilment_status with documented values tells a consumer what decision it can make.

Be conservative with enums. Clients routinely generate exhaustive switches from OpenAPI definitions. Adding a new value can break them even when the JSON remains valid. Document that consumers must tolerate unknown values, and consider an unknown fallback in generated SDKs.

Nullability deserves the same attention. Missing, null, and empty are three different states. If the distinction matters, specify it. If it does not, pick one representation and enforce it.

Compatibility is a daily habit

Versioning will not rescue careless changes. A /v2 path gives you somewhere to put a new contract, but now you operate two contracts, two documentation sets, and often two sets of bugs.

Prefer additive evolution inside a stable version. Add optional response fields. Accept old request forms during a migration. Never change the meaning of an existing field because its name still looks convenient.

Run contract checks in the pipeline. Tools such as oasdiff can compare an OpenAPI document against the version on the main branch:

oasdiff breaking \
 https://raw.githubusercontent.com/acme/platform/main/openapi.yaml \
 ./openapi.yaml

The repository URL here is illustrative, but the practice is concrete. A pull request that removes a response field, makes an optional property required, or narrows an accepted type should fail before deployment.

Generated specifications also need review. Framework annotations are useful, but they capture what the code exposes, not necessarily what the team intended to promise. Treat openapi.yaml as a product artifact. Give it owners. Review examples, error responses, authentication requirements, and deprecation notes alongside the implementation.

Errors are part of the contract

Returning 400 with {"message":"bad request"} is an invitation to open a support ticket.

Clients need a stable machine-readable code, a human-readable explanation, and enough context to fix their request. They should also receive a request identifier that an operator can search in logs and traces.

{
 "type": "https://api.example.com/problems/invalid-address",
 "title": "The delivery address is invalid",
 "status": 422,
 "code": "INVALID_DELIVERY_ADDRESS",
 "request_id": "req_01J9QM1E7A",
 "errors": [
 {"field": "postcode", "reason": "unsupported_format"}
 ]
}

RFC 9457 problem details provides a useful base shape. You still need to define your own error catalogue and decide which details are safe to expose.

Avoid making clients parse prose. Error messages change when someone fixes grammar. Error codes should change only when the condition itself changes.

Status codes should carry their ordinary meaning. Use 401 when authentication is missing or invalid, 403 when the identity lacks permission, 404 when a resource cannot be found, and 409 when current state conflicts with the requested operation. A blanket 200 response containing an error object makes gateways, metrics, retries, and humans work harder.

Design for the operator too

Cloud native software development adds more hops, more identities, and more places for a request to disappear. Every API should propagate trace context, emit structured logs, and expose metrics around latency, error classes, dependency failures, and throttling.

Do not log access tokens, full request bodies, or customer secrets just because debugging is easier. Redaction needs to happen before data reaches the logging backend. By then, copying has already begun.

Put limits in writing. Maximum body size, page size, timeout, rate limits, and concurrency rules are contract details. If clients only discover them from a production 413 or 429, the documentation has failed.

Security belongs in the same design review. Define scopes around actions, not job titles. Validate object-level authorization on every resource lookup. Set request size limits before parsing expensive payloads. For public and internal APIs alike, assume identifiers will be guessed and input will be hostile.

Internal does not mean trusted. It usually means the attacker needs one compromised workload instead of an internet connection.

Make the paved road usable

Good platform engineering reduces the number of API decisions each team must reinvent. A shared service template can include OpenAPI validation, standard problem responses, trace propagation, authentication middleware, idempotency support, and hardened pipeline checks.

The template should remain replaceable code, not a platform team mystery box. Product engineers need to see how requests are authenticated, where timeouts are configured, and what gets logged. Internal tooling earns adoption when it removes repetitive work without hiding operational truth.

This is where software development and platform engineering meet. The API contract, cloud native runtime, and delivery pipeline are one system. Designing them separately produces gaps that only appear under load or during an incident.

If you only do one thing this week

Take the busiest create endpoint you own and write down exactly what happens when the client sends the same request twice.

Then prove it with an integration test. Drop the connection after the server commits, retry with the same idempotency key, and verify that only one business operation exists. If the expected behaviour is unclear, you have found a contract problem before production finds it for you.

Your future self does not need a clever API. They need one whose behaviour remains boring when everything around it is misbehaving.

If this is on your plate, TecLeads does exactly this as part of our Software Development work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.