99.94%
Availability
SLO 99.9% · met
Architecture review
The order platform holds today. Peak season needs one change.
Current state, the bottleneck we found, three options and the change we propose.
Where we stand
Healthy today, close to the limit at peak
Last 90 days, production.
420 ms
p95 checkout latency
Budget 500 ms
2.1×
Peak traffic expected
Black Friday forecast
31
Deploys per week
From 18 last year
Source: monitoring dashboards, 90-day export
How an order flows
Three services, one synchronous chain
Each hop adds latency and a failure mode.
API gateway
Authenticates and validates the order.
Owner: edge team
Order service
Prices and reserves stock in one call to the inventory database.
Owner: order team
Fulfilment
Books the carrier and sends the confirmation.
Owner: logistics team
Failures retry synchronously from the gateway
The takeawayThe order service waits on the inventory database for every order.
Latency budget
The inventory lock takes half the budget
p95 per hop at today's peak, in ms.
Inventory lock
210 ms +160
Pricing
85 ms +40
Gateway
60 ms +10
Carrier booking
45 ms +15
Confirmation
20 ms
| Inventory lock | 210 ms | +160 Added at 2.1× traffic |
|---|---|---|
| Pricing | 85 ms | +40 Added at 2.1× traffic |
| Gateway | 60 ms | +10 Added at 2.1× traffic |
| Carrier booking | 45 ms | +15 Added at 2.1× traffic |
| Confirmation | 20 ms |
Added at 2.1× trafficSource: load test LT-48, 2.1× replay
Options
Reserve stock asynchronously for the best balance
Scored with the order and data teams.
| Option | Peak latency | Effort | Risk | Run cost |
|---|---|---|---|---|
| Scale the database | Weak | Low | Low | High |
| Async reservation | Good | Medium | Medium | Low |
| Split the inventory | Good | High | High | Medium |
RecommendationMove the stock reservation to an event, confirm the order first and compensate the rare failures.
ADR-021
Reserve stock after the order is accepted
Proposed · for approval today
Context
At 2.1× traffic the synchronous inventory lock pushes p95 latency to 560 ms, above the 500 ms budget.
Decision
Publish an order-accepted event and reserve stock asynchronously; cancel and notify when stock is gone.
Consequences
- +p95 latency back under 350 ms at peak
- +The database lock leaves the checkout path
- −About 0.2% of orders cancelled after acceptance
- −A new compensation flow to test and operate
Risks
Three risks, all with an owner
Reviewed with service owners.
| Risk | Impact | Likelihood | Owner | Mitigation |
|---|---|---|---|---|
| Oversold stock during flash sales | High | Medium | Order team | Hold a 2% safety stock on promoted items. |
| Event backlog at peak | Medium | Medium | Platform team | Autoscale consumers; alert on lag over 30 s. |
| Customers confused by late cancellations | Medium | Low | Customer care | Clear email and an instant refund. |
Before we build
Four points to confirm with the board
Decision needed today.
| Done | Owner | Due | |
|---|---|---|---|
| Approve ADR-021 and the compensation flow | Architecture board | Today | |
| Agree the 0.2% late-cancellation budget | Product owner | 10 Oct | |
| Load test at 2.5× before the code freeze | Platform team | 31 Oct | |
| Update the runbook and on-call training | SRE lead | 7 Nov |
Sources
- 02Healthy today, close to the limit at peakMonitoring dashboards, 90-day export
- Peak forecast 2026
- 04The inventory lock takes half the budgetLoad test LT-48 report