The collector can stay up while its destination rejects every request. To see what readers of your logs actually get, count events at the destination after it recovers. Keep track of how long the backlog takes to drain.
This lab runs Vector and a small Python HTTP receiver on one Linux host. It simulates a destination that is reachable but temporarily responds with HTTP 503. It needs Bash, Python 3, and a supported Vector release. Run it as an ordinary user. The receiver binds only to loopback, takes synthetic data, and uses no credentials. Confirm that port 18083 is free and that the temporary filesystem has room for the lab’s 300 MiB disk-buffer limit plus overhead.
1. Start a receiver you can switch offline
In the first terminal, create a fresh directory and record its absolute path. The receiver rejects requests while the unavailable marker exists. Accepted events get a receiver timestamp so you can measure the drain after recovery.
|
|
The receiver is disposable. It acknowledges after closing the capture file. That is adequate for this exercise, but it does not model a production service’s durability or authentication.
2. Configure the collector
In the second terminal, replace the placeholder with the path printed in step 1. The sink sends one event per batch to make its request and retry behavior easier to observe. The health check is disabled because this minimal receiver only implements the ingestion endpoint.
|
|
Vector retries eligible HTTP failures, including 503. A full buffer configured to block applies backpressure. It does not extend upstream file retention, and application logs can still rotate away while the collector is behind. The file source must be able to read any backlog you expect it to recover.
3. Inject the failure and restore the destination
In a third terminal, enter the same directory. Wait until the first marker is captured before continuing.
|
|
Confirm that the receiver logs at least one 503 and that no outage marker has reached the capture. Record collector errors and disk usage if you need evidence of the backlog. Leave Vector running and restore acceptance:
|
|
Wait for delivery. Retry backoff can delay the first successful request after the receiver is ready. Check for all ten IDs, then stop Vector cleanly before taking the final count.
4. Compare the expected IDs with the capture
|
|
A passing run captures each expected ID once in this experiment. The time is measured from just before the receiver starts accepting requests again. Interpret it as a backlog drain measurement only when all outage IDs arrived.
| Observation | First check |
|---|---|
| 503 responses continue | Check whether the marker file was removed in the receiver’s actual working directory. |
| The capture stays empty after recovery | Check Vector’s retry diagnostics and verify the URI, request format, and receiver status. |
| IDs are missing | Check source retention, retry limits, buffer policy, and whether the sink finished draining. |
| IDs repeat | Check retry and acknowledgement boundaries. A sender may retry after the receiver accepted data if the response was lost. |
The lab does not simulate power loss, process crashes, disk exhaustion, or a lost acknowledgement. For production sizing, start with measured ingress bytes per second and the outage interval you need to absorb. Add headroom for buffer overhead and continued ingress during recovery, then measure whether the destination drains faster than new events arrive. Keep retry limits, source retention, and the response behavior of your actual destination in the test plan.
Stop the receiver with Ctrl+C. Preserve the capture and configuration if you want to compare runs. Use a fresh temporary directory for a repeat so old checkpoints, disk-buffer files, and output cannot affect the results. For a separate writer-side check, see the NGINX log reopen guide.