It worked yesterday.
Nobody changed anything.
Both halves of that sentence are said in good faith, and at least one of them is usually wrong. Without a version history there is no way to tell which.
The only reliable way to answer this is a configuration history: a stored copy of every device's configuration over time, so two dates can be compared line by line. With one, the investigation is a diff and takes minutes. Without one, it is device-by-device recall and takes hours — during an outage, which is the worst possible time to be reconstructing what somebody did on Tuesday. The change that caused an outage is usually small, recent, and made for a good reason by someone who did not expect it to matter.
Six changes that break things quietly.
None of these look dangerous at the moment they are made, which is precisely why they are the ones that reach production.
A speed or duplex setting pinned by hand
Someone hard-sets one end of a link to solve an immediate problem. The other end stays on auto-negotiation. It works, passes traffic and looks healthy until the link gets busy.
Look for: an interface-level change on one device with no matching change on its neighbour.
An ACL line added in the middle
Access lists are ordered. A rule inserted above an existing one changes the behaviour of everything below it, and the diff is two lines while the effect is enormous.
Look for: line-number changes within an ACL, not just added or removed entries.
A VLAN allowed — or no longer allowed — on a trunk
Tidying a trunk's allowed list is routine housekeeping right up to the moment it removes the VLAN that one quiet but important system depends on.
Look for: changes to trunk allowed-VLAN lists in the days before the incident.
A routing metric or preference adjusted
Traffic silently moves to a different path. Everything still works, but now it is asymmetric, or crossing a firewall that was not in the path before.
Look for: metric, weight or local-preference changes; then check whether return traffic still takes the same route.
An automation run that did more than expected
A template applied to a group where one member was not quite like the others. The automation log says success, because from its point of view it was.
Look for: identical changes across many devices at the same second, and check the outlier.
A change made and then partly undone
Someone tried something, decided against it and reverted — nearly. The half that remains is the one nobody remembers making.
Look for: two changes close together on the same device. The second one is often incomplete.
Four steps, in this order.
- 1
Fix the window from the symptom, not from memory
Take the first hard evidence of the failure — a monitoring alert, a log entry, a user report with a timestamp — and work backwards from there.
- 2
List every device that changed in that window
This usually cuts the search from hundreds of devices to two or three, and it takes seconds when the history exists.
- 3
Diff the candidates against their last known-good version
Read the changed lines rather than the whole configuration. The line that caused it is almost always obvious once it is isolated.
- 4
Restore, then investigate
Roll back to the last working version to end the outage, and do the analysis afterwards. Debugging in production while people cannot work is how a ten-minute incident becomes a half-day one.
Every change, with who and when.
Discovered devices
| DEVICE | TYPE | PORTS | LOAD |
|---|---|---|---|
| core-1 | Router | 48/48 | 62% |
| core-2 | Router | 48/48 | 55% |
| dist-3 | Switch | 96/96 | 71% |
| acc-3 | Switch | 46/48 | 94% |
| acc-4 | Switch | 48/48 | 38% |
Physical map
WAN-1 utilisation
38%
WAN-2 utilisation
41%
Response time
480ms
Conversations on this link
LINK ONLY 38% USED
- Backup replication10.4.18.2234%
- File share (SMB)10.4.2.5124%
- Video conferencing10.4.9.14017%
- Database sync10.4.31.812%
- Everything else—13%
- Retransmit storm on Gi0/3acc-3
Sniffer recording packets · +412% vs baseline · duplex mismatch suspected
9614:36 - Unusual east-west conversation10.4.18.22
NetFlow · new peer pair, 3.1 GB in 20 min, off-hours
7114:22 - Latency drift on WAN-2core-2
SNMP + sniffer recording packets · RTT trending up 6 hours
6411:08 - Baseline re-learneddist-3
Model updated after sustained topology change
1209:41
Recent configuration changes
142 DEVICES BACKED UP · 2 DRIFTED
- OFF BASELINE
interface Gi0/3 speed 100 → auto
acc-3 · n.levi · 14:31
ACL 120 — 2 lines added
core-1 · automation · 11:55
VLAN 340 created
dist-2 · m.cohen · Yesterday
- OFF BASELINE
SNMP community updated
acc-5 · n.levi · Yesterday
Illustrative views representing Net-Monitor output.
What a configuration history changes
- Backups happen on a schedule and after every change. No script that somebody wrote once and nobody maintains.
- Every version is diffable against every other. Two dates, side by side, line by line — which is what turns recall into evidence.
- Rollback is restoring a file, not rebuilding from memory. That is the difference between a ten-minute incident and a half-day one.
Short answers.
How do I find out who changed a switch configuration?
You need stored versions of the configuration over time. With a version history you compare the current configuration against the last known-good one and read the differences directly. Where device access is logged centrally, the account that made the change lines up with the timestamp of the version.
Can I roll back a bad network change?
If the previous version was saved, yes — restoring it is straightforward and fast. If it was not, you are reconstructing the previous state from memory and documentation, which is slow and error-prone at exactly the moment when neither is affordable.
What is configuration drift?
Drift is the gap between what a device is configured to be and what your standard says it should be. It accumulates from one-off fixes, emergency changes and devices provisioned before the standard existed. Detecting it continuously is what stops "we are compliant" from being an annual claim rather than a fact.
Does this work across different vendors?
Yes. Backup and restore work over SSH and the standard protocols the equipment already supports, so a mixed estate is managed from one place rather than through a separate tool for each manufacturer.
How long does it take you to find a change today?
That is a good number to measure before a demo. We will show you what it becomes afterwards.