Backing up configs to Git: Oxidized/RANCID and review

Goal: keep configs and know who changed what

Network devices are usually altered in small steps: added a VLAN, tweaked an ACL, changed a route. Such edits are easy to lose after a failure, hardware replacement, botched upgrade, or simply because no one remembers the exact change a week later. Worse is when changes go unnoticed and only surface during an incident.

Backing up configs to Git solves two problems at once: you keep a copy of “how it was”, and you get a history showing “who, when, and what exactly changed”. Git is convenient because it shows line-by-line differences (diff). That's more illustrative than date-stamped archives or a folder named “final_final2”.

Typically the backup includes configs from access and core switches, routers, firewalls, load balancers, and Wi‑Fi controllers. Formats differ, but the goal is the same: regularly fetch the current running-config (or equivalent) and store it in a repository so changes are readable to humans.

If you design the process in advance, audit becomes real, not just formal. Define three things upfront: how you strip secrets, who has repository access, and who is responsible for investigating unexpected diffs (and confirming “this was planned”).

A minimal useful start is simple: daily automated config collection, a single repository, and a clear change history. Then for a disputed edit you can quickly see what changed and confidently roll back to the last working version.

Oxidized and RANCID: which to choose and why

If the goal is backing up configs to Git with readable change history, both tools solve the same task: regularly fetch device configs and normalize them for useful diffs.

Oxidized is often chosen for its simplicity and a “live” polling model. It works well with Git as a backend and can poll devices frequently (for example, every 5–15 minutes). This is useful where changes happen during the day and it’s important to quickly see what changed.

RANCID is the classic “collect once a day, commit, get history” approach. It fits well when changes are rare and collection is primarily for control and recovery.

Limits usually come from vendor support and output stability rather than Git. In practice diffs can be noisy because of lines with timestamps, counters, uptime, or random ordering of blocks. Another typical issue: on some devices show/run returns the full config, on others only part of it. It’s best to test on a pilot set.

Choice typically depends on your operating mode:

• Up to a few hundred devices and rare changes — RANCID with daily collection is often enough.
• Lots of changes and the need for quick capture — Oxidized with frequent polling is usually better.
• Mixed fleet and varying frequency needs — separate policies are reasonable: collect critical devices more often, others on schedule.

If the network core in a data center is changed by requests a couple of times a month, RANCID will run quietly for years. If the team often edits ACLs and NAT on the perimeter, Oxidized will show unexpected commits faster and simplify incident investigation.

Architecture: where the collector lives, where Git is, and how to connect

To make config backups to Git reliable, separate roles: the collector fetches configs, Git stores history, and the network grants only the necessary access to devices. The fewer extra paths and privileges, the easier auditing and safer operation.

You can host Git as a separate service or use an internal GitLab/Bitbucket/DevOps platform if the organization already uses one. It’s important that the repository is reachable by the collector over a stable channel and that branch protection and review rules are in place.

Run the collector (Oxidized or RANCID) on a dedicated host, not on an administrator’s workstation: a VM, dedicated server or container. That makes updates, backups of the collector itself, and access separation easier.

A minimal scheme usually includes the Git service, a host for the collector, a management network (or a dedicated segment), and logging of access at least to the collector host's system logs. If direct access to the management network is prohibited, add a jump-host and let the collector reach devices only through it.

Make device connections via the management network, and keep ACLs minimal: allow only the necessary ports (usually SSH) from the collector host to the device list. Create separate read-only accounts on devices for the collector, without rights to change configuration.

Store Git credentials as an SSH key or token with minimal rights (only push to the required repository). Agree on key rotation and who can replace them so the process doesn’t depend on a single person.

Access rights: who can read, change and approve

When configs live in Git, access to them becomes part of network security. Define roles early so backups don't turn into a shared “password archive” and change history remains useful for audits.

Three levels are usually enough:

• Collector (Oxidized/RANCID + service account in Git): adds commits to a dedicated branch or repository and nothing else.
• Reviewers (network engineers/on‑call): read everything, open MR/PRs, approve changes, and resolve conflicts.
• Readers (audit, InfoSec, adjacent teams): read-only access.

The principle of least privilege is simple: full history is not needed by everyone, only by those who investigate incidents or accept changes. Others typically only need read access and the ability to see diffs in MR/PR.

Separate access by zones. A practical approach is separate projects or at least separate directories/branches for prod, test, sites, and critical segments (for example core and perimeter). That way a regional reviewer doesn't see data center configs and auditors don't mix contexts.

Enable branch protection in Git: nobody pushes directly to main. Allow merges only via MR/PR. For critical zones require a minimum of two approvals (for example, a network engineer plus an InfoSec representative).

Secrets in configs: how not to store sensitive data in Git

If you back up configs to Git, adopt a simple rule: no secrets in the repository. Git stores history well, which is exactly why it's a poor place for passwords and keys — old commits remain for years and repository access often expands over time.

Do not commit: SNMP community strings, local passwords, enable secrets, SSH private keys, VPN pre-shared keys, API tokens. Often the issue is accidental: an engineer adds a user, the collector picks up the config, and a secret ends up in history.

Good practice is to strip secrets during export (filters/sanitizers) or store them separately in a secure vault, leaving only template strings in the config. Guideline: a config in Git should help with diffs and restoration but should not allow access to the device without additional data.

Add a check before commit: a pre-commit hook or a CI step that searches for suspicious lines with regex and blocks the change. False positives are preferable to a quiet leak.

If a secret has already made it into history, act according to plan: rotate the password/key/community/PSK, check dependencies (gateways, monitoring, VPN), temporarily restrict repository access, purge history and add rules to prevent recurrence.

How to configure collection so diffs are useful

The goal is not just to “save a file”, but to get readable diffs showing what changed in settings, not what “updated itself”. Normalize configs to a stable form.

Normalize output: remove noise

Many devices add lines that change constantly: generation time, uptime, counters, serial numbers in headers. If you don't remove them, every commit will look like a large change.

Filter out lines that don't help reviews. For example, remove ! Last configuration change ... or ! NVRAM config last updated ..., and dynamic statistic blocks.

Naming conventions and inventory

Diffs are easier to read when it's clear what device and where it is. Agree on a naming scheme so the repo doesn't contain sw1, sw2 or test.

A working format: site-role-model-hostname. For example: ala-access-c2960-ala-sch-01 or ast-core-n9k-ast-dc-01.

Keep inventory as the source of truth for the collector: lists of devices, groups and tags (site, role, criticality). Then you can apply different polling and filter rules without manual exceptions.

Polling schedule and different policies

Frequent polling means freshness but more load and more noise in history. A basic schedule plus exceptions helps: poll core and DC devices more often, access devices less often, run edge and branch polls at night, and trigger a forced collection after a change window.

Different policies by device role almost always pay off: less noise and faster detection of real config errors.

Repository structure template (starter)

A good repo for network configs solves two tasks: lets you quickly find a device and produces clean diffs. Below is a template suitable for both Oxidized and RANCID.

repo/
  inventory/
    prod/
      almaty/
        core.yaml
        access.yaml
      astana/
        core.yaml
    test/
      lab.yaml

  configs/
    prod/
      almaty/
        core/
          rtr-a1.cfg
          rtr-a2.cfg
        access/
          sw-a10.cfg
      astana/
        core/
          rtr-n1.cfg
    test/
      lab/
        access/
          sw-lab1.cfg

  docs/
    naming.md
    onboarding.md
    restore-playbook.md

  tools/
    sanitize-config.sh
    normalize-lines.sh

  policies/
    access.md
    review-rules.md
    retention.md

For filenames the scheme \u003cenv\u003e/\u003csite\u003e/\u003crole\u003e/\u003chostname\u003e.cfg is usually enough. If you have many identical hostnames (for example after migrations), add an inventory ID prefix: 00123-rtr-a1.cfg. The path should answer “where the device is and what role it performs”.

Store brief metadata alongside config in inventory: model, OS version, mgmt IP, owner (team or department), contact for approvals, criticality, maintenance window. This helps during review: you see the change and understand the context.

Don't put things in the repo that bloat history or increase leak risk: private keys and passwords, log dumps “just in case”, binaries (firmware, archives), and configs with dynamic counters that change on every run (clean those with tools/ before committing).

Review process before merging: branches, MR/PR and approval rules

To keep Git history suitable for review, agree that only reviewed changes land in main. Make any change (even “fixed interface description”) in a working branch and then open an MR/PR.

The scheme can be the same for planned work and for automatic collection after a change window. In both cases the goal is that a person confirms the diff is expected, not a result of an error.

Branches and minimum rules

A basic set is usually enough: protect main (no direct pushes), name branches meaningfully (for example, change/\u003cdevice\u003e-\u003cdate\u003e or ticket/\u003cid\u003e), require at least two approvals for critical devices, authors do not approve their own changes, and remove the branch after merge.

What to include in an MR/PR

Descriptions should be short but provide context: reason and ticket number, list of affected devices, maintenance window and expected effect, rollback plan, and a note that secrets check passed and the diff is readable.

Example: after replacing a switch, an engineer opens an MR/PR with VLAN and ACL changes, documents rollback, and confirms passwords and SNMP communities are not in the diff. The reviewer checks that only intended lines changed and after approval the change goes to main.

Change audit: making history useful for verification

Audit works only when commits read like an action journal, not random diffs. For config backups to Git, set a standard: what counts as a change, how it’s signed and where to look for approval.

What to capture in history

If Oxidized/RANCID collects and a service account commits, don't lose the human trace. A consistent commit message format and linking to a ticket helps.

Practical minimum: device name and method (oxidized/rancid), time (in a single timezone), executor (login/full name), reason (change or incident number). Then git log plus diff answers “who changed what and when”.

Checkpoints, reports and collector logs

For big changes create checkpoints with tags (for example, BEFORE-CHANGE-\u003cid\u003e and AFTER-CHANGE-\u003cid\u003e). That makes it easier to prove before/after and to roll back.

Daily summaries are useful: which devices changed, which failed to be polled, and which changes affected critical nodes (core, perimeter, DC). Don't hide access errors and unreachable devices in commits. Store collector logs separately: success/failure, reason, time, and which host performed polling. That shows not only changes but coverage and gaps.

Common mistakes and pitfalls when storing configs in Git

The problem is almost never Git itself but the rules around it. Without rules the repo becomes a dump: unclear what changed, why and who approved it.

A typical pitfall is mixing manual and automatic commits. For example, Oxidized commits every 5–10 minutes while an engineer concurrently makes a manual commit after an incident. History fragments and an important change gets lost among background edits. Agree in advance: either manual commits are forbidden, or they must go through a separate branch and review.

Another common mistake is overly broad privileges. Configs include addresses, topology and sometimes lines that ease access. If everyone can read the repo you expand the attack surface. Minimum: read access only to those who need it, write only to automation and a small group, and approvals by a separate role.

Large batches of changes are another pain. If 300 files arrive overnight without proper description and without review, investigation becomes guesswork. Splitting by site/role and clear messages help to spot background noise vs real changes.

Have an incident plan: how to rotate passwords and tokens, how quickly to revoke collector keys, who freezes the repo during investigations, which logs to keep for forensics and when to rebuild the trusted access chain.

Quick checks: a daily control checklist

Daily checks are not “for show”. They help spot an issue before you urgently need a config after a failure or bad change.

A short 3–5 minute checklist:

• There are fresh backup commits from the last 24 hours, not a silent Git with no changes.
• The collector successfully connected to critical devices (core, edge, VPN, major switches) at least twice.
• No secrets surfaced in the repo (quick search for “password”, “secret”, “community”, “private key”).
• The main branch is protected: direct pushes are forbidden, changes flow via MR/PR and approvals.
• It's clear who is on duty and who reviews today so changes don't stall.

If something is red, fix the cause, not just the symptom. Two key devices not being backed up can be due to an expired read-only account, changed SSH algorithms, a new ACL, or a full disk on the collector host.

A useful team rule: every backup failure is turned into a short task with an owner and due date of “today”. If a secret appears in history react immediately: remove it from Git history, replace the secret on the devices, and add a rule to prevent recurrence.

Practical example: a device change and a transparent rollback

A branch reports: some employees can't reach the accounting system. A common cause is a change to a branch router's ACL. To avoid guessing in chats, backing up configs to Git and change discipline helps.

Typical flow: the engineer must not make a stealth change. They open a ticket, prepare an MR/PR describing what to change, why, the risk and how to verify.

Usual chain:

1. Engineer applies the change during the agreed window.
2. Oxidized/RANCID fetches the config and Git records the commit.
3. An MR/PR appears showing the diff.
4. Reviewer inspects the diff and asks questions.
5. InfoSec confirms that no access was widened inadvertently.
6. After approval the change is official and verification of connectivity is recorded in comments.

Git is useful not only as storage. The diff shows precisely what changed: one ACL line or also NAT, routes, VPN. History proves who did what, when and under which ticket and who approved it.

If connectivity breaks after rollout, rollback is quick. The engineer takes the last “good” commit, restores the previous config (or just the ACL block), applies it and returns to the expected state. Then investigate the root cause: rule ordering, missed subnet, or a deny conflict.

Next steps: run a pilot and scale carefully

To make config backups to Git work, start with a short pilot and clear rules. The pilot's goal is not “cover everything” but to verify collection stability, readable diffs, and that reviews don't become a formality.

1–2 week plan

Select 10–20 devices: one or two key models per vendor plus one “problem” device that is changed often. Then proceed: assign roles (collector admin, reviewers, merge approvers, read-only users), fix minimal rules (naming, polling windows, what counts as emergency), set up secret masking and verify nothing extra is committed.

Run 2–3 test changes on a lab or non-critical device and practice rollback. Collect feedback and refine filters and templates before wider rollout.

Documentation should be short: one page for engineers (“how to make a change and open an MR/PR”) and one for reviewers (“what to check before approval”). Aim for: a person who didn't participate in setup should follow the process without extra questions.

Later, small discipline and light automation bring more value than new features: alerts for unexpected changes, a daily report of devices without backups, and integration with ITSM so each change has a ticket number.

If you need to quickly deploy infrastructure for this process (collector host, Git service, logs server, basic resilience), that can run in parallel with the pilot. System integrators sometimes help: for example, GSE.kz (gse.kz) as a server manufacturer and systems integrator in Kazakhstan can select hardware for the collector and repository and provide 24/7 support within overall IT operations.