Network time synchronization without broken log timelines

Infrastructure clocks must differ by less than the interval between events whose causal order you need to prove. If a server is 47 seconds slow, a switch is 18 seconds fast, and the log collector adds its own receipt timestamp, the combined timeline may look convincing while telling a false story.

I do not treat NTP as an auxiliary setting that can be configured once and forgotten. Time belongs to the evidence system alongside logs, configuration, and chain of custody. A dependable design needs several sources, internal time nodes, an explicit clock correction policy, and observation of the actual offset. A green service status proves nothing by itself.

Clock differences destroy causality

During an incident, the order of dependent actions matters more than the calendar itself. An account signed in to a web server, the application queried a database, a firewall allowed the connection, and a switch changed a port state. When timestamps contradict that order, the analyst spends the first hours arguing with the logs instead of finding the cause.

The errors do more than sort lines incorrectly. A correlation system may place one episode into separate windows, a detection rule may fail to assemble the sequence, and certificate or authentication ticket validity may be judged by a different clock. With strict authentication protocols, a noticeable offset can cause access failures. With more permissive systems, the subtler outcome is worse: the operation succeeds, but its trace appears before its cause.

A single record often has several times. The device adds the event time, an agent adds the read time, the syslog receiver records arrival, and storage records indexing. These fields are not interchangeable. Receipt time helps estimate delivery delay, but it does not repair a bad source clock. If a switch sends buffered messages after connectivity returns, they will arrive almost together even though they occurred minutes apart.

A time zone is not synchronization. Two nodes may display 12:00 and 18:00 yet refer to the same instant when both include the correct UTC offset. Conversely, two screens with identical digits may differ by seconds or minutes. UTC is more practical for central storage, while local time is better applied at display time. Daylight saving changes, historical zone changes, and manual settings then do not create repeated or missing intervals.

Finally, distinguish timestamp precision from clock accuracy. A record with six fractional digits looks precise to a microsecond, but the source may be two seconds wrong. The number of digits describes format resolution, not closeness to a reference. An investigation needs a measured error for each device class, not an impressive timestamp.

Investigation needs should define time requirements

Derive the allowed offset from the closest events you may need to distinguish. If a system must prove that a network rule took effect 200 milliseconds before a connection, a one-second tolerance is already too large. If an old controller logs only whole seconds, you cannot promise an order for two operations within one second even with perfect NTP.

List the log sources and record four properties for each one: timestamp resolution, expected clock offset, synchronization method, and behavior after a restart. Mark devices that lack a battery-backed hardware clock separately. They may boot with a factory date and then step sharply after the first NTP response. You cannot mix those early boot records with later records without an adjustment.

RFC 5424 allows syslog to send UTC with the Z suffix, an explicit numerical offset, and fractional seconds. It also allows NILVALUE when the source cannot obtain system time. This distinction matters: no trustworthy timestamp is more honest than an invented local date. Require year, month, day, hour, minute, second, zone, and enough fractional precision. A zone name such as ALMT is worse than a numerical offset for machine parsing because zone rules change and the string does not explain which rule was applied.

Set an error budget for every group. Application servers might have a 50 millisecond limit, access network devices 250 milliseconds, and standalone equipment that polls infrequently 1 second. Do not copy these numbers. Network delay, equipment capability, and investigation requirements determine them. What matters is a written boundary beyond which a log is marked temporarily unreliable.

Document the correction method as well. Slewing changes the system clock rate until the error disappears. A step moves the clock immediately and can repeat a time range or jump forward. A practical policy usually permits a step early in boot, before applications start working, and allows only gradual correction afterward. An exception needs an owner: sometimes a large error is more dangerous than a step, but that decision cannot remain an unknown default.

Monotonic clocks solve another problem. An application can measure operation duration with a monotonic counter that does not jump when wall time changes. Cross-system correlation still depends on shared UTC time. A monotonic clock cannot replace NTP because counters on different machines do not share a starting point.

A dependable design begins with several sources

Internal clients need at least two reachable time nodes, and those nodes need independent external references. RFC 8633 goes further and recommends at least four independent and diverse sources for operators who care about accuracy. The reason is not simple averaging. NTP algorithms compare candidates, reject clocks that are clearly wrong, and select an agreeing group. With one source, a client can detect lost connectivity but cannot tell whether the reachable server is lying.

A practical enterprise design looks like this:

1. Two or more internal time nodes serve servers and network equipment through stable addresses.
2. Each internal node sees several approved upstream sources over independent paths where possible.
3. Clients receive the same internal address set through configuration management instead of a random public pool.
4. Firewalls allow UDP 123 only between defined clients, internal nodes, and approved external addresses.
5. Monitoring polls every node separately and compares them with one another.

Do not make one server the sole corporate reference just because it has a low stratum. In RFC 5905, stratum describes position in the hierarchy: a source with a reference clock has stratum 1, the next level gets 2, and so on through 15, while 16 means unsynchronized. The number is not a quality certificate. A stratum 2 server across an unstable path may deliver worse time than a carefully run stratum 3 server near the client.

Local intermediate nodes are useful in branches with unreliable WAN links. They can continue steering clocks gradually from the measured oscillator drift during a temporary loss of upstream sources. Holdover does not create accurate time from nothing: uncertainty grows, and monitoring must show the age of the last good update. After a long isolation, the local clock cannot silently be declared trustworthy.

An isolated network needs its own authoritative source, such as a satellite time receiver with PPS, or an approved path to the corporate service. The choice depends on required accuracy and the threat model. A command that declares a router's local clock authoritative only distributes its error across the network. This mode can serve as documented holdover for applications that need mutual agreement more than UTC, but the logs must carry that qualification.

Do not mix sources on regular UTC with sources that smear a leap second across an interval. RFC 8633 explicitly warns that a client must not use such a mixture. The servers will legitimately disagree during the smear, and the response contains no standardized marker for the policy. Choose one policy inside the managed environment and document it.

Initial setting and gradual correction need different rules

An NTP client should fix a gross startup error quickly and steer the clock carefully after that. The chrony configuration below is a starting point for two internal servers, but replace the addresses, thresholds, and allowed networks with values from your design:

server 10.20.0.10 iburst
server 10.20.0.11 iburst
makestep 1.0 3
rtcsync
driftfile /var/lib/chrony/drift

The iburst option accelerates initial measurements after a source becomes available. makestep 1.0 3 lets chronyd step the clock when the offset exceeds one second, but only during the first three updates. After that, the service adjusts frequency gradually. On supported systems, rtcsync helps the kernel periodically copy system time to the hardware clock, while the driftfile preserves an oscillator drift estimate between starts.

I disagree with advice to run chronyc makestep whenever any warning appears. It is popular because it instantly makes command output look good. On a running database, queueing system, or metrics collector, a backward step can repeat a wall-clock interval, while a forward step can expire timers early. First determine the error size, application state, and the cause of lost synchronization. A manual step belongs in a controlled emergency procedure that records time before and after.

After boot, dependent services must not treat a running process as proof that time is ready. With chrony, you can add this check:

chronyc waitsync 60 0.010

The command makes up to 60 checks at intervals of about ten seconds and succeeds when chronyd is synchronized and the remaining correction is no more than 10 milliseconds. The threshold must match the error budget of the specific service. If an application can start with less accurate time, do not block boot forever. Record a degraded state and prohibit operations that require trustworthy time.

Keep one clear control loop on a virtual machine. When the guest NTP service and the hypervisor independently move the same clock, an operator sees unexplained oscillation and cannot identify who made a correction. Hypervisor synchronization can help during resume or migration, but its role must be coordinated with the guest service and tested on the specific platform. Two simultaneous hidden corrections are worse than either deliberate choice.

A container usually uses its host kernel's clock, so a separate NTP daemon in the container does not repair the source problem. Check the node that runs the container, and put node and instance identifiers in the log. That ties a record to the clock that actually supplied its timestamp.

A switch should be a client, not an accidental reference

Point network equipment at the same internal time nodes used by servers. Analysts then compare logs against one time scale, and the network does not depend on public DNS or internet access. Configure two or more addresses, set the NTP source interface where routing and filters depend on it, and keep UTC as the system zone if the platform supports it.

Syntax differs between vendors. On equipment with Cisco IOS style commands, a minimal fragment may look like this:

clock timezone UTC 0 0
ntp server 10.20.0.10 prefer
ntp server 10.20.0.11

Do not copy this block blindly to another operating system. Check the manual for the exact firmware version. The meanings and syntax of server, peer, source-interface, VRF, and authentication parameters vary. A configuration that accepts commands without an error but sends requests from the wrong VRF or from an address denied by an ACL is particularly dangerous.

The show ntp associations command on Cisco IOS displays associations, but a row does not prove synchronization. An asterisk marks the selected system peer, reach keeps an eight-bit response history in octal, and 377 means the last eight polls received replies. A zero reach means there were no successful replies in the window. A large offset with 377 points away from simple packet blocking and toward source quality, path behavior, or the local clock.

address         ref clock       st  when  poll  reach  delay  offset  disp
*10.20.0.10     192.0.2.10       2    34    64    377  1.82    0.41  2.10
+10.20.0.11     192.0.2.20       2    29    64    377  2.04   -0.18  2.34

Read the whole output. st shows the hierarchy level, delay estimates the path, offset shows the difference relative to the peer, and disp reflects estimated uncertainty. Field names and units can change between platforms, so a metric collector cannot rely on one regular expression for every device.

An SNTP client in an inexpensive switch may be simpler than full NTP and may not perform sophisticated source selection. That is not a reason to exclude the device from the policy. Give it several supported servers, reduce network uncertainty, and verify the actual result externally. If the device logs only whole seconds, include that boundary in the investigation model.

Time-source authentication reduces the risk of spoofing, but it does not prove that the clock itself is correct. RFC 5905 makes this distinction: a trusted party can still have a failed receiver or bad configuration. Use a modern mechanism supported by the platform, protect secrets, and restrict the network, while retaining several independently observed sources.

Verification must measure state, not process existence

A working check answers five questions: whether a source is selected, whether enough candidates are available, what the current offset is, how large the uncertainty is, and when the clock last stepped. systemctl is-active chronyd answers only the process question. The service can be running, see zero sources, and methodically free-run the machine from an old driftfile.

On Linux with chrony, begin with two commands:

chronyc -n tracking
chronyc -n sources -v

An abbreviated healthy tracking output looks like this:

Reference ID    : 10.20.0.10
Stratum         : 3
Ref time (UTC)  : Mon Jul 27 10:42:18 2026
System time     : 0.000004321 seconds slow of NTP time
Last offset     : -0.000006102 seconds
RMS offset      : 0.000021443 seconds
Leap status     : Normal

In chrony, System time means the remaining correction between the system clock and NTP's estimate, not merely the last measured offset. Leap status: Normal confirms a normal protocol state, but an acceptance test must still check the numerical limit. Reference ID shows the selected source, yet monitoring also needs to see the spare candidates.

In sources -v, ^* identifies the selected server, ^+ a usable additional source, ^- a source not selected by the algorithm, and ^? a source without enough measurements. One ^* with three permanently unreachable addresses gives accurate time now but no resilience at the next failure. After a configuration change, wait through several polling cycles and confirm that candidates agree within your budget.

On switches, use the pair of status and association commands provided by the vendor. For Cisco IOS, these are usually show ntp status and show ntp associations detail. The first reports whether the system clock is synchronized and at which stratum. The second explains candidate selection and rejection. Save the full output with a timestamp from the checking node, not just a screenshot of a green management indicator.

Test the path through an intentional but safe failure. On a test client, temporarily remove one internal server from configuration or block it with a precise rule, then confirm that the client selects another source without leaving tolerance. Do not run the experiment on both time nodes at once, and do not start with a production switch that lacks backup access. After restoring the source, confirm that it becomes a candidate instead of remaining rejected because of a large error.

Comparing date on two terminals is useful only for rough diagnosis. A person cannot press Enter simultaneously, output is rounded, and remote console latency is unknown. Use protocol metrics and an independent query. Visually matching seconds do not prove a millisecond budget.

Acceptance of a new site should cover the full time-service lifecycle, not one quiet moment. Reboot a test client with deliberately wrong hardware time and record when it gets its first source, makes a step, and enters tolerance. Then restart only the NTP service while applications keep running. There should be no unexpected step outside the startup window. Separately test loss of one source, recovery after a WAN interruption, and behavior when DNS is unavailable if names remain in configuration.

An acceptance result needs numbers and raw output. The phrase "time synchronizes" cannot be reproduced. Record initial offset, time to enter tolerance, maximum offset during failover, number of available candidates, and the configuration version. For each failure, state which limit was violated. This protocol helps distinguish a firmware regression from a long-standing model limitation.

Also verify that logs actually use the corrected system clock. Some applications cache their time zone until restart, and some network devices have separate settings for system time and syslog timestamps. Generate a controlled event, such as a test account login or a state change on an unused port, and compare the original device record, receiver record, and NTP metric. The difference between event and receipt time should be explained by measured delay, not an unknown constant adjustment.

Do not discard failed test results after a repair. They reveal the error's shape and provide material for alerts. If blocking the second source left status green while the candidate count fell, that transition should become a separate monitoring condition. Acceptance ends when the reproducible defect is visible automatically, not while one engineer still remembers where to look.

Time needs service-level monitoring

Monitoring must retain a time series, or a brief disruption will disappear before the investigation begins. Collect offset, usable source count, current source, stratum, root delay or its equivalent, dispersion, reachability, frequency correction, age of the last update, and clock-step events. For equipment that exposes fewer fields, collect what is available and add an external comparison.

One offset threshold is insufficient. Create separate alerts for these states:

• no selected source;
• usable source count below the accepted minimum;
• offset exceeds the budget for several consecutive polls;
• uncertainty or update age grows while offset is still small;
• the service steps the clock after the startup window closes.

An alert delay keeps one lost UDP packet from waking the on-call engineer. Averaging must not hide a sudden 30-second step, however. Keep the maximum absolute offset in each interval, a source-change counter, and a separate correction event. The mean of -30 and +30 is zero and says nothing useful about the logs.

Check internal nodes from different network zones. A monitor beside the NTP server will not notice that a branch ACL blocks replies or that return routing sends them elsewhere. Client-side metrics matter more than server availability: a server can answer its local monitor while remaining unreachable to half the network.

Synchronization status should enter the logging context. You do not have to append offset to every line if that inflates the stream. A periodic event with the current source, offset, uncertainty, and configuration identifier is enough. During an investigation, it then shows which clock served a device in a given interval even if the monitoring panel has already discarded detailed metrics.

Compare configuration with its approved baseline periodically. Server address, VRF, ACL, zone, polling interval, and local-reference mode change during upgrades and emergency work. State checks catch the effects, while configuration checks explain the cause. Both are necessary because a correct file on a broken network and a working network with accidental configuration require different repairs.

An alert needs an explicit owner. The network team owns UDP availability, routing, and switch settings, the platform team owns internal servers and client policy, and application owners define the permitted budget. This does not require three separate alert systems. It requires an escalation path that prevents a large-offset signal from moving between queues for a week as "not our problem."

A synthetic check from every major segment also controls the common time scale. The observer queries both internal nodes, calculates their difference, and compares the replies with a separate approved reference. If both corporate servers drift together because they share a bad upstream source, their mutual comparison shows zero while the external control detects the common shift. The control reference must not be the only source used by the servers under test.

Time metrics should be retained for as long as the logs that may enter an investigation, although detail can be reduced over time. A single hourly average is not enough. Preserve extremes, source changes, synchronization state, and steps without aggregation. Otherwise, a month later, only a smooth line remains, with the one-minute error at the suspicious operation removed.

Typical failures leave recognizable traces

A complete loss of UDP 123 usually produces reach 0, no selected source, and an increasing update age. Check direction, source address, VRF, and return path rather than testing only the port. NTP uses a request and response. An outbound rule without returning traffic looks like a dead server to the client.

Partial filtering is more deceptive. One internal address works and the other does not, so status stays green for months. When the only reachable server enters maintenance, the whole segment loses time. An alert on usable source count and a planned failover test find this defect, while a selected-source check does not.

After a reboot, a device without a healthy hardware clock writes several messages with an old date, receives NTP, and jumps to the present. If the collector sorts only by source time, the boot cause lands far away from its continuation. Preserve receipt time and a message sequence number if the device provides one. During analysis, treat the synchronization point as the boundary between two clock modes.

Gradual drift with good reach often points to an overloaded host, an unstable virtual environment, conflicting correction mechanisms, or a highly variable network path. Compare Last offset, RMS offset, frequency correction, and source changes. A single current value may look normal immediately after the service repairs a long period of bad time.

Another recognizable case begins when an administrator manually sets time on a switch but does not repair NTP. The log shows a step, then the service slowly returns the clock to its source or makes a second step. Record the manual action in the change log and restrict the command by role where the platform supports it. Manual setting repairs the display, not the time service.

A bad local reference distributes failure with unusual neatness. All clients agree, dashboards are green, and offset is small, but the entire network differs from UTC. Compare internal nodes with independent external references, and do not close the alerting loop around the same hierarchy. Agreement within a group and correctness relative to UTC are separate tests.

Mixing local time and UTC looks like a constant shift by a whole number of hours, while an NTP error is usually smaller and changes gradually. Do not repair a time zone by moving the system clock. First inspect the raw timestamp, its numerical offset, and collector display settings. Otherwise, you may add a second error on top of the first.

Reconstruct an incident together with clock error

If the incident has already happened, do not edit the source logs or add an adjustment directly to the archive. Preserve originals, checksums, receipt data, and time configuration. Build a normalized timeline as a separate view in which every adjustment has a source and an interval of validity.

A practical reconstruction has five actions:

1. Capture NTP state on each important node, including selected source, offset, stratum, reach, and last correction time.
2. Find events with independent time: a packet capture on an observed interface, a central receiver record, a transaction from a trusted system, or a physical action with its own timestamp.
3. Divide the period at reboots, manual settings, time steps, and source changes.
4. Estimate an adjustment and uncertainty range for each interval without pretending that one point describes the entire drift.
5. Sort dependent events with that range and mark pairs whose order cannot be proved.

Suppose a web server sent a request at 10:00:00 by its clock, the firewall logged the connection at 10:00:31, and the receiver got both records around 10:00:34. A check shows that the server was 35 seconds slow and the firewall was 2 seconds fast. After normalization, the estimates become 10:00:35 and 10:00:29. That does not justify calling the network event the request's cause because measurement intervals and delivery delay may overlap. The honest finding is that the clocks were not accurate enough to prove the order of these two records.

With two control points for one node, you can estimate linear drift between them, but first check for a step. A step makes the correction function discontinuous, and a straight line creates times that never existed. Keep an uncertainty range even for gradual correction, because network offset is an estimate with delay and dispersion, not a perfect measurement of wall time at every microsecond.

After the analysis, repair the lack of evidence as well as the broken NTP address. Add metrics, correction events, a configuration baseline, and a failover test. For a new server room or network upgrade, GSE.kz can include time architecture and collection of its state in a systems integration project, drawing on the server infrastructure and round-the-clock technical support described by the company. The organization must still approve specific thresholds and sources because its applications, network, and threat model determine them.

A log with incorrect time cannot be made more accurate after the fact than the measurements that survived. You can stop producing new ambiguous records: assign an owner for the time service, test backup sources under failure, and make a budget violation visible before the next incident.