Add a time synchronization page

[gounthar]

Before submitting the pull request, you must agree with the following statements by checking both boxes with a 'x'.

• "I accept that my contribution is placed under the CC BY-SA 2.0 license [1]."
• "My contribution complies with the Developer Certificate of Origin [2]."

[1] https://creativecommons.org/licenses/by-sa/2.0/
[2] https://docs.xcp-ng.org/project/contributing/#developer-certificate-of-origin-dco

@stormi asked for this in #504:

Wrong dates can cause a lot of different issues. We must make sure we have a dedicated page or section somewhere in the docs that gives all details about NTP and why it must be set correctly, and this specific troubleshooting topic about yum would then link to it for details.

NTP was documented in three places, each in passing: the installer walkthrough tells you to always use an NTP server, the answer file reference lists <ntp-server>, and the pool requirements mention clock synchronization. Nothing said what to do when the clock is already wrong, or how to tell whether chrony is actually doing anything.

This gathers it into one page: why the date matters, how to set the sources at install time and afterwards, how to check them, and how to correct a host that has already drifted.

The part I had not seen documented anywhere is that chronyd can be running and enabled while having no time sources configured at all. systemctl status chronyd looks healthy, the network is fine, and the host simply keeps its installation date. The only check that reveals it is chronyc sources returning Number of sources = 0, which nobody runs until they already suspect the clock. So the page puts that check first, and splits the correction into two paths, because chronyc makestep reports success and moves nothing when there is no source to step to.

#504 will link here for detail once this lands, and sheds its NTP how-to in the process. I have deliberately not linked from here to that troubleshooting section yet, since it does not exist on master and onBrokenLinks is set to throw.

[rzr]

Suggested change

	server 0.centos.pool.ntp.org iburst
	server 2.pool.ntp.org iburst

I think it's preferable to not use centos vendor zone. since XCP-ng is a different vendor.

I think it's acceptable for users to use a public pool, as suggested at:

https://community.ntppool.org/t/oracle-linux-vendor-zone-request-for-a-clear-decision-and-a-usable-vendor-zone-process/4503/5?u=rzr

FYI xcpng vendor zone is still pending (for months):

xcp-ng-rpms/chrony#1 (comment)

[gounthar]

Taken, and thanks for the links. The vendor zone was not ours to use and I had not thought about it at all. Switched to the generic pool.

Worth someone chasing xcp-ng-rpms/chrony#1 so there is an XCP-ng zone to point at eventually, but this page should not wait on that.

[stormi]

As I regularly repeat, conventional commits is not a convention to apply to this repository.

[stormi]

A reference to a numbered step in another page won't age well.

[gounthar]

I dropped the number. It now reads that the installer prompts for the time, the time zone, and either NTP servers or a manual time, and links to the install page without pointing to a position in it.

[stormi]

I think this would be worth expanding on. Same time when joining is necessary, but we also want it not to drift later, as this can cause subtle synchronization issues, and I'm not sure it's entirely covered by the next bullet point about certificates (which are used by hosts to authenticate to each other, notably).

[gounthar]

I expanded it. The bullet now covers staying synchronized as well as joining, and says why the second is the harder one to catch: a failed join tells you what is wrong; drift afterwards does not. It surfaces as hosts disagreeing about when things happened, each convinced by its own clock.

I kept it separate from the certificates bullet, since that is a different mechanism, and folded the old Pools section into this bullet as well, per your comment further down.

[stormi]

"It will keep whatever date it was installed with." might need rewording, as it may let some think that the date will not increment every day.

[thomas-dkmt]

Also, "it" seems ambiguous to me, in that sentence.

[gounthar]

I reworded it, and "it" is gone. It now says the clock still ticks, but nothing ever checks it, so it drifts from the date you typed, and if that date was wrong, it stays wrong.

Testing another of your comments turned up a reason to say more here: this is the one case that does not fix itself at boot. A host with sources gets stepped back into line within seconds of chronyd starting, so no sources at all is what makes this state persist. The warning says that now.

[stormi]

⚠️ We discourage manually modifying configuration files that are maintained by XAPI.

This should be of last resort if no other way works, such as xsconsole, xe or Xen Orchestra (does it support setting up NTP?).

[gounthar]

I changed it. xsconsole is the documented route now, and editing the file by hand is marked as a last resort in both places it appeared.

Regarding your question, I could be wrong, but I think XO does not configure NTP on hosts. xoa network ntp sets the time servers of the XOA appliance, which is a Debian VM keeping its own clock through systemd-timesyncd, and the NTP field in the deployment form does the same for the appliance being deployed:

@xen-orchestra/lite/src/pages/xoa-deploy.vue:447
  xapi.call('VM.add_to_xenstore_data', [vmRef.value, 'vm-data/ntp', ntp.value])

I grepped a clone of the tree rather than trusting a code search. Every hit outside docs/ is that one locale string repeated per language plus the form using it, and chrony appears nowhere. 🤷
The page now states the distinction, because appliance NTP and host NTP are easy to conflate.

[stormi]

Suggestion: I'd replace the server names with placeholders, so that no one starts wondering why their results differ from the example.

[gounthar]

Done: ntp1.example.net through ntp4.example.net.

One tension worth naming: @rzr asked earlier for the real expected output, and this trades against that a little.
I kept every column value real and changed only the names, so the shape someone compares against is still a genuine listing. 🤷

I'm happy to put the real hostnames back if you would rather.

[stormi]

I'm not sure about this. Either the service is healthy and this is unneeded, or it's the service that needs to be fixed, isn't it? Manually setting the clock once doesn't look like a long term solution and may just postpone the actual resolution by weeks or months.

I think it could be mentioned as an emergency fix, but not as a solution. And to add steps to go towards an actual solution (re-run xsconsole? check chronyd logs?).

It can also be useful as a one time check: "is chronyc able to fetch the date?", but I think it should be included in a higher level resolution process.

Unless I'm mistaken, it's XAPI that manages chronyc (worth verifying though), so the resolution could be at several layers:

• configuration layer (no sources configured, user mistake initially)
• XAPI layer (xsconsole being a XAPI client)
• chronyd service
• network, firewall (I don't see the later mentioned, but I would be surprised if that isn't a common cause).

Btw, I'm pretty sure Vates' support team would have interesting input on this.

[gounthar]

Testing your other comment showed that a host with reachable sources does not usually have a wrong clock at all: stock config has makestep 1.0 3, and a 45-day offset was corrected within ten seconds of chronyd starting. So a clock that is still wrong is evidence that something upstream is broken, and stepping it by hand mostly hides that. 🤔

Restructured along the layers you listed, working outward:

1. Any sources at all, chronyc sources, with xsconsole to add them.
2. Sources reachable. Everything stuck at ? is configured but unanswered, so UDP 123 and the firewall get named here, along with the no-route-to-the-pool case.
3. Daemon health: systemctl status chronyd and journalctl -u chronyd.
4. Configuration changed underneath you, which rpm -V chrony will show.

Stepping the clock is its own short section now, described as a stopgap that buys time for the diagnosis rather than ending it, including the detail that it reports success while moving nothing when there are no sources.

One correction on XAPI managing chronyc, since you flagged it as worth verifying: /etc/chrony.conf is owned by the chrony package, and nothing under /opt/xensource writes it.
xsconsole is an XAPI client that edits the file and restarts the daemon, which comes to the same thing for a user, but it does mean rpm -V chrony reports its edits as local modifications.

On the support team, I have not asked them. Their ordering of these layers would be better than mine, since they see which one actually turns out to be the cause.

[stormi]

again, editing the configuration file directly should be last resort.

[gounthar]

Same change as above, applied here too. Both places now put xsconsole first and mark editing the file by hand as a last resort.

[stormi]

What consequences does it have? Doesn't chronyd fix the date shortly after initial startup? Does it cause problems at startup that do not self-heal?

[gounthar]

Yes, chronyd fixes it quickly, as long as it has a source.

The stock /etc/chrony.conf carries makestep 1.0 3, so for the first three updates after the service starts, it jumps the clock instead of easing it into place. I stopped chronyd, put the clock 45 days into the past, and started it again:

skew    2026-07-03T12:17:41Z   chronyd started
t+05s   2026-07-03  still wrong
t+10s   2026-08-17T12:17:46Z   corrected

Under ten seconds for 45 days. The page claimed chrony only slews and so never converges for offsets of months or years, which is true once those three updates are spent and wrong at startup. Fixed, and it changes the shape of the page: a host with sources heals itself, so the manual correction is for the daemon that has been up a while, or the host that never had a source at all.

What does not self-heal is Number of sources = 0, since makestep needs something to measure against. That is now said where the manual-time-at-install warning is.

[stormi]

If date is wrong again after every power cycle

This part implies, to me, that we you get a wrong date again after every power cycle DESPITE ntp, and I don't see how that is possible thanks to makestep, unless there's a problem with the ntp configuration which must then be fixed and the battery then doesn't matter. Or does it?

It's still unclear to me what the problem is if you have a dead battery, outside the fact that the clock needs to be fixed at each boot (implying a part of the boot process with wrong log times and maybe worse consequences that I can't foresee).

[stormi]

"a host installed with a wrong date still holds a certificate that covers the corrected date"

Can we affirm that? It's likely, not guaranteed.

Also, this note brings up a topic that many users wouldn't have thought of initially, only to dismiss it directly. I'd rather 1. tell them that they're probably fine but 2. also tell them how to verify and 3. how to fix it if really necessary (if we're able to provide the appropriate steps, validated).

[gounthar]

I read one off an 8.3 host:

notBefore=Jun  2 15:17:54 2026 GMT
notAfter=May 30 15:17:54 2036 GMT

Ten years wide, anchored on the clock at the moment it was generated. A host that installs itself believing the year is 2010 gets a certificate valid from 2010 to 2020, so correcting the clock to now leaves it expired. Set the clock far forward at install, and you get the mirror image: not valid yet once you correct backwards. My note leaned on the width, and the anchor is what actually decides it.

Rewritten along your three lines. Most corrections are fine, and it says so. To check:

openssl x509 -in /etc/xensource/xapi-ssl.pem -noout -dates
date -u

To fix, if the date falls outside that window, xe host-refresh-server-certificate host=<host>, with xe host-emergency-reset-server-certificate for a host already being rejected and therefore not reachable the usual way. Both were checked against xe help on the host.

[gounthar]

Correction to my own reply above, and a better answer than the one I gave.

I reproduced it on a lab host, and the result contradicted part of what I wrote. On an 8.3 host, chronyd stopped and the clock moved to 2016:

xe host-refresh-server-certificate host=<uuid>

/etc/xensource/xapi-pool-tls.pem
  notBefore=Aug 17 10:00:00 2016 GMT
  notAfter=Aug 15 10:00:00 2026 GMT

Ten years, anchored on the fake clock. Restoring the real date left that certificate expired by two days.

What I had wrong: xapi-ssl.pem never moved during any of this. host-refresh-server-certificate renews the internal pool certificate, and xe help says so plainly once read, "Refresh internal server certificate of host", against "Deletes the current TLS server certificate in the host" for host-reset-server-certificate. So my note told the reader to check the dates on one certificate and then run the command that renews the other. It would have reported success and changed nothing, which is the worst shape for a troubleshooting instruction.

Fixed in c848a9b. The note now lists both certificates with the command that regenerates each, tells the reader to check both windows, and warns against exactly the substitution I had just made.

The host is back to a 2026 to 2036 pool certificate and healthy. I will open an issue on xcp-ng/xcp for the underlying behaviour, since a certificate anchored to a wrong install-time clock is a product question rather than a documentation one, and link it here.

[stormi]

I'm not sure this section brings much value besides what's already described in "Why the clock must be correct", at least in its current state.

[gounthar]

Agreed, it was saying the same thing twice. Folded into the Pool membership bullet in "Why the clock must be correct" and dropped the section.

The merge also gave that bullet somewhere to make your point from the other comment, that we care about drift after joining and not only about the clock at join time. Nothing referenced the #pools anchor, so no links break.

[thomas-dkmt]

Depending on whether or not this other PR gets merged, we might want to remove the emojis in the section headings.

[gounthar]

Removed in c2b19bb. Six headings, text, and anchors untouched, so no links break.

I went with removing rather than waiting, because the choice looks settled: #522 and #524 are both closed, so #525 is the only live option left, and it is already approved. A new page arriving with six emoji headings would put back a small piece of exactly what that PR takes out, and whoever merges it would have to come back for this file.

If #525 ends up not merging, this page is then the odd one out, and I will put them back.

[stormi]

Suggested rewording. Straight out of my fingers, so feel free to improve the language correctness.

Suggested change

	- **Pool membership.** A host joining a pool must have its clock synchronized with the pool
	master, and must stay synchronized afterwards. A failed join tells you what is wrong. Drift
	that sets in later does not: it surfaces as hosts disagreeing about when things happened,
	each of them convinced by its own clock. Keep every host in a pool on the same time sources.
	See [Pool Requirements](../../installation/requirements#pool-requirements) for the other
	conditions a host has to meet before it can join.
	- **Hosts synchronization inside a pool.** Desynchronized time references between hosts
	in a pool can cause a variety of subtle issues for operations that require the hosts to agree
	on the current time. As such, it is necessary for hosts to synchronize time from a common
	source. Disagreeing clocks can also prevent adding a new host to an existing pool. The correct
	way to fix it is to ensure synchronization works, rather than a one time fix which would let
	them drift apart in the future.

[stormi]

Here let's add a word about Xen Orchestra, which, when used, must also have a synchronized clock?

[stormi]

Let's remove this log-like precision.

[stormi]

I didn't know verb "to slew", and think that "gradually adjusts" who be easier to understand by non-native english speakers like me.

[stormi]

I'm not so sure about this claim. As I understand it, a reboot fixes the clock if ntp is correctly configured, so:

• either the hosts haven't been rebooted in a very long time and they're likely behind on updates, and should probably be updated+rebooted.
• if rebooting doesn't fix the problem, it's probably elsewhere and not caused by the fact that chronyd can only gradually adjust time after first three times since last reboot, but by another, bigger cause (heavily drifting hardware clock, misconfigured ntp or failure to join it due to firewall...).

[stormi]

"This is also the one case that does not fix itself at boot, since makestep needs a source to measure against."

=> this could be removed, if the sentence just above would say that nothing adjusts it (instead of checks it).

[stormi]

I have doubts about this one. Can you verify?

1. If xsconsole changes chrony.conf, then of course it will differ from the base file provided by the package
2. more generally, rpm -V checks differences between a file as provided by the RPM package and what's on the system. For a configuration file, it's natural that there's a difference. It's there for this purpose. I don't know what rpm -V says here but I wouldn't be surprised if either it would always show it different (with a flag stating that it's configuration), or always ignore it. It also depends whether the file is declared %config or %config(noreplace) in the RPM spec file. Here, we have %config(noreplace) %{_sysconfdir}/chrony.conf which means that the RPM must not overwrite the file with future updates if the sysadmin (XAPI/xsconsole being considered an extension of the sysadmin for this purpose) has made any change to the file.

So, the output of rpm -V tells you nothing here, except maybe in some cases "it was never configured", but you would already have found out with earlier checks.

What you probably really want is to determine if something has modified the configuration outside the supported way (installer, xsconsole, xe/XAPI), and I'm not sure there's an answer to this one, but I haven't digged into the details.

[stormi]

It could also be that the ntp server they have chosen doesn't exist anymore

[stormi]

To understand this, one must have read #how-xcp-ng-keeps-time

[stormi]

Same comment about "slew" that earlier.

[stormi]

I think this should just be a short note explaining how to verify the certificate validity periods. How to renew certificates is a topic that doesn't belong here. There's an old draft on this whole topic, which stalled: #216

The certificate topic is a big one in itself, we should cover it independently.

[stormi]

using xsconsole is one way to do it, for an already installed host. But here we're not in the troubleshooting part of the document anymore, so the best course of action is to set it right at installation time.