diff --git a/docs/management/hosts-pools.md b/docs/management/hosts-pools.md index 24d8011a..2f83820a 100644 --- a/docs/management/hosts-pools.md +++ b/docs/management/hosts-pools.md @@ -192,6 +192,8 @@ Lost the root password? See the [reset procedure](../troubleshooting/common-prob Correct and consistent clocks across the pool matter more than on ordinary servers: XAPI coordination, live migration, HA heartbeats, logs correlation and Windows guests (which get their initial time from the host) all rely on it. NTP servers are normally configured at [installation time](../installation/install-xcp-ng.md). +See [Time synchronization](time-synchronization.md) for what a wrong clock breaks, how to check whether `chronyd` has any time sources, and how to correct a clock that is already wrong. + To change them afterwards on XCP-ng 8.3, edit `/etc/chrony.conf`, then: {` diff --git a/docs/management/time-synchronization.md b/docs/management/time-synchronization.md new file mode 100644 index 00000000..2081304d --- /dev/null +++ b/docs/management/time-synchronization.md @@ -0,0 +1,167 @@ +--- +sidebar_position: 8 +--- + +# Time synchronization + +XCP-ng keeps the dom0 clock synchronized using `chronyd`. A host whose date is wrong keeps +working normally until something needs to trust that date. When that happens, the resulting +failures rarely point to the clock as the cause. This page explains why the clock matters, how +to configure NTP, and how to check and correct it. + +## ⏰ Why the clock must be correct {#why-the-clock-must-be-correct} + +- **Access to the repositories.** Updates are fetched over HTTPS, and TLS validates the + server's certificate against the host's own date. If the host's clock is set too far in the + past, the TLS handshake fails because, from the host's perspective, the mirror's certificate + is not yet valid. The error `yum` prints in that situation does not mention time. +- **Pool membership.** A host joining a pool must have its clock synchronized with the pool + master. See [Pool Requirements](../../installation/requirements#pool-requirements). +- **Certificates.** Certificates are only valid inside a validity window. A host whose date + falls outside that window will reject, or be rejected by, connections that would otherwise + succeed. +- **Logs and scheduled operations.** Correlating events across the hosts of a pool, and + anything that runs on a schedule such as backups, depend on the hosts agreeing on what + time it is. + +## 🕰️ How XCP-ng keeps time {#how-xcp-ng-keeps-time} + +The time daemon on dom0 is `chronyd`, configured through `/etc/chrony.conf`. + +The default configuration enables `rtcsync`, allowing chronyd to keep the hardware clock +synchronized with the system clock automatically. There is no need to manually update the +hardware clock after correcting the system time, and the correction persists across reboots. + +## 🔧 Configuring the time sources {#configuring-the-time-sources} + +### During installation + +At step 11, the installer prompts you to enter the time, select a time zone and either +configure NTP servers or set the time manually. Always configure at least one NTP server. See +[Install XCP-ng](../../installation/install-xcp-ng). + +:::warning +If you set the time manually at this step, the host can end up with `chronyd` running but no +time source configured at all. It will keep whatever date it was installed with. See +[Checking the time sources](#checking-the-time-sources) below. +::: + +### During an automated installation + +The installation answer file accepts one or more NTP servers through the `` +element. See [Answer file](../../appendix/answerfile). + +### On an installed host + +You can configure NTP from `xsconsole`, or by editing `/etc/chrony.conf` directly and +restarting the service. + +## 🔍 Checking the time sources {#checking-the-time-sources} + +Check that the service is running, and that it actually has something to synchronize with: + +```bash +systemctl status chronyd +chronyc sources +``` + +Both matter, and the second is the one that gets skipped. `chronyd` can be running and +enabled while having **no time sources configured at all**: + +``` +# chronyc sources +210 Number of sources = 0 +``` + +Then, the service behaves exactly as configured, and does nothing. Every other check an +operator would normally run looks healthy, which is what makes this one worth running early +rather than late. + +When sources are configured, `chronyc sources` lists them. In the `MS` column, the second +character is the state of that source: + +- `*` marks the one currently being used. +- `+` marks another acceptable source being combined with it. +- `?` means the source is unreachable. + +At least one source should reach the `*` state. + +To see the offset chrony believes it has, and whether it has settled: + +```bash +chronyc tracking +``` + +## 🛠️ Correcting a wrong clock {#correcting-a-wrong-clock} + +Start with `chronyc sources`, because the answer decides which of the two paths below +applies. + +### If sources are listed + +Correct the clock and check the result: + +```bash +chronyc makestep +date +``` + +`chronyc makestep` is what does the work here. By default, chrony corrects time offsets by +gradually slewing the system clock, which never converges for an offset of months or years. + +### If `Number of sources = 0` + +Do not start with `makestep`. It will report success and move nothing, because chrony has no +measured offset to step to. + +1. Add time sources to `/etc/chrony.conf`: + + ``` + server 0.pool.ntp.org iburst + server 1.pool.ntp.org iburst + server 2.pool.ntp.org iburst + server 3.pool.ntp.org iburst + ``` + +2. Restart the service and confirm that a source has become reachable, as described in + [Checking the time sources](#checking-the-time-sources): + + ```bash + systemctl restart chronyd + chronyc sources + ``` + +3. Once a source is reachable, correct the clock: + + ```bash + chronyc makestep + date + ``` + +:::note +Correcting the date does not invalidate the host's own certificate. XAPI issues it with a +ten-year validity, so a host installed with a wrong date still holds a certificate that +covers the corrected date. `xe host-refresh-server-certificate` is not needed for this. +::: + +:::tip +If `date` is wrong again after every power cycle, the motherboard's RTC battery is probably +dead and should be replaced. `rtcsync` can only keep the hardware clock in step while the +host is running; it cannot help a clock that loses its value when the power goes. +::: + +## 🌐 Networks without internet access {#networks-without-internet-access} + +Hosts without Internet access cannot reach the public NTP pool. The result is the same as +having no sources at all: `chronyc sources` lists servers that never leave the `?` state. + +On such networks, point `/etc/chrony.conf` to a time source that hosts can actually reach, +such as an appliance on the same network, or a local server that is itself synchronized and +acts as the reference time for the network. + +## 🎱 Pools {#pools} + +Keep every host in a pool synchronized, ideally against the same sources. A host whose clock +differs from the pool master is not just inconvenient: clock synchronization is one of +the requirements for joining a pool, along with the other requirements listed in +[Pool Requirements](../../installation/requirements#pool-requirements). diff --git a/docs/management/updates.md b/docs/management/updates.md index cda86f50..115bdcb5 100644 --- a/docs/management/updates.md +++ b/docs/management/updates.md @@ -28,6 +28,10 @@ If your version is lower than `8.3`, it will not receive updates anymore. To kee Your dom0 system must either have access to the internet, or to a local mirror. In the second case, make sure to update the `baseurl` values in `/etc/yum.repos.d/xcp-ng.repo` to make them point at the local mirror, and keep the mirror regularly synced. +:::warning +The host clock must also be correct. Repository access uses HTTPS, and a host whose date is wrong cannot validate the mirrors' TLS certificates. The error `yum` reports in that case is misleading, because it names a redirect instead of the certificate: see [yum fails with "HTTPS Error 301 - Moved Permanently"](../../troubleshooting/common-problems#yum-fails-with-https-error-301---moved-permanently). +::: + #### Proxy configuration If your hosts require a proxy to access the repositories, you have several options depending on your needs: diff --git a/docs/troubleshooting/common-problems.md b/docs/troubleshooting/common-problems.md index 094b4979..1c4e8133 100644 --- a/docs/troubleshooting/common-problems.md +++ b/docs/troubleshooting/common-problems.md @@ -119,6 +119,60 @@ chmod +x /etc/rc.d/rc.local --- +## 📦 yum fails with "HTTPS Error 301 - Moved Permanently" {#yum-fails-with-https-error-301---moved-permanently} + +Every `yum` command on dom0 fails, most often on a freshly installed host: + +``` +Loaded plugins: fastestmirror + * xcp-ng-base: mirrors.xcp-ng.org +http://mirrors.xcp-ng.org/8/8.3/base/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 301 - Moved Permanently +Trying other mirror. +... +failure: repodata/repomd.xml from xcp-ng-base: [Errno 256] No more mirrors to try. +``` + +### Cause + +In almost every reported case, **the system clock on dom0 is wrong**, usually set far in the past. + +`mirrors.xcp-ng.org` redirects HTTP to HTTPS, then to a mirror close to you. That redirect is normal and is not the problem. But if the host's date is earlier than the start of the mirror's TLS certificate validity period, the HTTPS connection cannot be established: from the host's point of view, the certificate is not yet valid. + +The message names the redirect rather than the certificate because of the way `yum` reports errors. When a transfer fails *after* a redirect, it prints the redirect's status code and discards the underlying error. The `301` is genuine, and it is the last thing that succeeded. + +To confirm the diagnosis before changing anything, you can temporarily change the repository URL in `/etc/yum.repos.d/xcp-ng.repo` from `http://` to `https://`. This does not fix the download, but it removes the redirect, so `yum` reports the real error: + +``` +https://mirrors.xcp-ng.org/8/8.3/base/x86_64/repodata/repomd.xml: [Errno 14] curl#60 - "SSL certificate problem: certificate is not yet valid" +``` + +### Solution + +Check the date on dom0: + +{` +date +`} + +If it is wrong, fix the time synchronization: see [Time synchronization](../management/time-synchronization.md#correcting-a-wrong-clock) for how to check the chrony sources and step the clock. On a host whose date was set manually at installation, `chronyd` is often running with no time source at all, and correcting the clock then takes an extra step. + +Once the clock is correct: + +{` +yum clean all +yum check-update +`} + +:::warning +Revert any change you made to `/etc/yum.repos.d/xcp-ng.repo` while investigating. With a correct clock, the default `http://mirrors.xcp-ng.org` URL works. Pinning a single mirror by hand takes the host out of the automatic mirror selection, and out of failover if that mirror becomes unavailable. See [Mirrors](../project/mirrors.md). +::: + +:::note +Correcting the date does not invalidate the host's own certificate. XAPI issues it with a ten-year validity, so a host installed with a wrong date still holds a certificate that covers the corrected date. `xe host-refresh-server-certificate` is not needed here. +::: + +--- + ## 🐌 Async Tasks/Commands Hang or Execute Extremely Slowly {#async-taskscommands-hang-or-execute-extremely-slowly} ### Cause