panic: freepages: failed to get all reachable pages

[remmen-io]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

I've setup a bare-metal (Inte NUC) k8s cluster with talos using 1 control-plane/worker node and 2 additional worker nodes.
After a week etcd is no longer responding due to this panic.
This happened already twice after a week of operation. I've checked the resources and there are plenty of free CPU/MEMORY/DISK available.

What did you expect to happen?

Having etcd up and running for longer than a week

How can we reproduce it (as minimally and precisely as possible)?

Wait for a week...

Anything else we need to know?

Talos support files
support.zip

Etcd version (please run commands below)

$ etcd --version
1.3.9


### Etcd configuration (command line flags or environment variables)

<details>

# paste your configuration here

</details>


### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

<details>

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[tjungblu]

$ etcd --version
1.3.9

Are you sure that's the right etcd version?

[remmen-io]

no.. sorry

It's v3.5.13 according to the release notes of talos 1.7.6

[jmhbnz]

Hi @remmen-io - Can you please provide more details on your environment, what storage are you using? Some metrics over time would be really useful. Thanks

[remmen-io]

Hi @jmhbnz,
It's a small lab cluster (low traffic) based on 3 Intel NUC. The control-plane/worker node (10.10.40.90) is a NUC7i5BNB while the two others are NUC5i (.91/.92). Each of them has 32Gb of memory and a 1TB NVMe disk.

disk

❯ talosctl --talosconfig=./clusterconfig/talosconfig -e 10.10.40.90 -n 10.10.40.90 disks 
NODE          DEV            MODEL                 SERIAL            TYPE   UUID   WWID                   MODALIAS   NAME   SIZE     BUS_PATH                                                   SUBSYSTEM          READ_ONLY   SYSTEM_DISK
10.10.40.90   /dev/nvme0n1   Samsung SSD 980 1TB   S649NU1W916629H   NVME   -      eui.002538d931a2c66d   -          -      1.0 TB   /pci0000:00/0000:00:1d.0/0000:3a:00.0/nvme/nvme0/nvme0n1   /sys/class/block  

memory

❯ talosctl --talosconfig=./clusterconfig/talosconfig -e 10.10.40.90 -n 10.10.40.90 memory   
NODE          TOTAL   USED   FREE    SHARED   BUFFERS   CACHE   AVAILABLE
10.10.40.90   31964   2907   23631   70       4         5420    28805

The cluster is based on the minimal immutable distro talos (https://www.talos.dev/) I first opened an issue there, but was redirected to here.

What type of metrics do you would like to see?

[jmhbnz]

On second read this issue is looking like a potential instance of etcd-io/bbolt#778.

@ahrtr FYI.

@remmen-io - Does Talos allow you to retrieve a copy of the database file, for example: etcd-io/bbolt#778 (comment)

[remmen-io]

Hi @jmhbnz,

I do have a copy of the “corrupted” DB file as from the instructions

[ahrtr]

It seems like a data corruption issue. Please feel free to reach out if you know how to reproduce it reliably.

Also FYI etcd-io/bbolt#789