ETCD Defragmentation - Cronjob vs Event-Driven Job

[ugur99]

Hi,

I'm looking for the best practices for defragmenting an etcd cluster. I've read the official documentation, but haven't seen any suggested methods to automate this process. I've seen some cronjobs or systemd.timer with systemd.service to defrag each instance one by one in a for loop or asynchronously respectively.

I thought that starting to defragmentation from other etcd instances and lastly defragmenting the leader one would be safer if we can assume the defragmentation may not be complete in election_timeout time. Also, it would be useful to change the leader with etcdctl move-leader to the defragmentation completed etcd node. Such a cronjob running on the k8s is one option.

What about instead of running such a defragmentation job as a cron, does it sound good to run it on an event-driven structure? When the defragmentation alert became active the job can be triggered.

I believe running a defragmentation job regularly could make much more sense rather than running on an event-driven format, because depending on the db size that you configured; defragmentation can take longer to complete; but yes, at the end of the day, it would not cause any problem if you start from the other etcd instances except the leader one to defrag.

I would welcome to hear any ideas/practices about it.

[serathius]

KISS - keep it stupid simple.

Cron job that avoid running when not needed based on some fragmentation threshold should work good enough.

[guettli]

There is now https://github.com/ahrtr/etcd-defrag which handles etcd defragmentation

[guettli]

I suggest to add an example script to the docs, and created a feature request: #15477