Starting Galera Cluster node fails due to Systemd `TimeoutStartSec`

I recently had a Problem where adding a Galera Cluster node, after a failure, failed and left me with a broken and out of Sync node that was not able to recover by simply starting the Service via systemctl start mysql.service

The Problem was that the Node that fell out of the Cluster had to synchronize about 30GB of Data and Systemd killed the rsync SST Synchronisation after the standard start timeout of 90 seconds.

To resolve the issue I had to increase the TimeoutStartSec for the mysql.service.

To check the currently set start timeout run

systemctl show mysql.service -p TimeoutStartUSec

Which should return something like this:

TimeoutStartUSec=10min

To alter the TimeoutStartSec for the MySQL Service first run

sudo systemctl edit mysql.service

A Text editor will open, in which you overwrite the TimeoutStartSec option by adding a configuration like this:

[Service]
TimeoutStartSec=300s

For the change to take effect you need to run the command

sudo systemctl daemon-reload

After that you can, again, check the if your change was successful by running systemctl show mysql.service -p TimeoutStartUSec

Be aware that setting the TimeoutStartSec of a Service to a ridiculous high number, or even to infinity could result in it blocking the boot process.

After this you can start the MySQL Service as usual by running:

sudo systemctl start mysql.service

POSTSCRIPT

Recently I stumbled upon an issue with a Mariadb Installation that aliased mariadb/mysql with a difrent Service script/file, please be aware that your changes might not work as expected if you alter a aliased configuration/script/service.

Resources:
https://blog.neoflow.jp/oss/mariadb-galera-cluster/4894/
https://mariadb.com/kb/en/library/systemd/