Recently one of our app servers stopped serving, the server was running, but it had run out of disk space. The database wasn’t too large, and the Laravel apps log files weren’t massive.
The first stop then was to find any large files, to see what was clogging up the system.
find / -size +10M -ls
This revealed a 17Gb log file: /var/log/mysql/mysql-slow.log
It appeared that MySQL was taking it’s time with some queries.
We sync our database with various external APIs every day, these are inserted in bulk, so I don’t mind them taking over 0.1 second to run (the default slow-query log threshold set in our mysql config). What I do mind is having the server run out of space due to large log files.