I’m pretty new to selfhosting, but one thing that I know to take seriously is log collection. Since there are a lot of different type of logs (kernel log, application logs, etc) and logs come in many different formats (binary, json, strings) - it’s no easy task to collect them centrally and look through them whenever neccessarly.
I’ve looked at grafana and tried the agent briefly, but it wasn’t as easy as I thought (and it might be a too big tool for my needs). So I thought to ask the linuxlemmy community to get some inspiration.
Honestly? I just ignore them. Something seems to be happening to them, as they’re not growing infinitely, but no idea why.
It was a bit complicated to set this up, but it took me almost no time at all.
You can use rsyslog and rsyslogd for OS log. For app use flat file, collect using ansible. 😂
Well I’m quite interested in msg stack like grafana, but haven’t tried it.
Do you push your logs regularly to a central storage, or do you just SSH into the machines regularly to look at the logs?
If it’s OS log,it’s pushing https://serverfault.com/questions/522341/how-do-i-setup-rsyslog-to-send-all-logs-to-multiple-remote-servers
If it’s laravel/apache, php, then use ansible to pull the log. Or using sentry as I remember. 😂
Systemd does all that for you. Just set a limit in journal.conf
So does syslog; and really well. Logging is just another thing Lennart tried to take over from a working system, that he and Kay half-delivered, and wandered off. See also: nfsroot, fast boots, easy init config, nfs mounts in general, and cron.
I use the standard system syslog with logrotate every 7 days with 1 month of gzip archive.
However on production systems, I run a central rsyslog server which archives once a week and a year of archives. Considering ELK in the future but for simple retention syslog is fine.
Nothing. I always use the standard configuration from the services I’ve installed. If something doesn’t work I go into those logs and look what’s wrong.
