I’m pretty new to selfhosting, but one thing that I know to take seriously is log collection. Since there are a lot of different type of logs (kernel log, application logs, etc) and logs come in many different formats (binary, json, strings) - it’s no easy task to collect them centrally and look through them whenever neccessarly.
I’ve looked at grafana and tried the agent briefly, but it wasn’t as easy as I thought (and it might be a too big tool for my needs). So I thought to ask the linuxlemmy community to get some inspiration.
In terms if logging: sys journal does the job for me …
I do run prometheus + grafana for some services but that is mostly for some fancy looking graphics nothing really usefull.
I would recomend you to monitor updates of you apps so you are well informed when und what to update (i just have subsribed to all the diffrent release git rss feeds)
You can use rsyslog and rsyslogd for OS log. For app use flat file, collect using ansible. 😂
Well I’m quite interested in msg stack like grafana, but haven’t tried it.
Do you push your logs regularly to a central storage, or do you just SSH into the machines regularly to look at the logs?
If it’s OS log,it’s pushing https://serverfault.com/questions/522341/how-do-i-setup-rsyslog-to-send-all-logs-to-multiple-remote-servers
If it’s laravel/apache, php, then use ansible to pull the log. Or using sentry as I remember. 😂
Honestly? I just ignore them. Something seems to be happening to them, as they’re not growing infinitely, but no idea why.
It was a bit complicated to set this up, but it took me almost no time at all.
My logs were actually filling up my disk! I edited journald conf and limited them to 500mb . disk is only 10gb for reference
You could try deploying an ELK stack and manage log collection using Elastic Agents and integrations, works pretty well for us and, as long as your log is supported by an integration, it’s dead simple to manage
