Home

Heya.

I’m still pretty new to the homelab scene, so the more detail you can add the better. I’d like to add some sort of log aggregation tool, something like Elastic, where I can go to look at logs from any of my systems that aren’t working, or just make sure I don’t miss any errors.

Pretty much everything I run is set up as a Proxmox LXC from Proxmox helper scripts, which most of the time means it’s running as a systemctl service. Sometimes they run in Alpine instead, and a few of my apps also run in Docker.

What’s a good app to aggregate logs from those sources? I’ve heard of Prometheus, Grafana and Loki but not sure if they do what I’m after, they seem pretty overwhelming and more focused on metrics, whereas I want to be able to search for and view logs. I’d appreciate if you also mention the basic steps to send the logs from each container to said app.

Greylog is a syslog aggregator that might do what you’re looking for.

I use rsyslog directly into postgres. Grafana nativly can read from that. Super lightweight, super flexible. (I use 1 cpu and 2G mem and it works better than graylog which needed 4 cpu’s and 16 G ram)

Do you have pointers on how to get that up an going?

this is the main pointer

On every device (but the central syslogserver or you create a loop that fills the drive in mere seconds)

/etc/rsyslog.d/99-centralsyslog.conf

$PreserveFQDN on
*.*  @192.168.1.66

then on the central syslog server 192.168.1.66

/etc/rsyslog.d/01-syslog_receiver.conf

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

and also

/etc/rsyslog.d/20-save2postgresql.conf

# Load the PostgreSQL output module
module(load="ompgsql")

# Template for inserting logs
template(name="pgsql-template" option.sql="on" type="string" string="INSERT INTO system_events (hostname, facility, priority, tag, message) VALUES ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, '%syslogtag%', '%msg%' )") 

# Send logs to PostgreSQL
*.emerg    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.panic    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.alert    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.crit     :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.error    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.err      :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.warning  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.warn     :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.notice  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.info  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.debug  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template

make sure you install postgres, the rsyslog-psql module and create the databases. Grafana can run on the same or any other server.