Sending journalctl logs to a centralized logging system

Introduction

So you have a Linux system and you want to export your logs to a centralized logging system. There is a good chance your Linux distribution is running systemd which comes with journaltctl.

journalctl is a command-line tool provided by systemd, which is the system and service manager for Linux operating systems. It is used to query and display messages from the systemd journal. It allows you to view logs from various components of the system, including the kernel, system services, and applications, all in one place. Think of it as a local centralized logging system.

Output structured logs as JSON

One useful property of journalctl is that it can output log messages as JSON which is generally the most common way to send and store structured log messages. Central Logging’s CLAgent takes advantage of this.

You can try this.

Log into your Linux host and do:

journalctl -b -o json

This will display all log messages since last boot and output them as JSON.

You could output this to a file like: journalctl -b -o json > /tmp/out.txt

An example log row pretty printed might look like:

{
    "_CAP_EFFECTIVE": "1ffffffffff",
    "_EXE": "/usr/bin/sudo",
    "__MONOTONIC_TIMESTAMP": "130890256435",
    "_AUDIT_SESSION": "921",
    "MESSAGE": "pam_unix(sudo:session): session closed for user root",
    "_MACHINE_ID": "b2a7e89023c14c6e8cc357174818438a",
    "_BOOT_ID": "cef4b788fae442b68d64fae0026cfd2d",
    "_COMM": "sudo",
    "_GID": "0",
    "_PID": "378643",
    "_TRANSPORT": "syslog",
    "_SYSTEMD_USER_SLICE": "-.slice",
    "SYSLOG_TIMESTAMP": "Feb 22 21:01:40 ",
    "_SELINUX_CONTEXT": "unconfined\n",
    "__REALTIME_TIMESTAMP": "1708664500248295",
    "_SOURCE_REALTIME_TIMESTAMP": "1708664500248265",
    "_SYSTEMD_UNIT": "session-921.scope",
    "_RUNTIME_SCOPE": "system",
    "__CURSOR": "s=ca35ebb8029548c8aedb4ddc90c7fb64;i=4172f5;b=aef...",
    "_SYSTEMD_INVOCATION_ID": "8d7a23f2c2b54f73a1302b18755c8c94",
    "_SYSTEMD_CGROUP": "/user.slice/user-1000.slice/session-921.scope",
    "_HOSTNAME": "myhost",
    "_CMDLINE": "sudo journalctl -b -o json",
    "PRIORITY": "6",
    "_SYSTEMD_SLICE": "user-1000.slice",
    "_AUDIT_LOGINUID": "1000",
    "SYSLOG_IDENTIFIER": "sudo",
    "_UID": "1000",
    "_SYSTEMD_OWNER_UID": "1000",
    "_SYSTEMD_SESSION": "921",
    "SYSLOG_FACILITY": "10"
}

We now have a way to get all of the system logs in a standard structured output. Next, we need to be able to output only new logs since we last checked so we don’t send duplicate logs. This is where --cursor-file comes in.

Sending only new logs

From the journalctl -h:

--cursor-file=FILE Show entries after cursor in FILE and update FILE

By using the --cursor-file=/tmp/my-cursor.txt option, we can maintain a record of the log entries that have been processed and sent. This works by storing the unique identifier (ID) of the last log entry retrieved by journalctl in the specified file /tmp/my-cursor.txt. When journalctl is run with this flag, it reads the last known position from this file and begins displaying new log entries from that point onward, updating the file with the ID of the new last entry after the command completes.

From here you have all the parts you need. Your solution could be as simple as setting up a 1 minute cron job that pipes the output through curl to your Centralized Logging system over HTTPS. Or run a script.

One minute cron:

* * * * * journalctl --since -o json --cursor-file=/path/to/your/cursor-file | curl -X POST -H "Content-Type: application/json" -d @- https://my-central-logging-server.com/

Here is a snippit in Go:

// send logs from journalctl
func doJournalctl() {
	// kick things off
	// journalctl -r -n 1 -o json --cursor-file="/tmp/foo.txt"
	out, err := exec.Command("journalctl", "-r", "-n", "1", "-o", "json", "--cursor-file", cursorFilePath).Output()
	if err != nil {
		log.Fatal(out, err)
	}

	// continuously call
	// journalctl -o json --cursor-file="/tmp/foo.txt"
	ticker := time.NewTicker(maxTimeToFlush)
	done := make(chan bool)
	for {
		select {
		case <-done:
			return
		case <-ticker.C:
			// log.Println("checking for new logs at", t)

			out, err := exec.Command("journalctl", "-o", "json", "--cursor-file", cursorFilePath).Output()
			if err != nil {
				log.Fatal(err)
			}
			if len(out) < 1 {
				continue
			}
			outStr := string(out)
			if strings.TrimSpace(outStr) == "" {
				continue
			}
			// log.Println(string(out))
			client := http.Client{
				Timeout: 5 * time.Second,
			}

			// log.Printf("sending new logs (%d bytes) at %s", len(out), t.String())
			req, err := http.NewRequest("POST", conf.URL, bytes.NewBuffer(out))
			if err != nil {
				log.Fatal(err)
			}
			req.Header.Set("Content-Type", "text/plain")

			req.Close = true

			resp, err := client.Do(req)
			if err != nil {
				log.Println(err)
				if resp != nil {
					log.Println(resp.Header)
				}
			}
		}
	}
}