Logging mit Sytemd im Raspberry Pi Cluster

2016-08-20 @ FrOSCon 2016

Christian Prior

Slides available

http://www.helotism.de \ ⏎
/business/marketing/presentation/FrOSCon_2016-08-20

## about me ![](/business/marketing/presentation/images/about-cpr.svg) https://github.com/cprior
## The Problem ![](/business/marketing/images/complex-solution.svg) The German Industry does not produce enough interesting Linux-jobs ;)
## Rootcause ![](/business/marketing/images/spareparts.svg) The strengths of \*ix and F/LOSS are not applied to Mechanical Engineering / Automation / ...
## Solution ![](/business/planning/images/PRDV_TOGAF-Cube_BusinessDataApplicationTechnology.jpg) tbd
## Logging trends - structured logging - client-side aggregation - metrics
## Logging, support, development - traditionally logs are for developers - ideally a bug only reaches development once, with a qualified root cause analysis - follow-the-sun customer support must forward incidents appropriately - gap between monitoring capabilities and logfile content - between fix and software update support must go on -- as efficient as possible
## Logfiles as part of Data Streams - Monitoring/alerting/logging must provide data - for root cause analysis and bug fixes - and generate input for the Product Lifecycle Complex machines, especially with operators, need data analysis.

Logging with systemd

If you like it

- future-proof skills

- context for services

- central logging

If not: Nice concepts

- structured logging, multi-lines

- sealing

- message IDs

## systemd logging and status > along with the output of systemctl status we wanted to show the last 10 log messages of the daemon ```bash [cpr@axle ~]$ sudo systemctl status sshd * sshd.service - OpenSSH Daemon Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2016-08-07 12:35:29 UTC; 13min ago Main PID: 305 (sshd) Tasks: 1 (limit: 512) CGroup: /system.slice/sshd.service `-305 /usr/bin/sshd -D Aug 07 12:35:29 axle.wheel.prdv.de systemd[1]: Started OpenSSH Daemon. Aug 07 12:35:30 axle.wheel.prdv.de sshd[305]: Server listening on 0.0.0.0 port 22. Aug 07 12:48:21 axle.wheel.prdv.de sshd[866]: Accepted publickey for cpr from 10.166.15.116 port 35028 ssh2: RSA Aug 07 12:48:22 axle.wheel.prdv.de sshd[866]: pam_unix(sshd:session): session opened for user cpr by (uid=0) ```
## Basic Usage ```bash journalctl journalctl -b journalctl -f journalctl -f -u sshd journalctl --since=2012-10-15 --until="2011-10-16 23:59:59" ``` Structured logging ```bash journalctl --fields journalctl -F FIELD2 journalctl PRIORITY=5 #logger "test" journalctl PRIORITY=6 #logger -p 6 "Hi" ```
## Custom logging with systemd - own fields - writing from Python into the journal - bonus complexity: listening on the journal Usecase in the Helotism project: 1 Power-off button to poweroff all boards
## Minimal Example ```Python from systemd import journal journal.send('Hello world') journal.send('Hello, again, world', FIELD2='Greetings!', FIELD3='Guten tag') ```
## What comes next - Python script is run as a service - listens to RPi's GPIO pin - if debounced button press detected, logs to journal
##Powersupply - prevent brown-out - one switch for all boards
##Debouncing ![](/technology/logical/images/ElectricalSwitchDebounce.svg)
##Python script ```Python #!/usr/bin/env python2 import time from gpiozero import Button from systemdream import journal #virtualenv-capable systemd journal (only journal!) Python code import salt.client journal.send('power-button script initialized') ``` ```Python def do_shutdown(): local = salt.client.LocalClient() journal.send('Button on GPIO24 pressed, DEBOUNCED.', FIELD2='GPIO24') local.cmd_async('not ', 'cmd.run', ['shutdown -h 1'], expr_form='compound') local.cmd_async('', 'cmd.run', ['shutdown -h 2']) journal.send('Commands were sent.', FIELD2='GPIO24') button = Button(24, bounce_time=2) button.when_released = do_shutdown ```
##…and systemd daemonization. ```INI [Unit] Description=An GPIO interrupt listener After=local-fs.target [Service] Type=simple ExecStart=/bin/bash -c 'cd /opt/helotism/powersupply-env; \ source bin/activate; \ python ./mraa_interrupt.py' [Install] WantedBy=multi-user.target ```
## Listening to journal with Saltstack - with the journal being part of PID=1 it is a source of important events - journald can be a source of events like inotify - implemented in SaltStack as "beacon"/"reactor"
##SaltStack Ecosystem ![](/application/physical/images/saltstack-ecosystem.svg)
## listen and fire event ```bash beacons: journald: sshd: SYSLOG_IDENTIFIER: sshd PRIORITY: 6 ```
## react centrally ```bash reactor: - salt/beacon/*/journald/*/GPIO24 - /srv/reactor/scary-shutdown.sls ```
## Remote sink, and local upload ```bash systemctl status systemd-journal-remote.service systemctl status systemd-journal-upload.service ``` - Port 19531 (socket.wants) - http or https - [libmicrohttpd](https://github.com/rboulton/libmicrohttpd)
## Config remote ```bash [Remote] #master ServerKeyFile=/etc/ssl/private/axle.wheel.prdv.de.key.pem ServerCertificateFile=/etc/ssl/certs/axle.wheel.prdv.de.cert.pem TrustedCertificateFile=/etc/ssl/helotism-ca-chained-public-certs.cert.pem ```
## Config uplad ```bash Upload] #URL=http://axle.wheel.prdv.de URL=https://axle.wheel.prdv.de ServerKeyFile=/etc/ssl/private/axle.wheel.prdv.de.key.pem ServerCertificateFile=/etc/ssl/certs/axle.wheel.prdv.de.cert.pem TrustedCertificateFile=/etc/ssl/helotism-ca-chained-public-certs.cert.pem ``` Hostname and certificate must match.
## Generate certificates [link](/application/physical/building-blocks/images/pki_ca-intermediate-server.svg)
## Troubleshoot SSL - openssl has s_server and s_client - or a Python https server and curl Just testing: ``` openssl s_server -cert /etc/ssl/certs/axle.wheel.prdv.de.cert.pem -key /etc/ssl/private/axle.wheel.prdv.de.key.pem -www ``` Or even as webserver: ``` openssl s_server -cert /etc/ssl/certs/axle.wheel.prdv.de.cert.pem -key /etc/ssl/private/axle.wheel.prdv.de.key.pem -accept 443 -WWW ```
## With Python ```Python https://www.piware.de/2011/01/creating-an-https-server-in-python/ import BaseHTTPServer, SimpleHTTPServer import ssl httpd = BaseHTTPServer.HTTPServer(('axle.wheel.prdv.de', 4443), SimpleHTTPServer.SimpleHTTPRequestHandler) #httpd.socket = ssl.wrap_socket (httpd.socket, certfile='server.pem', server_side=True) httpd.socket = ssl.wrap_socket (httpd.socket, certfile='/etc/ssl/certs/axle.wheel.prdv.de.cert.pem', keyfile='/e tc/ssl/private/axle.wheel.prdv.de.key.pem', server_side=True) httpd.serve_forever() ``` and then ```bash python2.7 https.py ``` and connecting with curl: ```bash curl --cacert /etc/ssl/helotism-ca-chained-public-certs.cert.pem --cert /etc/ssl/certs/axle.wheel.prdv.de.cert.pem --key /etc/ssl/private/axle.wheel.prdv.de.key.pem https://axle.wheel.prdv.de:4443 ```
## Default SSL directories - cpr/opensuseleap OPENSSLDIR: “/etc/ssl” - cpr/fedora24 OPENSSLDIR: “/etc/pki/tls” - cpr/debianjessie OPENSSLDIR: “/usr/lib/ssl” - cpr/archlinuxlatest OPENSSLDIR: “/etc/ssl”
## text vs binary - logfiles as data streams - archiving / auditing - eng. vs pol.
## binary [GitHub systemd src](https://github.com/systemd/systemd/blob/master/src/journal/journal-def.h) ```C struct Header { uint8_t signature[8]; /* "LPKSHHRH" */ ... sd_id128_t file_id; sd_id128_t machine_id; sd_id128_t boot_id; /* last writer */ sd_id128_t seqnum_id; le64_t header_size; ... le64_t n_objects; le64_t n_entries; ... /* Size: 240 */ } _packed_; ```
## content of the journal file ```bash strings /var/log/journal/*/system.journal ```
## Roadmap Helotism project as data producer and data analysis "appliance". Automating the sysadmin stuff as much as possible. Contributions welcome!