## about me
https://github.com/cprior
## The Problem
The German Industry does not produce enough interesting Linux-jobs ;)
## Rootcause
The strengths of \*ix and F/LOSS are not applied to Mechanical Engineering / Automation / ...
## Solution
tbd
## Logging trends
- structured logging
- client-side aggregation
- metrics
## Logging, support, development
- traditionally logs are for developers
- ideally a bug only reaches development once, with a qualified root cause analysis
- follow-the-sun customer support must forward incidents appropriately
- gap between monitoring capabilities and logfile content
- between fix and software update support must go on -- as efficient as possible
## Logfiles as part of Data Streams
- Monitoring/alerting/logging must provide data
- for root cause analysis and bug fixes
- and generate input for the Product Lifecycle
Complex machines, especially with operators, need data analysis.
Logging with systemd
If you like it
- future-proof skills
- context for services
- central logging
If not: Nice concepts
- structured logging, multi-lines
- sealing
- message IDs
## systemd logging and status
> along with the output of systemctl status we wanted to show the last 10 log messages of the daemon
```bash
[cpr@axle ~]$ sudo systemctl status sshd
* sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2016-08-07 12:35:29 UTC; 13min ago
Main PID: 305 (sshd)
Tasks: 1 (limit: 512)
CGroup: /system.slice/sshd.service
`-305 /usr/bin/sshd -D
Aug 07 12:35:29 axle.wheel.prdv.de systemd[1]: Started OpenSSH Daemon.
Aug 07 12:35:30 axle.wheel.prdv.de sshd[305]: Server listening on 0.0.0.0 port 22.
Aug 07 12:48:21 axle.wheel.prdv.de sshd[866]: Accepted publickey for cpr from 10.166.15.116 port 35028 ssh2: RSA
Aug 07 12:48:22 axle.wheel.prdv.de sshd[866]: pam_unix(sshd:session): session opened for user cpr by (uid=0)
```
## Basic Usage
```bash
journalctl
journalctl -b
journalctl -f
journalctl -f -u sshd
journalctl --since=2012-10-15 --until="2011-10-16 23:59:59"
```
Structured logging
```bash
journalctl --fields
journalctl -F FIELD2
journalctl PRIORITY=5 #logger "test"
journalctl PRIORITY=6 #logger -p 6 "Hi"
```
## Custom logging with systemd
- own fields
- writing from Python into the journal
- bonus complexity: listening on the journal
Usecase in the Helotism project: 1 Power-off button to poweroff all boards
## Minimal Example
```Python
from systemd import journal
journal.send('Hello world')
journal.send('Hello, again, world', FIELD2='Greetings!', FIELD3='Guten tag')
```
## What comes next
- Python script is run as a service
- listens to RPi's GPIO pin
- if debounced button press detected, logs to journal
##Powersupply
- prevent brown-out
- one switch for all boards
##Debouncing
##Python script
```Python
#!/usr/bin/env python2
import time
from gpiozero import Button
from systemdream import journal #virtualenv-capable systemd journal (only journal!) Python code
import salt.client
journal.send('power-button script initialized')
```
```Python
def do_shutdown():
local = salt.client.LocalClient()
journal.send('Button on GPIO24 pressed, DEBOUNCED.', FIELD2='GPIO24')
local.cmd_async('not ', 'cmd.run', ['shutdown -h 1'], expr_form='compound')
local.cmd_async('', 'cmd.run', ['shutdown -h 2'])
journal.send('Commands were sent.', FIELD2='GPIO24')
button = Button(24, bounce_time=2)
button.when_released = do_shutdown
```
##…and systemd daemonization.
```INI
[Unit]
Description=An GPIO interrupt listener
After=local-fs.target
[Service]
Type=simple
ExecStart=/bin/bash -c 'cd /opt/helotism/powersupply-env; \
source bin/activate; \
python ./mraa_interrupt.py'
[Install]
WantedBy=multi-user.target
```
## Listening to journal with Saltstack
- with the journal being part of PID=1 it is a source of important events
- journald can be a source of events like inotify
- implemented in SaltStack as "beacon"/"reactor"
##SaltStack Ecosystem
## listen and fire event
```bash
beacons:
journald:
sshd:
SYSLOG_IDENTIFIER: sshd
PRIORITY: 6
```
## react centrally
```bash
reactor:
- salt/beacon/*/journald/*/GPIO24
- /srv/reactor/scary-shutdown.sls
```
## Remote sink, and local upload
```bash
systemctl status systemd-journal-remote.service
systemctl status systemd-journal-upload.service
```
- Port 19531 (socket.wants)
- http or https
- [libmicrohttpd](https://github.com/rboulton/libmicrohttpd)
## Config remote
```bash
[Remote]
#master
ServerKeyFile=/etc/ssl/private/axle.wheel.prdv.de.key.pem
ServerCertificateFile=/etc/ssl/certs/axle.wheel.prdv.de.cert.pem
TrustedCertificateFile=/etc/ssl/helotism-ca-chained-public-certs.cert.pem
```
## Config uplad
```bash
Upload]
#URL=http://axle.wheel.prdv.de
URL=https://axle.wheel.prdv.de
ServerKeyFile=/etc/ssl/private/axle.wheel.prdv.de.key.pem
ServerCertificateFile=/etc/ssl/certs/axle.wheel.prdv.de.cert.pem
TrustedCertificateFile=/etc/ssl/helotism-ca-chained-public-certs.cert.pem
```
Hostname and certificate must match.
## Generate certificates
[link](/application/physical/building-blocks/images/pki_ca-intermediate-server.svg)
## Troubleshoot SSL
- openssl has s_server and s_client
- or a Python https server and curl
Just testing:
```
openssl s_server -cert /etc/ssl/certs/axle.wheel.prdv.de.cert.pem -key /etc/ssl/private/axle.wheel.prdv.de.key.pem -www
```
Or even as webserver:
```
openssl s_server -cert /etc/ssl/certs/axle.wheel.prdv.de.cert.pem -key /etc/ssl/private/axle.wheel.prdv.de.key.pem -accept 443 -WWW
```
## With Python
```Python
https://www.piware.de/2011/01/creating-an-https-server-in-python/
import BaseHTTPServer, SimpleHTTPServer
import ssl
httpd = BaseHTTPServer.HTTPServer(('axle.wheel.prdv.de', 4443), SimpleHTTPServer.SimpleHTTPRequestHandler)
#httpd.socket = ssl.wrap_socket (httpd.socket, certfile='server.pem', server_side=True)
httpd.socket = ssl.wrap_socket (httpd.socket, certfile='/etc/ssl/certs/axle.wheel.prdv.de.cert.pem', keyfile='/e
tc/ssl/private/axle.wheel.prdv.de.key.pem', server_side=True)
httpd.serve_forever()
```
and then
```bash
python2.7 https.py
```
and connecting with curl:
```bash
curl --cacert /etc/ssl/helotism-ca-chained-public-certs.cert.pem --cert /etc/ssl/certs/axle.wheel.prdv.de.cert.pem --key /etc/ssl/private/axle.wheel.prdv.de.key.pem https://axle.wheel.prdv.de:4443
```
## Default SSL directories
- cpr/opensuseleap OPENSSLDIR: “/etc/ssl”
- cpr/fedora24 OPENSSLDIR: “/etc/pki/tls”
- cpr/debianjessie OPENSSLDIR: “/usr/lib/ssl”
- cpr/archlinuxlatest OPENSSLDIR: “/etc/ssl”
## text vs binary
- logfiles as data streams
- archiving / auditing
- eng. vs pol.
## binary
[GitHub systemd src](https://github.com/systemd/systemd/blob/master/src/journal/journal-def.h)
```C
struct Header {
uint8_t signature[8]; /* "LPKSHHRH" */
...
sd_id128_t file_id;
sd_id128_t machine_id;
sd_id128_t boot_id; /* last writer */
sd_id128_t seqnum_id;
le64_t header_size;
...
le64_t n_objects;
le64_t n_entries;
...
/* Size: 240 */
} _packed_;
```
## content of the journal file
```bash
strings /var/log/journal/*/system.journal
```
## Roadmap
Helotism project as data producer and data analysis "appliance".
Automating the sysadmin stuff as much as possible.
Contributions welcome!