Musing about logservers.
Having a LAN where all computers send their syslog entries to a
particular IP allows for better observation, as even without NTP
synchronized clocks, the entries in the log files are just a slight
delay and written in order. So, the proper order of log entries is
almost the same as the order you might observe them in. But having all
clocks on a LAN agreeing on the time is better.
So, the logserver is useful; but if someone bad breaks in they notice
that either no logs are kept on the machine they broken into, or there
are only trivial logs. So, they go looking for a logserver (which is
easy to find, it is in the configuration of every machine that is
sending logs to the logserver).
They broke into one of your LAN machines, they can probably break into
your logserver and get rid of evidence there. It is after all, on your
Enter the stealth logserver. It is a computer connected to your LAN,
but it has no assigned IP address for its NIC. The NIC is in
promiscuous mode, and it gathers its log information by sniffing the
network of all packets, and saving the logging information to logs.
The original stealth logserver, was probably a printer printing logs,
with a box of fanfold paper. It would be difficult for someone to
erase logs, since they are printed on paper as received.
In order for your stealth logserver to be on a LAN segment that log
entries would be delivered to, I think you need a machine on that same
LAN segment to actually send log entries to. I will suggest a RPi with
a read only filesystem that is configured to write all log entries
to /dev/null. If that LAN segment really only contains this read-only
device to pretend to receive them, and a real device which is only
promiscuous; it is much more difficult for some bad person to cleanse
the logs. They could DOS the logging system, but to delete logs would
not be easy.
If someone was to break into your house (and now they have access to
the hardware); one suggestion I seen for a stealth logserver is to find
a dead UPS which has a RJ45 port for catching surges on the Ethernet.
You take out the dead guts and stick a RPi (or similar) inside the
shell, using the RJ45 as a connection to the LAN. It might be
subterfuge which works.
But, another device that is ubiquitous to the LAN, is the router. And
some routers have USB ports, and can make use of a SSD in a USB box
(typically a SATA SSD). A router connected to an external SATA box by
USB isn't quite as invisible as a UPS, but maybe people will miss it?
I think to be most useful, you have multiple USB SATA SSD boxes. You
run a shell script to divert logging to /tmp (or something), umount the
SSD, you swap the SSD devices, and then the shell script mounts the SSD
and starts logging again (starting with anything saved in /tmp).
The recently used SSD can then be analyzed for anything interesting,
and posssibly archived.
There is not much sense going for the smallest SSD on this. If things
get busy, be able to handle as much logs as it takes to get through
your busy season. In terms of archiving, I think writing a MDisc (long
lived DVD) is probably the best. I don't know if they come in 100 or
125 GB versions, but who generates 100 GB of logs per week? (Or per
month, or ...?)
I haven't looked, but there might be someone who is storing Linux logs
in PostgreSQL. I would guess you take that code, and morph it to save
into SQLite 3. An objective of SQLite was to be compatible with