Now that we have KMS, it happens things go bad enough that when X goes down, one can no longer access any console to see what happened. Mainly, one can distinguish between two cases.

• The “good” one: the machine is still accessible over the network. Unfortunately, that requires having a spare machine to ssh into the first machine, but that means one can run dmesg, look into X logs, possibly run X inside gdb and see where it breaks (because glibc’s backtrace isn‘t exactly an excellent tool to get the idea of what’s going on — and that’s the tool used in X to print a backtrace in the log through a signal handler).
• The “bad” one: the machine is no longer accessible over the network. Likely, the kernel is in such a bad shape that one is never going to know what happened. Usually, writing logs to disc didn’t happen, so logs won’t magically show up at next reboot.

I’m going to discuss the second case. There’s a nice tool in the kernel which makes it possible to send kernel logs over the network. It’s called netconsole. As far as limitations are concerned, one shall note that it’s UDP only, and over Ethernet (in other words: no wireless). The good news is that it can usually make the last crucial lines available.

Let’s call the crashy machine a patient and the logging machine a doctor.

One should keep in mind that if something goes wrong while configuring the netconsole module, it can be unloaded at anytime through:

sudo modprobe -r netconsole

Easy case: on a local network

Here’s an example: Patient’s IP is 192.168.0.1, doctor's IP is 192.168.0.2.

Doctor setup

What one needs on the receiving side: only netcat. A tiny warning, there are several implementations. Here’s the appropriate syntax:

# netcat-traditional:
nc -l -u -p 6666 | tee ~/netconsole.log

# netcat-openbsd:
nc -l -u 6666 | tee ~/netconsole.log

The nc command is handled through the alternatives system, one can use update-alternative --config nc to pick the appropriate one if several netcat-* packages are installed. To list the current one, an easy way is readlink -f $(which nc).

Using tee means that incoming messages are going to be printed, both to the standard output and to the specified file.

Running as non-privileged user is sufficient.

Patient setup

Now, to have the patient send stuff to the doctor, a simple modprobe is needed:

sudo modprobe netconsole netconsole=@/,6666@192.168.0.2/

What happens here? One requests the netconsole module to be loaded, and one specifies the parameters. Details can be read in the Linux kernel documentation (Documentation/networking/netconsole.txt), but concentrating on the points of interest here:

• 192.168.0.2: The doctor’s IP.
• 6666: The UDP port to which packets are sent. 6666 is the default and can be omitted.

That’s all! On the doctor side, one should see something along those lines:

[2920549.188090] console [netcon0] enabled
[2920549.188102] netconsole: network logging started

Slightly harder case: over internet

Because one might not have a second machine handy, it’s also possible to go through a router and send stuff across the internet.

Here’s an example: Patient’s IP is 192.168.0.1, connecting to internet through a router, which IP is 192.168.0.254. The doctor, available over the internet, is 192.0.32.10.

Doctor setup

Exactly the same as in the previous case. The main issue one can run into is firewalls at this point. That’s why one may want to switch from the default 6666 port to another one which wouldn’t be filtered. One can think of 53 (DNS), but that means running nc as root since it’s a “privileged” port (below 1024).

Patient setup

Routing is also possible, but that requires an extra parameter: the MAC address of the router. To obtain it, one can use the arp command:

/usr/sbin/arp 192.168.0.254

Supposing it returned the 01:02:03:04:05:06 address, loading the module becomes:

sudo modprobe netconsole netconsole=@/,6666@192.0.32.10/01:02:03:04:05:06

Now, if you’re running into firewall-related issues, you can change the source for the UDP packets. The default is 6665, but assuming you want to send from an unfiltered 1234 port, that becomes:

sudo modprobe netconsole netconsole=1234@/,6666@192.0.32.10/01:02:03:04:05:06