Alas, after all these while of "stable" operations, Raspberry Pi still breakdown. Today the issue seemed major and beyond me. This is the first time I have seen it dumped this huge amount of error messages in a few log files such as daemon.log, kern.log, and syslog. I pinpoint the onset to this error:
Jan 31 17:36:12 raspberrypi kernel: [30960.788346] smsc95xx 1-1.1:1.0: eth0: Failed to read register index 0x00000114
Flashback
Today Raspberry was not running transmission as usual. It was running motion and probably ffmpeg while it was encoding captured video from a webcam. The motion and ffmpeg was custom compiled from the latest source code with full optimization for Raspberry Pi.
During some tests of motion/ffmpeg some time ago, motion is not known to cause problems running more than 12 hours at one go, while encoding mpeg4 videos during the time. There was no observed stress to the hardware with maximum CPU utilization below 20% at peak (average at 3%) with plenty of memory available. Maximum temperature recorded was 52C. It is pretty confident that motion/ffmpeg is stable.
Investigation
This is the first time such situation was encountered or at least been caught by me. Since the onset of the error message, communication via SSH was lost, not surprising since the Raspberry Pi connects to the network through a USB Wifi adapter, and the error points to USB related issue. (Note: smsc95xx and eth0 indicates the LAN port which is architecturely connected to the USB subsystem in Raspberry Pi's design) The webcam lost its connection too, with an uncompleted video timestamped at 17:37:09, barely a minute after the first error message.
So, it is apparent that the USB subsystem crashed.
Surprisingly, the rest of the Raspberry Pi system was still working. This is shown by the executions of my bash script by the cron daemon. I had written a "keep-alive" script that will attempt to recover the wlan0 interface if it is unable to ping the gateway. The script had logged its activities in a custom log file too.
Now, it is shown that a crash in the USB subsystem will not necessarily break down the whole system.
Solution
This problem is beyond me. The only way was to update the firmware and kernel on Raspberry Pi using the Hexxeh rpi-update.
Before rpi-update: Kernel version 3.2.27+
After rpi-update: Kernel version 3.6.11+
Rpi-update also updates the bootloader image and the kernel modules. Let's see if this solve the difficult problem.
This is the kern.log (cropped) for reference. Download here (139KB).
