Suddenly /dev/null became a regular file

I was doing a routine task – preparing a new CentOS server – today and ran into quite obscure problem.

I was at the point where I needed to configure VPN link but OpenVPN wouldn’t let me daemonize itself. It complained in the logs basically saying that the problem was this:

openvpn[4738]: daemon() failed: No such device (errno=19)

That’s weird. After an hour of troubleshooting this issue on the server I took it to #openvpn@irc.freenode.net where dazo, the channel operator, pointed out that some people previously had have a similarly looking problem, and that if /dev/null was involved it might be a similar or exactly that kind of problem.

I checked /dev/null with stat utility and it was indeed just a regular file. WHOA. This is a production server that doesn’t see software updates, tested and works for the most part as a clock. Utterly inexplicable at this point to me but I don’t have time to research this right now. I just wanted to make a post about it to remember to look into this later, because this is quite interesting and doesn’t happen very often. In fact, I’ve been working with Linux for at least 5 years now, and I’ve never seen anything of the sort. Not even my more than I am experienced colleagues.

So, luckily the fix for this problem is easy and painless. All you have to do is the following:

% rm -rf /dev/null
% mknod /dev/null c 1 3

To check whether it’s a character device, and not a regular file run stat like this:

% stat /dev/null

Access: (0644/crw-r–r–)

Look for the line that starts with Access: and if it has crw then it’s a proper character device.

I didn’t even have to reboot the system. Which is important if you’re running a production server.

Now, the most time-consuming part of this adventure was actually pinpointing the source of the problem. If you ever run into mysterious issues where proven and tested ways of doing things on the server behave as if they’re on LSD check the system logs, maybe daemons are crying out that there’s something wrong with /dev/null and you just don’t know about it.

% grep null /var/log/* |grep -v “\/dev\/null 2”
/var/log/maillog:Jan 27 14:55:01 prodbox-02 sendmail[9219]: q0RDt1XZ009218: disconnect: sm_io_reopen(“/dev/null”) failed: No such device or address
/var/log/maillog:Jan 27 14:55:01 prodbox-02 sendmail[9219]: q0RDt1XZ009218: disconnect: open(“/dev/null”) failed: No such device or address
/var/log/maillog:Jan 27 14:55:01 prodbox-02 sendmail[9218]: q0RDt1Xa009218: disconnect: sm_io_reopen(“/dev/null”) failed: No such device or address
/var/log/maillog:Jan 27 14:55:01 prodbox-02 sendmail[9218]: q0RDt1Xa009218: disconnect: open(“/dev/null”) failed: No such device or address
/var/log/maillog:Jan 27 14:55:01 prodbox-02 procmail[9220]: Error while writing to “/dev/null”
/var/log/maillog:Jan 27 14:55:01 prodbox-02 procmail[9222]: Error while writing to “/dev/null”
/var/log/maillog:Jan 27 14:55:01 prodbox-02 procmail[9224]: Error while writing to “/dev/null”
/var/log/maillog:Jan 27 15:33:21 prodbox-02 sm-msp-queue[30847]: q0REXLr8030847: disconnect: open(“/dev/null”) failed: Permission denied
/var/log/maillog:Jan 27 16:33:21 prodbox-02 sm-msp-queue[4214]: q0RFXLr8004214: disconnect: open(“/dev/null”) failed: Permission denied
/var/log/maillog:Jan 27 17:33:21 prodbox-02 sm-msp-queue[6375]: q0RGXLr8006375: disconnect: open(“/dev/null”) failed: Permission denied

Note that I tell grep to ignore “/dev/null 2” which will help to weed out bazillion of cron jobs and other commands that simply try to redirect output to /dev/null 2>&1.

If you happen to know what, and why, makes a character device /dev/null become a regular file, please, do let me know.

Advertisements

3 thoughts on “Suddenly /dev/null became a regular file

  1. I found the cause on my archlinux system.

    If you use bash and HISTFILE=/dev/null is in environment, you shouldn’t execute more commands than $HISTFILESIZE or $HISTSIZE.

    If you executed more commands than $HISTFILESIZE on bash while HISTFILE is /dev/null and exited bash, bash moves /dev/null somewhere else and recreates /dev/null as a regular file with permission 600.

    If you use tramp on emacs 24.4, tramp-sh.el sets HISTFILE to /dev/null.
    Thus, if bash is the shell for root and if you do a lot of operations with tramp on emacs 24.4, when you kill emacs, tramp makes bash delete /dev/null.

    Please check if HISTFILE is set to /dev/null in .bashrc or in programs like emacs 24.4.

    • If you change the shell to zsh, you can work around the problem.
      If tramp caused your problem, you could also downgrade emacs instead of changing the shell to zsh.

    • Interesting. Somehow WordPress failed to notify me of your comment (and other’s too) earlier. It was years ago and I can’t really be certain if that particular server ran too many tasks or what $HISTFILESIZE etc. was set to but it’s interesting to know what it was in your case.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s