Chris LaPlante (chrisla) wrote,
Chris LaPlante

3AM in SysAdmin land

Part of my job doing SysAdmin is to provide escalation support to the admins in India. They do some things, if they can't figure it out they call me (or my co-worker; we rotate) at all hours. This morning I got a call that was the non-technical equivalent of, "I'm trying to drive the most important car in the fleet off of a cliff, but my key is not working, please fix the lock." I kept trying to tell the guy, "It's a good thing your key is not working, b/c you're about to drive off a cliff and the owner of the car is not really going to like that! Call the owner." He kept just repeating about the key. Come to find out, he found another way, drove the car off the cliff, and today we're picking up the parts.

A log location on the production SAP system was full. He said he wanted to "clean some stuff up; move some files and directories some place else" His login was not working (after a recent upgrade) I asked, which files, and where are you going to move them? He said to /dev/shm (shared memory/RAM; he picked this b/c it "had 20G free"). I said, that's a really bad idea, they would be gone if the system rebooted 10 seconds later, and we don't have it setup that way even (not mounted to a tmpfs filesystem). Also that running applications don't often take kindly to having files pulled from underneath them.

He found another system with the log location NFS moutned, moved the files to the /dev/shm device on that host. On top of that said files were locked open by a process on the server. That left them in never never land, and in a quasi state on the server. At any rate, thankfully the files were not really needed, and now we have to have an outage to get the space to free from the real filesystem. LSOF shows the open inodes, ls by inode does not.
  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your IP address will be recorded