Accidentally server backups
Tags: Domino Outsourcing
Outsourcing IT services to a 3rd party company seems to be very popular. I'm working for some customers who did this and I don't know one where the service provided to the end users is equal to the level it was before. In all cases it is worse. But talking about outsourcing and it's pros and cons is worth its own entry or two or three. Today I made a really unbelievable experience with the service provider of one customer, I have to talk about.
At this customer the service provider is hosting the hardware and does the OS administration. The customer does have the responsibility for the application, which is Domino, and I am working for the customer. The application backup is done by the service provider also. But because he isn't able to do a online backup we need to shutdown the Domino server before the backup starts. And as you can guess there is a reponsibility problem with that, because the service provider can't shutdown a Domino server he isn't responsible for. So we created a cmd file to stop the Domino server and the service provider tried to start that script before the backup. Yes, he tried. After several days (nights) and several backup attempts, where sometimes the Domino was stopped and the backup hanged and sometimes the backup application wasn't able to start our script, the service provider told us today, that our script is the problem. Our argument, that exactly the same script runs on exactly the same machines (which are btw. also backed up by the same service provider) was refused. The administrator of the service provider told us that:
This is obvious the most likely conclusion. 5 servers are shutdown and backed up every night and because one server doesn't work, the function for the other 5 is happens "accidentally".
I'm not talking about rocket sience. The main lines in the code are very simple and easy:
Our conclusion is to shutdown the server with a scheduled job and start it with another an hour later. Not nice but it is working.
Outsourcing IT services to a 3rd party company seems to be very popular. I'm working for some customers who did this and I don't know one where the service provided to the end users is equal to the level it was before. In all cases it is worse. But talking about outsourcing and it's pros and cons is worth its own entry or two or three. Today I made a really unbelievable experience with the service provider of one customer, I have to talk about.
At this customer the service provider is hosting the hardware and does the OS administration. The customer does have the responsibility for the application, which is Domino, and I am working for the customer. The application backup is done by the service provider also. But because he isn't able to do a online backup we need to shutdown the Domino server before the backup starts. And as you can guess there is a reponsibility problem with that, because the service provider can't shutdown a Domino server he isn't responsible for. So we created a cmd file to stop the Domino server and the service provider tried to start that script before the backup. Yes, he tried. After several days (nights) and several backup attempts, where sometimes the Domino was stopped and the backup hanged and sometimes the backup application wasn't able to start our script, the service provider told us today, that our script is the problem. Our argument, that exactly the same script runs on exactly the same machines (which are btw. also backed up by the same service provider) was refused. The administrator of the service provider told us that:
I assume the script runs accidentally on the other servers.
This is obvious the most likely conclusion. 5 servers are shutdown and backed up every night and because one server doesn't work, the function for the other 5 is happens "accidentally".
I'm not talking about rocket sience. The main lines in the code are very simple and easy:
sc STOP "Lotus Domino Server (DominoData)"
ping 1.0.0.0 -n 1 -w 120000 >NUL
nsd -kill -nolog
Our conclusion is to shutdown the server with a scheduled job and start it with another an hour later. Not nice but it is working.
