IT Musings

[isaacnickell-blog]

Windows 2003 Server - LSASS.EXE - System Error

I received a call last night where a client had a physical Active Directory server that blue screened during a routine reboot and update cycle and then got caught in a loop with the following error:

LSASS.EXE - System Error, security accounts manager initialization failed because of the following error: Directory Services cannot start. Error status 0xc00002e1

I found a KB article referencing the issue here http://support.microsoft.com/kb/258062 

Based on feedback from other sources I decided to do the following:

Boot in to Directory Services Restore Mode

Open a cmd prompt and run "ntdsutil files info" to determine the location of the AD database (it was good I did this as the client had an old install at C:\Windows and a newer install at D:\Windows)

Run a repair on the database "esentutl /p D:\Windows\System32\NTDS\ntds.dit"

Delete all log files from the NTDS folder where ntds.dit resides and then reboot the server

After the server reboot the client was able to login normally. There were still a few issues to clean up. The DHCP service was failing. The client had 25+ scopes so recreating the scopes would be a huge hassle. 

We were able to grab a backup of the dhcp folder and rename the existing folder dhcp.old, copy the restored folder over and start the service normally.

One last item to check was AD replication. That consisted of the following:

From a cmd prompt run "repadmin /replsum >c:\replsum.txt && c:\replsum.txt" (this will generate a replication summary and output to a file and then open that file for easy viewing)

Push out replication from each Active Directory Domain Controller using the command "repadmin /syncall /AedP" from the cmd prompt

Verify the replication status using the command from step 1

After this we rebooted to verify everything was working and then ran a system state backup.

[isaacnickell-blog]

Run a SQL backup to a mapped drive

I recently ran in to an issue where I needed to perform some maintenance on a database running on SQL 2008 R2 and I wasn't sure if working backups existed. I decided to backup to a network location due to a lack of local drive space.

I mapped a drive in Windows and went through the process of running a manual backup through the SSMS interface. Much to my chagrin the newly mapped Z drive was not an option. 

SQL server runs as a service and as such only sees the local drives that a console session would see and not the additional mapped drives a user may have. There is a nifty command to allow SQL to map a network drive that can then be used and seen through the SSMS interface.

The command can be found here http://blogs.msdn.com/b/sqlserverfaq/archive/2009/02/27/how-to-backup-sql-server-databases-to-a-mapped-drive.aspx and involves the following steps.

1. Run the following command from a query window EXEC xp_cmdshell 'net use <drivename> <share name>'

-- where

<drive name>: Letter used to map the drive

<share name>: UNC path to the share

2. After that you should be able to backup using the mapped drive letter

3. Your Management Studio Object Explorer logical drive enumeration window should be able to list the above mapped drive

Now when I ran this command I got a security related error saying this procedure was not available. You may have to enable this procedure on your SQL server. 

I found instructions for that here http://blog.sqlauthority.com/2007/04/26/sql-server-enable-xp_cmdshell-using-sp_configure/ and the command is as follows.

---- To allow advanced options to be changed.

EXEC sp_configure ‘show advanced options’, 1

GO

—- To update the currently configured value for advanced options.

RECONFIGURE

GO

—- To enable the feature.

EXEC sp_configure ‘xp_cmdshell’, 1

GO

—- To update the currently configured value for this feature.

RECONFIGURE

GO

Once you have enable the procedure and mapped your drive through SQL you should see it as an option when running a SQL backup.

[isaacnickell-blog]

vSphere Web Client Login SSL Error

I experienced the following error when trying to access our vCenter Server Appliance (vCSA) through the vSphere Web Client interface.

VMware vSphere Web Client

Failed to connect to VMware Lookup service https://hostname:7444/lookupservice/sdk - SSL Certificate verification failed

This can happen if the IP of your vCSA changes and the certificate is not updated. You can generate a new certificate by following this KB article.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033338

[isaacnickell-blog]

vCenter Server Appliance 5.5 Unable to Authenticate

I ran in to this problem while trying to fix another issue. When logging in to vCenter Server Appliance (vCSA) 5.5 I found I was unable to login with my credentials. Apparently the root account password automatically expires after 90 days.

The message I received was "Unable to authenticate user. Please try again."

VMware has a KB article for removing the password expiration and a resolution to resetting the password. It requires console access to the appliance so you will need to access this through the host your vCSA is on due to the maintenance being on the vCSA itself.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069041

[isaacnickell-blog]

XenServer - Recover "Destroyed" VDI

We recently had an issue where one too many disks was pulled from a storage array (no bueno) and we lost a storage repository running a production file server. To further complicate the matter, the command "xe vdi-destroy" was accidentally run against the vdi disk hosted on the offline SR. (I know, right?)

Thankfully, we were able to recover the SR but we still had the matter of fixing the vdi. When we would attach the disk to the VM and try and start it we would receive the "VDI not available" error message. Here is what we did to resolve the orphaned disk.

Note: Try this with a dummy/test disk first to ensure your SR is working otherwise you might lose the disk. Also, make a backup of the pool metadata.

First we ran "xe vdi-list name-label=disk_name" in order to get the uuid and sr uuid of the disk we were working with. 

Next we ran "xe vdi-forget uuid=uuid_disk_number" in order to have Xen forget the disk.

Next we ran "xe sr-scan uuid=uuid_sr_number" in order to scan the SR with the disk and have it show back up.

Now at this point you should be able to see your disk in your storage repository with no name or description. You can edit the properties and give it the original name.

After this, we were able to attach the disk to the VM and boot it up without issue.