One of the more agitating problems I’ve dealt with recently was a project to install Dell’s OpenManage Server Administrator (OMSA) on a Dell R720xd server running ESXi 5.5. I had installed OMSA at another clients offices just the week before and had no issues whatsoever, but this time would prove different.

As you may know, in order to access OMSA on an ESXi installation, you have to install OMSA on another machine as well and remotely connect to OMSA on the ESXi host (using the ESXi username and password). Accordingly, I installed OMSA on an old server in the environment so I could remotely connect to the OMSA ESXi install. I opened OMSA and entered the information as below:

After trying to connect, I got the following error message:

[callout] Login failed…. connection error[/callout]

I went through some of the traditional troubleshooting steps, such as checking and rechecking that I was using the right IP address, username and password. I confirmed several times over that they were correct, but every connection attempt yielded the same error message “Login failed…. connection error”. So I started digging deeper.

Troubleshooting steps attempted

First, I tried uninstalling and reinstalling OMSA several times, including various patterns of rebooting. Every time, the install seemed to go perfectly, yet I continued to get the connection error message.

Next, I enabled SSH and tried downloading the 5.5 OMSA build directly from the Dell OMSA ESXi 5.5 download location using the wget command on the host, just to make sure the installer wasn’t corrupted. Same result.

I decided it was time to dig a little deeper so I enabled SSH on the ESXi hosts again and started looking for the Dell OMSA related files.

I ran the following command to make sure OpenManage was installed:

~ # esxcli software vib list | grep -i open
OpenManage 7.3.0.2-0000 Dell PartnerSupported 2014-02-21

It showed that OpenMange was indeed installed.

I found files that looked promising under /opt/dell/srvadmin/ and started researching. I tried to start the OMSA services as follows:

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # cd /opt/dell/srvadmin/bin
/opt/dell/srvadmin/bin # ./dataeng restart
grep: /etc/cim/dell/srvadmin/srvadmin-omilcore/install.ini: No such file or directory
Stopping Systems Management Data Engine:
Stopping dsm_sa_snmpd: Not started FAILED
Stopping dsm_sa_eventmgrd: Not started FAILED
Stopping dsm_sa_datamgrd: Not started FAILED
root: snmpd has not been enabled.
Starting Systems Management Data Engine:
Starting dsm_sa_datamgrd: ./dataeng: line 1397: /opt/dell/srvadmin/sbin/dsm_sa_datamgrd: not found
FAILED
Starting dsm_sa_eventmgrd: ./dataeng: line 1397: /opt/dell/srvadmin/sbin/dsm_sa_eventmgrd: not found
FAILED
Starting dsm_sa_snmpd: ./dataeng: line 1397: /opt/dell/srvadmin/sbin/dsm_sa_snmpd: not found
FAILED

These error messages were very confusing, because those files that are “not found” definitely didn’t exist in those directories. The baffling part was, those files didn’t exist on the working hosts at the other client either, but OMSA still worked!

At this point, I was thoroughly confused. I called Dell Pro Support and started working through troubleshooting the issue with them. In the middle of having them troubleshoot the problem, without success, I decided to try something different on a whim that ultimately led me to the answer.

Epiphany

I got to thinking about it and wondered if for some reason the problem was possibly with the Windows Server 2003 server that I installed OMSA on for remotely accessing the ESXi OMSA install. I decided to load up my Windows 8.1 VM inside of Hyper-V on my laptop and install OMSA there. I did that and and then tried connecting to the ESXi Host, and low and behold, IT WORKED!

I kept investigating because I wanted to make sure it wasn’t a browser issue. On the Windows Server 2003 server, I installed the latest version of Chrome. I honestly expected that would solve the problem, but it didn’t. As it turns out, there is apparently a KB article inside of Dell that states that OMSA remote won’t work with Server 2003 because Server 2003 is missing pieces required for WS-MAN to work. Possibly .NET framwork pieces missing as well.

In the end, if you’re getting this error, make sure you check what operating system you’re installing OMSA on. Make sure it’s at least Server 2008 or later.