Restarting the Equallogic web interface

Posted by Alex Brett on November 27, 2012

It appears to be possible in some situations to overload the management interfaces of a Dell Equallogic filer. The situation we experienced with a client was that after running an internal vulnerability scan (required for PCI:DSS compliance), the monitoring system picked up that the web interface had stopped responding.

The vulnerability scan essentially attempts to determine what software is in use on each device, and look for known issues, either in terms of misconfiguration (e.g. insecure ciphers in use for SSL), or vulnerabilities (e.g. version X of package Y is known to be insecure). In doing so it does generate a fairly large number of requests, so the assumption is that this somehow overloaded the device.

Further investigation showed that even SSHing to the array or accessing the serial console was not possible – the login would succeed, but then no prompt would appear.

While it continued to serve iSCSI requests perfectly (and interestingly SNMP remained operational for monitoring), not having access to the management interfaces is a serious problem, as it means it is impossible to perform routine tasks such as managing snapshots, and in the event of a drive failure it will make it very difficult to determine what has occurred and take the necessary actions.

After some research, some discussions with Dell technical support, and some further investigation, we discovered a very simple solution. While CLI access to the array (via SSH / serial console) as either the ‘grpadmin’ user or a RADIUS account was not operational, we discovered it was possible to log in to the array as the ‘root’ user and get a root prompt. The password for the ‘root’ user will likely have been set when the array was first commissioned – if you don’t know what it is then one suggestion would be to try your ‘grpadmin’ password.

Once there, it was then simply a matter of running the following command to restart the management interfaces (note this does not restart the filer, just the management tools so there should be no interruption in service):

eqlinit restart-snap netmgtd

This command completed essentially immediately, and the management interfaces became operational once more, allowing us to carry out some checks to ensure everthing was as expected.

While we need to point out that this information should be used at your own risk (and in particular if you have a support contract then we would always advise talking to Dell technical support prior to carrying out any unusual actions such as this), we hope it might come in useful in avoiding having to take more drastic actions such as causing a controller failover, or hard power cycling the filer, with all the associated risks of data corruption etc.


Windows DHCP server problem on Xen VM hosts

Posted by Christian Ashby on May 6, 2010

If your XenServer hosted Windows DHCP server(s) are running on the same physical host as linux DHCP clients, then they will not receive a DHCP address.

To work around this problem, turn off checksum offloading on the network adapter. To do this, follow these steps:

  • Click Start, click Run, type regedit, and then click OK.
  • Locate and then click the following registry key:
  • In the right pane, make sure that the DisableTaskOffload registry entry exists. If this entry does not exist, follow these steps to add the entry:
    • On the Edit menu, point to New, and then click DWORD value.
    • Type DisableTaskOffload, and then press ENTER.
    • Click DisableTaskOffload.
    • On the Edit menu, click Modify.
    • Type 1 in the Value data box, and then press ENTER.
    • Exit Registry Editor.

Note that this doesn’t effect Open-Source Xen as it is specifically related to the XenServer-supplied PV drivers.


Remote desktop ‘Exceeded the number of connections’

Posted by Christian Ashby on May 4, 2010

You can optionally connect to the ‘console’ which is a screen 0 (by default screens 1 and 2 are the screens you connect to with a remote desktop client). This option is not in the Windows UI, you need to use the following command from Start->Run:

mstsc -v:{machine address} /f -console

Windows Vista / Windows 7 use the syntax /admin rather than -console.

If you use a mac, you can put /console at the end of the machine name in the machine box.

If you use linux and rdesktop then add -0 to the rdesktop command line.


BASH if statements: ‘too many arguments’ error

Posted by Christian Ashby on April 28, 2010

If you are writing a BASH script which searches for a file pattern in a folder using this syntax:

[-f {pattern}]

Instead, use the following syntax:

files=$(ls {pattern} 2> /dev/null | wc -l)
if [ "$files" != "0" ]

This can be replaced with a similar command using find if required.


Debian: apt-get interrupted, leaves files in bad state

Posted by Christian Ashby on April 19, 2010

For those using Debian or Ubuntu Linux, there are situations where the package management system can leave files open if a certain package fails, or if you cancel the installation using CTRL+C, you may see the following error when you try another package installation:

debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unavailable

Using the following command will tell you what’s using it:

fuser -v /var/cache/debconf/config.dat

This will give you the process ID in the right-hand column. You can then use kill {process ID} to remove this process and re run your apt-get command.


Converting open-source Xen Windows VMs to XenServer

Posted by Christian Ashby on April 13, 2010

A number of people are looking to migrate from the open-source Xen to Citrix XenServer itself, and it’s not immediately obvious how to migrate Windows VMs between the two platforms. Citrix have a tool called ‘XenConvert’ designed to move machines between environments, and the way you can use this to move from open-source Xen is as follows:

  • Download XenConvert from the Citrix downloads centre.
  • Add a virtual disk to the guest machine (at least twice the size of the existing disk) to receive the exported VM.
  • Run the original guest machine and partition / format the new virtual disk (NTFS / FAT, as long as you can mount these filesystems in the host).
  • Install & Run XenConvert on the source guest machine.
  • Select ‘XVA’ and select the new virtual disk as the destination for the XVA image.
  • Shut down the source guest machine.
  • Work out where the new virtual partition starts – something like this:
    root@host:/etc/xen# parted /dev/host-vm/name-XVA
    GNU Parted 1.7.1
    Using /dev/mapper/host--vm-name--XVA
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) unit B
    (parted) print
    Disk /dev/mapper/host--vm-name--XVA: 26843545599B
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Number  Start   End           Size          Type     File system  Flags
    1      32256B  26839088639B  26839056384B  primary  ntfs
  • Mount the new virtual disk – change ‘offset’ to be the value in the Start column above, replace ntfs with vfat if FAT32 was used:
    mount -o ro,offset=32256 -t ntfs /dev/host-vm/name-XVA /mnt/t
  • Copy the *.xva folder from this mounted drive to somewhere accessible to the new XenServer host.
  • Run XenCenter on a machine and connect to the new host.
  • Import a new VM, select the ova.xml file from the *.xva folder, and follow the import – you’ll need to setup a new Network Interface.
  • If required, reactivate Windows on the new guest as it starts up, and install XenServer Tools.


Oracle XE: Clearing out unwanted trace files

Posted by Christian Ashby on April 9, 2010

If left unchecked, Oracle XE installations can balloon in size quite quickly – this is due to the trace files being written by the server. The following run in a cron script can be used to remove files more than 7 days old.

find $ORACLE_HOME/../../../admin/$ORACLE_SID/bdump -name "*.trc" -mtime +7 -exec rm "{}" \;
find $ORACLE_HOME/../../../admin/$ORACLE_SID/udump -name "*.trc" -mtime +7 -exec rm "{}" \;
find $ORACLE_HOME/../../../admin/$ORACLE_SID/cdump -name "*.trc" -mtime +7 -exec rm "{}" \;

This tip was found and modified for XE in the following a useful article ‘Oracle Linux – Using the “find” command to manage files’.


Blocking Internet Explorer updates

Posted by Christian Ashby on April 2, 2010

If you are developing websites, or simply don’t want to change browsers, you can stop Microsoft automatic update from installing IE 7 or IE 8 by installing the following updates from Microsoft:

Once run, they will extract files to the location of your choice, then you have to use an elevated command prompt and run the command with a /B switch (block) as follows:

IE70Blocker.cmd /B

You can revert to the normal state (allowing Microsoft / Windows Update to replace your browser) as follows:

IE70Blocker.cmd /U

Note that in order to keep IE6 you need to run both 7 and 8 blockers.

Then, you can keep your system completely up to date whilst maintaining the old-school browsers!


Linux: Arrow keys don’t work?

Posted by Christian Ashby on March 26, 2010

Some Linux applications such as sqlplus still use readline to read what you type, which doesn’t properly support arrow keys, home, end, etc in most terminals.

You can fix this with a small wrapper called rlfe (for older Ubuntu/Debian variants) or rlwrap (for RedHat and most new distributions).

Once installed (apt-get install rlwrap for Debian/Ubuntu) you can enable it by default for your user by doing the following:

echo "alias {command}='rlwrap {command}'" >> ~/.bashrc
source ~/.bashrc


Linux and Seagate FreeAgent Drives

Posted by Christian Ashby on March 24, 2010

If you’ve bought a Seagate FreeAgent drive with a view to running it on Linux, be aware that it won’t necessarily work out-of-the-box.

The issue is that after about 1min of use the drive spins down and closes the USB connection, causing mounts to fail and the device to be unusable.

It turns out that the drives include some cunning/annoying (delete as applicable) power saving logic which only correctly works in Windows. The solution is to run the following command as root as soon as the drive is plugged in.

sdparm -clear STANDBY -6 /dev/sdX

If you use your drive regularly then you can make this command run whenever the drive is used, by editing the file /etc/udev/rules.d (you can create the file if it doesn’t exist) and adding the following line:

-c 'echo 1 > /sys/class/scsi_disk/%k/allow_restart'"