Restarting the Equallogic web interface

Posted by Alex Brett on November 27, 2012

It appears to be possible in some situations to overload the management interfaces of a Dell Equallogic filer. The situation we experienced with a client was that after running an internal vulnerability scan (required for PCI:DSS compliance), the monitoring system picked up that the web interface had stopped responding.

The vulnerability scan essentially attempts to determine what software is in use on each device, and look for known issues, either in terms of misconfiguration (e.g. insecure ciphers in use for SSL), or vulnerabilities (e.g. version X of package Y is known to be insecure). In doing so it does generate a fairly large number of requests, so the assumption is that this somehow overloaded the device.

Further investigation showed that even SSHing to the array or accessing the serial console was not possible – the login would succeed, but then no prompt would appear.

While it continued to serve iSCSI requests perfectly (and interestingly SNMP remained operational for monitoring), not having access to the management interfaces is a serious problem, as it means it is impossible to perform routine tasks such as managing snapshots, and in the event of a drive failure it will make it very difficult to determine what has occurred and take the necessary actions.

After some research, some discussions with Dell technical support, and some further investigation, we discovered a very simple solution. While CLI access to the array (via SSH / serial console) as either the ‘grpadmin’ user or a RADIUS account was not operational, we discovered it was possible to log in to the array as the ‘root’ user and get a root prompt. The password for the ‘root’ user will likely have been set when the array was first commissioned – if you don’t know what it is then one suggestion would be to try your ‘grpadmin’ password.

Once there, it was then simply a matter of running the following command to restart the management interfaces (note this does not restart the filer, just the management tools so there should be no interruption in service):

eqlinit restart-snap netmgtd

This command completed essentially immediately, and the management interfaces became operational once more, allowing us to carry out some checks to ensure everthing was as expected.

While we need to point out that this information should be used at your own risk (and in particular if you have a support contract then we would always advise talking to Dell technical support prior to carrying out any unusual actions such as this), we hope it might come in useful in avoiding having to take more drastic actions such as causing a controller failover, or hard power cycling the filer, with all the associated risks of data corruption etc.


 

An update from Loho

Posted by Alex Brett on September 15, 2011

We’ve been a bit quiet recently, and therefore we wanted to provide an update as to what we’ve been up to and what our plans are for the future…

What have we been up to?

It’s been a very busy time for us over the past year, mostly with behind the scenes changes such as:

  • Relocating our equipment to the tier-4 specification BlueSquare:MK facility in Milton Keynes (May 2011)
  • Reviewing and improving our backup and disaster recovery procedures to ensure we can provide the service you expect
  • Lots of development work on our services (see later)

We’ve also been taking on board a number of new clients, for example Spektrix Limited, who provide an excellent web-based box office solution – we host and manage their hardware from our rackspace in BlueSquare:MK, allowing them to concentrate on further development of their application.

In terms of development we have been working on a number of aspects of our VoIP solutions, both back end code to automate many aspects such as integration with carriers and billing improvements, and front end code to improve the user experience and allow you to configure the system as required at your convenience. This is all ongoing, but here are a few screenshots of some of the things we’ve been working on (feel free to get in touch if you want more information on any of the below):

To ensure our development has been going in the right direction, we have been performing significant market research by contacting potential future partners and customers to discuss their experiences with VoIP and ensure our service is tailored precisely to our customers needs.

We’ve also been busy developing the next generation of our SitePBX devices, which are now based on Intel Atom processors rather than the embedded Blackfin hardware used in the past. These new devices are significantly more powerful, so can be used in much larger organisations, and even for smaller organisations they allow us a great deal more flexibility, with only a small increase in cost. The design of these units are still based around the same principles as our previous devices (no / minimal moving parts, low power usage etc) so there is no change there, however we are now able to offer the units in two different case styles – desktop or (by popular demand) rackmount.

What are our plans?

Over the next few months we expect to reach a suitable stage in development of our Distributed PBX service to roll it out on a wider basis, and more information on this will be coming soon. In addition we have a number of smaller projects in the pipeline, and we will be releasing more information on these as soon as we can.

We are also in the process of rolling out the capability to take payments by Direct Debit, and we will be contacting current customers over the next month or two to ascertain if you wish to switch to use this for future invoices – this has the advantage for you that you do not have to worry about posting a cheque / sending an online payment as we handle everything, and for us reduces the amount of time we have to spend checking and chasing payments, allowing us to concentrate on developing and maintaining our services.

As ever, if you have any queries about our services (either as a new or existing customer), or think we might be able to help with an upcoming project, please don’t hesitate to get in touch!

Alex Brett
Director, Loho Ltd


 

SIP load testing with SIPp

Posted by Alex Brett on August 31, 2011

When developing our VoIP services naturally we need to test them. In addition to the obvious functional testing (e.g. making sure that when you tell the system to send calls to your mobile it really does, and not to someone else’s!), we also need to perform load or stress testing to verify the system will not fail under large amounts of traffic.

One of the most useful tools we use for doing this is the open source project SIPp. This is a very powerful tool, and internally we use it with a number of custom scenarios designed to create load on the areas we know to be potential bottlenecks within our system. In this article however I am going to focus on using its built-in scenarios for simple load testing, and for simplicity the example is based on installing onto a fresh Debian Squeeze installation – for other systems similar packages etc will be required.

For best results, install SIPp on another host on the same network as your VoIP server (otherwise any CPU load introduced by your VoIP server might affect how much load SIPp can generate).

1. Install required packages
$ sudo apt-get install build-essential libncurses5-dev

2. Download, extract and compile SIPp
$ wget "http://downloads.sourceforge.net/project/sipp/sipp/3.2/sipp.svn.tar.gz?r=&ts=1314783436&use_mirror=puzzle" -O sipp.svn.tar.gz
$ tar -xzf sipp.svn.tar.gz
$ cd sipp.svn
$ make

3. Set up the SIP server
Note these instructions are for configuring the Asterisk open source PBX, for other platforms you will need to consult the documentation.

First, define the SIP peer by adding to the end of sip.conf:
[sipp]
type=friend
context=sipp
host=dynamic
port=6000
user=sipp
canreinvite=no
disallow=all
allow=alaw
allow=ulaw

Next set up some extensions that we will use to test by adding to the end of extensions.conf (this assumes a default Asterisk installation where the demo context exists, if not then point calls at some other context that has e.g. an IVR menu or similar):
[sipp]
exten => 1001,1,Answer
exten => 1001,n,SetMusicOnHold(default)
exten => 1001,n,WaitMusicOnHold(20)
exten => 1001,n,Hangup
exten => 1002,1,Answer
exten => 1002,n,Goto(demo,s,1)
exten => 1002,n,Hangup

Finally load the new configuration into asterisk:
$ asterisk -rx 'module reload'

4. Start testing

There are various simple tests that can be done without creating your own scenarios, such as:

1. Simple concurrent call test

$ ./sipp -sn uac -d 10000 -s 1001 <asterisk's IP address> -l 10
This will execute 10 concurrent calls (the -l parameter) with each call lasting 10s (the -d parameter in ms) to extension 1001. Note that this simple test does not actually establish an RTP connection, and thus does not actually place full load on the system.

2. Testing with media

$ ./sipp -sn uac -d 10000 -s 1002 <asterisk's IP address> -l 10 -mp 5606
This executes 10 concurrent calls, each lasting 10s to extension 1002 using the ulaw codec.

When running SIPp will display a screen showing various statistics such as the number of calls in progress, the number completed and some information about the SIP messages it has sent. It also shows any errors it has received. To stop a test, simply press ‘q’.

By playing around with the duration (-d) and limit (-l) parameters you can normally find the limit of your system’s scalability. It is also often an idea to leave the test running at a reasonable call level for a long period of time, this will help identify any memory leaks or similar that will likely cause problems over time.

Note that while SIPp will verify that an RTP connection is established, it will not check the quality – the simplest way to do this is to set up your call load using SIPp, then make a manual call through the system to check the quality is acceptable.

Notes

  • If the machine you are running SIPp on has multiple network interfaces, it may not correctly identify which interface to use for the outbound traffic – to correct this use the -bind_local option, e.g. to use the IP address 192.168.1.1 for outbound traffic you would add “-bind_local 192.168.1.1
  • If you stop a test without letting all the calls clean up, and then attempt to start another, the new one may report errors as it receives SIP messages from the server relating to calls initiated by the previous test – it’s always best to let a test fully clean up

 

Windows DHCP server problem on Xen VM hosts

Posted by Christian Ashby on May 6, 2010

If your XenServer hosted Windows DHCP server(s) are running on the same physical host as linux DHCP clients, then they will not receive a DHCP address.

To work around this problem, turn off checksum offloading on the network adapter. To do this, follow these steps:

  • Click Start, click Run, type regedit, and then click OK.
  • Locate and then click the following registry key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
  • In the right pane, make sure that the DisableTaskOffload registry entry exists. If this entry does not exist, follow these steps to add the entry:
    • On the Edit menu, point to New, and then click DWORD value.
    • Type DisableTaskOffload, and then press ENTER.
    • Click DisableTaskOffload.
    • On the Edit menu, click Modify.
    • Type 1 in the Value data box, and then press ENTER.
    • Exit Registry Editor.

Note that this doesn’t effect Open-Source Xen as it is specifically related to the XenServer-supplied PV drivers.


 

Remote desktop ‘Exceeded the number of connections’

Posted by Christian Ashby on May 4, 2010

You can optionally connect to the ‘console’ which is a screen 0 (by default screens 1 and 2 are the screens you connect to with a remote desktop client). This option is not in the Windows UI, you need to use the following command from Start->Run:

mstsc -v:{machine address} /f -console

Windows Vista / Windows 7 use the syntax /admin rather than -console.

If you use a mac, you can put /console at the end of the machine name in the machine box.

If you use linux and rdesktop then add -0 to the rdesktop command line.


 

BASH if statements: ‘too many arguments’ error

Posted by Christian Ashby on April 28, 2010

If you are writing a BASH script which searches for a file pattern in a folder using this syntax:

[-f {pattern}]

Instead, use the following syntax:

files=$(ls {pattern} 2> /dev/null | wc -l)
if [ "$files" != "0" ]

This can be replaced with a similar command using find if required.


 

Debian: apt-get interrupted, leaves files in bad state

Posted by Christian Ashby on April 19, 2010

For those using Debian or Ubuntu Linux, there are situations where the package management system can leave files open if a certain package fails, or if you cancel the installation using CTRL+C, you may see the following error when you try another package installation:

debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unavailable

Using the following command will tell you what’s using it:

fuser -v /var/cache/debconf/config.dat

This will give you the process ID in the right-hand column. You can then use kill {process ID} to remove this process and re run your apt-get command.


 

Converting open-source Xen Windows VMs to XenServer

Posted by Christian Ashby on April 13, 2010

A number of people are looking to migrate from the open-source Xen to Citrix XenServer itself, and it’s not immediately obvious how to migrate Windows VMs between the two platforms. Citrix have a tool called ‘XenConvert’ designed to move machines between environments, and the way you can use this to move from open-source Xen is as follows:

  • Download XenConvert from the Citrix downloads centre.
  • Add a virtual disk to the guest machine (at least twice the size of the existing disk) to receive the exported VM.
  • Run the original guest machine and partition / format the new virtual disk (NTFS / FAT, as long as you can mount these filesystems in the host).
  • Install & Run XenConvert on the source guest machine.
  • Select ‘XVA’ and select the new virtual disk as the destination for the XVA image.
  • Shut down the source guest machine.
  • Work out where the new virtual partition starts – something like this:
    root@host:/etc/xen# parted /dev/host-vm/name-XVA
    GNU Parted 1.7.1
    Using /dev/mapper/host--vm-name--XVA
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) unit B
    (parted) print
    
    Disk /dev/mapper/host--vm-name--XVA: 26843545599B
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    
    Number  Start   End           Size          Type     File system  Flags
    1      32256B  26839088639B  26839056384B  primary  ntfs
  • Mount the new virtual disk – change ‘offset’ to be the value in the Start column above, replace ntfs with vfat if FAT32 was used:
    mount -o ro,offset=32256 -t ntfs /dev/host-vm/name-XVA /mnt/t
  • Copy the *.xva folder from this mounted drive to somewhere accessible to the new XenServer host.
  • Run XenCenter on a machine and connect to the new host.
  • Import a new VM, select the ova.xml file from the *.xva folder, and follow the import – you’ll need to setup a new Network Interface.
  • If required, reactivate Windows on the new guest as it starts up, and install XenServer Tools.

 

Oracle XE: Clearing out unwanted trace files

Posted by Christian Ashby on April 9, 2010

If left unchecked, Oracle XE installations can balloon in size quite quickly – this is due to the trace files being written by the server. The following run in a cron script can be used to remove files more than 7 days old.

#!/bin/bash
find $ORACLE_HOME/../../../admin/$ORACLE_SID/bdump -name "*.trc" -mtime +7 -exec rm "{}" \;
find $ORACLE_HOME/../../../admin/$ORACLE_SID/udump -name "*.trc" -mtime +7 -exec rm "{}" \;
find $ORACLE_HOME/../../../admin/$ORACLE_SID/cdump -name "*.trc" -mtime +7 -exec rm "{}" \;

This tip was found and modified for XE in the following a useful article ‘Oracle Linux – Using the “find” command to manage files’.


 

Blocking Internet Explorer updates

Posted by Christian Ashby on April 2, 2010

If you are developing websites, or simply don’t want to change browsers, you can stop Microsoft automatic update from installing IE 7 or IE 8 by installing the following updates from Microsoft:

Once run, they will extract files to the location of your choice, then you have to use an elevated command prompt and run the command with a /B switch (block) as follows:

IE70Blocker.cmd /B

You can revert to the normal state (allowing Microsoft / Windows Update to replace your browser) as follows:

IE70Blocker.cmd /U

Note that in order to keep IE6 you need to run both 7 and 8 blockers.

Then, you can keep your system completely up to date whilst maintaining the old-school browsers!