IP Office Server Edition/App Server/UCM Can't escalate to root

Avaya, in their infinite wisdom, decided to take away root access over SSH connections. This was put into place for versions 10.1 SP7, 10.1 SP8,, and 11.1. Of course, this makes it very difficult to provide remote support!

Naturally Avaya's customer base revolted at this, so Avaya will be bringing back root access for upcoming releases. Well, that doesn't help if you're stuck now! To that end I've figured out what was blocking it and created a simple workaround. It will require direct access to the system. If you're using VMWare this can be through the VMWare console, otherwise you will need physical access to the server. If you're working on a UC Module remember that the HDMI port will not be active until you reboot. You should be able to execute the commands without a monitor, but being able to see what you're doing is a lot easier. Follow these steps to allow SSH access to root. You will still need to log in as Administrator when connecting via SSH, but you will no longer get the dreaded 'incorrect password' error.

  1. Log in to the server (direct-connect) as root, using the Security password
  2. enter the command usermod -a -G wheel Administrator 
That's all it takes! Now you should be able to SSH to the system and log in as Administrator. Once logged in enter the command admin, then log in again. Now you can enter root and log in with the security password.

IP Office Manager Upgrade Fix

IP Office Manager 11.0 FP 4 SP 1 has a bunch of 0-byte files. When Manager tries to transfer these files to the IP Office the IP Office’s internal file server rejects the files because they are invalid. This should be fixed in future releases, but until it is there is one published workaround from Avaya: recreate the SD Card. This is a pain and requires physical access to the system, so I have come up with a better way to fix the problem.

The error I see most often is 400 - Bad Request

I have created a zip file containing a number of replacement files. You can download the file from my Google Drive here.

Once you have downloaded the fix you can extract the contents of the IPO11Fix.zip file:

Copy the folder system (right click and copy, or left click and then CTRL+C):
Navigate to the IP Office installation directory and find the common memory card folder (in a default installation that would be C:\Program Files (x86)\Avaya\IP Office\Manager\MemoryCards\Common\). Paste the system folder that you had copied into this folder and overwrite all the files when prompted:

You may be prompted for Administrative access. Provide the access if requested:

Upgrading the IP Office

I'm not going to tell you how to upgrade your IP Office, that's what the official training is for. I am, however, happy to discuss the various issues that I have experienced while performing upgrades, and the steps I take to ensure that the upgrade goes smoothly.

File Write Errors

I often see people struggling when transferring files to the IP Office during an upgrade. More times than I'd like to count I've seen the system pop up with an error stating "405 Method Not Allowed".  While there are a number of things that can cause this issue the most common, by far, is the File Writer IP field in the system tab. If you're getting this error just check there and see if there is an address populated. I usually set the address to, which allows the writing of files from any address. While this may present a security risk, as long as you follow strong password management and don't leave any of the default passwords in place the risk should be next to zero. Alternatively you could specify the address of the PC being used to perform the upgrade, restricting the ability to write to the memory card to just that device. The third option is to set the address to, which will cause the system to update the address in this field the next time embedded file management is used to reflect the address of the system that is using embedded file management.

In some cases, even after making the change to the File Writer IP address, some installations still don't proceed. In these cases I've seen a couple of solutions. In some instances the system will allow the completion of the upgrade to the core equipment, and not the update of the files on the SD Card. When this happens it is often possible to simply log in to Embedded File Management and upload the system files from there. If that doesn't work, however, the cause is usually a corrupted SD Card. This happens most often on systems that have been through a number of upgrades, or potentially if you've forgotten that you have to step up from any release below 8.1(65) to 8.1(65) or 9.0 prior to upgrading to the current release. When this happens the best plan is to pop out the SD Card, put it in your laptop, and recreate the SD Card. Once this is complete pop it back in to the system and push your backed up config (and any embedded voicemail files, if applicable) back to the system. Once this is complete you should be able to access the system just fine.

Server Edition and Application Server Upgrades

Server Edition and Application Server upgrades appear to take forever, frequently appearing to stop around the 92% complete mark. This has caused panic for so many of my colleagues over the years. The first time I encountered this I panicked and rebooted the server. Let me tell you, this is a mistake! Now I've learned a thing or two and I have found a better way to keep an eye on the progress than simply looking at that 92% for what seems like an eternity. Log in to the server as root and type the following command:
tail -f /var/log/yum.log
This command will continuously scroll the output of the YUM application. YUM, or the Yellowdog Updater, Modified, is the Linux program that the IP Office uses to manage the installation of software. The yum.log file will scroll frequently even if the Web Manager progress indicator hasn't moved. This should keep your mind at ease while you wait.

It is possible that the updater may fail. Avaya has documented issues with the upgrade failing, and has even written code into the update software to identify issues with YUM failing to complete the upgrade. If you see the error "yum process died before completing its job" you have experienced an issue with the /boot/ partition being full. This happens, again, on systems that have been upgraded multiple times. If your system has been upgraded to at least 9.1.7 you should not experience this issue. If you do have this issue after 9.1.7 engage Avaya support right away, and anticipate a rebuild. If you are at a release older than 9.1.7 there is a quick fix. Download the file UpgradeKernelFix_v2.zip from Avaya. This zip file contains a patch that will allow you to expand the /boot/ partition. Extract UpgradeKernelFix_v2.sh from the zip file and upload it to your server using WinSCP, or your favorite SFTP client. Browse to the location of the file as root and type chmod +x UpgradeKernelFix_v2.sh - this makes the file executable. Run the file with ./UpgradeKernelFix_v2.sh and you should be off to the races. Of course, you will have to start the upgrade again. This issue should only affect systems running on VMWare, however I wouldn't rule it out for bare metal servers. 
You can download the fix from Avaya's web site here.

I can not stress enough the importance of making a full backup before you begin. For an application server this means assessing the applications used and ensuring that all application data is backed up off the system before you begin. For systems using embedded voicemail this means ensuring that you save all the dynamic contents of the SD Card before you even think of starting. 

Let me know if you've experienced any other errors, and if so, what is your resolution. I'll try to help out anyone who is stuck if I can!

SIP Standard

The VoIP world is dominated by SIP lately. SIP is a standard that is defined by a number of RFCs but searching through here for answers to questions is ridiculously difficult. I would like to use this post to answer any SIP questions that I come across in my daily travels. If you have a question that has not been answered feel free to drop it in the comments. I'll do my very best to find an answer for you!

Q: What is the maximum size of a SIP packet?
A: The maximum allowed size is equal to the maximum size of a UDP packet, or 65,507 bytes (65,535B - 20 B IP Header -8 B UDP Header). Some software may not accept packets of this size. Often software will limit the packet size to 4096B.

Q: What is a SIP User Agent (UA)?
A: A User Agent refers to a component in the SIP communication between two devices. There are two types of User Agents: Client and Server. A UA Client (UAC) sends a SIP request and a UA Server (UAS) sends a response. Most of the time a SIP server or endpoint will act as both a UAC and a UAS, depending on the circumstance. A call may terminate to a phone in which case the SIP server is the UAC and the phone is the UAS. The phone may then place the call on hold or conference another user, at which point the phone sends the request and becomes the UAC.

Q: What is a B2BUA?
A: A B2BUA (Back to Back User Agent) is a device that receives a request from a UAC and forwards that request out to another device, acting initially as the UAS and as the second leg as a UAC. A B2BUA will remain in the middle of the conversation so that the endpoints do not communicate directly for signaling purposes. It is still possible to have a B2BUA with direct media path for RTP packets.

Q: What is the maximum speed for faxing with SIP?
A: Contrary to what most people believe, there is no difference in the maximum speed of a fax machine on analog or PRI versus SIP. The SIP standard doesn't care - this is a function of the codec negotiation. But that doesn't answer the real question here. When I'm setting up a PBX I always suggest the customer limit their fax speed to 14,400bps. The PBX will see the 14,400 speed (or lower) and will use the T.38 codec for faxing. Most phone systems will only allow T.38 if the PBX will be holding up the RTP (i.e. no direct media path) as the system will switch to T.38 when it detects the fax tones being sent from the originating fax machine. Modern fax machines (also known as Super G3 fax machines) support speeds of up to 38,400bps, although the realized throughput is often lower. When using a Super G3 fax machine the codec will negotiate to G.711. Network reliability is incredibly important when using Super G3 speeds. These machines are incredibly intolerant to jitter and packet loss.