Adding a CertBot Certificate to IP Office

The IP Office generates a self-signed certificate for the purpose of presenting its identity. Any Avaya elements will trust this certificate, but modern web browsers will not by default. One option we have is to add the Avaya Certificate Authority to our list of trusted issuers, but a better way is to ensure that the IP Office presents a certificate that is universally trusted. The Electronic Frontier Foundation (EFF) is providing free publicly trusted certificates with a three month expiration. The short duration is intentional - any malicious use of the certificates wouldn't last longer than three months, making the Internet a safer place. 

The EFF has a utility called CertBot that can be used to generate these certificates on your local machine. They use Public DNS to validate that the correct machine is requesting the certificate. The whole process takes only a few minutes, and when run on a web server can automatically apply the certificates to the hosted pages.Unfortunately Avaya has not yet integrated CertBot functionality into their finished product, so we will need to manually create a certificate, extract the portion that the IP Office needs, and apply the certificate. This will need to be done on a regular basis because of the short certificate duration. Please reach out to your Avaya partner or account manager and ask them to add CertBot integration to the IP Office in future releases! As more people request this functionality Avaya becomes more likely to include it in a future software release. You can download CertBot here. We can't install CertBot directly on the IP Office, so a Linux PC or server will be required. Once CertBot is installed on your Linux PC or Server we can proceed with generating a new certificate. 

The next step is to use CertBot to create a certificate. For the purpose of this example I will redact all my screenshots, hiding the actual FQDN I used. For reference I will use the domain in all of the commands. Simply change out the FQDN for the FQDN you want to assign to your IP Office. Since the CertBot utility uses DNS records you need to be sure to have a DNS A Record for the FQDN you are using. This should point to the public IP Address of the router that the Linux machine is connected to. You will need to enable port forwarding for Port 80 to the Linux machine to get CertBot to work properly since it checks for an HTTP request/response. 

My Linux machine already has a web server running, so I chose to leverage that for the generation of the certificate. To generate the certificate I ran the following command:

certbot run --domains

If you don't have a web server running on your Linux machine you can run the commmand in Certificate Only mode:

certbot certonly --domains

Once the certificate chain has been created you may be prompted to install it on your web server - since we aren't generating the certificate for this machine we should not be applying it to this server. 

Now we have a new certificate. It will be located in the CertBot folder under a subfolder for that FQDN. In our example this full path would be:


The IP Office is unable to use the certificate in this format so we will need to convert it. Change to the folder containing the certificate and extract the PKCS#12 Certificate from the full chain with these commands:

cd /etc/letsencrypt/live/
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out ipo.p12 -CAfile chain.pem

The second line will prompt for an export password. This password will be used to add the certificate back to the IP Office. Once complete the file ipo.p12 will be created. We can apply this certificate to the IP Office. Copy the ipo.p12 file back to your local PC and launch a web browser. The certificate is easiest to apply using Web Manager, so that's the method I will cover here. 

Log in to Web Manger using the Administrator account (other accounts can be used as long as the permissions allow the manipulation of certificates). Select Security from the top menu, then Certificates. 

This will take you to the Certificates portion of IP Office Web Manager. Here you will see a list of all the trusted certificate authorities, as well as some options to manipulate the IP Office certificate. 

The button for Set will allow us to upload our new certificate. Click Set then locate the ipo.p12 file and enter the password set during the extraction process.

The IP Office will advise you that the system and applications will stop responding while the certificate is applied. Click OK and wait for the IP Office to apply the certificate. Once applied you can view the certificate by clicking View.

Now that the IP Office is presenting the certificate to users we need to ensure that this resolves properly. We will still get an error if we try to browse to the IP Office by IP Address, since the certificate doesn't contain the internal IP. The internal DNS Server needs to point to the internal IP of the IP Office. As with many Avaya applications the IP Office performs a reverse DNS lookup to validate the FQDN. If you skip this step the IP Office will server an error page when you browse to it. Make sure that the IP Office is set to use the DNS Server that hosts your internal DNS entries. If you want your IP Office to be accessible to the Internet you can enable port forwarding on your router for ports 80, 443, 7070, 7071, and 8443. Otherwise, disable the port forwarding that was set up for the certificate creation. 

This process is a manual process that can't be automated at this time. You will need to update the certificate before it expires, following the entire process again. While this is not an ideal scenario, it is the easiest way to get free TLS certificates for your IP Office. The same process can be applied to a UC Module or Application Server - even Server Edition. Just follow the steps and remember that each element will require a unique, DNS-resolvable FQDN.

One final piece of housekeeping - the IP Office will, by default, generate an alarm when the certificate is within 60 days of expiring. To change this open the security settings, navigate to System, then Certificates. Change the Certificate Expiration Warning (days) to 30. This is the lowest number possible, so a System Status Alarm (or SNMP alarm if you've configured that) will be generated when the certificate is set to expire. IP Office Manager will also generate an alarm when logging in. I haven't found a way to eliminate this that doesn't do away with all the application security altogether. If you come up with something please let me know in the comments. I will update my post and give credit where it is due.    

IP Office Server Edition/App Server/UCM Can't escalate to root

Avaya, in their infinite wisdom, decided to take away root access over SSH connections. This was put into place for versions 10.1 SP7, 10.1 SP8,, and 11.1. Of course, this makes it very difficult to provide remote support!

Naturally Avaya's customer base revolted at this, so Avaya will be bringing back root access for upcoming releases. Well, that doesn't help if you're stuck now! To that end I've figured out what was blocking it and created a simple workaround. It will require direct access to the system. If you're using VMWare this can be through the VMWare console, otherwise you will need physical access to the server. If you're working on a UC Module remember that the HDMI port will not be active until you reboot. You should be able to execute the commands without a monitor, but being able to see what you're doing is a lot easier. Follow these steps to allow SSH access to root. You will still need to log in as Administrator when connecting via SSH, but you will no longer get the dreaded 'incorrect password' error.

  1. Log in to the server (direct-connect) as root, using the Security password
  2. enter the command usermod -a -G wheel Administrator 
That's all it takes! Now you should be able to SSH to the system and log in as Administrator. Once logged in enter the command admin, then log in again. Now you can enter root and log in with the security password.

IP Office Manager Upgrade Fix

IP Office Manager 11.0 FP 4 SP 1 has a bunch of 0-byte files. When Manager tries to transfer these files to the IP Office the IP Office’s internal file server rejects the files because they are invalid. This should be fixed in future releases, but until it is there is one published workaround from Avaya: recreate the SD Card. This is a pain and requires physical access to the system, so I have come up with a better way to fix the problem.

The error I see most often is 400 - Bad Request

I have created a zip file containing a number of replacement files. You can download the file from my Google Drive here.

Once you have downloaded the fix you can extract the contents of the file:

Copy the folder system (right click and copy, or left click and then CTRL+C):
Navigate to the IP Office installation directory and find the common memory card folder (in a default installation that would be C:\Program Files (x86)\Avaya\IP Office\Manager\MemoryCards\Common\). Paste the system folder that you had copied into this folder and overwrite all the files when prompted:

You may be prompted for Administrative access. Provide the access if requested:

Upgrading the IP Office

I'm not going to tell you how to upgrade your IP Office, that's what the official training is for. I am, however, happy to discuss the various issues that I have experienced while performing upgrades, and the steps I take to ensure that the upgrade goes smoothly.

File Write Errors

I often see people struggling when transferring files to the IP Office during an upgrade. More times than I'd like to count I've seen the system pop up with an error stating "405 Method Not Allowed".  While there are a number of things that can cause this issue the most common, by far, is the File Writer IP field in the system tab. If you're getting this error just check there and see if there is an address populated. I usually set the address to, which allows the writing of files from any address. While this may present a security risk, as long as you follow strong password management and don't leave any of the default passwords in place the risk should be next to zero. Alternatively you could specify the address of the PC being used to perform the upgrade, restricting the ability to write to the memory card to just that device. The third option is to set the address to, which will cause the system to update the address in this field the next time embedded file management is used to reflect the address of the system that is using embedded file management.

In some cases, even after making the change to the File Writer IP address, some installations still don't proceed. In these cases I've seen a couple of solutions. In some instances the system will allow the completion of the upgrade to the core equipment, and not the update of the files on the SD Card. When this happens it is often possible to simply log in to Embedded File Management and upload the system files from there. If that doesn't work, however, the cause is usually a corrupted SD Card. This happens most often on systems that have been through a number of upgrades, or potentially if you've forgotten that you have to step up from any release below 8.1(65) to 8.1(65) or 9.0 prior to upgrading to the current release. When this happens the best plan is to pop out the SD Card, put it in your laptop, and recreate the SD Card. Once this is complete pop it back in to the system and push your backed up config (and any embedded voicemail files, if applicable) back to the system. Once this is complete you should be able to access the system just fine.

Server Edition and Application Server Upgrades

Server Edition and Application Server upgrades appear to take forever, frequently appearing to stop around the 92% complete mark. This has caused panic for so many of my colleagues over the years. The first time I encountered this I panicked and rebooted the server. Let me tell you, this is a mistake! Now I've learned a thing or two and I have found a better way to keep an eye on the progress than simply looking at that 92% for what seems like an eternity. Log in to the server as root and type the following command:
tail -f /var/log/yum.log
This command will continuously scroll the output of the YUM application. YUM, or the Yellowdog Updater, Modified, is the Linux program that the IP Office uses to manage the installation of software. The yum.log file will scroll frequently even if the Web Manager progress indicator hasn't moved. This should keep your mind at ease while you wait.

It is possible that the updater may fail. Avaya has documented issues with the upgrade failing, and has even written code into the update software to identify issues with YUM failing to complete the upgrade. If you see the error "yum process died before completing its job" you have experienced an issue with the /boot/ partition being full. This happens, again, on systems that have been upgraded multiple times. If your system has been upgraded to at least 9.1.7 you should not experience this issue. If you do have this issue after 9.1.7 engage Avaya support right away, and anticipate a rebuild. If you are at a release older than 9.1.7 there is a quick fix. Download the file from Avaya. This zip file contains a patch that will allow you to expand the /boot/ partition. Extract from the zip file and upload it to your server using WinSCP, or your favorite SFTP client. Browse to the location of the file as root and type chmod +x - this makes the file executable. Run the file with ./ and you should be off to the races. Of course, you will have to start the upgrade again. This issue should only affect systems running on VMWare, however I wouldn't rule it out for bare metal servers. 
You can download the fix from Avaya's web site here.

I can not stress enough the importance of making a full backup before you begin. For an application server this means assessing the applications used and ensuring that all application data is backed up off the system before you begin. For systems using embedded voicemail this means ensuring that you save all the dynamic contents of the SD Card before you even think of starting. 

Let me know if you've experienced any other errors, and if so, what is your resolution. I'll try to help out anyone who is stuck if I can!