Network failures and the Dell R620

So we recently installed a Dell R620 server with IP Office Server Edition 9.1. While troubleshooting an unrelated issue we came up with a big problem: The device could no longer be reached on the network! Well that's a huge problem since we have IP phones and all the bells and whistles on this server. Avaya support wasn't able to resolve the issue because...guess what? They couldn't get into it remotely...

So we replaced the server and I took a look once we had it back in the lab. It turns out that the devices had been renamed. ETH0 switched with ETH2 and ETH1 switched with ETH3. By plugging in to Port 3 (which should have been ETH2) and enabling the port (#ip link set up) I was able to get connectivity right away. But how could this have happened, and what can I do to make it work with the correct network port? It turns out there is an issue with the way CentOS uses UDEV to create network interfaces. So how do you fix it? Well it's not too hard. I was able to make it work exactly as it did before by looking at a few files and changing one.

In the root folder there are four interesting files. They are ifcfg-eth0, ifcfg-eth1, ifcfg-eth2, and ifcfg-eth3. These files each contain a single line of text with the MAC address of the server:

HWADDR="b8:2a:72:dc:17:28"

Trust these files. They are the gospel. I was also able to find the MAC address of eth0 (Port 1) printed on the circuit board inside the server. The four addresses should be sequential, but keep in mind that they are sequential in HEXADECIMAL. So...29 does not go to 30, 29 goes to 2a.

So you may wonder, now that we have these MAC addresses, what are we going to do with them?

That's easy. Use your favourite text editor (I personally like NANO) to edit the file /etc/udev/rules.d/70-persistent-net.rules. You'll see your interfaces and, I bet, incorrect MAC addresses. Or at least MAC addresses that are not associated with the correct device name. Simply change the MAC addresses to the correct address for each device and you're on your way.

For example, change:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="b8:2a:72:dc:17:2a", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

 to:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="b8:2a:72:dc:17:28", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Security Certificates - 9.1

With the release of IP Office 9.1 there have been enhancements made to the way security is handled. When deploying an IP Office Server Edition or Select Server Edition for a customer it is best practice to have them provide a fully qualified domain name or a machine name to use for the security certificate. The IP Office can be configured with a valid host name and the certificate can be imported into the Trusted Root Certification Authority certificate store. When accessing the system by the proper host name with the certificate properly stored there will be no security warnings while accessing the page.

When defining the hostname for the IP Office you need to either enter the FQDN that will be used to access the system or the IP address that will be used to access the system. In the case of my example I used the IP address of 192.168.11.11 as the host name. If you are using a fully qualified domain name (FQDN) or a server name (NetBIOS) you will want to make sure it resolves with your DNS server or you will see a certificate mismatch error.

To use a self-signed certificate we will select “Generate New”:

After you click Next you will see the following warning:

The certificate will now be generated. 

Once the certificate has been created it is available for download. For a Windows Certificate Store you need to download the DER-Encoded certificate:

Once you have downloaded the certificate click Apply. The process will take several minutes, after which you will be logged out of the system.  Be sure to add the certificate you downloaded to your Trusted Root Certification Authority. If you're working with a domain this can be pushed to client systems using a group policy, or it can be added to machines individually using the Microsoft Management Console.

IP Office Registry Hacks

It has come to my attention that Avaya often hides functionality in a registry entry. I'm going to keep a list here. As I learn more I will add them. Feel free to comment any that you have tucked away under your cap!

Maximum UMS Users (166 by default):
Under HKEY_LOCAL_MACHINE/SYSTEM/CurentControlSet/Services/MSExchangeIS/ParametersSystem, add a new key MaxObjsPerMapiSesion. Under the new key, create a new DWORD Value objtMesageView, and set the value to three times the required users. For example, to support 500 users, set the value to 1500.

SIP Line Template:
In IP Office 9.1 the option for SIP Line Templates was included out of the box. In 7.0 and up the option is still there, but it's hidden. There is a two-part step to enable this:
Navigate to File --> Preferences on the IP Office Manager and select the Visual Preferences tab. Check the Enable Template Options box.
Under HKEY_CURRENT_USER/Software/Avaya/IP400/Manager and add a DWORD value TemplateProvisioning and set its value to 1. Reboot the server hosting the IP Office Manager.
You can now generate a SIP Trunk template

Upgrade: File Access Error

I am in the middle of programming up an IP Office and I came across an issue with the phones. I plugged a phone in and I received the following message:

Upgrade: File Access Error

I did some digging and found the solution. I decided to share it here.

This issue comes from the fact that the phone is not able to get the firmware from the system or wherever it is configured to go for the update. In my case I had configured the IP Office to retrieve its firmware from Manager, however I had the wrong IP Address in Manager. Basically what happened is the phone went looking for the firmware and couldn't retrieve it from the identified location. The solution is quite easy. In my case I just changed the Phone File Server Type to Memory Card. The other options would be to configure the Manager PC IP Address to reflect the location of the correct version of Manager (Making sure the IP Office can reach that PC), or to change the type to Custom and specify the IP Address of the server that contains your BIN files.

Hopefully this helps!

IP Office BCM CS1000 Integration with NRS

I recently had to network some BCMs, and IP Office 8.1, and a CS1000 together for four-digit dialing. I figure I will share some of my knowledge here. With the release of IP Office 9.0 a Session Manager is required which greatly simplifies the implementation, as long as you know how to route calls using Session Manager.

For the purpose of this article I'm going to assume that everyone knows the basics of how to build a SIP trunk on all three platforms. I will also assume that the appropriate licenses have been purchased and that the BCM is SIP capable.

Let's start with the CS1000. The NRS needs to be configured You will need to create endpoints for the BCMs and IP Offices, as well as any CS1000 locations that you have. The IP Office and BCM endpoints must be configured as static SIP endpoints.Routing entries need to be created for each endpoint to correspond with your dialing plan. It may seem obvious, but you need to build SIP trunks and routes from the CS1000. It is very important to note that the IP Office must be configured in Proxy Mode. The IP Office does not support Redirect Mode. Once you have finished with this you are done with the easy part.

The BCM configuration is also quite easy. Build your SIP trunks on the BCM. Configure your SIP domain and enable RTP keepalives. Calls will be routed on the Private network, so navigate to the Private tab under SIP Trunking. Ensure that your URI map matches the URI Map on your node:

Next you need to configure your routing table. For any BCM or CS1000 endpoint you need to enter the destination digits and direct it to the NRS IP. I also selected MCDN Protocol CSE. Ensure that the port number matches up with what the CS1000 is expecting. For each IP Office configure the destination digits per your dialing plan and use the IP address of the IP Office.

And now on to the fun piece. Configuring the IP Office is the most time consuming part of the whole setup. You need to configure three items: SIP Lines, Incoming Call Routes, and Short Codes.
For your SIP Lines you need a connection to each networked system. For simplicity I chose to use the same incoming call route, however the Line Group ID needs to be different for each site. In my example I used SIP Lines 17-20. All my Incoming Call Groups were 17, however I matched up the Line Group ID with the line number. Since these are IP trunks you can configure them however you like as long as they don't interfere with any other lines in the system. For the SIP line to the CS1000 use the Node IP address. For the BCM SIP Lines use the IP address of each individual BCM. Configure each SIP line with the maximum number of calls equal to the maximum number of SIP Line licenses in the system. This way each site is capable of having the maximum number of VoIP calls, as long as no other licenses are in use.
If you used the same incoming call route for each SIP trunk you only need to create a single incoming call route for IP calls. For the destination use a period (.) to have the call sent to whichever digits are being sent from the far end.
In order to make outgoing calls work you need to create short codes on the system to match the dialing plan. In my case we had four-digit dialing with each site having a unique first digit. This made it easy, I configured a short code for each first digit (i.e. 1XXX, 2XXX, 3XXX, etc.). Each short code was configured as Dial 3K1 using the appropriate line group for the far end site. The BCMs will not understand the standard dial string sent by the IP Office so you need to configure the phone context in the telephone number. In my example I showed the Private/CDP URI as cdp.udp so I needed the following Telephone Number in my short code:

.";phone-context=cdp.udp"

Once all of this is done just go ahead and commit your changes to the IP Office. It will probably require a reboot as you are changing IP information. Once everything comes up go ahead and make your test calls and give the customer a big smile on your way out the door, you're their hero.

As always feel free to comment if you have any questions. I'll always do my best to help!

IP Office Initial Startup

When powering up an IP Office for the first time a number of things are configured automatically. It is important to remember to power up all expansion cabinets first as the IP Office will reach out to all connected modules to identify them. Without power these devices can't be identified and the IP Office will not configure them.

The IP Office will automatically build extensions and users for each extension port. This does not require the phones to be connected, simply that the port is installed and has power. The numbering starts at 201 and goes up sequentially, going from left to right across each port on the main unit and then sequentially up through the expansion modules. A default hunt group is created called MAIN with the number 200. The first ten users are added to this hunt group. By default all voice calls are directed to this hunt group. Data calls are routed to the RAS Access DialIn.

All lines are assigned to Line Group 0 by default. A Short code of 9 is created to provide access to this line group.

Embedded voicemail is configured by default. Every user will be assigned a mailbox and they will be active by default.

The IP Office will configure IP Networking. LAN 1 will have the IP Address of 192.168.42.1 and LAN 2 will be 192.168.43.1. Both networks will have the Subnet Mask of 255.255.255.0 and will have the DHCP server configured with a 200 address range. When connecting to the network the IP Office will check to see if there is another DHCP server on the network. If another DHCP server is active the IP Office will disable the DHCP server for that interface.

The IP Office System Name is configured using the MAC address of the IP Office control unit.

IP Office 8 Licensing

IP Office licensing is rather complicated. I’m not going to spend too much time on it but I would like to point out a few rather important details.

The IP Office Basic Edition is a starter-edition of IP Office that only supports one PRI (Up to 64 total trunks with analog and SIP) and up to 9 Auto Attendants. It does not support any applications or IP telephones.  It can be configured through a web interface or phone-based administration. Up to 100 telephones are supported on IP Office Basic Edition. Six concurrent connections to embedded voicemail are supported with approximately 25 hours of storage.

IP Office Essential Edition offers more flexibility over the Basic Edition. It supports up to 40 Auto Attendants and 8 PRIs. This version offers the benefit of IP telephony, including One-X Communicator, mobile twinning, and remote worker support. The Essential Edition comes with 2 concurrent voicemail connections and 15 hours of storage. This can be upgraded to 4/20 or 6/25. There is a built-in conference bridge that supports up to 128 simultaneous connections. Up to 64 connections are supported in a single conference call.

An IP Office Preferred license requires the Essential license and adds some additional functionality. Instead of the embedded voicemail Preferred Edition allows the use of Voicemail Pro. This can be on a standalone server or on a Unified Communications Module (UCM). Voicemail Pro adds significant functionality to the IP Office, including unlimited multi-level auto attendants and up to 40 simultaneous VM connections. The amount of storage available for VM Pro depends on the size of the hard drive on the VM server. A good benchmark is that each minute of voice stored will take up about one MB of hard drive space. VM Pro also includes call recording and an enhanced conference bridge. A single IP Office Preferred License can supply VM Pro to multiple sites in a Small Community Network configuration. VM Pro also adds support for web-based voicemail retrieval. The VM Pro server can be installed on a Linux-based or Windows-based server. The UC Module is Linux-based. When installing on a Windows server all Microsoft licensing requirements must be met for the server platform.

IP Office Advanced Edition provides all of the features of the Preferred license and adds a number of contact center features. Advanced Edition supports up to 150 agents and 30 supervisors in a contact center environment. The Advanced Edition can provide access to real-time and historical reporting to manage efficiency. Recording of all calls can be configured with sufficient storage on your server.  You can configure your menus to interact with an SQL database to provide a better interactive customer experience or to offer self-serve options. Visual Basic scripting is supported with a Windows-based Advanced Edition server.

In addition to platform licensing you must also have a license for all your IP endpoints and trunks.  A basic user (Digital or analog station) does not require a license. Any IP set requires an IP Endpoint License. There are two types of IP endpoint licenses: Avaya or third party. The appropriate license must be purchased for each endpoint. Additional licenses must be purchased for PRI B-channels above the eight that are included with the PRI daughterboard.

Currently license files are provided through http://adi.avaya.com/ but this will eventually be transitioned over to https://plds.avaya.com/

IP Office Startup and default passwords

When the IP Office is powered on it will look for any attached hardware. This means that any attached hardware should be powered on prior to connecting power to the IP Office. When started up for the first time the IP Office will automatically build extensions and users for any recognized extension port, starting with extension/user 201. The IP Office will number extension from left to right on the IP Office then left to right on any attached modules, starting at Module 1 and working up. A hunt group (number 200) will be created with the first ten users as members. All detected lines are included in Line Group 0 and a short code on 9 is created to provide access to the default routing table. Embedded voicemail is also configured on startup. Every user on the IP Office receives a mailbox. This is also true when Voicemail Pro is enabled.

The system name for the IP Office will be the MAC Address of the LAN1 port. As I previously mentioned the default IP addresses are 192.168.42.1 and 192.168.43.1 for LAN1 and LAN2, with a netmask of 255.255.255.0. A DHCP Server is built into the IP Office that is automatically configured to assign up to 200 IP addresses. The range is 192.168.42.2-201 and 192.168.43.2-201 for LAN1 and LAN2 respectively. If the IP Office detects that there is another DHCP server on the network it will disable the internal DHCP server.

The IP Office also builds a few default usernames and passwords. They are as follows:

IP Office Administration: Administrator / Administrator

IP Office Security Settings: security / securitypwd

Remote Access Dialin: RemoteManager / password

System Password (for upgrades): password (no username)

A number of other defaults are created as well. These include usernames that are used by the system for various functions as well as a few different levels of administration access. You can see the full list of users in the Security Settings. For obvious reasons you should consider changing some of these passwords. Do not change any of the passwords for system-level users (EnhTcpaService, SCN_Admin, IPDECTService, and SMGRB5800Admin). To change these passwords you need to open Manager and click File -> Advanced -> Security Settings. Log in using the Security Settings information. You can change the default passwords and create new users with various permissions from this screen.  In the event that you do not have any passwords for your IP Office you can use a physical connection to the RS-232 port on the back of the IP Office control unit. You can connect to the RS-232 port by configuring a terminal to connect at 38,400/8/N/1, Flow Control Off, TTY or VT100. The command type to use is at, followed by the type at-securityresetall. The IP Office will prompt for a complex response after which all passwords will be defaulted.

IP Office Telephones

The IP Office supports analog, digital, and IP sets. Avaya uses the second digit of the phone model to indicate the phone type. A phone with the numbering scheme x4xx or x5xx is a digital set and a numbering scheme of x6xx is an IP set. Avaya uses the same sets for the IP Office as they do for Avaya Call Manager.

Avaya offers expansion modules known as Button Modules for some of their digital and IP sets.

1400 series sets are digital sets using the Avaya Digital Control Protocol. Sets in this series include the 1403, 1408, and 1416 sets.  The 1416 set supports up to three DBM32 expansion modules with external power. The 1400 series sets have a red light that shows up next to the first line appearance on the phone. These sets do not have self-labeling buttons and require a paper label for the buttons. Labels can be printed from within IP Office Manager using the DESI software available for free download from http://www.desi.com/

9500 series sets are digital sets that have soft-labels. The button labels that are programmed into the IP Office are passed through using the Digital Control Protocol to the phones. Available sets are the 9504 and 9508.

1600 Series sets use the H.323 protocol and are visually similar to the 1400 series sets. They also require the use of DESI labels for button labeling. In addition to the 1600 sets there are 1600i sets that have a larger screen that support non-English lettering.

9600 series sets include the 9620, 9630, 9640, and 9650 IP telephones. There are letters after the model number to indicate functionality. C indicates a colour display, L indicates low power consumption (PoE Class 1), G indicates Gigabit Ethernet. These sets support the SBM32 expansion module and require Professional Edition licensing. 9608, 9611, 9621, and 9641 sets are newer IP phones that are also supported. The 9600 series sets have displays that support soft labels so no DESI strips are required.

The IP Office supports Nortel (now Avaya) 1120 and 1140 sets, as well as the newer 1220 and 1240 sets.

Avaya has three conference phone offerings. The B149 is an analog conference phone. The B159 is an analog conference phone with a USB connector for attaching to a PC or cell phone. The B179 is a SIP-enabled PoE conference phone that supports the G.722 Codec for HD voice quality. The B179 requires an Avaya IP Endpoint license to function and is supported on IP Office 7.0 and higher.

Avaya supports DECT R4 sets with the IP Office. Each base station can support up to eight simultaneous calls. DECT sets use the x7xx numbering scheme.

Avaya Video phones (1010, 1020, 1030, 1040, and 1050) are available. These video phones support internal video calls only (no, your customers won’t be able to see you). The Avaya softphone also supports video calls. The softphone can be installed on a Windows or MAC computer. The installer is located on the Admin CD.

Legacy IP Office phones include the 4600, 5600, 4400, and 5400 series sets that are also supported. These sets are not available for new sales. 

IP Office Hardware

The IP Office 500 v2 is the base system for all new IP Office installations. It supports up to 384 extensions (ports) and up to 1000 subscribers (users).

The IPO500v2 has a LAN and WAN port, however in the IPO programming these are referred to as LAN1 and LAN2 respectively. The default IP addresses for these ports are 192.168.42.1 and 192.168.43.1 respectively, with a netmask of 255.255.255.0.

Up to four base cards can be installed in the system. Each base card can be fitted with a trunk daughterboard, with the exception of the 4-port expansion module.

A Digital Station card supports up to   eight digital stations. Up to three DS card can be included in the IP Office.

A TCM8DS Module supports up to eight Nortel 7000-series digital stations on the IP Office.

Voice Compression Module (VCM) provides resources for use with IP transcoding (same as DSP resources in the Nortel world). There are two VCM base cards – 32- or 64-VCM cards.

A Combo Card supports six digital stations, 2 analog stations, four analog trunks, and 10 VCMs. You can have up to two combo cards in an IP Office. Port 8 on a combo card can also be configured as a powerfail line.

A P-2 or P-8 base card supports 2 or 8 phone sets. The P2 is often used if a trunk module is required without any other functionality as it is the least expensive card available for the IP Office.

The 4-port expansion module allows for additional expansion modules to be connected. It must be plugged into Slot 4 of the IP Office.

The Unified Communications Module is an embedded Linux server that can host Voicemail Pro (up to 200 users and 40 ports) and the One-X Portal. It is managed separately from the IP Office through a web interface on Port 7070. The UCM is not supported with IP Office Basic Edition.

The IP Office supports up to twelve expansion modules. The IPO500v2 has eight ports built into the back for expansion and an optional 4-port expansion module can be installed into slot 4 of the IP Office to add the extra four expansion modules.

Expansion Modules for the IP Office include 16-Port Analog Trunk, BRI So8, 16- or 30-port Analog Station, 16- or 30-port Digital Station, and 16- or 30-port Amphenol Digital Station for Nortel 7000 series phones.

Daughterboards are connected to the IPO Base Cards to allow for trunking. The possible options are 4-port analog trunk, BRI (not usually used in North America), or PRI (1- or 2-port PRI). By default the IPO PRI module only comes licensed for eight B-Channels. Additional licenses will be required to open channels 9-23.

The IP Office has two slots for SD Cards in the back of the unit. The System SD card contains a feature key that is tied to licensing. The IP Office will run for up to two hours with the System SD Card removed in unlicensed mode. IP Office voicemail is hosted on the SD card so voicemail will not function while the card is removed. A secondary SD Card can be used to back up data and host voicemail. The secondary slot supports SDHC Cards 4GB and greater. 

IP Office Music on Hold

I've seen a lot of questions regarding Music on Hold on the IP Office. It's not nearly as simple as music was on the BCM but it's still quite easy. Starting with IP Office 3.1 you can use an external source There's always the easiest solution - plug it in to an audio source. There is a 3.5mm jack on the IP Office Control Unit. This will allow you to play anything and is a frequent solution for people who are happy to have the radio play as their hold music. There is no real effort required here - you just plug it in and point the IP Office to use this source and PRESTO! you have hold music. You can also configure an internal analog trunk as a music source. I've never seen this implemented myself, however my experience is quite limited. I'm guessing this would be an easy way to integrate a third party MoH source. This is supported on IP Office 6.1 and up. The way that seems to confuse people is using the internal source. This is a WAV file that is housed on the IP Office. There are some rather severe limitations here. The IP Office 500 and 500 V2 can support up to 90 seconds of recording time. The IP401 does not support an internal music source. Up to Release 4.2 there was only one supported source. In release 4.2 and up there are up to four supported sources, This is also the release in 90 second length is supported. Prior to Release 4.2 you can only have up to 30 seconds. Now on to loading the hold music. The IP500 V2 system can have the hold music loaded during the initial setup. I'm copying this right out of the Avaya documentation:

By default the IP Office will use internal music on hold by uploading a music file from the IP Office Manager PC. For IP500v2 systems, you can load a file onto the System SD card prior to installing it in the IP Office. The file must be of the following format and must be called holdmusic.wav. 1. Rename the music file holdmusic.wav. 2. Using a card reader, copy the file into the /system/primary folder on the System SD memory card. 3. If the IP Office is or will be configured for additional hold music files (up to 3 additional files), copy those files to the same location. The name of the additional files must match those specified in the IP Office system's configuration.

The format of the Wav file is important as well. In order to be played by the IP Office the file must be recorded with a bitrate of 128kbps and be recorded in a single channel (mono). The sample size must be 16-bit with an 8Khz sample rate. 

Post-install the Music on Hold can be added using Embedded File Management. Just switch to Embedded File Management mode and copy the file to the IP Office SD Card and upload holdmusic.wav to the PRIMARY folder on the SD Card. The other method of loading the IP Office hold music is to use the TFTP Server. When the IP Office is rebooted it will always look for a copy of holdmusic.wav on a TFTP server (if present). The IP Office Manager acts as a TFTP Server while running so this is an easy solution that requires no additional software. The file can just be placed in the working directory of the IP Office Manager software. If you need assistance in configuring additional MoH source files (IPO 4.2+) it is in the documentation. I hope this little bit helps people with their questions.

The IP Office supports multiple hold music selections, based on the incoming call route. You can use a WAV file or an analog extension to provide the music. In order to use different WAV files you need to add them to the control unit using embedded file management. You can enter a friendly name for the source. The source number is specified by the system, starting with 2.

Specify a WAV file using WAV:filename.wav. Specify an extension using XTN:222 (replace 222 with the extension number). When using an extension you need to change the extension to show that it is a music on hold source.

Once you have added your sources you can add them to the incoming call route: