Category Archives: voice

How to automatically extract data from an Avaya Communication Manager

You might call this post “the secret of my success” as a PBX admin for Avaya Communication Manager. Avaya provides some utilities for their techs and business partners, but customers don’t really have a lot of tools to easily extract data. In this post, I’ll show you how to extract ANYTHING from Avaya from a command line so it’s suitable for scripting. In my case, I parse the data further and generate a list of all stations and report of changes to my PBX every 15 minutes. Have you ever wondered who used to have extension 4438? How about that phone for the intern over the summer named “Bodkin Van Horn” that you deleted but now he has been hired and you want to give him the same extension? How about the question “how long has this extension been unplugged”?

Back in mid 00s, I wrote some PHP integrate with CM using telnet. Since I didn’t know much about PHP’s ANSI support, I ended up writing my own ANSI parser of the raw data. Anyway, it was great, but many customers don’t support telnet and want all communication to CM to be through ssh, which is reasonable, right?

Then I found this post from Benjamin Roy where he provided a perl module to do it through ssh. Also, he meticulously reverse-engineered Avaya’s OSSI protocol, which I think is impressive and sounds like fun. But I didn’t use OSSI; I just wanted a way to get data using ssh. So I took Ben’s code and tweaked it a little bit. It’s straight perl code – it’s not a module or anything like that. It does have dependencies on a couple other modules though. You do not need to be a perl developer to use this. It’s really easy. Here is a link to the script. Just save this as “av.pl”. Try to run it with “perl -c av.pl” (the -c means just check to make sure it compiles but do not run it). You might get some errors about missing modules. There are only three, and this is how you install them if you need to:

perl -MCPAN -e "install Expect"
perl -MCPAN -e "install Data::Dumper"
perl -MCPAN -e "install Term::VT102"

I should probably mention here – if you don’t have a Linux server in your environment, you should really get one. Your Virtual Machine team can make one for you – just about any distribution of Lunux will do and they take almost no resources. If you need to justify it, just say something like “utility server for PBX monitoring” or something like that.

Anyway, back to the script. Naturally, it needs to connect to your PBX, so create a little text file like “mypbx.txt” with the contents

10.10.40.89,5022,cmusername,Passw0rd,ssh

Obviously, replace the IP address, username, and password of your PBX.

And then create another file called “commands.txt” with the contents

list trunk
display station 3100

This can contain whatever commands you’d like. list reg, list station, list trunk, list locations, list measurements, stat cdr, etc. It should be any command that can be “paged” through, you know?

And then simply run this line

perl av.pl mypbx.txt commands.txt

The script will connect to the PBX defined in mypbx.txt, run the commands in commands.txt and output the results to STDOUT!

This is the core script to a bunch of stuff I automate in my PBXs. In a later post, I’ll show you how I parse the results with Perl and make use of the data.

I call this “the secret of my success” because I use this data to build a simple html page of all extensions in all of my PBXs every 15 minutes. I keep this page up all the time in a browser tab, which enables me to simply “control-f” for any name in the PBX and I have the extension number and the IP address of the station (or ‘unregistered’ if it’s unplugged). I can see unregistered ‘guest’ numbers, see the station’s history, etc. There’s a lot more I do with this information and I look forward to showing you.

Please let me know what you think! And if there’s anything I can do to help you get it working.

Lastly, if any of you need to do any telephone system testing (capacity, QA, DID number ports, call-flow, queuing, etc), please check out my post about CallsByCloud. If you sign up and use the promotion code ‘roger2015’, you’ll get $10 in credit and a 20% discount on the per-minute rate.

Also, if you hate unsolicited outbound telemarketing, consider subscribing to the Jolly Roger Telephone Company. Your telemarketers will entertain and delight you. Sample recordings here.

Thanks all!

Roger

 

Roger speaks at TEDx Naperville about disrupting the telemarketing industry!

Way back in July of 2014, I wrote about how I stopped telemarketing at my house. Little did I know that my efforts to combat and disrupt unsolicited telemarketers would eventually lead to me giving a TEDx talk about it!

Some of you may know that, in an effort to stop telemarketing, I created a robot to talk to them. It’s an interesting story and I posted some audio at the beginning of 2016. My story was picked up by Gizmodo on February 1, 2016 and things got really crazy from there.

Anyway, Fast forward to November 4, 2016 and I find myself on stage at TEDxNaperville in Illinois speaking in front of about 500 people at a beautiful venue. I’ve never spoken in front of a crowd like that, and it was amazing.

Don’t get me wrong – as a phone guy, I love call centers. But as a phone guy I also love the telephone network. The Public Switched Telephone Network has been around for about 140 years and it is crushed under the burden of unsolicited telemarketing. Blocking doesn’t work. Their predictive dialers simply call the next sucker. I think the only solution (short of abandoning the entire network) is the Jolly Roger Telephone Company. Here is me story about how I’m disrupting the scammers, spammers, and crooks, which make up most of the unsolicited telemarketing industry.

By the way, this goes for cold callers too. If your business is struggling to handle cold callers (ask your receptionists, Investor Relations team, or any admins for your CxOs) then I also have a solution for you. It’s called a Biz-Bot and it will handle all your pesky cold callers for all your employees.

But anyway, here is the talk!

Avaya Call Detail Recording to Kiwi syslog server

In a previous post about busy/release of CDR, I asked if anyone was curious about capturing CDR from Avaya Communication Manager using a Kiwi Syslog server. At least one of you was, so here it is.

There are two parts of course. One part is programming the phone system to send CDR. And the other part is programming Kiwi to capture CDR. These can be done in either order, but if you program the phone system first, the CDR buffer will fill as is tries to establish the connection with Syslog.

So let’s start with Kiwi. Your version may differ. In my case, the network team already had a rather powerful installation of Kiwi on some great hardware. All I had to do was configure it to capture Avaya CDR. So when you launch the Kiwi dashboard, click File->Setup and in the “Server Setup” window, scroll to the bottom for “Inputs”.

kiwi-config-0

In the TCP section, tell Kiwi to listen for TCP messages on whatever port you wish. I’m using 5013 – probably because I saw it in an Avaya doc somewhere. But you can use any port.

Then scroll up to the “Rules” section and configure a new rule to match for messages from your Avaya Communication Manager.

kiwi-config-1

In my case, I named my rule “Call Records from LA PBX”. I set a filter by IP address. If you also send any other information to Kiwi from the CMs, then you’ll want to filter by the Priority field. You want the Local7 field to be Debug. I don’t recall why this works. I think I was looking at the data one day to try to separate the CDR from the other syslog messages from Avaya and this seemed the easiest way to do it.

kiwi-config-3

Once you have those two filters – IP address and Local7.Debug, you need to create an action. My action above is a little misleading. All you really need is one that logs to a file, then stops processing. If you’d like, you can also display it to one of the virtual displays in Kiwi. I log to a text file on the local file system:

kiwi-config-2

Now, if we program our PBX correctly, we should get CDR to these text files!

Step 2 – Program the Avaya Communication Manager

Programming the Communication Manager is three steps: Add the node-name, set the ip-services, and change the system-parameters cdr.

To set the node name, ‘change node-name ip’ and create an entry for the IP address of your Kiwi server. This is basically the DNS lookup for the CM. This sets the name and IP address. Any name will do. Enter the IP address of your Kiwi server.

change node-names ip                                            Page   1 of   2
                                  IP NODE NAMES
    Name              IP Address
SL_LSP              10.10.60.10
CHA_LSP             10.10.1.10
Chicago             10.10.148.2
Chicago_LSP         10.30.10.10
DO_LSP              10.20.16.10
Smoot               10.87.176.11
Gateway006          10.46.12.1
LA                  10.10.19.6
LA_ESS              10.44.192.10
LE_ESS              10.58.200.10
LA_SM               10.59.29.16
( 16 of 44   administered node-names were displayed )
Use 'list node-names' command to see all the administered node-names
Use 'change node-names ip xxx' to change a node-name 'xxx' or add a node-name

Next, set the ip-services. I have squeezed all three screens here. The important thing here is page three. We need to set the “Reliable Protocol” to n. I think Avaya defaults this to y and it works great for their pre-packaged CDR partners. For home-grown syslog, we set this to n. It may not hold up in court if there’s ever a dispute.

change ip-services                                              Page   1 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port
CDR1                 procr            0       la_syslog       5013
CDR2                 procr            0       prognosis       50000

change ip-services                                              Page   2 of   3

                                   IP SERVICES
 Service     Enabled     Local        Local       Remote      Remote
  Type                   Node         Port        Node        Port


change ip-services                                              Page   3 of   3

                              SESSION LAYER TIMERS
  Service     Reliable  Packet Resp   Session Connect  SPDU  Connectivity
   Type       Protocol     Timer       Message Cntr    Cntr     Timer

 CDR1            n         10                3          3         10
 CDR2            n         10                3          3         10


And last step – set the system-parameters for CDR. Here’s the thing. You have lots of options here. You can define fields, separators, etc. But I discovered that there is a line length limit that is not obvious. Also, the caller-id name is surprisingly absent. Rather than customize the CDR, I have gone back to “unformatted” and let further processing figure it out.

change system-parameters cdr                                    Page   1 of   1
                            CDR SYSTEM PARAMETERS

 Node Number (Local PBX ID): 2                     CDR Date Format: month/day
      Primary Output Format: unformatted   Primary Output Endpoint: CDR1
    Secondary Output Format: unformatted Secondary Output Endpoint: CDR2
           Use ISDN Layouts? n                   Enable CDR Storage on Disk? n
       Use Enhanced Formats? n      Condition Code 'T' For Redirected Calls? n
      Use Legacy CDR Formats? y                 Remove # From Called Number? n
Modified Circuit ID Display? n                             Intra-switch CDR? y
                  Record Outgoing Calls Only? n     Outg Trk Call Splitting? y
  Suppress CDR for Ineffective Call Attempts? n       Outg Attd Call Record? y
      Disconnect Information in Place of FRL? y      Interworking Feat-flag? n
 Force Entry of Acct Code for Calls Marked on Toll Analysis Form? n
                                    Calls to Hunt Group - Record: member-ext
Record Called Vector Directory Number Instead of Group or Member? n

     Inc Trk Call Splitting? n
  Record Non-Call-Assoc TSC? n           Call Record Handling Option: warning
      Record Call-Assoc TSC? n   Digits to Record for Outgoing Calls: dialed
   Privacy - Digits to Hide: 0               CDR Account Code Length: 15
Remove '+' from SIP Numbers? y


I have a perl script that parses unformatted cdr. Anyone want to see it?

Anyway, when you have this set up, you can status CDR to make sure the link is up:

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: up                           up

      Date & Time: 2016/04/17 08:43:29          2016/05/09 10:12:20
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer % Full:   0.00                         0.00
      Reason Code: OK                           OK





The good news here is the link state(s) are “up”  and the CDR buffer % full is 0. If you have trouble, you can “busy cdr primary” and “release cdr primary” to close and open the socket. You should see a blip in the Kiwi server when this happens.

Make a call and watch Kiwi! You may need to check the log files rather than rely on the Kiwi display. The extra CR/LF will sometimes make Kiwi look funky but the text file is usually just fine.

Let me know how it goes for you!

Roger

 

How to load-test your telephone system, IVR, or DID number port

When you work on telephone systems, at some point you need to make some test calls into your trunks. And sometimes it’s not realistic to use the same phone at your desk, since the call may not truly leave your PBX or your carrier’s network. So you end up using your mobile phone,or if you’re remote, then your home phone. However,

  • What if you need to make dozens of calls?
  • What if you need to load up a trunk group?
  • What if you need to test a number port of hundreds of individual DIDs?
  • What if you need to simulate calls from different areas of the country?
  • What if you need to dial an extremely specific sequence of digits once connected, and you need to do this over-and-over-and-over?

About a year ago, I found a great tool called CallsByCloud.com. This is an automated testing framework – completely self service using your browser. I liked it so much I became a business partner recommend it highly to all of you telecom administrators. I even recorded the training videos for them.

It’s very straight-forward to use. You build a “test scenario” that contain simple “scripts” that are use cases for each type of call. If you’re testing a call center IVR, this could be “make payment” or “renew contract” or “queue for customer service”. And if you’re just testing a trunk group, you can dial into your PBX and “hold” the channel for x seconds. This lets you load the trunk group and test overflow.

cbc-script-commands

Once the tests are complete, you can see the call history, channel usage, a spectrogram of the call, and you can listen to or download the audio.

cbc-history

Once you set up a test, you can schedule it to run at a recurring time and alert you if there are any problems with the call.

cbc-qa-schedule

There are other advanced features such as

  • Caller-id manipulation to simulate geographic routing.
  • Detailed logs of the call processing.
  • Number pooling for testing multiple numbers.
  • The ability to set a “disposition” or “goto” a different spot in the script based upon the timing of noise, silence, or DTMF in the channel.
  • The ability to export call records for the tests.

It’s flexible, cost effective, and has made my job so much easier. I recommend it highly to all of you. If you sign up and use the promotion code ‘roger2015’, you’ll get $10 in credit when you sign up and a 20% discount on the per-minute rate.

Check out the training videos and let me know what you think!

Thanks for watching!

Roger

 

How to reboot the Avaya S8300D from the G-Series gateway

I recently had a problem logging into System Platform. I could not connect to the CDOM via the web, nor SSH into DOM0. Fortunately, there is a way to reboot the S8300D from the gateway itself. Log into the gateway as root, then type “configure” to get to the config menu. Then simply type “reset mm v1”, which will reboot the media module in slot V1. Confirm, and the card will reboot. From my understanding, it will cleanly shut down each VM and reboot.

As always, it will take just long enough for you to panic, plus 45 seconds or so.

 

How to “kick” an Avaya Local Survivable Processor without rebooting

I recently had an Avaya Local Survivable Processor (LSP) unregister. There was no indication why – it just went down. I was able to get to its web interface, and it had a minor alarm indicating that it was unregistered. I suppose I could have restarted the whole thing, but my tech talked me through this procedure, which worked great.

Please note that the LSP is typically registered and inactive. This LSP only activates if network connectivity is lost to the site. So if the home site is available through the WAN, it should look like this:

list survivable-processor
                             SURVIVABLE PROCESSORS
 Record Name/            Type         Reg Act         Translations      Net
 Number  IP Address                                   Updated           Rgn
  1     THE_LSP          LSP           y   n          2:02 12/30/2013   10
         10.2.10.110
        No V6 Entry

However, in my case it looked like this!

list survivable-processor
                             SURVIVABLE PROCESSORS
 Record Name/            Type         Reg Act         Translations      Net
 Number  IP Address                                   Updated           Rgn
  1     THE_LSP          LSP           n                                10
         10.2.10.110
        No V6 Entry

No entry to indicate the active status nor the last translation date. I was able to SSH to the LSP and I confirmed the LSP wasn’t active. The media gateway was still connected to my home site. The LSP could ping the home site. Somehow, the LSP was simply lost.

So I confirmed all CM processes were up with the “statapp” command:

telecom@pbx-cm01> statapp
Watchdog         9/ 9 UP SIMPLEX
TraceLogger      3/ 3 UP SIMPLEX
LicenseServer    2/ 2 UP SIMPLEX
SME              6/ 6 UP SIMPLEX
MasterAgent      1/ 1 UP SIMPLEX
MIB2Agent        1/ 1 UP SIMPLEX
MVSubAgent       1/ 1 UP SIMPLEX
LoadAgent        1/ 1 UP SIMPLEX
FPAgent          1/ 1 UP SIMPLEX
INADSAlarmAgent  1/ 1 UP SIMPLEX
GMM              4/ 4 UP SIMPLEX
SNMPManager      1/ 1 UP SIMPLEX
filesyncd        1/ 1 UP SIMPLEX
MCD              1/ 1 UP SIMPLEX
CommunicaMgr    86/86 UP SIMPLEX
telecom@pbx-cm01>

Sure enough, all processes were running fine. So I stopped CM with the command

stop -afcn

This will stop the processes listed above, refreshing as it goes. It’s pretty nice. When all processes have stopped, you can start CM again with

start -ac

Again, it refreshes the screen as the processes start back up. When it’s done, the LSP should re-register. It’s also a good idea to watch the home CM for this register event. About 30 seconds after the CM comes back, you should see it. In your home site, do a

list trace ras ip-a 10.2.10.110

while the LSP processes restarts.

Here is the register event

list trace ras ip-address 10.2.10.110

                                LIST TRACE

time            data

17:45:39 TRACE STARTED 12/30/2013 CM Release String cold-03.0.124.0-21166
17:46:37   rcv RRQ ext
                   endpt  [10.2.10.110]:1719
                   switch [10.1.10.110]:1719
17:46:52   rcv KARRQ ext
                   endpt  [10.2.10.110]:1719
                   switch [10.1.10.110]:1719
17:47:07   rcv KARRQ ext
                   endpt  [10.2.10.110]:1719
                   switch [10.1.10.110]:1719
17:47:20   RAS TRACE COMPLETE ext

Those KARRQ messages are the keepalives.

After this restart, the LSP was registered fine. No service impact at the site, as the LSP is inactive unless the WAN goes down. And much less scary (for me anyway) than restarting the whole LSP.

Happy New Year everyone!

How to integrate Avaya Communication Manager and Session Manager 6.1 with Asterisk

As with most technical documents of this type, I wrote this because I could not find this information no matter how many Google searches I tried. There’s information about connecting Avaya Communication Manager directly with Asterisk using the H.323 channel drivers, there’s information regarding integration with the Avaya SIP Enablement Server, and I did find some information about some older versions of System Manager.

However, even the information I did see was a simple lab assignment. There was no information on actually using the new integration to do cool things. Some of you may have been tasked with integrating the Avaya PBX with Microsoft Lync. This tutorial provides a great proof-of-concept for Lync integration. We can test everything with Asterisk and it all in the telecom sandbox at your company.

This tutorial is for those of you who already have an Avaya Communication Manager and one or more Session Managers already set up – probably because you’re using Modular Messaging 6.x. Therefore, you have all the ingredients you need to add Asterisk to the mix.

Okay, so let’s assume your Communication Manager is already connected to Session Manager. Often this is the case due to the fact that Modular Messaging is now SIP based and uses Session Manager to connect to Communication Manager. I wanted to connect my Avaya Communication Manager to an Asterisk system. To do so, I need to know a few things about my existing infrastructure. You may already know these things about your PBX. If so, skim this part and skip to “Build the Route to Asterisk”

Part 1 – Confirming the current integration

So how is your Communication Manager connected to Session Manager? Let’s follow the rabbit:

First, look at the pilot number for voicemail, you’ll probably see that it’s a hunt group pilot number. Let’s say your voicemail pilot extension is 2000:

list usage extension 2000
                             LIST USAGE REPORT
Used By
Hunt Group           Group Number         5                      Group Ext

Now display that hunt group

display hunt-group 5                                            Page   2 of  60
                                  HUNT GROUP

                      Message Center: sip-adjunct

     Voice Mail Number        Voice Mail Handle         Routing Digits
                                                 (e.g., AAR/ARS Access Code)
     3992000                  sipmm-la                  801

This pilot number dials the AAR feature code, then 3992000. Now list aar:

list aar analysis                                                      Page   1
                           AAR DIGIT ANALYSIS REPORT
                            Location:  all
               Dialed            Total        Route    Call      Node
               String          Min    Max    Pattern   Type     Number
         1010                   4      4      13       aar
         14                     6      6      40       aar
         3990022                7      7      29       aar
         3992000                7      7      92       aar
         4010                   4      4      19       aar
         4099                   4      4      44       aar

And you can see that 3992000 goes to route pattern 92. Now look at route 92

display route-pattern 92                                        Page   1 of   3
                    Pattern Number: 92  Pattern Name: MM -LA
                             SCCAN? n     Secure SIP? n
    Grp FRL NPA Pfx Hop Toll No.  Inserted                             DCS/ IXC
    No          Mrk Lmt List Del  Digits                               QSIG
                             Dgts                                      Intw
 1: 91   0                    3                                         n   user
 2: 92   0                    3                                         n   user
 3: 93   0                    3                                         n   user
 4:                                                                     n   user
 5:                                                                     n   user
 6:                                                                     n   user

In my case, I have three Session Managers. I just care about the first one for now. So note that the first trunk group is 91. Now display trunk group 91:

display trunk-group 91                                          Page   1 of  21
                                TRUNK GROUP

Group Number: 91                   Group Type: sip           CDR Reports: y
  Group Name: MM SIP to LA                COR: 22       TN: 1        TAC: 815
   Direction: two-way        Outgoing Display? n
 Dial Access? n                                   Night Service:
Queue Length: 0
Service Type: tie                   Auth Code? n
                                              Member Assignment Method: auto
                                                       Signaling Group: 91
                                                     Number of Members: 96

And note that this trunk group uses signaling group 91. Now display signaling group 91:

display signaling-group 91 
                                SIGNALING GROUP

 Group Number: 299            Group Type: sip
  IMS Enabled? n        Transport Method: tls
        Q-SIP? n                                             SIP Enabled LSP? n
     IP Video? n                                   Enforce SIPS URI for SRTP? y
  Peer Detection Enabled? y  Peer Server: SM

   Near-end Node Name: procr                 Far-end Node Name: sipmm-la
 Near-end Listen Port: 5061                Far-end Listen Port: 5061
                                        Far-end Network Region: 5

Far-end Domain: corp.abc.com
                                             Bypass If IP Threshold Exceeded? n
Incoming Dialog Loopbacks: eliminate                  RFC 3389 Comfort Noise? n
         DTMF over IP: rtp-payload            Direct IP-IP Audio Connections? y
Session Establishment Timer(min): 3                     IP Audio Hairpinning? n
         Enable Layer 3 Test? y                   Initial IP-IP Direct Media? n
H.323 Station Outgoing Direct Media? n            Alternate Route Timer(sec): 6

And note that this signaling group connects the Communication Manager main processor (procr) to a node name called sipmm-la. Now list node-names:

list node-names all                                                    Page   4

                        NODE NAMES

Type     Name              IP Address
IP       cm-oc01p          10.5.20.131
IP       default           0.0.0.0
IP       iolan-la          10.2.4.21
IP       medpro-la1        10.2.4.22
IP       medpro-la2        10.2.4.23
IP       medpro-la3        10.2.4.24
IP       medpro-la4        10.2.4.25
IP       medpro-ny1        10.7.4.26
IP       medpro-ny2        10.7.4.27
IP       medpro-oc1        10.5.20.51
IP       medpro-oc2        10.5.20.52
IP       procr             10.2.20.20
IP       procr6            ::
IP       sipmm-la          10.5.20.139

And note that the node name is IP address 10.5.20.139. So in a nutshell, when I dial the voicemail pilot number, the call gets routed to my local session manager 10.5.20.139. That’s pretty much all we needed to know from the Communication Manager point of view.

Part 2 – Building the Route to Asterisk via Session Manager

What we really want is to set up a new route with new extensions between Communication Manager and Session Manager. Next we’ll configure Session Manager to understand a new number range and route it over to Asterisk.

In the case of a single pilot number, I could duplicate the hunt group method that is already set up. However, for more flexibility, I wanted a range of numbers to route to Asterisk. Or even if I do want to point individual numbers, I don’t want to create hunt groups for each one – I just want the number to route over to Asterisk through AAR routing.

So in my case, I had a spare DID number range ending in 44xx that I wanted to send over to Asterisk. So in CM, any four-digit number starting with 44 should go to Session Manager. Note that my route pattern 92 above deletes three digits. I don’t want to delete the first three digits of my 44xx extensions, so I will create a new route pattern that points to the same 91 trunk group. In my case, I used 90:

display route-pattern 90                                        Page   1 of   3
                    Pattern Number: 90  Pattern Name: SIP to SM
                             SCCAN? n     Secure SIP? n
    Grp FRL NPA Pfx Hop Toll No.  Inserted                             DCS/ IXC
    No          Mrk Lmt List Del  Digits                               QSIG
                             Dgts                                      Intw
 1: 91   0                                                              n   use

Now I want to point an entire internal number range to this route using AAR. So the first thing I need to do is edit my uniform dial plan so CM will use AAR routing when I dial this range:

list uniform-dialplan start 41                                         Page   2
                      UNIFORM DIAL PLAN TABLE

 Matching Pattern   Len   Del   Insert Digits   Net    Conv   Node Num

 4129                4     0                    ext     n
 4152                4     0                    ext     n
 4156                4     0                    ext     n
 4163                4     0                    ext     n
 4171                4     0                    ext     n
 4232                4     0                    ext     n
 44                  4     0                    aar     n
 4541                4     0                    ext     n
 4552                4     4      2089          ext     n
 4583                4     0                    ext     n
 4596                4     0                    ext     n
 4598                4     0                    ext     n

Fortunately CM is very forgiving with aar vs. ext. If I can digress for one screenshot, note that my dialplan analysis table defines all four-digit patterns that begin with 4 as extensions. However, I am able to override this in the uniform dialplan table and point these to aar in the table entry above.

display dialplan analysis                                       Page   1 of  12
                             DIAL PLAN ANALYSIS TABLE
                                   Location: all            Percent Full: 5

    Dialed   Total  Call     Dialed   Total  Call     Dialed   Total  Call
    String   Length Type     String   Length Type     String   Length Type
   0           4   ext      37          4   ext      77          4   ext
   20          4   ext      4           4   ext      78          4   ext
   21          6   ext      5           4   ext      79          4   ext
   22          6   ext      51          6   ext      888         3   fac
   23          6   ext      64          4   ext      
   24          6   ext      65          4   ext      
   25          6   ext      66          4   ext      
   26          6   ext      67          4   ext      
   27          4   ext      72          4   ext      
   29          6   ext      73          4   ext

So back to CM routing. Now that I have defined 44xx to point to the aar table, let’s create the aar entry we need to send this to Session Manager:

list aar analysis                                                      Page   1
                           AAR DIGIT ANALYSIS REPORT
                            Location:  all
               Dialed            Total        Route    Call      Node
               String          Min    Max    Pattern   Type     Number
         1010                   4      4      1        aar
         14                     6      6      400      aar
         3990022                7      7      298      aar
         3992000                7      7      299      aar
         4010                   4      4      11       aar
         4099                   4      4      44       aar
         44                     4      4      90       aar
         63xx                   4      4      25       aar
         74                     4      4      24       aar
         750                    6      6      299      aar

Now any four digit numbers starting with 44 will route (via route pattern 90) over to my session manager. Next, we need to configure session manager to route these to Asterisk!

Part 3 – Configuring Session Manager via System Manager

Now, I don’t know your experience with System Manager, but it seems that if I turn my back for a few weeks, the Admin password stops working and I have to reset it via SSH. Hopefully you have better luck than I do.

Major Gotcha: The first time I worked on this routing in System Manager, I just couldn’t get the calls to work. They would die in Session Manager with extremely unhelpful errors. After some troubleshooting, I discovered my changes weren’t synchronizing between System Manager and Session Manager. I had to go into Home->Replication and “repair” the replica group. I have done this several times since then and all of my calls during replication route fine. I don’t think it is service-affecting (with my configuration anyway). Your results may vary. Be careful.

avaya-smgr-replica1

 

I’ve worked on PBXs for a long time. Naturally, it can sometimes take a while to get comfortable with the GUI or command line of new systems. System Manager was tough for me. It’s such an abstraction from the actual routing engine (Session Manager) and I don’t get a chance to use a command line. I guess web interfaces drive me crazy for that reason – they’re a front-end to the actual magic, and I like to be closer to the soul of a PBX.

Enough rant – the actual configuration all takes place within Home->Routing. I just went in order down the list on the left. First, I configured a location called “LA-Asterisk”.

avaya-location-asterisk

 

avaya-location-asterisk-details

 

Next I created an Adaptation also called LA-Asterisk. Note in the screenshot below, there is a 4-digit extension pattern defined. We will do this later.

avaya-adaptation1 avaya-adaptation-asterisk

Next, create a SIP Entity. This is where you define your Asterisk server

avaya-sip-entity1

 

Next, create the Entity Link to “connect” your Asterisk and Session Manager together:

avaya-sip-entity-link

 

In my case, I went for simple UDP connectivity. Next I create the Routing Policy for Asterisk:

avaya-routing-policy1

 

Don’t worry about dial patterns or regular expressions on this screen. Just set up a SIP Entity and Time of Day and commit.

Then the Dial Pattern:

avaya-dialpattern-asterisk

 

 

Note in my case, the PBX is sending the full E.164 number to Session Manager. Well, this isn’t entirely accurate. To digress again, the PBX is sending the regular extension, but the Adaptation for Communication Manager defines several digit conversion rules to convert these to E.164 numbers. This allows for a very flexible system within Session Manager. As a result of this digit conversion, the internal routing of the Session Manager uses the E.164 numbers and this is what you see when you trace. Here are my Adaptation rules:

avaya-adaptation-digit-conversion

 

I’ve erased the DID numbers there – I’m not much of a photo editor. You can see how the various extension ranges and lengths translate into E.164. Note that four-digit non-DIDs are translated to +1000000xxxx. This is how Avaya set up this site and I assume the best practice. I work at another site (set up by an Avaya business partner) that was not normalized to E.164 and it’s been kind of a pain to manage as we’ve added DIDs and extensions.

So back to Communication Manager routing: do you remember way back when I mentioned you could use the “hunt group” routing to send calls to Asterisk? I created a hunt group in CM that looks like this:

display hunt-group 9                                            Page   1 of  60
                                  HUNT GROUP

            Group Number: 9                                ACD? n
              Group Name: Asterisk                       Queue? n
         Group Extension: 2022                          Vector? n
              Group Type: ucd-mia                Coverage Path:
                      TN: 1          Night Service Destination:
                     COR: 21                   MM Early Answer? n
           Security Code:               Local Agent Preference? n
 ISDN/SIP Caller Display: mbr-name

display hunt-group 9                                            Page   2 of  60
                                  HUNT GROUP

                      Message Center: sip-adjunct

     Voice Mail Number        Voice Mail Handle         Routing Digits
                                                 (e.g., AAR/ARS Access Code)
     3992022                  ast-la                    888

This hunt group will send calls to 2022 over to Session Manager with a dial pattern of ast-la. You can then create a regular expression in System Manager to match this pattern:
avaya-regular-expression

Why would you want to do this? Well, this is how the Avaya technicians originally set up the routing to Modular Messaging. I suspect it allows the SIP header to retain the original dialed number as it is passed across. There’s a “diversion” header in the SIP invite, and this is how the originally-dialed number gets to Modular Messaging. I happened to set this up in a desperate attempt to get Session Manager to route for me. When I use the traceSM utility in Session Manager, I noticed that Session Manager was not trying to find a match for this pattern. This is when I discovered that the replication wasn’t working between System Manager and Session Manager. Once I fixed the replication, the rest of the routing worked and I left this in place rather than pull it all out. More on traceSM later.

---------------------------------------------------------------------------------------------------------------
         cm01(.30)             192.168.204.39
                       SM100
---------------------------------------------------------------------------------------------------------------
16:03:51,290 |   Dial Pattern route parameters   | URI Domain: abc.com  Location: LA-MAIN-CM
16:03:51,290 |   Dial Pattern route parameters   | URI Domain: null  Location: LA-MAIN-CM
16:03:51,290 |     Trying Dial Pattern route     | Domain: null  Location: LA-MAIN-CM
16:03:51,290 |   Dial Pattern route parameters   | URI Domain: corp.abc.com  Location: null
16:03:51,290 |     Trying Dial Pattern route     | Domain: corp.abc.com  Location: null
16:03:51,290 |   Dial Pattern route parameters   | URI Domain: abc.com  Location: null
16:03:51,290 |   Dial Pattern route parameters   | URI Domain: null  Location: null
16:03:51,290 |     Trying Dial Pattern route     | Domain: null  Location: null
16:03:51,290 |  Request Regular Expression rout  | for: sip:ast-la@corp.abc.com
16:03:51,291 |  Trying Regular Expression route  | pattern: sipmm-ny.*  for: sip:ast-la@abc.corp
16:03:51,291 |  Trying Regular Expression route  | pattern: sipmm-la.*  for: sip:ast-la@abc.corp
16:03:51,291 |  Trying Regular Expression route  | pattern: sipmm-oc.*  for: sip:ast-la@abc.corp
16:03:51,291 |  Trying Regular Expression route  | pattern: ast-la.*  for: sip:ast-la@abc.corp
16:03:51,291 |  Trying Regular Expression route  | pattern: sipmm-ny.*  for: ast-la@corp.abc.com
16:03:51,291 |  Trying Regular Expression route  | pattern: sipmm-la.*  for: ast-la@corp.abc.com
16:03:51,291 |  Trying Regular Expression route  | pattern: sipmm-oc.*  for: ast-la@corp.abc.com
16:03:51,291 |  Trying Regular Expression route  | pattern: ast-la.*  for: ast-la@corp.abc.com
16:03:51,291 |     Regular Expression found      | pattern: ast-la.*  for: ast-la@corp.abc.com  RoutePolicyList
16:03:51,291 |            Route found            | for: sip:ast-la@corp.abc.com  SIPEntity: asterisk-la01p
16:03:51,291 |         Entity Link found         | SIPEntity: asterisk-la01p  EntityLink: sip-oc1->UDP, b
16:03:51,291 |     Entity Link to another SM     | To: sip-oc1  MyInstance: sip-la1
16:03:51,291 |         Entity Link found         | SIPEntity: sip-oc1  EntityLink: sip-la1->TLS, biD
16:03:51,291 |        Request Adaptation         | Adapter: LA-MAIN-CM
16:03:51,292 |     Applied egress Adaptation     | P-Asserted-Identity="Roger Ramjet" <sip:2135552245@abc.corp 16:03:51,292 |        Routing SIP request        | SipEntity: asterisk-la1  EntityLink: sip-oc01->UDP:50
16:03:51,292 |         Entity Link found         | SIPEntity: sip-oc1  EntityLink: sip-la1->TLS, biD
16:03:51,293 |  No hostname resolution required  | Routing to: sip: 10.5.20.139;transport=tls;lr;sm-routethru
16:03:51,294 |           |--INVITE-->|           | (8) T:ast-la F:+12135552245 U:ast-la P:terminating
16:03:51,342 |           |<--Trying--|           | (8) 100 Trying
16:03:52,419 |           |

Once you create a regular expression in System Manager, you should see an attempt to match it in traceSM. I wasn’t seeing my new patterns in this “Trying…” list. If you don’t see your patterns, it may be time to repair the replication.

Part 4 – Configure Asterisk to accept inbound calls from Communication Manager

Ok, so you finally have CM configured, and you think you have Session Manager/System Manager configured. Now for the last step. In my case, I was happy to finally get to Asterisk. Getting back to my rant about Session Manager, I like Asterisk because it’s completely command line and config files. This allows you to get very close to the inner workings of Asterisk.

Asterisk was a simple two step configuration: sip.conf and extensions.conf.

Just add these lines to /etc/asterisk/sip.conf

[avayaLA]
type=peer
qualify=yes
fromdomain=corp.abc.com
host=192.168.24.39
disallow=all
allow=ulaw
allow=g729
dtmfmode=rfc2833
canreinvite=yes

And add these lines to /etc/asterisk/extensions.conf. Note that we’re creating a context for our Avaya Session Manager and adding it to the default context. Later you can pull this out, but for now it provides a nice way to test incoming calls by sending them to the “congratulations” demo that came with Asterisk.

[avaya-la]
exten => ast-la,1,noop
exten => ast-la,n,goto(demo,1000,1)
exten => 4498,1,goto(demo,1000,1)
exten => 4499,1,meetme(4499,1)

[default]
;
; By default we include the demo.  In a production system, you
; probably don't want to have the demo there.
;
include => demo
include => internal
include => avaya-la

So now let’s test. We should have these things in place:

  1. Connectivity between Communication Manager and Session Manager
  2. Connectivity between Session Manager and Asterisk
  3. Routes built from Communication Manager through Session Manager to Asterisk

Now when I call a 44xx extension, the Communication Manager will send it via AAR to Asterisk. Let’s look at that call within Communication Manager:

list trace station 2245                                                Page   1
                                LIST TRACE
time            data
13:12:11 TRACE STARTED 03/19/2012 CM Release String cold-00.1.510.1-19100
13:12:13     active station      2245 cid 0x11c3
13:12:13     G711MU ss:off ps:20
             rgn:1 [172.25.114.8]:11394
             rgn:1 [172.25.24.124]:59628
13:12:15 SIP>INVITE sip:4498@corp.abc.com SIP/2.0
13:12:15     Call-ID: 801ecff7b275e11f7724f24def200
13:12:15     dial 4498 route:UDP|AAR
13:12:15     term trunk-group 91      cid 0x11c3
13:12:15     dial 4498 route:UDP|AAR
13:12:15     route-pattern  90 preference 1 location 1/ALL  cid 0x11c3
13:12:15     seize trunk-group 91 member 6    cid 0x11c3
13:12:15     Setup digits 4498
13:12:15     Calling Number & Name *12135552245 Roger Ramjet
13:12:15 SIP<SIP/2.0 100 Trying
13:12:15     Call-ID: 801ecff7b275e11f7724f24def200
13:12:15     Proceed trunk-group 91 member 6    cid 0x11c3
13:12:15 SIP<SIP/2.0 180 Ringing
13:12:15     Call-ID: 801ecff7b275e11f7724f24def200
13:12:15     Alert trunk-group 91 member 6    cid 0x11c3

We can see from this trace that the call to 4498 goes over route 90 to trunk group 91. Perfect. Now let’s watch Session Manager. If you haven’t had the chance, you can trace calls within Session Manager via a tool called traceSM. At first I wasn’t sure about it, but I’ve come to really like it. SSH into your session manager (the management interface, not the traffic interface) and login as ‘craft’. If you know your Avaya systems, you’ll know the default password. Then type ‘traceSM’. After a few seconds of loading the log file, you’ll be ready to trace.

If your system has a lot of traffic, you’ll probably want to filter your results. Type ‘f’ at the screen and filter by your test extension.

/----------------------------------------------------------------------\
|Filter Usage:                                                         |
|  -u   Filter calls that contain  in          |
|                   the 'From' or 'To' field.                          |
|  -i           Filter SIP messages from/to  address.                  |
|  -c      Filter based on the SIP 'Call-ID' header field.             |
|  -g = Filter SIP header field  for value .                           |
|  -or              Use a logical OR operator instead of the implicit  |
|                   AND when using multiple filter options.            |
|  -nr              Do not display REGISTER messages.                  |
|  -ns              Do not display SUBSCRIBE/NOTIFY messages.          |
|  -no              Do not display OPTIONS messages.                   |
|  -na              Do not display SM related messages.                |
|Filter examples:                                                      |
| To display a call to/from 3035556666 and not REGISTER messages:      |
|    -u 3035556666 -nr                                                 |
| To display SIP messages from/to 1.1.1.1 and 2.2.2.2:                 |
|    -i "1.1.1.1|2.2.2.2"                                              |
|                                                                      |
|Current Filter:                                                       |
|New Filter: -u 4498                         |
|                                                                      |
\----------------------------------------------------------------------/

Press enter and traceSM will re-process the log file. Then press ‘c’ to clear the results, then press ‘s’ to start the trace. Now when I re-dial extension 4498 and I can see the call pass through Session Manager:

---------------------------------------------------------------------------------------------------------------------------------
       172.25.204.30           172.25.14.131
                       SM100
---------------------------------------------------------------------------------------------------------------------------------
13:39:33,115 |--INVITE-->|           |           | (1) T:4498 F:+12135552245 U:4498 P:terminating
13:39:33,117 |<--Trying--|           |           | (1) 100 Trying
13:39:33,118 |      Remote host is trusted       | Trusted
13:39:33,118 |        Request Adaptation         | Adapter: LA-MAIN-CM
13:39:33,119 |    Applied ingress Adaptation     | Request-URI=sip:+12135554498@corp.abc.com, History-Info=<sip:+12135554498@corp.abc.com>;index=1,"4498" UDP, biDirId=null:5060
13:39:33,121 |        Request Adaptation         | Adapter: LA-MAIN-CM
13:39:33,121 |     Applied egress Adaptation     | P-Asserted-Identity="Roger Ramjet" <sip:2135552245@corp.abc.com>, Request-URI=sip:4498@corp.abc.com;rout
13:39:33,121 |        Routing SIP request        | SipEntity: asterisk-la01p  EntityLink: sm-sip-la01p->UDP:5060
13:39:33,123 |  No hostname resolution required  | Routing to: sip:172.25.4.231;lr;phase=terminating
13:39:33,124 |           |--INVITE-->|           | (1) T:4498 F:+12135552245 U:4498 P:terminating
13:39:33,195 |           |<--Trying--|           | (1) 100 Trying
13:39:33,209 |           |<--Ringing-|           | (1) 180 Ringing
13:39:33,211 |        Request Adaptation         | Adapter: LA-MAIN-CM
13:39:33,211 |        Request Adaptation         | Adapter: LA-MAIN-CM
13:39:33,212 |<--Ringing-|           |           | (1) 180 Ringing 13:39:34,455 |--CANCEL-->|           |           | (1) sip:4498@corp.abc.com
13:39:34,455 ||           | (1) sip:4498@corp.abc.com
13:39:34,496 |           ||           | (1) sip:4498@corp.abc.com
13:39:34,499 |<--Request-|           |           | (1) 487 Request Terminated 13:39:34,548 |----ACK--->|           |           | (1) sip:4498@corp.abc.com

You can see the SIP INVITE from Communication Manager. You can see Session Manager process it using the various rules in the routing engine, and you can see the INVITE passed along to Asterisk. You can use the arrow keys to highlight the INVITE and press enter to see the details:

/--------------------------------------------------------------------------------\
|INVITE sip:4498@corp.abc.com;routeinfo=0-0 SIP/2.0                              |
|Record-Route: <sip:192.168.214.8:15060;lr;sap=865602204*1*016asm-callprocessing | 
|.sar634103744~1332189573117~-1715763735~1>                                      |
|From: "Roger Ramjet" <sip:+12135552245@corp.abc.com>;tag=0f617ceb675e1123764f2  |
|4def200                                                                         |
|To: <sip:4498@corp.abc.com>                                                     |
|Call-ID: 0f617ceb675e1124764f24def200                                           |
|CSeq: 1 INVITE                                                                  |
|Via: SIP/2.0/UDP 192.168.24.8:15070;branch=z9hG4bKC0A8CC2608E7B337012365250     |
|Via: SIP/2.0/UDP 192.168.24.8:15070;branch=z9hG4bKC0A8CC2608E7B337112365248     |
|Via: SIP/2.0/UDP 192.168.24.8:15070;branch=z9hG4bKC0A8CC2608E7B337112365247     |
|Via: SIP/2.0/TLS 192.168.24.9;branch=z9hG4bK0f617ceb675e1125764f24def200-AP;ft  |
|=23826                                                                          |
|Via: SIP/2.0/TLS 172.25.24.30;branch=z9hG4bK0f617ceb675e1125764f24def200        |
|Supported: 100rel,histinfo,join,replaces,sdp-anat,timer                         |
|Allow: INVITE,ACK,OPTIONS,BYE,CANCEL,SUBSCRIBE,NOTIFY,REFER,INFO,PRACK,PUBLISH  |
|User-Agent:  Avaya CM/R016x.00.1.510.1 AVAYA-SM-6.1.1.0.611023                  |
|Contact: "Roger Ramjet" <sip:+12135552245@172.25.24.30:5061;transport=tls>      |
|Alert-Info: <cid:internal@corp.abc.com>;avaya-cm-alert-type=internal            |
|Min-SE: 1200                                                                    |
|Record-Route: <sip:559ae004@192.168.24.39;transport=tls;lr>                     |
|Record-Route: <sip:172.25.24.30:5061;transport=tls;lr>                          |
|Session-Expires: 1200;refresher=uac                                             |
|P-Charging-Vector: icid-value="AAS:5279-ce17f6001e175b6244f7622f2de"            |
|Content-Type: application/sdp                                                   |
|Content-Length: 210                                                             |
|P-Asserted-Identity: "Roger Ramjet" <sip:2135552245@corp.abc.com>               |
|History-Info: <sip:2135554498@corp.abc.com>;index=1,"4498" <sip:2135554498@corp.abc.com>;index=1.1                                                              |
|Route: <sip:192.168.204.39;lr>                                                  |
|Route: <sip:172.25.4.31;lr;phase=terminating>                                   |
|P-AV-Transport: AP;fe=172.25.24.30:34534;ne=192.168.24.39:5061;tt=TLS;th;timer  |
|B=4                                                                             |
|P-Location: SM;origlocname="LA-MAIN-CM";termlocname="LA-Asterisk"               |
|Max-Forwards: 68                                                                |
|                                                                                |
|v=0                                                                             |
|o=- 1332189580 1 IN IP4 172.25.24.30                                            |
|s=-                                                                             |
|c=IN IP4 172.25.24.54                                                           |
|b=AS:64                                                                         |
|t=0 0                                                                           |
|a=avf:avc=n prio=n                                                              |
|a=csup:avf-v0                                                                   |
|m=audio 65044 RTP/AVP 0 127                                                     |
|a=rtpmap:0 PCMU/8000                                                            |
|a=rtpmap:127 telephone-event/8000                                               |
\--------------------------------------------------------------------------------/

This confirms that your dial request to 4498 is passing through the Session Manager over to Asterisk. Now let’s look at Asterisk:

exten => 4498,1,goto(demo,1000,1)

Because our extensions.conf file contains the line above, the calls that come in should go to the Asterisk demo application. Let’s watch from the CLI when I call 4498 again:

localhost*CLI>
  == Using SIP RTP CoS mark 5
    -- Executing [4498@default:1] Goto("SIP/avayaLA-00000037", "demo,1000,1") in new stack
    -- Goto (demo,1000,1)
    -- Executing [1000@demo:1] Goto("SIP/avayaLA-00000037", "default,s,1") in new stack
    -- Goto (default,s,1)
    -- Executing [s@default:1] Wait("SIP/avayaLA-00000037", "1") in new stack
    -- Executing [s@default:2] Answer("SIP/avayaLA-00000037", "") in new stack
    -- Executing [s@default:3] Set("SIP/avayaLA-00000037", "TIMEOUT(digit)=5") in new stack
    -- Digit timeout set to 5.000
    -- Executing [s@default:4] Set("SIP/avayaLA-00000037", "TIMEOUT(response)=10") in new stack
    -- Response timeout set to 10.000
    -- Executing [s@default:5] BackGround("SIP/avayaLA-00000037", "demo-congrats") in new stack
    -- <SIP/avayaWLA-00000037> Playing 'demo-congrats.slin' (language 'en')
  == Spawn extension (default, s, 5) exited non-zero on 'SIP/avayaWLA-00000037'
localhost*CLI>

And if you could hear what I hear, you’d enjoy Allison Smith’s congratulations message also! We have inbound calls to Asterisk.

Part 5 – Configure Asterisk to send outbound calls to Communication Manager

This one is a little easier. To send internal calls to Avaya, just create an extension match in extensions.conf to match your internal number ranges. However, don’t forget that you are probably sending some numbers from Communication Manager to Asterisk. Be sure not to send these number ranges back to Communication Manager! In my case, Communication Manager is sending 44xx to Asterisk so I make sure I keep those numbers internal to Asterisk by dialing any registered SIP phones (more on this later).

[losangeles]
;these are LA extensions registered through softphones
exten => _44XX,1,Dial(SIP/${EXTEN}) ; these stay internal to Asterisk
exten => _91XXXXXXXXXX,1,dial(SIP/${EXTEN}@avayaLA) ; external – leave the 9 in place
exten => _[245]XXX,1,dial(SIP/${EXTEN}@avayaLA) ; 4 digits starting with 0, 4, or 5

To test this, you can issue an ‘originate’ command right from the Asterisk CLI.

localhost*CLI> originate SIP/2245@avayaWLA extension 1000
  == Using SIP RTP CoS mark 5
    -- Executing [1000@default:1] Goto("SIP/avayaLA-00000049", "default,s,1") in new stack
    -- Goto (default,s,1)
    -- Executing [s@default:1] Wait("SIP/avayaLA-00000049", "1") in new stack
    -- Executing [s@default:2] Answer("SIP/avayaLA-00000049", "") in new stack
    -- Executing [s@default:3] Set("SIP/avayaLA-00000049", "TIMEOUT(digit)=5") in new stack
    -- Digit timeout set to 5.000
    -- Executing [s@default:4] Set("SIP/avayaLA-00000049", "TIMEOUT(response)=10") in new stack
    -- Response timeout set to 10.000
    -- Executing [s@default:5] BackGround("SIP/avayaLA-00000049", "demo-congrats") in new stack
    -- <SIP/avayaWLA-00000049> Playing 'demo-congrats.slin' (language 'en')
  == Spawn extension (default, s, 5) exited non-zero on 'SIP/avayaWLA-00000049'
localhost*CLI>

The command above causes Asterisk to launch a call to 2245 through the avayaLA peer and send the call to extension 1000. You should get a call and when you answer, you’ll hear the demo. Likewise, external calls should include the ‘9’ when sent to Communication Manager. Your system will probably pass this through Session Manager to Communication Manager just fine since Modular Messaging needs to dial out sometimes (for find-me/follow-me features).

So that’s it! You now have an Avaya Communication Manager sending and receiving calls to Asterisk via Session Manager! This is a great proof-of-concept for you to start your Lync integration. I’ll write up a separate article for Lync. There’s an amazing amount of politics involved in a Lync project. I’d love to hear from you about it.

Avaya 9630 Locks / Reboots up when registering

I recently had an Avaya phone in a reboot cycle. It would boot up, then when it registered it would lock up and after a couple minutes reboot again. The display looked like it was up and running fine, but when you press the SPEAKER button I would get three beeps. This is typically what happens when a phone cannot get TCP/IP signalling traffic to the call server. And the night before we had some maintenance on that Ethernet switch so I immediately suspected a network problem.

Just in case, I did a “CLEAR” procedure on the phone. Then I swapped out the phone. Then I swapped out the patch cables (at both ends). Then I moved the phone to a different port in the Ethernet switch. No matter what I did, the phone locked up. Then I tried something I probably should have tried earlier – I logged in a different extension and it worked fine! Then I logged the “bad” extension into a different phone and it locked up!

Turns out the config file on the web server (1234_96xxdata.txt) was incomplete. Apparently it was related to the network after all! When the phone was writing its data file to the web server the previous night, the write operation was interrupted as the network guy shut off the Ethernet switch. The resulting data file was incomplete – it had about half the call log entries and a partial line at the end. But none of the important lines you’d expect in the file such as:

LOGTDFORMAT=128
Redial=1
Edit Dialing=1
Go to Phone Screen on Calling=0
Go to Phone Screen on Ringing=1
Call Timer=1
Visual Alerting=0
History Active=1
Log Bridged Calls=1
Audio Path=1
Personalized Ring=0
Handset AGC=1
Headset AGC=1
Speaker AGC=1
Error Tone=1
Button Clicks=0
Text Size=1
Contacts Pairing=0
Voice Initiated Dialing=1
Voice Dialing Help Counter=0
Personalized Ring Menu=0
Go to Phone Screen on Answer=0
Voice Initiated Dialing Language=

If the file were missing, the phone would use default values and create the file at the next backup. However, since the file was there, the phone processed it but ended up locking up because it was incomplete. In all my years working with these phones, I’ve never seen that before. I wouldn’t have thought it possible for the phone’s interrupted “HTTP PUT” operation to result in an incomplete file on the web server, but there you go. Hopefully this helps you.

MPLS Network and QOS – Traffic Policing vs. Traffic Shaping

As mentioned in my last post, I had some trouble with resets in my PBX due to “some kind of network event”. It was very enlightening when we discovered the root cause, so I wanted to share my experience here in hopes that it saves you from similar sleepless nights.

The symptom was quite simple – the PBX was losing connectivity with its host processor for some reason. Avaya called it a “network event” but couldn’t be more specific. I was fortunate enough to work on the same team as the LAN/WAN techs. If we had been in different departments, this would have been even more of a nightmare. So the Avaya PBX loses heartbeats – after 15 of them it caused a reset. Heartbeats are about one second apart, so a failure of 15 heartbeats is a very long time. Surely a WAN outage of 15 seconds would be noticed by other systems, right? So when Avaya says there was a network event, my response was “uh, you gotta give me more than that. Nothing else on the network noticed”.

I’ll spare you the details here – they’re in my previous post anyway. The issue is the PBX is marking all IP traffic with “Expedited Forwarding” (EF), or Diffserv 46, or High Priority, or QOS. There are plenty of synonyms but it just means all IP packets are tagged with high priority and should be preferred over other packets in the LAN and WAN. Our MPLS carrier honors this tag through their network. Perfect, right?

So when you purchase QOS in the MPLS network, you are given a certain amount of bandwidth allowed. Obviously you’re not allowed to mark all traffic as high priority. MPLS is a shared cloud and you’ll pay a premium for expedited delivery of QOS packets. This bandwidth allotment is called the Committed Access Rate or CAR. What happens when you exceed this CAR? Well, as a telephone guy, I would assume the packets are delivered as “best effort” after that. But I was very wrong. Packets over the CAR are discarded by the MPLS carrier. Think about it – if you have high-priority packages to deliver overnight and the you’re only allowed to send ten per day. The eleventh package isn’t held for best effort. It’s thrown in the dumpster. Sorry. You exceeded your ten packages today, I’m throwing this one out.

The fix is simple and elegant. And probably crystal clear to you network routing engineers out there. It’s up to your edge router to strip the QOS tag from any packets that exceed the CAR. It sounds dangerous to me – what if my edge router and the carrier’s MPLS router disagree slightly on the current amount of traffic at this particular second? Especially if we’re using different brands of router? However, Mbps seems to be an agreed-upon measurement across all vendors, so having my edge router strip the QOS tag from these packets simply allows them to be delivered at best-effort across MPLS.

What I discovered, is if this setting is not correct, it only affects high-priority traffic. In my experience, most companies use a different network for video and everything else on the network such as Internet, email, chat, file servers, replication, database synchronization, etc. are not marked with QOS. So guess what? It looks like your phone system is hosed! And nothing else on the network is complaining! This gets back to a trend I’ve been noticing: Being a phone guy doesn’t have much to do with telephones anymore. It’s almost always the network. And to be a really good phone tech, you have to know networking well. I hope this helps.

I recently went through an ordeal with a PBX resetting. It’s an Avaya system using an IPSI to connect a port network back to its host, but this situation applies to anyone out there using QOS on their MPLS network. I’ve often said that being a “phone guy” is rarely about phones anymore. Most of my work – certainly troubleshooting – involves IP networking.

So I had a PBX with one IPSI that would occasionally reset. Since there was only one IPSI, the reset would cause all cards in the port network to reset as well, which would drop all calls in progress. Now this is about the worst thing that can happen when you’re responsible for the telephones. Full system outages are easier to understand. This is a reset, calls drop, users get frustrated and re-establish their calls, then it would reset again. It was a really bad situation.

What is causing the resets? Avaya said the heartbeats were failing to the IPSI. For any of you with an IPSI-connected port network, you should occasionally look for these. SSH to your Communication Manager and cd to /var/log/ecs. Then list the log files. Assuming you’re in Feburary 2013, you would look for missed heartbeats in your ecs log with the command:

cat 2013-02*.log|grep checkSlot
:pcd(5561):MED:[[3:0] checkSlot: sanity failure (1)]
:pcd(5561):MED:[[3:0] checkSlot: sanity failure (2)]
:pcd(5561):MED:[[3:0] checkSlot: sanity failure (3)]
:pcd(5561):MED:[[3:0] checkSlot: sanity failure (4)]
:pcd(5561):MED:[[3:0] checkSlot: data received replacing sanity message; socket delay is 14 secs]

I have stripped the date/time; you’ll see those on the left. Port networks and IPSIs are zero indexed, so the messages above apply to port network 4 and IPSI number 1.

I have been told that occasional sanity failures are just a part of life. These heartbeat messages are part of the Avaya protocol, not ICMP. So if you’re missing heartbeats, it’s not because ICMP is being dropped.

However, after a certain number of sanity failures, the IPSI will reset in order to re-esablish communication. How many sanity failures? That depends upon a system parameter setting:

display system-parameters ipserver-interface
IP SERVER INTERFACE (IPSI) SYSTEM PARAMETERS

SERVER INFORMATION
Primary Control Subnet Address:
Secondary Control Subnet Address:

OPTIONS

Switch Identifier: A
IPSI Control of Port Networks: enabled
A-side IPSI Preference: disabled
IPSI Socket Sanity Timeout: 15

QoS PARAMETERS
802.1p: 6
DiffServ: 46

The IPSI Socket Sanity Timeout determines how many sanity failures will cause an IPSI failover (if you have two in your port network), or a reset(!) if you only have one. The reset is the IPSI’s way of trying to re-establish communication. If you get too many sanity failures, you’ll get this message:

:pcd(5561):MED:[[3:0] checkSlot: too many sanity failures (15)]

Unfortunately, this means my CM lost connectivity to the first IPSI on port network 4. If I only have one IPSI, then the IPSI and all cards in the port network will reset. If I have a redundant IPSI, then the port network will failover and everything should be okay. In my particular case a second IPSI would not have helped me. It turns out, my MPLS carrier (who had also set up our edge routers) was policing the committed access rate. I’ll explain with more detail in my next post. The resolution was to shape the traffic rather than police it.