Category Archives: Avaya

Avaya System Manager – unable to login

There are many reasons for the login to fail on System Manager. Version 6.3.x seems better but this thing still feels like it’s built on a house of cards. Tonight I just found another reason and it was new to me so I want to tell you all about it.

Earlier today I tired to login and was confronted with this error

avaya-system-manager-internal-error

Some internal error has occurred in the service. Please contact the support team.

Guess who the support team is? Oh yeah, that’s us!

Well, I didn’t really know what to do and it was a Sunday so I figured I would reboot it. Now my System Manager is a virtual machine and I don’t have access to the VMWare console. But I was able to SSH into System Manager as admin. Then su and enter the root password. From there, I was able to issue a reboot command. It dropped my session, and then it never did come back to ping. So I contacted my VM team and they rebooted it. Once they did that, I was able to ping and SSH to it. But the web page said “This page has a redirect loop”. I knew it could take a long time to come back up, so I gave it a while. But no dice. I contemplated rebooting it again, but rather than that, I went through the “ask Ava” chat pages on support.avaya.com. I was hoping with such a specific error like “redirect loop” I would find something and sure enough, Ava pointed me to SOLN270630, which was exactly my problem. According to this page, the Postgres certificate had expired. Avaya provided the command to confirm it and the command to fix it. All I had to do was run that command and then restart jboss. Then wait. Don’t forget, it can take 15-20 minutes for the restart to finish. Don’t panic. Actually, I think you do have to panic. Three minutes after you start to panic it comes back up. Don’t forget to re-enable Geographic Redundancy!

root >openssl x509 -text -in /var/lib/pgsql/data/server.crt | grep "Not After"
 Not After : Oct 22 15:14:31 2015 GMT
root >/opt/Avaya/Postgres/*/utils/securePostgres.sh
Mon Oct 26 00:13:18 EDT 2015 : Restarting Postgres service...
Stopping postgresql service: [ OK ]
Starting postgresql service: Waiting for Postmaster PID file creation .
Waiting for Postmaster DB connection availability .
Starting postgresql service: [ OK ]
Mon Oct 26 00:13:21 EDT 2015 : Startup of Postgres done
root >service jboss restart
Perform cleanup ....
Stopping System Manager JBoss - PID: 6580
.............................................
Stopped System Manager JBoss
Starting System Manager JBoss
.
Service started successfully - PID: 21296
root >

Avaya CM – will I lose the CDR buffer if I busy/release the cdr link?

I have an Avaya Communication Manager sending CDR to a kiwi syslog server. For some reason, the link is down and the buffer is filling up. What happens if I busy/release the CDR link? Will I lose the buffer? Scary!

Short answer is no. I just tried it. I performed a busy/release on the primary CDR link and the buffer is still at 40%. Unfortunately, the link is still down so I’ll have to check with the syslog team. But I just wanted to let all of you know that it’s safe to restart the link.

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: down                         up
Number of Retries: 18
      Date & Time: 2015/09/23 06:35:59          2015/09/12 07:46:56
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer % Full:  41.21                         0.00
      Reason Code: CDR connection is closed     OK


I ended up disabling and then enabling the TCP listener in Kiwi and the buffer immediately cleared. Is anyone interested in a post on how to capture CDR to a SYSLOG server?

Flowchart of Avaya Communication Manager Routing

Click Here for the flowchart PDFs

One might argue that telephone systems do only one thing. They route calls. In the process they turn lamps on and off, and they make telephones buzz and beep. But the only point is to route voices from point A to point B.

If you’ve been working with Avaya Communication Manager for a while, you have been exposed to AAR and ARS routing. And you hopefully have a fair grasp of how it all fits together. But there are a whole lot of decisions that are made when routing outbound calls. And if you’re new to Avaya, it might seem overwhelming to troubleshoot routing issues. So here are four pages of flowcharts that, based on my experience, are the most common configuration issues you’ll see when managing your telephone system. Click Here for the flowchart PDFs

Page 1 is a flowchart of the digit collection. This is what the CM does to collect the dialed digits, translate them as necessary, and select a network (AAR vs. ARS). Digits can be translated many times and re-pointed to different networks. Once this process is complete, the logic continues to page 2.

avaya_call_routing_flowchart-page_1_of_4

Page 2 is the process of using the final dialed digits (and station’s location) to select a route and a trunk group. There are many location and facilities restriction level checks here. And there is even more digit translation that can take place during this process. Once a trunk is selected and the call delivered, this may not be the end of it. Look Ahead Routing may pull the call back and try again.

avaya_call_routing_flowchart-page_2_of_4

Those two pages compose the primary routing logic that you will use in your typical troubleshooting. However, many decisions are based upon the calling party’s location. It may be obvious to you how the CM know’s the station’s location. But if things get strange, you can see how the CM associates a station to a location on page 3.

avaya_call_routing_flowchart-page_3_of_4

And lastly, there is one major headache associated with the “Prefix Mark” in a route pattern. Prior to 1995, there were many, many dialing rules associated with seven digit dialing, one plus seven digits, ten digits, etc. I remember living in a small town in the California Bay Area, and to dial Concord (over the hill), I had to dial 1+seven digits. Then when the area code 415 was split in to 415/510, we could dial 10 digits between them. Unfortunately, if you dialed 1+10 digits, it would be charged as a long distance call. All this went away in 1995 when the North America Numbering Plan was implemented. It is much easier to route calls now. But there is still the dreaded “Prefix Mark” that may still be in your routes. This causes havoc with the dialed-digits that are sent to your carrier. Page 4 is a summary of how this prefix mark works.

avaya_call_routing_flowchart-page_4_of_4

Please note there are two extremely important troubleshooting tools when dealing with Avaya routing. One, is the ‘list ars route-chosen’ and ‘list aar route chosen’ commands, and the other is the ‘list trace station’ and ‘list trace tac’ commands. If you use Avaya Site Administrator, you will not be able to use the ‘list trace’ command. I encourage you to use telnet or ssh in your day-to-day management of your PBX so you can use these commands when necessary. It may not sound like much, but being able to list trace at a moment’s notice is very helpful.

I will save the “list aar/ars route-chosen” and “list trace station/tac” for another posting. There’s a lot of information available in these commands.

Please let me know if you find these flowcharts helpful. Also, if you find any typos please let me know. Please keep in mind there are many more routing decisions the CM makes that are not in these flowcharts. If your CM uses other routing (for example tenants or time-of-day) and you think I should incorporate them into these flowcharts, please let me know! Happy routing everyone. This is really the core of any PBX, so I personally really enjoy working with routing.

Roger

Click Here for the flowchart PDFs

How to reboot the Avaya S8300D from the G-Series gateway

I recently had a problem logging into System Platform. I could not connect to the CDOM via the web, nor SSH into DOM0. Fortunately, there is a way to reboot the S8300D from the gateway itself. Log into the gateway as root, then type “configure” to get to the config menu. Then simply type “reset mm v1”, which will reboot the media module in slot V1. Confirm, and the card will reboot. From my understanding, it will cleanly shut down each VM and reboot.

As always, it will take just long enough for you to panic, plus 45 seconds or so.

 

TraceSM is already running – how to stop it

In the new version of Session Manager (6.3 and above), the traceSM process is supposed to timeout and stop on its own. However, I was just trying to run it today and got this error

[cust@la-sessionmanager ~]$ traceSM
ERROR: traceSM is already running. Only one instance is allowed.

Uh oh. I don’t know the root password so I cannot kill it. However, Avaya thought of this for me. Just run traceSM with the -k switch and all is well:

traceSM -k

This kills the old process and runs a new instance for me. Just another thing that sounds simple once you know it.

How to build a T1/PRI from scratch on Avaya Communication Manager

This is almost an oxymoron. PRI interfaces are so common on PBXs that you’d think we would all be familiar with the process of building them. However, since these interfaces don’t change very often, sometimes we will go several years between PRI configuration. Here’s a quick reference for setting up a PRI. This one will use AT&T’s IPFlex BVoIP service, but most PRIs are configured the same way nowadays.

Here are the steps we will perform:

  1. Determine the physical location for the PRI (i.e. what port network or gateway and what slot?).
  2. Determine the trunk group number you will use and the TAC (trunk access code). Often, these will correspond with a site or media gateway. Just do a ‘list trunk’ and you’ll probably see a pattern where the trunk group number should fit and the TAC is almost always based upon the trunk group number.
  3. Find out the line coding and framing for the T1. Almost all T1s are now ‘b8zs’ and ‘esf’. Your carrier can confirm this, but I’ve never seen a voice PRI installed after the mid-1990s anything else.
  4. Find out the service protocol to use. This one is tricky and depends upon the emulation of the carrier’s switch. In fact, their planning docs will often ask you which one you want. And half the time it’s wrong anyway. If the D-channel doesn’t come up, this is usually the problem. It’s either ‘a’ for AT&T custom or ‘b’ for NI-2. It looks like this is used in three different places. The DS1 form, the signaling group form, and the trunk group form all have signaling protocol fields! However, the signaling and trunk group fields are different. When the carrier asks you “are you AT&T or NI2?”, they mean the ds1 itself. I have added some notes at the bottom of this post regarding the service protocols.

First, make sure you have a PRI slot available. I just pushed a PRI card into slot 5 of my G450 gateway (media gateway 10 in my case) so it looks like this:

list configuration media-gateway 10                                    Page   1

                              SYSTEM CONFIGURATION

Board                                                     Assigned Ports
Number   Board Type              Code     Vintage    u=unassigned t=tti p=psa

010V1    ICC MM                   S8300D  HW06 FW001
010V2    ANA MM                   MM716AP HW06 FW098 01 02 03 04 05 p  p  p
                                                     p  p  p  p  p  p  p  p
                                                     p  p  p  p  p  p  p  p
010V4    DS1 MM                   MM710BP HW11 FW052 01 02 03 04 05 06 07 08
                                                     09 10 11 12 13 14 15 16
                                                     17 18 19 20 21 22 23 24
                                                     u  u  u  u  u  u  u  u
010V5    DS1 MM                   MM710BP HW11 FW049 u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u



                press CANCEL to quit --  press NEXT PAGE to continue
ESC-x=Cancel Esc-e=Submit Esc-p=Prev Pg Esc-n=Next Pg Esc-h=Help Esc-r=Refresh

Slot 5 is nice and clean with 24 unassigned ports. First we need to create the DS1, which is really creating the D-channel on port 24. I type

add ds1 010v524

and the DS1 form comes up. It’s only two pages. Fortunately, I have a couple working DS1s from AT&T already so I’ll refer to those. By the time I’m done, my DS1 looks like this:

display ds1 010v524                                             Page   1 of   2
                                DS1 CIRCUIT PACK

            Location: 010V5                           Name: Atlanta BVoIP
            Bit Rate: 1.544                    Line Coding: b8zs
   Line Compensation: 1                       Framing Mode: esf
      Signaling Mode: isdn-pri
             Connect: network
   TN-C7 Long Timers? n                   Country Protocol: 1
Interworking Message: PROGress            Protocol Version: b
Interface Companding: mulaw                            CRC? n
           Idle Code: 11111111
                              DCP/Analog Bearer Capability: 3.1kHz

                                           T303 Timer(sec): 4


      Slip Detection? n                 Near-end CSU Type: integrated

   Echo Cancellation? n          Block Progress Indicator? n




display ds1 010v524                                             Page   2 of   2
                              DS1 CIRCUIT PACK

 ESF DATA LINK OPTIONS

                     Network Management Protocol: tabs
 Send ANSI-T1.403 One-Second Performance Reports? n
                             Far-end CSU Address: b

 INTEGRATED CSU OPTIONS
                                    Transmit LBO: 0db
                                    Receive ALBO: 26db
                                    Upon DTE LOS: loopback


 CPE LOOPBACK JACK OPTIONS
                  Supply CPE Loopback Jack Power? n




Now we create the signaling group, which is the communication layer on the D-channel. Signaling groups can be shared (rarely), so these are split up. They’re very simple. The form is 5 pages long, but page 1 is all that matters:

display signaling-group 110                                     Page   1 of   5
                                SIGNALING GROUP

 Group Number: 110            Group Type: isdn-pri
                    Associated Signaling? y          Max number of NCA TSC: 0
                       Primary D-Channel: 010V524     Max number of CA TSC: 0
                                                   Trunk Group for NCA TSC:
       Trunk Group for Channel Selection:         X-Mobility/Wireless Type: NONE
      TSC Supplementary Service Protocol: a


There is a chicken-and-egg problem here. The signaling group wants to know what trunk group to use, but we haven’t built the trunk group yet. We will need to modify this after we create the trunk group.

Now create the trunk group. After the first page, almost everything is fine with default values. The only fields I end up changing are “send name”, “send number”, and “show ANSWERED BY” on page 3. I typically submit the form before assigning channels on page 5 to make sure the form submits.

display trunk-group 110                                         Page   1 of  21
                                TRUNK GROUP

Group Number: 110                  Group Type: isdn          CDR Reports: y
  Group Name: Atlanta BVoIP               COR: 1        TN: 1        TAC: 8900
   Direction: two-way        Outgoing Display? n         Carrier Medium: PRI/BRI
 Dial Access? n              Busy Threshold: 255  Night Service:
Queue Length: 0
Service Type: public-ntwrk          Auth Code? n            TestCall ITC: rest
                         Far End Test Line No:
TestCall BCC: 4


display trunk-group 110                                         Page   2 of  21
      Group Type: isdn

TRUNK PARAMETERS
         Codeset to Send Display: 6     Codeset to Send National IEs: 6
        Max Message Size to Send: 260   Charge Advice: none
  Supplementary Service Protocol: a     Digit Handling (in/out): enbloc/enbloc

            Trunk Hunt: cyclical
                                                   Digital Loss Group: 13
Incoming Calling Number - Delete:     Insert:                 Format:
              Bit Rate: 1200         Synchronization: async    Duplex: full
 Disconnect Supervision - In? y  Out? n
 Answer Supervision Timeout: 0
          Administer Timers? n        CONNECT Reliable When Call Leaves ISDN? n
             XOIP Treatment: auto    Delay Call Setup When Accessed Via IGAR? n


display trunk-group 110                                         Page   3 of  21
TRUNK FEATURES
          ACA Assignment? n            Measured: none      Wideband Support? n
                                                          Maintenance Tests? y
                               Data Restriction? n     NCA-TSC Trunk Member:
                                      Send Name: y      Send Calling Number: y
            Used for DCS? n                            Send EMU Visitor CPN? n
   Suppress # Outpulsing? n    Format: natl-pub
 Outgoing Channel ID Encoding: preferred     UUI IE Treatment: service-provider

                                                 Replace Restricted Numbers? n
                                                Replace Unavailable Numbers? n
                                                      Send Connected Number: n
                                                  Hold/Unhold Notifications? n
             Send UUI IE? y     Modify Tandem Calling Number: no
               Send UCID? n
 Send Codeset 6/7 LAI IE? y                         Ds1 Echo Cancellation? n

    Apply Local Ringback? n               US NI Delayed Calling Name Update? n
 Show ANSWERED BY on Display? n      Invoke ID for USNI Calling Name: variable
                             Network (Japan) Needs Connect Before Disconnect? n


We still have two things to do – we need to set the trunk group for channel selection in the signaling group, and we need to assign the channels. Before I forget, change the signaling group to tell it what trunk group it will control:

change signaling-group 110                                      Page   1 of   5
                                SIGNALING GROUP

 Group Number: 110            Group Type: isdn-pri
                    Associated Signaling? y          Max number of NCA TSC: 0
                       Primary D-Channel: 010V524     Max number of CA TSC: 0
                                                   Trunk Group for NCA TSC:
       Trunk Group for Channel Selection: 110     X-Mobility/Wireless Type: NONE
      TSC Supplementary Service Protocol: a

Now, for the last step, let’s assign channels! Change the trunk group and set these on page 5 and 6. Note how we assign the signaling group along with the channel. When I get the circuit ID from AT&T, I will put it in the “Name” column along with the DIDs that were provided on this circuit.

change trunk-group 110                                          Page   5 of  21
                                 TRUNK GROUP
                                      Administered Members (min/max):   1/23
GROUP MEMBER ASSIGNMENTS                  Total Administered Members:  23

       Port    Code Sfx Name        Night           Sig Grp
  1: 010V501  MM710  B                                110
  2: 010V502  MM710  B                                110
  3: 010V503  MM710  B                                110
  4: 010V504  MM710  B                                110
  5: 010V505  MM710  B                                110
  6: 010V506  MM710  B                                110
  7: 010V507  MM710  B                                110
  8: 010V508  MM710  B                                110
  9: 010V509  MM710  B                                110
 10: 010V510  MM710  B                                110
 11: 010V511  MM710  B                                110
 12: 010V512  MM710  B                                110
 13: 010V513  MM710  B                                110
 14: 010V514  MM710  B                                110
 15: 010V515  MM710  B                                110


change trunk-group 110                                          Page   6 of  21
                                 TRUNK GROUP
                                      Administered Members (min/max):   1/23
GROUP MEMBER ASSIGNMENTS                  Total Administered Members:  23

       Port    Code Sfx Name        Night           Sig Grp
 16: 010V516  MM710  B                                110
 17: 010V517  MM710  B                                110
 18: 010V518  MM710  B                                110
 19: 010V519  MM710  B                                110
 20: 010V520  MM710  B                                110
 21: 010V521  MM710  B                                110
 22: 010V522  MM710  B                                110
 23: 010V523  MM710  B                                110
 24:
 25:
 26:
 27:
 28:
 29:
 30:

And now it’s a matter of troubleshooting. I plugged in my PRI into the BVoIP router and the first thing is to make sure the D-channel came up. Do this by statusing the signaling group:

status signaling-group 110
                         STATUS SIGNALING GROUP

        Group ID: 110                             Active NCA-TSC Count: 0
      Group Type: isdn-pri                         Active CA-TSC Count: 0
  Signaling Type: facility associated signaling
     Group State: in-service


                           Primary D-Channel


            Port: 010V524        Level 3 State: in-service


                          Secondary D-Channel


            Port:                Level 3 State: no-link

The “Level 3 State” is the state of the D-Channel. If you’re on the phone with your carrier and they ask about the D-Channel, you can give them that status.

Next, status the trunk group itself:

status trunk 110

                             TRUNK GROUP STATUS

Member   Port     Service State      Mtce Connected Ports
                                     Busy

0110/001 010V501  in-service/idle    no
0110/002 010V502  in-service/idle    no
0110/003 010V503  in-service/idle    no
0110/004 010V504  in-service/idle    no
0110/005 010V505  in-service/idle    no
0110/006 010V506  in-service/idle    no
0110/007 010V507  in-service/idle    no
0110/008 010V508  in-service/idle    no
0110/009 010V509  in-service/idle    no
0110/010 010V510  in-service/idle    no
0110/011 010V511  in-service/idle    no
0110/012 010V512  in-service/idle    no

This gives you the channel state within the trunk group. These channels will only come up in-service/idle if the D-channel is good. If you have any problems, you can try to “bounce” the D-channel by perform a busyout then release on the port of the D-channel with “busy port 010v524” in my case:

busyout port 010v524

                             COMMAND RESULTS

  Port      Maintenance Name  Alt. Name          Result           Error Code

  010V524   ISDN-PLK                             PASS

release port 010v524

                             COMMAND RESULTS

  Port      Maintenance Name  Alt. Name          Result           Error Code

  010V524   ISDN-PLK                             PASS

And if the D-channel still won’t come up, that’s when you tweak the service protocol values between ‘a’ and ‘b’.

When you’re on the phone with your carrier trying to turn up service, the ability to bounce the D-Channel, monitor the D and B channel status, and change from AT&T custom to NI-2 is typically enough to get the carrier to work with you and get this working. On rare occasions you may need to tweak the line coding (b8zf/esf) but this has only happened to me once.

As for service protocols, a good friend of mine provided a nice summary from his notes. Remember these apply to the DS1! (i.e. “change ds1 010v524”), not the sig or trunk group forms.

Protocol “a” – AT&T Custom

  • For the layer 3 messaging or 4ESS/5ESS messaging type (Maintenance Protocol Discriminator 03) (TR41449/TR41459)
  • Normally used by Long Distance Companies
  • 4ESS is strictly AT&T Custom for FAS/NFAS
  • DMS 100/250 is AT&T Custom for FAS/NFAS (software load BCS.36 or later)
  • DSC DEX600E is AT&T Custom for FAS/NFAS (used BY MCI and Frontier as central offices)

Protocol “b” – National ISDN 2

  • Bellcore protocol (5ESS LEC messaging for NFAS and DCBU applications, the only way the 5ESS can be set up for NFAS and DCBU) (Maintenance Protocol Discriminator 43) (TR1268)
  • Normally used by LECs
  • DMS 100 will do NI-2 FAS/NFAS (started in the BCS.37 load of software)
  • Starting with the 5E9.0 software load in the 5ESS
  • Normal protocol for connection to Siemen’s EWSD
  • Normal protocol for connection to GTE GTD5

Thanks as always for your time and attention. If you’re interested, I can include a post on routing incoming calls from this trunk group, routing calls out this trunk group, and setting the caller-id for calls over this trunk group. Please let me know your own experiences with setting up and using PRI and IP trunks.

How to clear PLAT-ALM alarms in Avaya Communication Manager

When gateways unregister due to various network blips or maintenance, it can cause a PLAT-ALM in your Communication Manager. These will linger forever until you clear them, It’s very simple but you don’t do it from SAT, you do it right from the shell. To see all of the alarms, login to CM’s shell and type

almdisplay -v

To clear them all, just type

almclear -a

If you still have a problem, the alarm will re-propagate so I haven’t seen the downside.

How to remove an extension when there are messages waiting

Just a quick note regarding the message

Message(s) waiting; please delete message(s) first

If you try to remove a station with messages waiting, the Communication Manager will reject it if there are messages waiting. In my case, I wanted to convert a virtual station to a physical 9630, but the system won’t let me remove the station. And of course, I cannot change the type of station from Virtual to 9630 – I get this message:

"9630" Station must be removed and re-added to change type

As usual, the solution is simple once you know it. You can clear the message waiting indicator with “clear amw all xxxx“. This clears the messages flag (not the messages!) and turns off the message waiting indicator. This allows you to remove the station.

How to update firmware in individual Avaya 96xx phones

If you have an Avaya telephone system, you probably have a web server to manage firmware. If not, it’s pretty easy to set up so let me know if I should post the procedures. What is not easy is testing firmware before deploying it to the whole network. What if you just want to update one particular phone? What if someone is complaining about something (e.g. continuous reboots) and you want to update their firmware to see if it fixes it? You could set up a development web server and statically assign the file server address to the phone. But then you have to statically assign all addresses on the phone or the DHCP server will overwrite the file server address. And this isn’t a permanent fix because the phone will forever have a statically assigned address. You could do tricky things in Apache using the IP address of the phone in the http.conf file or .htaccess, but there’s another way.

The trick is in the 96xxupgrade.txt file on your web server. As with most of these posts, we need to assume you have a web server and have access to the root html folder. Find the 96xxupgrade.txt, open it in any text editor and you should see something similar to this (if you’re using the sample file from Avaya):

############################################################
## Check backup application version                       ## 
############################################################
IF $MODEL4 SEQ 9610 goto BACKUPAPP96XX
IF $MODEL4 SEQ 9620 goto BACKUPAPP9640
IF $MODEL4 SEQ 9630 goto BACKUPAPP96XX
IF $MODEL4 SEQ 9640 goto BACKUPAPP9640
IF $MODEL4 SEQ 9650 goto BACKUPAPP96XX
IF $MODEL4 SEQ 9670 goto BACKUPAPP9670
goto END

This is the stuff Avaya put in the sample file so individual types of phones can download different versions of firmware. Note how there’s a check against a “$MODEL4” variable? This allows the phone to jump to a new section after a very basic check. Unfortunately, there are only a few variables available. I’ve never been able to find a definitive list of all available variables, but we can make this work with the GROUP variable.

So let’s say you want to test version ha96xxua3_2_0_S.bin of the 9600 firmware on your 9630 phone at your own desk before deploying to the entire network. In the 96xxupgrade.txt file, go to the section for the 9630s (probably BACKUPAPP96XX if you have the default Avaya file)

Your file will probably have a section like this:

# BACKUPAPP96XX

IF $BOOTNAME SEQ hb96xxua3_1_03_S.bin goto PHONEAPP96XX
SET APPNAME hb96xxua3_1_03_S.bin
goto GETSET

This just tells the phone to check its firmware version and if it’s already 3_1_03_S, then go to PHONEAPP96XX. If not, set its version to 3_1_03_S, which will cause the phone to download the new file, reboot, write it to flash, and reload the 96xxupgrade.txt file. Then when it gets to this line, it will have the version 3_1_03_S and jump to PHONEAPP96XX. Why is there a goto GETSET? Well, that’s there in case the phone cannot find the file. It will just jump to GETSET.

Within the PHONEAPP96XX section, my file says

 # PHONEAPP96XX
SET APPNAME ha96xxua1_1_03_S.bin
goto GETSET

Maybe yours does too. This just sets the application file to the right name. Again, causing the phone to load the file, reboot, write to flash, and reboot again. Then when it gets to this section, it jumps to GETSET. When you download new firmware, the phone reboots four times as it works through all these files.

In GETSET, my file says

GET 46xxsettings.txt
goto END

And this again is an Avaya default. I don’t have any 4600 series phones. I guess I should rename mine to 96xxsettings.txt. But it doesn’t matter. The settings file contains tons of options. Tons.

So back to my point. How do you load different firmware for a single phone? What you can do is perform a check against the phone’s GROUP variable. My 96xxupgrade file now has this line in the BACKUPAPP96XX section:

# BACKUPAPP96XX
IF $GROUP SEQ 30 goto BACKUPAPP30
IF $BOOTNAME SEQ hb96xxua3_1_03_S.bin goto PHONEAPP96XX
SET APPNAME hb96xxua3_1_03_S.bin
goto GETSET

When the phone gets to that line in red, it will check against its own internal GROUP variable and jump to BACKUPAPP30 if it matches. Now you can add the firmware version you want in that section by adding the following lines to the file:

# BACKUPAPP30
IF $BOOTNAME SEQ hb96xxua3_2_0_S.bin goto PHONEAPP30
SET APPNAME hb96xxua3_2_0_S.bin
goto GETSET

# PHONEAPP30
SET APPNAME ha96xxua3_2_0_S.bin
goto GETSET

So the phone will jump to a new section of the file and load specific firmware based upon its GROUP variable. Note that you can also “goto GETSET30” for example if you wanted to load a different settings file for this phone. In my case, I just want to test version ha96xxua3_2_0_S of firmware.

So how to you set the phone’s GROUP variable? Two ways:

Method 1 – Page 3 of the station form within SAT of your Communication Manager:

display station 4799                                            Page   3 of   7
                                     STATION

             Conf/Trans on Primary Appearance? n
   Bridged Appearance Origination Restriction? n


               Call Appearance Display Format: disp-param-default
                            IP Phone Group ID: 30
Enhanced Callr-Info Display for 1-Line Phones? n

                              ENHANCED CALL FORWARDING
                                       Forwarded Destination         Active
 Unconditional For Internal Calls To:                                   n
                   External Calls To:                                   n
          Busy For Internal Calls To:                                   n
                   External Calls To:                                   n
      No Reply For Internal Calls To:                                   n
                   External Calls To:                                   n

            SAC/CF Override: n

This method will “follow the station” so if you log a new extension into the phone, the GROUP will follow the new extension and will likely change. Then when the phone reboots, it will load new firmware. If you log your extension into another phone and then reboot that phone, it will load new firmware.

Method 2 – The “GROUP” menu item on the config screen of the 9630 telephone (Usually accessed by pressing Mute-CRAFT-# on the 9630, or pressing *-CRAFT-# at boot-up).

GROUP-item-in-craft-menu

This method will “follow the phone” so if you log a new extension into the phone, it will retain the phone’s setting. However, it gets over-ridden by the station’s group if there is one. Interaction gets complicated. I always use the station form.

And lastly, this is how we load SIP firmware in the 9630 using the same network and DHCP server we already have. We can check the value of the SIG variable, which is set in the CRAFT menu. But that’s a different post. For now, I hope this helps and I look forward to hearing from you.

Updating firmware on Avaya TN799DP Circuit Packs

There are tons of documents that describe the firmware update procedures for the Avaya circuit packs and media modules. This document is meant to be simple, and as I often do for these posts, it’s is also written for myself for future reference. I have found that I update firmware just a few months after I’ve forgotten how to do it.

I will assume you know what firmware to load. There’s a hard-to-find matrix on support.avaya.com that will tell you and I’ll include this procedure in a future post. For now, let’s say you have the firmware you want for your card. This post is for the TN799DP (C-LAN) cards but the procedures are similar on other cards as well.

Step 1 – Make some notes

You’ll want to have the following handy – paste into notepad or something:

  • The exact card type including the suffix (TN799DP in my case),
  • The board location
  • The current firmware version. “list config all” will tell you these first three.
  • The IP address of the C-LAN. “list ip-interface clan” will tell you this.
  • The username and password you will use for file transfers. This is temporary – File transfer will get disabled automatically when you’re done so don’t worry about crazy passwords.
  • The exact filename of the firmware.
list configuration all                                                 Page   1

                              SYSTEM CONFIGURATION

Board                                                     Assigned Ports
Number   Board Type              Code     Vintage    u=unassigned t=tti p=psa

01A00    POWER SUPPLY              655A
01A01    IP SERVER INTFC         TN2312BP HW36 FW056 01 02 03 04 05 06 07 08
01A03    CONTROL-LAN              TN799DP HW01 FW040 u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u
                                                     17
01A04    DS1 INTERFACE            TN464GP HW02 FW025 01 02 03 04 05 06 07 08
                                                     09 10 11 12 13 14 15 16
                                                     17 18 19 20 21 22 23 24
                                                     25 26 27 28 29 30 31 u
01A05    CONTROL-LAN              TN799DP HW16 FW040 u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u
                                                     17
01A06    ANALOG LINE              TN746B  000013     01 02 03 04 05 06 07 08
                                                     09 10 11 12 13 14 15 16
list ip-interface clan

                                IP INTERFACES

                                                            Skts  Net       Eth
ON  Slot   Code/Sfx Node Name/       Mask  Gateway Node     Warn  Rgn VLAN  Link
                    IP-Address
--  ----   -------- ---------------  ----  ---------------  ----  --- ---- ----
 y  01A03  TN799  D CLAN_01A03       /24   Gateway001       400   6   n    1
                    10.10.11.5
 y  01A05  TN799  D CLAN_01A05       /24   Gateway001       400   6   n    2
                    10.10.11.9
 y  01B10  TN799  D CLAN_01B10       /24   Gateway001       400   6   n    4
                    10.10.11.8

At the time of this posting, I want those C-LANs to say FW043. Note there’s a special case with C-LANs. I have one at hardware version HW01, and one at hardware version HW16. There are two different version 43 firmware files for these!

Step 2 – Enable file transfer on the C-LAN

From the CLI, type “enable filexfer” and fill out the form. If file transfer is already enabled for the board you want, you may need to disable it and re-enable it. If there are files already on the board, you will need to remove them. You can “list directory board xxxx” to see the files, and “remove directory board xxxxx” to remove them. Then you can disable the filexfer and re-enable it with the password you want.

enable filexfer                                                 Page   1 of   1


                                 ENABLE FILE TRANSFER

                    Login: roger
                 Password:
         Reenter Password:
                   Secure? y
            Board address: 01a05

Step 3 – Transfer the file to the C-LAN

Use whatever file transfer method (WinSCP, ftp, ssh, sftp, etc) to move the firmware file to the card. When it’s done, you can “list directory board xxxx” to make sure the file transferred correctly. Avaya is very good at confirming the file transferred correctly and is appropriate to your card. In my humble opinion, Avaya’s deep roots in telecom have made this process smooth. It’s designed for remote work and verification. Of course now I have jinxed myself and will brick-a-clan tonight when I update my system.

list directory board 01A05

                                LIST DIRECTORY

Board                File/Directory                    Creation       Size  Pro-
Loc                      Name                       Date       Time    Kb   tect
01A05 F:tn799dp-h13-f43-sig.bin                   2014/04/24 15:53:54 4729   n

Step 4 – Start the update

This is the part that affects service. If your C-LAN is serving any phones (“status socket” to see), then you should wait until after-hours. C-LANs serve 400 phones maximum, so you could disable phone registrations on the C-LAN after hours and update the card mid-day if you’d like. If you “change ip-interface xxxxx” you’ll see a parameter called “Allow H.323 Endpoints”. The scary part is you have to set “Enable Interface” to N before you can change that. The first time I did this, I was worried that I wouldn’t be able to manage the C-LAN but don’t worry – you are managing it via the G650 backplane so you can disable the interface and still manage the card.

Open a separate command prompt and continuously ping the C-LAN’s IP address.

Now type “change firmware download 1” and fill out the form. For the source board, use your C-LAN, file server is “none”. The image file name is the exact name of your firmware file including the “.bin” extension. The target board codes is TN799 and the Suffix is DP. Vintage is blank. I never schedule the updates, so I say “n” to schedule. According to my docs, the C-LAN can only be updated from its own file system, so you might as well “Remove Image File After Successful Download” so you don’t have to delete it yourself (this also disables file transfer for you). The target location is the same as the Source Board Location (i.e. itself), and submit.

change firmware download 1                                      Page   1 of   1
                               FIRMWARE DOWNLOAD          Schedule: 1
   Source Board Location: 01a05        File Server: none

Firmware Image File Name: tn799dp-h13-f43-sig.bin
       Target Board Code: TN799   Suffix: DP  Firmware Vintage:
       Schedule Download? n


    Target          Target          Target          Target          Target
   Location        Location        Location        Location        Location
 1. 01a05       11.             21.             31.             41.
 2.             12.             22.             32.             42.
 3.             13.             23.             33.             43.
 4.             14.             24.             34.             44.
 5.             15.             25.             35.             45.
 6.             16.             26.             36.             46.
 7.             17.             27.             37.             47.
 8.             18.             28.             38.             48.
 9.             19.             29.             39.             49.
10.             20.             30.             40.             50.

This is where Avaya confirms the firmware file is complete, signed correctly, and is appropriate for your card. In the past, I have intentionally tried to update with the wrong firmware and I get an Abort code. Anyway, submit the correct filename and then do a “stat firm down 1”

Step 5 – Monitor the update

Now wait. At some point the pings will drop then come back. You can check the status of the update with “status firmware download 1”. The status code will be Pending, Complete, Failed, or Aborted. If failed or aborted – check the documentation for the reason code. If you’re like me, you have tons of PDFs of Avaya documentation in your hard drive. I end up using Google anyway.

status firmware download 1
                            STATUS FIRMWARE DOWNLOAD      Schedule: 1
   Source Board Location: 01A05        File Server: none

Firmware Image File Name: tn799dp-h13-f43-sig.bin
       Target Board Code: TN799   Suffix: DP  Firmware Vintage: 43
       Schedule Download? n
   Start Date/Time: 05/02/2014  00:01    Stop Date/Time:   /  /        :

   Target    Status    Reason              Target    Status    Reason
   Location            Code                Location            Code
 1. 01A05    Pending                    11.

You should see “Pending” for the download state. Soon your pings will drop. It takes a few minutes. In my experience it will take just long enough for me to panic, plus 30 seconds. Not worried, but actually concerned. After 30 seconds of “oh crap I’m going to have to drive the site”, the update completes. I have noticed the HW01 vintage was MUCH slower than the HW16.

If you have a lot of updates and trust the system, you can run these in parallel with “change/status firmware download x” where x is between 1 and 4. So you can have up to four firmware updates running at a time. I’ve done it – it works. And by the time you’re done with the fourth one, the first one is probably done.

 Step 6 – Verify

When the update is done, the “status firm down 1” will blank out. When you “list config” you should see your new firmware version. However, the ping did not come back! I am embarrassed to say I panicked. I did a “busy board” If you disabled the ip-interface then enable it. Type “status sock” and check for registered phones. Phones should move back to it as the CM balances the traffic. you can also disable other ip-interfaces momentarily to encourage phones to move to it immediately.

status socket-usage
                           SOCKET USAGE

                                 Total Registered IP Endpoints: 872
 Registered IP Endpoints with TCP Signaling Socket Established: 872
                                            Total Socket Usage: 00926/02900

Intf         Board Socket      Net    Intf          Board Socket      Net
Type   Loc    Sfx  Usage       Rgn    Type   Loc    Sfx   Usage       Rgn

procr              442/1700    250
CLAN   01A03   D   162/400     1
CLAN   01A05   D   161/400     1
CLAN   01B10   D   161/400     1

 What if the ping doesn’t come back?

Strangely, the ping doesn’t stop when you disable the ip interface. But once the firmware is done loading and the “stat firm down 1” shows nothing and the “list config” shows the right SW version, what if the pings still timeout? Well first, give it a good long time. You really need to panic before the firmware will finish. If that doesn’t seem to work, then “change ip-i xxxxx” and enable the ip-interface. The pings respond? Mine did. If yours don’t, you can try “busy board xxxxx” then “release board xxxxx”. That always works for me. If it still doesn’t work, Avaya would probably recommend pulling the board and re-seating it. There are a bunch of test procedures though. I assume performing a busy/release would Abort if you try it while the firmware is loading, but I don’t want to try. Let me know if you do it.

Summary

That’s it. The first time is scary, then it gets easier. Especially since this is done so infrequently that, as usual I start to forget the exact procedures. Good luck all and let me know how it goes. Thanks as always, Roger.