Monthly Archives: October 2015

Avaya System Manager – unable to login

There are many reasons for the login to fail on System Manager. Version 6.3.x seems better but this thing still feels like it’s built on a house of cards. Tonight I just found another reason and it was new to me so I want to tell you all about it.

Earlier today I tired to login and was confronted with this error

avaya-system-manager-internal-error

Some internal error has occurred in the service. Please contact the support team.

Guess who the support team is? Oh yeah, that’s us!

Well, I didn’t really know what to do and it was a Sunday so I figured I would reboot it. Now my System Manager is a virtual machine and I don’t have access to the VMWare console. But I was able to SSH into System Manager as admin. Then su and enter the root password. From there, I was able to issue a reboot command. It dropped my session, and then it never did come back to ping. So I contacted my VM team and they rebooted it. Once they did that, I was able to ping and SSH to it. But the web page said “This page has a redirect loop”. I knew it could take a long time to come back up, so I gave it a while. But no dice. I contemplated rebooting it again, but rather than that, I went through the “ask Ava” chat pages on support.avaya.com. I was hoping with such a specific error like “redirect loop” I would find something and sure enough, Ava pointed me to SOLN270630, which was exactly my problem. According to this page, the Postgres certificate had expired. Avaya provided the command to confirm it and the command to fix it. All I had to do was run that command and then restart jboss. Then wait. Don’t forget, it can take 15-20 minutes for the restart to finish. Don’t panic. Actually, I think you do have to panic. Three minutes after you start to panic it comes back up. Don’t forget to re-enable Geographic Redundancy!

root >openssl x509 -text -in /var/lib/pgsql/data/server.crt | grep "Not After"
 Not After : Oct 22 15:14:31 2015 GMT
root >/opt/Avaya/Postgres/*/utils/securePostgres.sh
Mon Oct 26 00:13:18 EDT 2015 : Restarting Postgres service...
Stopping postgresql service: [ OK ]
Starting postgresql service: Waiting for Postmaster PID file creation .
Waiting for Postmaster DB connection availability .
Starting postgresql service: [ OK ]
Mon Oct 26 00:13:21 EDT 2015 : Startup of Postgres done
root >service jboss restart
Perform cleanup ....
Stopping System Manager JBoss - PID: 6580
.............................................
Stopped System Manager JBoss
Starting System Manager JBoss
.
Service started successfully - PID: 21296
root >

Avaya CM – will I lose the CDR buffer if I busy/release the cdr link?

I have an Avaya Communication Manager sending CDR to a kiwi syslog server. For some reason, the link is down and the buffer is filling up. What happens if I busy/release the CDR link? Will I lose the buffer? Scary!

Short answer is no. I just tried it. I performed a busy/release on the primary CDR link and the buffer is still at 40{0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0}. Unfortunately, the link is still down so I’ll have to check with the syslog team. But I just wanted to let all of you know that it’s safe to restart the link.

status cdr-link
                                CDR LINK STATUS
                   Primary                      Secondary

       Link State: down                         up
Number of Retries: 18
      Date & Time: 2015/09/23 06:35:59          2015/09/12 07:46:56
  Forward Seq. No: 0                            0
 Backward Seq. No: 0                            0
CDR Buffer {0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0} Full:  41.21                         0.00
      Reason Code: CDR connection is closed     OK


I ended up disabling and then enabling the TCP listener in Kiwi and the buffer immediately cleared. Is anyone interested in a post on how to capture CDR to a SYSLOG server?

Flowchart of Avaya Communication Manager Routing

Click Here for the flowchart PDFs

One might argue that telephone systems do only one thing. They route calls. In the process they turn lamps on and off, and they make telephones buzz and beep. But the only point is to route voices from point A to point B.

If you’ve been working with Avaya Communication Manager for a while, you have been exposed to AAR and ARS routing. And you hopefully have a fair grasp of how it all fits together. But there are a whole lot of decisions that are made when routing outbound calls. And if you’re new to Avaya, it might seem overwhelming to troubleshoot routing issues. So here are four pages of flowcharts that, based on my experience, are the most common configuration issues you’ll see when managing your telephone system. Click Here for the flowchart PDFs

Page 1 is a flowchart of the digit collection. This is what the CM does to collect the dialed digits, translate them as necessary, and select a network (AAR vs. ARS). Digits can be translated many times and re-pointed to different networks. Once this process is complete, the logic continues to page 2.

avaya_call_routing_flowchart-page_1_of_4

Page 2 is the process of using the final dialed digits (and station’s location) to select a route and a trunk group. There are many location and facilities restriction level checks here. And there is even more digit translation that can take place during this process. Once a trunk is selected and the call delivered, this may not be the end of it. Look Ahead Routing may pull the call back and try again.

avaya_call_routing_flowchart-page_2_of_4

Those two pages compose the primary routing logic that you will use in your typical troubleshooting. However, many decisions are based upon the calling party’s location. It may be obvious to you how the CM know’s the station’s location. But if things get strange, you can see how the CM associates a station to a location on page 3.

avaya_call_routing_flowchart-page_3_of_4

And lastly, there is one major headache associated with the “Prefix Mark” in a route pattern. Prior to 1995, there were many, many dialing rules associated with seven digit dialing, one plus seven digits, ten digits, etc. I remember living in a small town in the California Bay Area, and to dial Concord (over the hill), I had to dial 1+seven digits. Then when the area code 415 was split in to 415/510, we could dial 10 digits between them. Unfortunately, if you dialed 1+10 digits, it would be charged as a long distance call. All this went away in 1995 when the North America Numbering Plan was implemented. It is much easier to route calls now. But there is still the dreaded “Prefix Mark” that may still be in your routes. This causes havoc with the dialed-digits that are sent to your carrier. Page 4 is a summary of how this prefix mark works.

avaya_call_routing_flowchart-page_4_of_4

Please note there are two extremely important troubleshooting tools when dealing with Avaya routing. One, is the ‘list ars route-chosen’ and ‘list aar route chosen’ commands, and the other is the ‘list trace station’ and ‘list trace tac’ commands. If you use Avaya Site Administrator, you will not be able to use the ‘list trace’ command. I encourage you to use telnet or ssh in your day-to-day management of your PBX so you can use these commands when necessary. It may not sound like much, but being able to list trace at a moment’s notice is very helpful.

I will save the “list aar/ars route-chosen” and “list trace station/tac” for another posting. There’s a lot of information available in these commands.

Please let me know if you find these flowcharts helpful. Also, if you find any typos please let me know. Please keep in mind there are many more routing decisions the CM makes that are not in these flowcharts. If your CM uses other routing (for example tenants or time-of-day) and you think I should incorporate them into these flowcharts, please let me know! Happy routing everyone. This is really the core of any PBX, so I personally really enjoy working with routing.

Roger

Click Here for the flowchart PDFs