Category Archives: Uncategorized

Unable to log into the Avaya Utility Server as cust or admin on System Platform

I recently had a strange issue with my utility servers. I have four of them on System Platform on the S8300D servers. But I was unable to log into any of them as cust nor admin. I could log into my standalone utility servers, but not the ones on system platform.

It turns our the ldap config was corrupted on the servers and Avaya needed to fix it. Once complete, the tech sent me a nice summary:

If this were to happen again, you can verify if the account works by attempting login to SSH of Dom0 or Cdom or webconsole of Cdom.  If it works there, then we know that LDAP configuration is corrupted and needs to be corrected.  If it happens again, have a ticket open stating that LDAP is corrupted on VUS and that /etc/ldap.conf and /etc/nsswitch.conf need to be fixed.

no VNS resource administered

Here’s one of those times when I’m in the Avaya Communication Manager and I get an error. Happens to me all the time. What do you do when you get an error? Google it? Me too!

In my case, I wanted to see if I could get a report of packet loss from one of my network regions. Sure enough, there’s a packet loss report. You can perform this “list measurements” command:

list measurements ip voice-stats hourly pktloss 6

No VNS resource administered

Hmm, no VNS resource. So I type that into google with quotes and there are no results! How can there be no results? I opened a ticket with Avaya and their expert systems didn’t return anything either. Later I was looking through some documentation and I recalled that you sometimes have to tell Communication Manager what you want to measure. Sure enough, if you “change measured selection network region”, you can add the network regions you want monitored.

Now when I type “list measurements ip voice-stats hourly pkloss 6” I get an empty report. Hopefully tomorrow I’ll have some stats. Do you ever wish you could automate this report? More on that later.

Thanks all!


What happens if you try to “traceSM” when the Avaya Session Manager hasn’t fully booted?

If you try to “traceSM” before the Avaya session manager is completely booted up, you get this. But only when you “start” the trace.

The complete log files are located:
        SIP messages: /var/log/Avaya/trace/tracer_asset.log*
        Call Processing: /var/log/Avaya/trace/call_proc.log*
        PPM: /var/log/Avaya/trace/trace_ppm.log*
Error enabling SM100 trace.

Preparing SCRUCH network filter num SD address:0
Preparing SCRUSH network filter num SH address:0
*** in client update res=28 ***

[cust@nj-pbx-sm02 ~]$

How to update Avaya System Manager from 6.3.3 to 6.3.9

Updating System Manager should not be scary. As I write this, I am updating my Avaya Aura System Manager from 6.3.3 to 6.3.9. I just have to keep the faith in the Avaya engineers.

So here are the steps to perform upgrade this upgrade. It is definitely something you, as a customer, can do yourself. These instructions are from page 22 of this document

  1. Download the patch System_Manager_6.3.9_r4602482.bin from
  2. Use WinSCP or (or any SCP program) to upload it to System Manager. I used the admin account (with the same password as the web interface).
  3. SSH to the System Manager using the same account as the file upload. The admin account didn’t give me a $ prompt at the shell, but it worked fine. When you login, you should see the .bin file in the default folder (/home/admin/ in my case)
  4. Before you start, you should disable geo-redundancy if you use it. Go to Home->Georedundancy and disable it.
  5. Simply type “SMGRPatchdeploy System_Manager_6.3.9_r4602482.bin” and the patch manager will take over.
  6. The Patch manager will ask you to review and accept the license agreement, then start running. I was warned that it will take about an hour and I should be patient. Unfortunately, I need to test this before I do the redundant session manager, then I need to do my system managers. So I’m looking at a late night and I’d like this to finish. I think my System Manager can tell so it’s taking its time. As I’ve said before – just long enough to panic, plus 30 seconds. In this case, I think it’ll be plus 10 minutes or so. Here is the output from my patching:
admin >SMGRPatchdeploy System_Manager_6.3.9_r4602482.bin
Verifying the patch binary.....
mv: cannot stat `/var/log/Avaya/SMGR_Patch.log.3': No such file or directory
Extracting files to /tmp/patchsfx.p18665...

Read the following Avaya Software License Terms carefully. The Chinese, German, Portugese, Spanish, French, Japanese, and Russian versions of the Avaya Software License Terms are posted at . If you do not have direct Internet access, please access a device that has Internet access to view, download and print the applicable license.

Press the [Enter] key to read the Avaya Software License Terms.

If you agree with Terms and Conditions of the applicable Avaya Software License Terms, please enter "Y" for yes. If you do not agree with the Terms and Conditions, please enter "N" for No.

If you enter "Y", you have read, understand, and agreed to the Terms and Conditions of the Avaya Software License Terms (Avaya Products).  You further understand that by entering "Y" you will initiate the process to upgrade, and/or install the Avaya Product; and continued installations is acceptance of the Terms and Conditions of the applicable Avaya Software License Terms (Avaya Products).

If you do not agree to these terms and enter "N" it will abort the installation/upgrade of this Software.

Do you accept the Avaya Software License Terms? (Y)es/(N)o: y

You have accepted the Avaya Software License Terms. The Software Installation will now begin.
Executing checkForPatchRequisite
Executed checkForPatchRequisite successfully, proceed to install the patch.
Thu Oct 16 22:10:09 EDT 2014 Continue the patch installation in normal way.
Perform cleanup ....
Stopping System Manager JBoss - PID: 6928
Stopped System Manager JBoss
/tmp/patchsfx.v29174 /tmp/patchsfx.v29174
Enter the Encryption Key :
Please Wait

Log file has been created at location: /tmp/avaya_aura_install-log_2014-10-16_10.11.42.txt
[ Starting automated Installation ]

Checking if the system has sufficient space as required by the selected packs.

Available Space: 13.9 GB
Total space Required: 1.54 GB

[ Starting to unpack ]

[ Processing package: System Manager - Update Patch (1/15) ]
[ Processing package: Conferencing - Update Patch (2/15) ]
[ Processing package: PS SysMgr Extensions (3/15) ]
[ Processing package: IPTCM - 6.2 FP4 S4 (4/15) ]
[ Processing package: IP Office - 6.2 FP3 S4 (5/15) ]
[ Processing package: SUM - 6.2 FP4 S4 (6/15) ]
[ Processing package: MESSAGING - 6.2 FP4 SP9 (7/15) ]
[ Processing package: INVENTORY - 6.2 FP4 S4 (8/15) ]
[ Processing package: MMCS (9/15) ]
[ Processing package: CS1000 - 6.3 FP4 (10/15) ]
[ Processing package: Session Manager Element Manager (11/15) ]
[ Processing package: Collaboration Environment Element Manager (12/15) ]
[ Processing package: AMM - Update Patch (13/15) ]
[ Processing package: System Manager on VMware (14/15) ]
[ Processing package: System Manager Update (15/15) ]
Parsing files
File execution started for pack : System Manager - Update Patch
File execution started for pack : Conferencing - Update Patch
File execution started for pack : PS SysMgr Extensions
File execution started for pack : IPTCM - 6.2 FP4 S4
File execution started for pack : IP Office - 6.2 FP3 S4
File execution started for pack : SUM - 6.2 FP4 S4
File execution started for pack : MESSAGING - 6.2 FP4 SP9
File execution started for pack : INVENTORY - 6.2 FP4 S4
File execution started for pack : MMCS
File execution started for pack : CS1000 - 6.3 FP4
File execution started for pack : Session Manager Element Manager
File execution started for pack : Collaboration Environment Element Manager
File execution started for pack : AMM - Update Patch
File execution started for pack : System Manager on VMware
File execution started for pack : System Manager Update
[ Unpacking finished ]
[ Writing the uninstaller data  ... ]
output.response.file /opt/Avaya/ generated
[Automated Installation Completed !]
Log file has been moved to location: /opt/Avaya/install_logs/avaya_aura_install-log_2014-10-16_10.11.42.txt from location: /tmp/avaya_aura_install-log_2014-10-16_10.11.42.txt
Executing Common Inventory Datamigration script for access point & attributes.
dos2unix: converting file /opt/Avaya/Mgmt/6.3.8//bulkadministration/exportutility/ to UNIX format ...
Thu Oct 16 23:18:14 EDT 2014 Restarting JBoss...
Service was not running
Starting System Manager JBoss
Service started successfully - PID: 21891
Thu Oct 16 23:18:42 EDT 2014 Patch installation completed and waiting for JBoss to finish start-up... this may take a few minutes
Thu Oct 16 23:39:45 EDT 2014 Patch installation completed and waiting for Quantum init to finish ... this may take a few minutes
QuantumElementType Version mis match in QuantumElementType.xml and DataBase so republishing the QuantumElementType.
dos2unix: converting file tmConfigProfile.xml to UNIX format ...
dos2unix: problems converting file tmConfigProfile.xml
log4j:WARN No such property [conversionPattern] in com.avaya.common.logging.client.CommonLoggingLayout. line 53: ${LOGFILE}: ambiguous redirect
Thu Oct 16 23:48:19 EDT 2014 : Check for the EP Status at the end of patch installation.
START : Execution of
END : Executed with status: 0
Thu Oct 16 23:48:19 EDT 2014 : Executed the domains and Adopter scripts successfully.
Thu Oct 16 23:48:19 EDT 2014 Changing the needful permission for various dirctories
Changed the ownerships of directories successfully..
Thu Oct 16 23:48:20 EDT 2014 Maintenance completed Successfully
admin >
admin >shutdown -r now
shutdown: you must be root to do that!
admin >su root

        * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
        *                                                                     *
        * !! Virtual Machine needs to be rebooted as System Manager Patch     *
        *    installation updated the Kernel. !!                              *
        *                                                                     *
        * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

root >shutdown -r now

Broadcast message from root (pts/0) (Fri Oct 17 00:34:51 2014):

The system is going down for reboot NOW!
root >

Troubleshooting Signaling Manipulation (SigMa) Scripts in Avaya Aura SBC

This is pretty obscure and I’m probably only talking to three people on the Internet right now, but if you have ever tried to troubleshoot SigMa scripts in the Avaya Aura Session Border Controller, you probably wish it had better logging. Actually, if you have ever gotten logging to work, please let me know. I had given up. However – I recently discovered a nice easy way to troubleshoot signaling manipulation, and consequently also discovered just how cumbersome this process really is.

The easy way to log is to set your own custom header in the SigMa script, then inspect the INVITE from the Session Manager’s point of view. For example, I added this SigMa script:

    act on request where {0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0}DIRECTION="INBOUND" and {0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0}ENTRY_POINT="PRE_ROUTING" {
        if({0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0}HEADERS["To"][1].URI.USER.regex_match("2025556199"))then {
            {0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0}HEADERS["roger-debug"][1] = {0ed28e3470e974017c124b0897303dd14e34b5245564abb28916e7d48d9b07c0}HEADERS["From"][1].URI.USER; 

This SigMa script checks for calls to 202-555-6199 and creates a new header. Then, within Session Manager, you can look at the full header and see this info:

INVITE sip:2027216199@ares.corp SIP/2.0 
From: "PHONEGUY ROGER " <sip:6265554799@ares.corp>;tag=6698455617080545_c1b09.2.2.1404967301186.0_7074379_15030130 
To: <sip:2025556199@telco.corp> 
Call-ID: 9959340302389262@ 
Contact: <sip:;transport=tcp> 
Record-Route: <sip:;ipcs-line=63646;lr;transport=tcp> 
Max-Forwards: 64 
Via: SIP/2.0/TCP;branch=z9hG4bK-s1632-000279738415-1--s1632- 
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed 
P-Asserted-Identity: "PHONEGUY ROGER " <sip:6265554799@telco.corp> 
Content-Disposition: session; handling=required 
Content-Type: application/sdp 
roger-debug: roger-debug: 6265554799< 
roger-debug: roger-debug: 62655547992135551212 
roger-debug: roger-debug: 6265554799@roger.corp> 
Content-Length: 241 
o=Sonus_UAC 11647 11015 IN IP4 
c=IN IP4 
t=0 0 
m=audio 19234 RTP/AVP 18 0 100 
a=rtpmap:18 G729/8000 
a=rtpmap:0 PCMU/8000 
a=rtpmap:100 telephone-event/8000 
a=fmtp:100 0-15 

For quick one-liners, this is pretty handy. However, you can see my attempt at creating ONE header with a new p-asserted-identity completely failed. I can generate a new header, but my attempts to append to it only created another header. I really hope the new version of software adds some better manipulation features. In the meantime, I’ll keep hacking away at it.

If anyone out there knows SigMa, please let me know!

So, how’d you end up managing telecom?

Broadly speaking, those who manage telephone systems come from either a voice background or a data background.

It is quite common for data engineers to be responsible for the company’s voice and data systems. It is very rare that voice engineers get the same privilege. It’s kind of a strange turn of events. I remember saying in the mid 90s “voice guys can learn data easier than data guys can learn voice.” It was true then, but data has gotten more complex. And you know what’s crazy? Voice systems are mostly data now. Most of my troubleshooting and end user support is related to VoIP, routing, dhcp, vlans, PoE, or a hardware problem at the endpoint. The same thing applies to design and installation. It’s all about VoIP assessments, address assignments, and routing. Modern PBXs need to join domains and often have dedicated Ethernet switches, routes, firewall rules. The PBXs are simply servers in the data cabinet. It’s enough to make any old-school voice engineer cringe. I’ve started seeing PBXs installed with patch panels instead of punchblocks. No punchblocks!

The tools of the trade used to be a test phone, punch tool, and toner. Now a laptop is almost all I need. I can learn more from a traceroute or a diagnostic utility within the pbx than I can from visiting the station. And if i do end up visiting the station, the telephone’s diagnostics often tells me the rest. It’s been quite a transition; and in retrospect, it has been all the industry promised us.

So if you come from voice, you are probably seeing some amazing changes to the pbx and peripheral equipment. If you come from data, you’re starting to see a bunch of unfamiliar equipment plugged into your network. At first, those are black boxes the vendors and just a couple people in your company know about. I will help you to break into those black boxes and see what is inside.

How to convert and play a wav file in an Avaya Modular Messaging Caller Application

One of my clients uses  Avaya Modular Messaging (aka Messaging Application Server). I recently needed to write a caller application to just hold a channel open for a few minutes. I was working in a quiet environment and didn’t want to sit in a cube and speak “blah blah testing 1 2 3 testing hello nothing to hear keep moving” into the phone for 30 seconds or more, so I figured I would use a wav file I had of some open source music. It’s not obvious, but Avaya Modular Messaging supports drag-and-drop of a wav file onto the “record a prompt” screen.

Drop the wav file onto the media controls

The problem is, the docs are pretty vague on the exact encoding that MAS supports for these files. Surprisingly, I couldn’t get Audacity to export a file in the right format. Nor could I use a variety of other GUI utilities.

I finally decided to try sox. This is a great open source utility that I typically use with Asterisk sound files. In the end, the following command converted the file to a format that MAS liked:

sox source_music_file.wav -eu-law -b8 -c1 -r8000 music4mas.wav

When I deployed the app, the audio was a little too loud, causing static and distortion on the channel. If you need to lower the volume, you can add a volume adjustment to the sox command line (on the input file!) with:

sox -v-0.1 music.wav -eu-law -b8 -c1 -r8000 music1v.wav

That did it for me. Happy caller-apping everyone.


“It all started with the smoke signal…”

I just finished a great book titled “The Deal of the Century – The Breakup of AT&T”. During final negotiations between the Justice Department and AT&T, a judge named Vincent Buinno started a two-hour-long diatribe with “It all started with the smoke signal…” Wow, you’re in for a long wait when a judge begins that way.

I owe my career to the breakup of the Bell System. It was a tough time in AT&T’s history, and even thirty years later there are arguments either way. But personally I love being a phone guy and I wouldn’t have the chance if Ma Bell owned and controlled everything.

Phone Guy Blog

I’m a “phone guy” who has made a successful transition to “IT Guy”. Between the 1990s and now, phone guys had to adapt or die. I have adapted. To the PBX administrator who has transitioned to VoIP, or the networking/LAN/WAN administrators who have to support the telecommunications network, this blog is for you.

And of course, being a pretty good telecom guy and computer programmer doesn’t necessarily mean I’m very good at WordPress.