TraceSM is already running – how to stop it

In the new version of Session Manager (6.3 and above), the traceSM process is supposed to timeout and stop on its own. However, I was just trying to run it today and got this error

[cust@la-sessionmanager ~]$ traceSM
ERROR: traceSM is already running. Only one instance is allowed.

Uh oh. I don’t know the root password so I cannot kill it. However, Avaya thought of this for me. Just run traceSM with the -k switch and all is well:

traceSM -k

This kills the old process and runs a new instance for me. Just another thing that sounds simple once you know it.

no VNS resource administered

Here’s one of those times when I’m in the Avaya Communication Manager and I get an error. Happens to me all the time. What do you do when you get an error? Google it? Me too!

In my case, I wanted to see if I could get a report of packet loss from one of my network regions. Sure enough, there’s a packet loss report. You can perform this “list measurements” command:

list measurements ip voice-stats hourly pktloss 6





















No VNS resource administered
Command:

Hmm, no VNS resource. So I type that into google with quotes and there are no results! How can there be no results? I opened a ticket with Avaya and their expert systems didn’t return anything either. Later I was looking through some documentation and I recalled that you sometimes have to tell Communication Manager what you want to measure. Sure enough, if you “change measured selection network region”, you can add the network regions you want monitored.

Now when I type “list measurements ip voice-stats hourly pkloss 6” I get an empty report. Hopefully tomorrow I’ll have some stats. Do you ever wish you could automate this report? More on that later.

Thanks all!

 

How to build a T1/PRI from scratch on Avaya Communication Manager

This is almost an oxymoron. PRI interfaces are so common on PBXs that you’d think we would all be familiar with the process of building them. However, since these interfaces don’t change very often, sometimes we will go several years between PRI configuration. Here’s a quick reference for setting up a PRI. This one will use AT&T’s IPFlex BVoIP service, but most PRIs are configured the same way nowadays.

Here are the steps we will perform:

  1. Determine the physical location for the PRI (i.e. what port network or gateway and what slot?).
  2. Determine the trunk group number you will use and the TAC (trunk access code). Often, these will correspond with a site or media gateway. Just do a ‘list trunk’ and you’ll probably see a pattern where the trunk group number should fit and the TAC is almost always based upon the trunk group number.
  3. Find out the line coding and framing for the T1. Almost all T1s are now ‘b8zs’ and ‘esf’. Your carrier can confirm this, but I’ve never seen a voice PRI installed after the mid-1990s anything else.
  4. Find out the service protocol to use. This one is tricky and depends upon the emulation of the carrier’s switch. In fact, their planning docs will often ask you which one you want. And half the time it’s wrong anyway. If the D-channel doesn’t come up, this is usually the problem. It’s either ‘a’ for AT&T custom or ‘b’ for NI-2. It looks like this is used in three different places. The DS1 form, the signaling group form, and the trunk group form all have signaling protocol fields! However, the signaling and trunk group fields are different. When the carrier asks you “are you AT&T or NI2?”, they mean the ds1 itself. I have added some notes at the bottom of this post regarding the service protocols.

First, make sure you have a PRI slot available. I just pushed a PRI card into slot 5 of my G450 gateway (media gateway 10 in my case) so it looks like this:

list configuration media-gateway 10                                    Page   1

                              SYSTEM CONFIGURATION

Board                                                     Assigned Ports
Number   Board Type              Code     Vintage    u=unassigned t=tti p=psa

010V1    ICC MM                   S8300D  HW06 FW001
010V2    ANA MM                   MM716AP HW06 FW098 01 02 03 04 05 p  p  p
                                                     p  p  p  p  p  p  p  p
                                                     p  p  p  p  p  p  p  p
010V4    DS1 MM                   MM710BP HW11 FW052 01 02 03 04 05 06 07 08
                                                     09 10 11 12 13 14 15 16
                                                     17 18 19 20 21 22 23 24
                                                     u  u  u  u  u  u  u  u
010V5    DS1 MM                   MM710BP HW11 FW049 u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u
                                                     u  u  u  u  u  u  u  u



                press CANCEL to quit --  press NEXT PAGE to continue
ESC-x=Cancel Esc-e=Submit Esc-p=Prev Pg Esc-n=Next Pg Esc-h=Help Esc-r=Refresh

Slot 5 is nice and clean with 24 unassigned ports. First we need to create the DS1, which is really creating the D-channel on port 24. I type

add ds1 010v524

and the DS1 form comes up. It’s only two pages. Fortunately, I have a couple working DS1s from AT&T already so I’ll refer to those. By the time I’m done, my DS1 looks like this:

display ds1 010v524                                             Page   1 of   2
                                DS1 CIRCUIT PACK

            Location: 010V5                           Name: Atlanta BVoIP
            Bit Rate: 1.544                    Line Coding: b8zs
   Line Compensation: 1                       Framing Mode: esf
      Signaling Mode: isdn-pri
             Connect: network
   TN-C7 Long Timers? n                   Country Protocol: 1
Interworking Message: PROGress            Protocol Version: b
Interface Companding: mulaw                            CRC? n
           Idle Code: 11111111
                              DCP/Analog Bearer Capability: 3.1kHz

                                           T303 Timer(sec): 4


      Slip Detection? n                 Near-end CSU Type: integrated

   Echo Cancellation? n          Block Progress Indicator? n




display ds1 010v524                                             Page   2 of   2
                              DS1 CIRCUIT PACK

 ESF DATA LINK OPTIONS

                     Network Management Protocol: tabs
 Send ANSI-T1.403 One-Second Performance Reports? n
                             Far-end CSU Address: b

 INTEGRATED CSU OPTIONS
                                    Transmit LBO: 0db
                                    Receive ALBO: 26db
                                    Upon DTE LOS: loopback


 CPE LOOPBACK JACK OPTIONS
                  Supply CPE Loopback Jack Power? n




Now we create the signaling group, which is the communication layer on the D-channel. Signaling groups can be shared (rarely), so these are split up. They’re very simple. The form is 5 pages long, but page 1 is all that matters:

display signaling-group 110                                     Page   1 of   5
                                SIGNALING GROUP

 Group Number: 110            Group Type: isdn-pri
                    Associated Signaling? y          Max number of NCA TSC: 0
                       Primary D-Channel: 010V524     Max number of CA TSC: 0
                                                   Trunk Group for NCA TSC:
       Trunk Group for Channel Selection:         X-Mobility/Wireless Type: NONE
      TSC Supplementary Service Protocol: a


There is a chicken-and-egg problem here. The signaling group wants to know what trunk group to use, but we haven’t built the trunk group yet. We will need to modify this after we create the trunk group.

Now create the trunk group. After the first page, almost everything is fine with default values. The only fields I end up changing are “send name”, “send number”, and “show ANSWERED BY” on page 3. I typically submit the form before assigning channels on page 5 to make sure the form submits.

display trunk-group 110                                         Page   1 of  21
                                TRUNK GROUP

Group Number: 110                  Group Type: isdn          CDR Reports: y
  Group Name: Atlanta BVoIP               COR: 1        TN: 1        TAC: 8900
   Direction: two-way        Outgoing Display? n         Carrier Medium: PRI/BRI
 Dial Access? n              Busy Threshold: 255  Night Service:
Queue Length: 0
Service Type: public-ntwrk          Auth Code? n            TestCall ITC: rest
                         Far End Test Line No:
TestCall BCC: 4


display trunk-group 110                                         Page   2 of  21
      Group Type: isdn

TRUNK PARAMETERS
         Codeset to Send Display: 6     Codeset to Send National IEs: 6
        Max Message Size to Send: 260   Charge Advice: none
  Supplementary Service Protocol: a     Digit Handling (in/out): enbloc/enbloc

            Trunk Hunt: cyclical
                                                   Digital Loss Group: 13
Incoming Calling Number - Delete:     Insert:                 Format:
              Bit Rate: 1200         Synchronization: async    Duplex: full
 Disconnect Supervision - In? y  Out? n
 Answer Supervision Timeout: 0
          Administer Timers? n        CONNECT Reliable When Call Leaves ISDN? n
             XOIP Treatment: auto    Delay Call Setup When Accessed Via IGAR? n


display trunk-group 110                                         Page   3 of  21
TRUNK FEATURES
          ACA Assignment? n            Measured: none      Wideband Support? n
                                                          Maintenance Tests? y
                               Data Restriction? n     NCA-TSC Trunk Member:
                                      Send Name: y      Send Calling Number: y
            Used for DCS? n                            Send EMU Visitor CPN? n
   Suppress # Outpulsing? n    Format: natl-pub
 Outgoing Channel ID Encoding: preferred     UUI IE Treatment: service-provider

                                                 Replace Restricted Numbers? n
                                                Replace Unavailable Numbers? n
                                                      Send Connected Number: n
                                                  Hold/Unhold Notifications? n
             Send UUI IE? y     Modify Tandem Calling Number: no
               Send UCID? n
 Send Codeset 6/7 LAI IE? y                         Ds1 Echo Cancellation? n

    Apply Local Ringback? n               US NI Delayed Calling Name Update? n
 Show ANSWERED BY on Display? n      Invoke ID for USNI Calling Name: variable
                             Network (Japan) Needs Connect Before Disconnect? n


We still have two things to do – we need to set the trunk group for channel selection in the signaling group, and we need to assign the channels. Before I forget, change the signaling group to tell it what trunk group it will control:

change signaling-group 110                                      Page   1 of   5
                                SIGNALING GROUP

 Group Number: 110            Group Type: isdn-pri
                    Associated Signaling? y          Max number of NCA TSC: 0
                       Primary D-Channel: 010V524     Max number of CA TSC: 0
                                                   Trunk Group for NCA TSC:
       Trunk Group for Channel Selection: 110     X-Mobility/Wireless Type: NONE
      TSC Supplementary Service Protocol: a

Now, for the last step, let’s assign channels! Change the trunk group and set these on page 5 and 6. Note how we assign the signaling group along with the channel. When I get the circuit ID from AT&T, I will put it in the “Name” column along with the DIDs that were provided on this circuit.

change trunk-group 110                                          Page   5 of  21
                                 TRUNK GROUP
                                      Administered Members (min/max):   1/23
GROUP MEMBER ASSIGNMENTS                  Total Administered Members:  23

       Port    Code Sfx Name        Night           Sig Grp
  1: 010V501  MM710  B                                110
  2: 010V502  MM710  B                                110
  3: 010V503  MM710  B                                110
  4: 010V504  MM710  B                                110
  5: 010V505  MM710  B                                110
  6: 010V506  MM710  B                                110
  7: 010V507  MM710  B                                110
  8: 010V508  MM710  B                                110
  9: 010V509  MM710  B                                110
 10: 010V510  MM710  B                                110
 11: 010V511  MM710  B                                110
 12: 010V512  MM710  B                                110
 13: 010V513  MM710  B                                110
 14: 010V514  MM710  B                                110
 15: 010V515  MM710  B                                110


change trunk-group 110                                          Page   6 of  21
                                 TRUNK GROUP
                                      Administered Members (min/max):   1/23
GROUP MEMBER ASSIGNMENTS                  Total Administered Members:  23

       Port    Code Sfx Name        Night           Sig Grp
 16: 010V516  MM710  B                                110
 17: 010V517  MM710  B                                110
 18: 010V518  MM710  B                                110
 19: 010V519  MM710  B                                110
 20: 010V520  MM710  B                                110
 21: 010V521  MM710  B                                110
 22: 010V522  MM710  B                                110
 23: 010V523  MM710  B                                110
 24:
 25:
 26:
 27:
 28:
 29:
 30:

And now it’s a matter of troubleshooting. I plugged in my PRI into the BVoIP router and the first thing is to make sure the D-channel came up. Do this by statusing the signaling group:

status signaling-group 110
                         STATUS SIGNALING GROUP

        Group ID: 110                             Active NCA-TSC Count: 0
      Group Type: isdn-pri                         Active CA-TSC Count: 0
  Signaling Type: facility associated signaling
     Group State: in-service


                           Primary D-Channel


            Port: 010V524        Level 3 State: in-service


                          Secondary D-Channel


            Port:                Level 3 State: no-link

The “Level 3 State” is the state of the D-Channel. If you’re on the phone with your carrier and they ask about the D-Channel, you can give them that status.

Next, status the trunk group itself:

status trunk 110

                             TRUNK GROUP STATUS

Member   Port     Service State      Mtce Connected Ports
                                     Busy

0110/001 010V501  in-service/idle    no
0110/002 010V502  in-service/idle    no
0110/003 010V503  in-service/idle    no
0110/004 010V504  in-service/idle    no
0110/005 010V505  in-service/idle    no
0110/006 010V506  in-service/idle    no
0110/007 010V507  in-service/idle    no
0110/008 010V508  in-service/idle    no
0110/009 010V509  in-service/idle    no
0110/010 010V510  in-service/idle    no
0110/011 010V511  in-service/idle    no
0110/012 010V512  in-service/idle    no

This gives you the channel state within the trunk group. These channels will only come up in-service/idle if the D-channel is good. If you have any problems, you can try to “bounce” the D-channel by perform a busyout then release on the port of the D-channel with “busy port 010v524” in my case:

busyout port 010v524

                             COMMAND RESULTS

  Port      Maintenance Name  Alt. Name          Result           Error Code

  010V524   ISDN-PLK                             PASS

release port 010v524

                             COMMAND RESULTS

  Port      Maintenance Name  Alt. Name          Result           Error Code

  010V524   ISDN-PLK                             PASS

And if the D-channel still won’t come up, that’s when you tweak the service protocol values between ‘a’ and ‘b’.

When you’re on the phone with your carrier trying to turn up service, the ability to bounce the D-Channel, monitor the D and B channel status, and change from AT&T custom to NI-2 is typically enough to get the carrier to work with you and get this working. On rare occasions you may need to tweak the line coding (b8zf/esf) but this has only happened to me once.

As for service protocols, a good friend of mine provided a nice summary from his notes. Remember these apply to the DS1! (i.e. “change ds1 010v524”), not the sig or trunk group forms.

Protocol “a” – AT&T Custom

  • For the layer 3 messaging or 4ESS/5ESS messaging type (Maintenance Protocol Discriminator 03) (TR41449/TR41459)
  • Normally used by Long Distance Companies
  • 4ESS is strictly AT&T Custom for FAS/NFAS
  • DMS 100/250 is AT&T Custom for FAS/NFAS (software load BCS.36 or later)
  • DSC DEX600E is AT&T Custom for FAS/NFAS (used BY MCI and Frontier as central offices)

Protocol “b” – National ISDN 2

  • Bellcore protocol (5ESS LEC messaging for NFAS and DCBU applications, the only way the 5ESS can be set up for NFAS and DCBU) (Maintenance Protocol Discriminator 43) (TR1268)
  • Normally used by LECs
  • DMS 100 will do NI-2 FAS/NFAS (started in the BCS.37 load of software)
  • Starting with the 5E9.0 software load in the 5ESS
  • Normal protocol for connection to Siemen’s EWSD
  • Normal protocol for connection to GTE GTD5

Thanks as always for your time and attention. If you’re interested, I can include a post on routing incoming calls from this trunk group, routing calls out this trunk group, and setting the caller-id for calls over this trunk group. Please let me know your own experiences with setting up and using PRI and IP trunks.

How to clear PLAT-ALM alarms in Avaya Communication Manager

When gateways unregister due to various network blips or maintenance, it can cause a PLAT-ALM in your Communication Manager. These will linger forever until you clear them, It’s very simple but you don’t do it from SAT, you do it right from the shell. To see all of the alarms, login to CM’s shell and type

almdisplay -v

To clear them all, just type

almclear -a

If you still have a problem, the alarm will re-propagate so I haven’t seen the downside.

What happens if you try to “traceSM” when the Avaya Session Manager hasn’t fully booted?

If you try to “traceSM” before the Avaya session manager is completely booted up, you get this. But only when you “start” the trace.

The complete log files are located:
        SIP messages: /var/log/Avaya/trace/tracer_asset.log*
        Call Processing: /var/log/Avaya/trace/call_proc.log*
        PPM: /var/log/Avaya/trace/trace_ppm.log*
Error enabling SM100 trace.

Preparing SCRUCH network filter num SD address:0
Preparing SCRUSH network filter num SH address:0
*** in client update res=28 ***


[cust@nj-pbx-sm02 ~]$

How to update Avaya System Manager from 6.3.3 to 6.3.9

Updating System Manager should not be scary. As I write this, I am updating my Avaya Aura System Manager from 6.3.3 to 6.3.9. I just have to keep the faith in the Avaya engineers.

So here are the steps to perform upgrade this upgrade. It is definitely something you, as a customer, can do yourself. These instructions are from page 22 of this document https://downloads.avaya.com/css/P8/documents/100175367

  1. Download the patch System_Manager_6.3.9_r4602482.bin from support.avaya.com.
  2. Use WinSCP or (or any SCP program) to upload it to System Manager. I used the admin account (with the same password as the web interface).
  3. SSH to the System Manager using the same account as the file upload. The admin account didn’t give me a $ prompt at the shell, but it worked fine. When you login, you should see the .bin file in the default folder (/home/admin/ in my case)
  4. Before you start, you should disable geo-redundancy if you use it. Go to Home->Georedundancy and disable it.
  5. Simply type “SMGRPatchdeploy System_Manager_6.3.9_r4602482.bin” and the patch manager will take over.
  6. The Patch manager will ask you to review and accept the license agreement, then start running. I was warned that it will take about an hour and I should be patient. Unfortunately, I need to test this before I do the redundant session manager, then I need to do my system managers. So I’m looking at a late night and I’d like this to finish. I think my System Manager can tell so it’s taking its time. As I’ve said before – just long enough to panic, plus 30 seconds. In this case, I think it’ll be plus 10 minutes or so. Here is the output from my patching:
admin >SMGRPatchdeploy System_Manager_6.3.9_r4602482.bin
Verifying the patch binary.....
mv: cannot stat `/var/log/Avaya/SMGR_Patch.log.3': No such file or directory
Extracting files to /tmp/patchsfx.p18665...
Adopter_list.txt
CAF.jar
EULA.txt
System_Manager_R6.3.9_4602482.txt
defConfParam.properties
geo_upgrade_util.jar
install.jar
installer_relno.txt
int_postgres_response.properties
mgmt-custom.xml
mgmt-custom.xsd
postPatchHook.sh
runAuditor.sh
runInstaller.sh

Read the following Avaya Software License Terms carefully. The Chinese, German, Portugese, Spanish, French, Japanese, and Russian versions of the Avaya Software License Terms are posted at https://support.avaya.com/LicenseInfo . If you do not have direct Internet access, please access a device that has Internet access to view, download and print the applicable license.

Press the [Enter] key to read the Avaya Software License Terms.

***THIS IS WHERE YOU READ AND ACCEPT THE LICENSE TERMS***
If you agree with Terms and Conditions of the applicable Avaya Software License Terms, please enter "Y" for yes. If you do not agree with the Terms and Conditions, please enter "N" for No.

If you enter "Y", you have read, understand, and agreed to the Terms and Conditions of the Avaya Software License Terms (Avaya Products).  You further understand that by entering "Y" you will initiate the process to upgrade, and/or install the Avaya Product; and continued installations is acceptance of the Terms and Conditions of the applicable Avaya Software License Terms (Avaya Products).

If you do not agree to these terms and enter "N" it will abort the installation/upgrade of this Software.

Do you accept the Avaya Software License Terms? (Y)es/(N)o: y

You have accepted the Avaya Software License Terms. The Software Installation will now begin.
Executing checkForPatchRequisite
Executed checkForPatchRequisite successfully, proceed to install the patch.
Thu Oct 16 22:10:09 EDT 2014 Continue the patch installation in normal way.
Perform cleanup ....
Stopping System Manager JBoss - PID: 6928
.............................................
Stopped System Manager JBoss
/tmp/patchsfx.v29174 /tmp/patchsfx.v29174
Enter the Encryption Key :
Please Wait


Log file has been created at location: /tmp/avaya_aura_install-log_2014-10-16_10.11.42.txt
[ Starting automated Installation ]


Checking if the system has sufficient space as required by the selected packs.

Available Space: 13.9 GB
Total space Required: 1.54 GB


[ Starting to unpack ]

[ Processing package: System Manager - Update Patch (1/15) ]
.
[ Processing package: Conferencing - Update Patch (2/15) ]
.......
[ Processing package: PS SysMgr Extensions (3/15) ]
.................
[ Processing package: IPTCM - 6.2 FP4 S4 (4/15) ]
....................
[ Processing package: IP Office - 6.2 FP3 S4 (5/15) ]
........
[ Processing package: SUM - 6.2 FP4 S4 (6/15) ]
......
[ Processing package: MESSAGING - 6.2 FP4 SP9 (7/15) ]
.............
[ Processing package: INVENTORY - 6.2 FP4 S4 (8/15) ]
............
[ Processing package: MMCS (9/15) ]
..
[ Processing package: CS1000 - 6.3 FP4 (10/15) ]
................
[ Processing package: Session Manager Element Manager (11/15) ]
..
[ Processing package: Collaboration Environment Element Manager (12/15) ]
..
[ Processing package: AMM - Update Patch (13/15) ]
...
[ Processing package: System Manager on VMware (14/15) ]
...
[ Processing package: System Manager Update (15/15) ]
...........
Parsing files
........
File execution started for pack : System Manager - Update Patch
..
File execution started for pack : Conferencing - Update Patch
.
File execution started for pack : PS SysMgr Extensions
.
File execution started for pack : IPTCM - 6.2 FP4 S4
.
File execution started for pack : IP Office - 6.2 FP3 S4
.
File execution started for pack : SUM - 6.2 FP4 S4
.
File execution started for pack : MESSAGING - 6.2 FP4 SP9
.
File execution started for pack : INVENTORY - 6.2 FP4 S4
.
File execution started for pack : MMCS
.
File execution started for pack : CS1000 - 6.3 FP4
.
File execution started for pack : Session Manager Element Manager
.
File execution started for pack : Collaboration Environment Element Manager
.
File execution started for pack : AMM - Update Patch
..
File execution started for pack : System Manager on VMware
..
File execution started for pack : System Manager Update
..
[ Unpacking finished ]
[ Writing the uninstaller data  ... ]
output.response.file /opt/Avaya/autoInstall_Avaya_AURA.properties generated
[Automated Installation Completed !]
Log file has been moved to location: /opt/Avaya/install_logs/avaya_aura_install-log_2014-10-16_10.11.42.txt from location: /tmp/avaya_aura_install-log_2014-10-16_10.11.42.txt
/tmp/patchsfx.v29174
Executing Common Inventory Datamigration script for access point & attributes.
dos2unix: converting file /opt/Avaya/Mgmt/6.3.8//bulkadministration/exportutility/epXmlArtifactProcessor.sh to UNIX format ...
Thu Oct 16 23:18:14 EDT 2014 Restarting JBoss...
Service was not running
Starting System Manager JBoss
.
Service started successfully - PID: 21891
Thu Oct 16 23:18:42 EDT 2014 Patch installation completed and waiting for JBoss to finish start-up... this may take a few minutes
Thu Oct 16 23:39:45 EDT 2014 Patch installation completed and waiting for Quantum init to finish ... this may take a few minutes
QuantumElementType Version mis match in QuantumElementType.xml and DataBase so republishing the QuantumElementType.
dos2unix: converting file tmConfigProfile.xml to UNIX format ...
dos2unix: problems converting file tmConfigProfile.xml
log4j:WARN No such property [conversionPattern] in com.avaya.common.logging.client.CommonLoggingLayout.
post_fix.sh: line 53: ${LOGFILE}: ambiguous redirect
Thu Oct 16 23:48:19 EDT 2014 : Check for the EP Status at the end of patch installation.
START : Execution of ep_process_state.sh
END : Executed ep_process_state.sh with status: 0
Thu Oct 16 23:48:19 EDT 2014 : Executed the domains and Adopter scripts successfully.
Thu Oct 16 23:48:19 EDT 2014 Changing the needful permission for various dirctories
Changed the ownerships of directories successfully..
Thu Oct 16 23:48:20 EDT 2014 Maintenance completed Successfully
admin >
admin >shutdown -r now
shutdown: you must be root to do that!
admin >su root
Password:

        * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
        *                                                                     *
        * !! Virtual Machine needs to be rebooted as System Manager Patch     *
        *    installation updated the Kernel. !!                              *
        *                                                                     *
        * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

root >shutdown -r now

Broadcast message from root (pts/0) (Fri Oct 17 00:34:51 2014):

The system is going down for reboot NOW!
root >

Troubleshooting Signaling Manipulation (SigMa) Scripts in Avaya Aura SBC

This is pretty obscure and I’m probably only talking to three people on the Internet right now, but if you have ever tried to troubleshoot SigMa scripts in the Avaya Aura Session Border Controller, you probably wish it had better logging. Actually, if you have ever gotten logging to work, please let me know. I had given up. However – I recently discovered a nice easy way to troubleshoot signaling manipulation, and consequently also discovered just how cumbersome this process really is.

The easy way to log is to set your own custom header in the SigMa script, then inspect the INVITE from the Session Manager’s point of view. For example, I added this SigMa script:

    act on request where %DIRECTION="INBOUND" and %ENTRY_POINT="PRE_ROUTING" {
        if(%HEADERS["To"][1].URI.USER.regex_match("2025556199"))then {
            %HEADERS["roger-debug"][1] = %HEADERS["From"][1].URI.USER; 
            append(%HEADERS["roger-debug"][1],"<");
            append(%HEADERS["roger-debug"][1],"2135551212");
            append(%HEADERS["roger-debug"][1],"@roger.corp>");
        }
    }

This SigMa script checks for calls to 202-555-6199 and creates a new header. Then, within Session Manager, you can look at the full header and see this info:

INVITE sip:2027216199@ares.corp SIP/2.0 
From: "PHONEGUY ROGER " <sip:6265554799@ares.corp>;tag=6698455617080545_c1b09.2.2.1404967301186.0_7074379_15030130 
To: <sip:2025556199@telco.corp> 
CSeq: 2 INVITE 
Call-ID: 9959340302389262@12.11.18.20 
Contact: <sip:10.10.20.71:5060;transport=tcp> 
Record-Route: <sip:10.10.20.71:5060;ipcs-line=63646;lr;transport=tcp> 
Allow: INVITE,ACK,CANCEL,BYE,INFO,PRACK 
Max-Forwards: 64 
Via: SIP/2.0/TCP 10.16.26.71:5060;branch=z9hG4bK-s1632-000279738415-1--s1632- 
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed 
P-Asserted-Identity: "PHONEGUY ROGER " <sip:6265554799@telco.corp> 
Content-Disposition: session; handling=required 
Content-Type: application/sdp 
roger-debug: roger-debug: 6265554799< 
roger-debug: roger-debug: 62655547992135551212 
roger-debug: roger-debug: 6265554799@roger.corp> 
Content-Length: 241 
 
v=0 
o=Sonus_UAC 11647 11015 IN IP4 10.16.26.71 
s=SIP 
c=IN IP4 10.16.26.71 
t=0 0 
m=audio 19234 RTP/AVP 18 0 100 
a=rtpmap:18 G729/8000 
a=rtpmap:0 PCMU/8000 
a=rtpmap:100 telephone-event/8000 
a=fmtp:100 0-15 
a=sendrecv 
a=maxptime:30

For quick one-liners, this is pretty handy. However, you can see my attempt at creating ONE header with a new p-asserted-identity completely failed. I can generate a new header, but my attempts to append to it only created another header. I really hope the new version of software adds some better manipulation features. In the meantime, I’ll keep hacking away at it.

If anyone out there knows SigMa, please let me know!

How I stopped one hundred percent of all telemarketers.

At the beginning of 2014, I got an aggressive telemarketer call at my house. Like most people, I really hate telemarketing calls so I let my 13-year-old son take the call. Well, my son said the caller said something really rude but he didn’t really hear. At that point, I said “hey, I’m a telephone guy. I should be able to stop this.” So I did.

Being a telephone guy, I cannot imagine NOT having a landline at home. I have AT&T U-Verse so it’s not like I have old-school POTS service anyway, but I like my home telephone numbers and don’t want to disconnect them. And hey, this was all an experiment anyway. So I got a new telephone number from a SIP provider and forwarded my home line to it, then via that SIP provider I routed it to my Asterisk server at a cloud provider. Then I added an Analog Terminal Adapter at home and plugged my home cordless phone into that. So I’m kinda like a Vonage or any other VoIP provider now. Incoming calls come into the SIP provider, then my Asterisk server, and then to my house via the ATA. Outgoing calls go out my Asterisk server then to my SIP provider. Pretty basic stuff. Yes, I tested 911.

Then I created a “white list” inbound app on Asterisk where all incoming callers get a greeting “hey, if you’re a real person press a digit and you’ll ring through to my house. If you don’t press a button, you’ll go to voicemail”. If a caller presses a button, their caller-id is added to the “white list” and rings to my house. From then on, that caller-id is allowed. After each call, I get an email with the caller-id and caller-id name of the caller, along with what happened to the call (e.g. rang through, went to voicemail, or hungup). It worked perfectly. It turns out that in the last six months, 100% of telemarketers are using predictive dialers; their machine thinks it got an answering machine and simply calls back later.

So on the email notification, I added a link to let me set the “disposition” of the caller-id next time. For friends, relatives, vendors, service providers, schools, etc, I can grant access next time. For obvious telemarketers (“SUN SOLAR”, “ELITE SERVICES”, etc), I can reject the call completely (“I’m sorry, your call did not go through”).

Sweet relief! It’s my telephone again! Now when the phone rings, I know it’s a real person and probably want to talk to him or her. I figured some telemarketers would punch through occasionally, but they don’t. I’ve stopped them all.

But here’s where it got interesting. I kept getting repeated calls from the same numbers – sometimes four within one minute. I get calls from particular numbers every night at about the same time. I got calls from fake numbers like “10000000000” or “16261346229”. That last one may not be obvious, but the first digit of an office code cannot be a 0 or 1. There’s no possible telephone number 134-6229 in the 626 area code. Now that my number is “protected”, I could even call the valid numbers back without worrying about tipping them off. When I did, I would get “Thank you for calling back. To be removed from our list…” or sometimes I would get a home-made “your call cannot be completed as dialed” message. Or sometimes just a fast busy or dead-air then disconnect. Obvious telemarketers.

So great, I’m blocking telemarketers and none are getting through, but I’m not causing them pain. I’m just blocking their dialing machine with my blocking machine. What if I send telemarketers to a greeting that says “hello” a few times, then tells them to hang on? Would that cause the predictive dialer to cut the agent through to me? If so, it would at least burn some “human time” and cost the telemarketing company some money, right? I tried it, and this was the result:

Not bad! I wasted 41 seconds of her time! So clearly I can get my telephone system to fool the predictive dialer into thinking it found a real person. I figured if I could combine some “detect noise” and “detect silence” algorithms into this thing, could I keep them on the phone longer? And how would I do it without any kind of speech recognition? Could I really keep a telemarketer on the phone just by detecting noise and silence? It turns out I could:

I really have to say thanks to Sofia for this call. Ironically, I listened more closely to Sofia’s pitch (and so have my friends) than if I had just hung up on her at the beginning. But anyway, this gave me the confidence to build a more elaborate parrot. At this point, I have collected several dozen calls of various entertaining quality. I will post them individually. I hope you enjoy. Perhaps next post about this subject, I’ll get a bit more technical how the algorithm works.

How to remove an extension when there are messages waiting

Just a quick note regarding the message

Message(s) waiting; please delete message(s) first

If you try to remove a station with messages waiting, the Communication Manager will reject it if there are messages waiting. In my case, I wanted to convert a virtual station to a physical 9630, but the system won’t let me remove the station. And of course, I cannot change the type of station from Virtual to 9630 – I get this message:

"9630" Station must be removed and re-added to change type

As usual, the solution is simple once you know it. You can clear the message waiting indicator with “clear amw all xxxx“. This clears the messages flag (not the messages!) and turns off the message waiting indicator. This allows you to remove the station.

How to update firmware in individual Avaya 96xx phones

If you have an Avaya telephone system, you probably have a web server to manage firmware. If not, it’s pretty easy to set up so let me know if I should post the procedures. What is not easy is testing firmware before deploying it to the whole network. What if you just want to update one particular phone? What if someone is complaining about something (e.g. continuous reboots) and you want to update their firmware to see if it fixes it? You could set up a development web server and statically assign the file server address to the phone. But then you have to statically assign all addresses on the phone or the DHCP server will overwrite the file server address. And this isn’t a permanent fix because the phone will forever have a statically assigned address. You could do tricky things in Apache using the IP address of the phone in the http.conf file or .htaccess, but there’s another way.

The trick is in the 96xxupgrade.txt file on your web server. As with most of these posts, we need to assume you have a web server and have access to the root html folder. Find the 96xxupgrade.txt, open it in any text editor and you should see something similar to this (if you’re using the sample file from Avaya):

############################################################
## Check backup application version                       ## 
############################################################
IF $MODEL4 SEQ 9610 goto BACKUPAPP96XX
IF $MODEL4 SEQ 9620 goto BACKUPAPP9640
IF $MODEL4 SEQ 9630 goto BACKUPAPP96XX
IF $MODEL4 SEQ 9640 goto BACKUPAPP9640
IF $MODEL4 SEQ 9650 goto BACKUPAPP96XX
IF $MODEL4 SEQ 9670 goto BACKUPAPP9670
goto END

This is the stuff Avaya put in the sample file so individual types of phones can download different versions of firmware. Note how there’s a check against a “$MODEL4” variable? This allows the phone to jump to a new section after a very basic check. Unfortunately, there are only a few variables available. I’ve never been able to find a definitive list of all available variables, but we can make this work with the GROUP variable.

So let’s say you want to test version ha96xxua3_2_0_S.bin of the 9600 firmware on your 9630 phone at your own desk before deploying to the entire network. In the 96xxupgrade.txt file, go to the section for the 9630s (probably BACKUPAPP96XX if you have the default Avaya file)

Your file will probably have a section like this:

# BACKUPAPP96XX

IF $BOOTNAME SEQ hb96xxua3_1_03_S.bin goto PHONEAPP96XX
SET APPNAME hb96xxua3_1_03_S.bin
goto GETSET

This just tells the phone to check its firmware version and if it’s already 3_1_03_S, then go to PHONEAPP96XX. If not, set its version to 3_1_03_S, which will cause the phone to download the new file, reboot, write it to flash, and reload the 96xxupgrade.txt file. Then when it gets to this line, it will have the version 3_1_03_S and jump to PHONEAPP96XX. Why is there a goto GETSET? Well, that’s there in case the phone cannot find the file. It will just jump to GETSET.

Within the PHONEAPP96XX section, my file says

 # PHONEAPP96XX
SET APPNAME ha96xxua1_1_03_S.bin
goto GETSET

Maybe yours does too. This just sets the application file to the right name. Again, causing the phone to load the file, reboot, write to flash, and reboot again. Then when it gets to this section, it jumps to GETSET. When you download new firmware, the phone reboots four times as it works through all these files.

In GETSET, my file says

GET 46xxsettings.txt
goto END

And this again is an Avaya default. I don’t have any 4600 series phones. I guess I should rename mine to 96xxsettings.txt. But it doesn’t matter. The settings file contains tons of options. Tons.

So back to my point. How do you load different firmware for a single phone? What you can do is perform a check against the phone’s GROUP variable. My 96xxupgrade file now has this line in the BACKUPAPP96XX section:

# BACKUPAPP96XX
IF $GROUP SEQ 30 goto BACKUPAPP30
IF $BOOTNAME SEQ hb96xxua3_1_03_S.bin goto PHONEAPP96XX
SET APPNAME hb96xxua3_1_03_S.bin
goto GETSET

When the phone gets to that line in red, it will check against its own internal GROUP variable and jump to BACKUPAPP30 if it matches. Now you can add the firmware version you want in that section by adding the following lines to the file:

# BACKUPAPP30
IF $BOOTNAME SEQ hb96xxua3_2_0_S.bin goto PHONEAPP30
SET APPNAME hb96xxua3_2_0_S.bin
goto GETSET

# PHONEAPP30
SET APPNAME ha96xxua3_2_0_S.bin
goto GETSET

So the phone will jump to a new section of the file and load specific firmware based upon its GROUP variable. Note that you can also “goto GETSET30” for example if you wanted to load a different settings file for this phone. In my case, I just want to test version ha96xxua3_2_0_S of firmware.

So how to you set the phone’s GROUP variable? Two ways:

Method 1 – Page 3 of the station form within SAT of your Communication Manager:

display station 4799                                            Page   3 of   7
                                     STATION

             Conf/Trans on Primary Appearance? n
   Bridged Appearance Origination Restriction? n


               Call Appearance Display Format: disp-param-default
                            IP Phone Group ID: 30
Enhanced Callr-Info Display for 1-Line Phones? n

                              ENHANCED CALL FORWARDING
                                       Forwarded Destination         Active
 Unconditional For Internal Calls To:                                   n
                   External Calls To:                                   n
          Busy For Internal Calls To:                                   n
                   External Calls To:                                   n
      No Reply For Internal Calls To:                                   n
                   External Calls To:                                   n

            SAC/CF Override: n

This method will “follow the station” so if you log a new extension into the phone, the GROUP will follow the new extension and will likely change. Then when the phone reboots, it will load new firmware. If you log your extension into another phone and then reboot that phone, it will load new firmware.

Method 2 – The “GROUP” menu item on the config screen of the 9630 telephone (Usually accessed by pressing Mute-CRAFT-# on the 9630, or pressing *-CRAFT-# at boot-up).

GROUP-item-in-craft-menu

This method will “follow the phone” so if you log a new extension into the phone, it will retain the phone’s setting. However, it gets over-ridden by the station’s group if there is one. Interaction gets complicated. I always use the station form.

And lastly, this is how we load SIP firmware in the 9630 using the same network and DHCP server we already have. We can check the value of the SIG variable, which is set in the CRAFT menu. But that’s a different post. For now, I hope this helps and I look forward to hearing from you.