Where is the ACI software repository located?

Excellent question.  I did some digging via SCP and finally found it.

But don’t get to excited you can’t copy files directly into it.

The repository is located in the following path:

There is a good write up on how to upgrade your software using the command line.

  1. the current version of the firmware.

Were the firmware uploaded via the APIC GUI? If this is true, the issue is related to defect CSCux40954. Please use ‘scp’ in APIC to copy the firmware into the controllers. Attached is the instructions: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/getting-started/b_APIC_Getting_Started_Guide/b_APIC_Getting_Started_Guide_chapter_011.html#concept_734A579133814A85813C9C5232BBE44C

Little Gotcha with APICs within ACI

This applies to versions of APIC controller software up to 1.1(4e)

Turns out the there is a bug that occurs when you connect an APIC to multiple leaves.  And yes that is stupid because you’re supposed to connect them to different leaves.  This bug manifests itself when integrating the VMM with the fabric.

Just be sure that you create an APIC policy in the policy groups. Fabric>Access Policies>Interface Policies>Policy Group

It’s a simple policy mine is:
CDP= Disabled
L2 Interface Policy= default

Then bind the policy to the leaves that the APIC’s are connected to.  Fabric>Access Policies>Interface Policies>Profiles>Leaf###.  The click the plus sign (+) and add in your newly created APIC Policy group for the interfaces that the APIC is connected to.

How to get code onto the APIC when the GUI upload fails

The following process worked when the GUI upload via http or scp failed for the 3.8GB APIC ISO file.

If SCP fails (or stalls), what you can do is use a program like Filezilla to connect to the APIC as admin and upload the image directly.

Once the image is in the admin’s home directory, you need to issue the command “firmware add <image_name>”. This adds the file to the firmware repository and should be seen in the GUI as well.

Can’t log into your APIC?

I ran into a split fabric issue setting up my test lab and got the following error trying to log into my 2nd APIC:

REST Endpoint user authorization datastore is not initialized – Check Fabric Membership Status of this fabric node

I was able to get logged into the APIC with the follow username and a blank password:


NOTE: as in the past physical access to a Cisco device equal total ownership.

Basically when installing the fabric for the 1st time you should only power on 1 APIC and discover the entire fabric, then add the other APICs 1 at a time.

How to find your ACI fabric serial numbers

Let’s say you need to open a TAC case and didn’t document all the serial numbers of your fabric upon installation.  You can get the membership information from the command line.

SSH to your APIC OOB management address and log in.

Then issue the following command:
acidiag fnvread

you will get output simular to the following:

For spines and Leaves

admin@apic1:~> acidiag fnvread
ID Name Serial Number IP Address Role State LastUpdMsgId
101 Leaf1 SALXXXXXXXX leaf active 0
102 Spine1 SALXXXXXXX spine active 0
103 Spine2 SALXXXXXXX spine active 0

For APIC’s

admin@apic1:~> acidiag verifyapic
openssl_check: certificate details
subject= CN=FCH1922V0L4,serialNumber=PID:APIC-SERVER-M1 SN:FXXXXXXXX
issuer= CN=Cisco Manufacturing CA,O=Cisco Systems
notBefore=Jul 14 14:52:07 2015 GMT
notAfter=Jul 14 15:02:07 2025 GMT
openssl_check: passed

Wierd 6500 interface output

So the question is what state is the following port in?

#sho run all | beg GigabitEthernet2/5
interface GigabitEthernet2/5
description LAB newtork
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 138
switchport mode trunk
no snmp trap link-status

The answer might not be what you think.

#sho int g2/5
GigabitEthernet2/5 is up, line protocol is up (connected)

#sho run int g2/5

Building configuration…

Current configuration : 230 bytes
interface GigabitEthernet2/5
description LAB newtork f
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 138
switchport mode trunk
no snmp trap link-status

#sho ver
Cisco IOS Software, s3223_rp Software (s3223_rp-IPBASEK9-M), Version 12.2(33)SXI14, RELEASE SOFTWARE (fc2)

The question is why?  I’m not sure other than it might be a code version bug.



So you really want to manually configure the 6800ia? It’s possible

Below is the command syntax to enable the fex configurations mode. First you will have to run “service internal” command, it will enable the hidden commands. However, do not forget to turn off the commands after making the changes using “no service internal”

6800-1#config t

6800-1(config)#service internal


6800-1#test platform software console fex 110 enable timeout 60  <<<<< This command will enable configuration mode on fex 110 for 60 minutes


Once done with the changes on the fex110, run the following command on the parent switch to turn off the service internal mode:

6800-1#config t

6800-1(config)#no service internal


How to see what traffic is hitting you CPU on a 6500

If you are having high CPU issues on your routers there is a way to see exactly what is causing it.

My issue was causing EIGRP to drop neighbors then come back online.

The problem is catching it fast enough to get the needed output especially when the spikes happen for only 1 second or 2.  I don’t know about you but i can’t type that fast.

To solve this issue we’ll use our friendly EEM script.

event manager session cli username “XXX” ——-This line may only be used if you have AAA configured and this “algo” must be a username that you already have in AAA

event manager applet HIGH_CPU
event snmp oid get-type exact entry-op ge entry-val 85 exit-op lt exit-val 75 poll-interval 7
action 1.01 syslog msg “——HIGH CPU DETECTED—-, CPU:$_snmp_oid_val%”
action 1.02 cli command “enable”
action 1.03 cli command “term length 0”
action 1.04 cli command “debug netdr cap rx”
action 1.05 cli command “show netdr cap | append disk0:HIGH_CPU.txt”
action 1.06 cli command “show proc cpu sort | append disk0:HIGH_CPU.txt”
action 1.07 cli command “Show users | append disk0:HIGH_CPU.txt”
action 1.08 cli command “Show proc cpu history | append disk0:HIGH_CPU.txt”
action 1.09 cli command “show logging | append disk0:HIGH_CPU.txt”
action 1.10 cli command “show spanning-tree detail | append disk0:HIGH_CPU.txt”
action 1.11 cli command “show ip traffic | append disk0:HIGH_CPU.txt”
action 1.12 cli command “show clock | append disk0:HIGH_CPU.txt”
action 1.13 cli command “undebug all”
action 1.14 cli command “term length 24”
action 1.15 cli command “exit”

Depending on your platform you may need to change disk0: to flash: or something else.

It will trigger when there is 85% CPU or greater and write a file to the destination.

With this output you can put it into a beta cisco tool https://cway.cisco.com/tools/netdr which will decode it for you.

Here is what one of mine looked like:


Cisco Anyconnect and Smart Tunnels

Cool feature that is available for SSL/WebVPN users.  When a process is started (Windows) or an application in a certain directory path is launched (MAC) you can have “smart tunnels” established.

This works really easily with the Windows platform and is very easy to configure.

Edit your Clientless SSL VPN Access Group policy


Select the Portal option on the left menu.

Go to the smart tunnel section and select your Tunnel Application.  In mine I had named mine RDPclientless


Click add


I added the windows on and it worked perfectly.  I also have tried many versions of the MAC configuration but I have not had any success.


One thing to note when ever you make changes to these profiles the Auto start check box becomes un-checked.