ACI hell part 1

When connecting access ports with static paths within an EPG that has trunking what a pain.

So basically if you have a static path binding using 802.1p then try and put an access port with 802.1p Access Untagged things may not work.

The reason is that the 802.1p Access Untagged setting it sets the vlan to 0 in the header, but it still has a vlan tag in there.  Some access devices don’t accept it because they are not  expecting a tag period.  This is especially meaningful with appliances.

If you set your mode to 802.1p Access Untagged and use the same Encapsulation VLAN tag as trunked ports, it will not work.  ACI will give you an error saying that you can’t have tagged and untagged in the same EPG.  Yet you can if you change the encapsulation VLAN ID to a different number it will work.

Remember that a VLAN in ACI is just bogus because ACI uses VXLAN, but endpoint devices care about that VLAN number.  Below is an example of 1 EPG with multiple endpoints in the same bridge domain with different VLAN encapsulations.




ACI software upload via GUI not working

FYI there is a bug in 1.4.1e that does not allow you to upload new code via the GUI.

What happens is the file is actually copied then immediately deleted after if finishes.  This was fixed in the version right before this version, but not ported.

Fun times

Good write up on the work around over at

Thanks to Jayson for this good write up.

Where is the ACI software repository located?

Excellent question.  I did some digging via SCP and finally found it.

But don’t get to excited you can’t copy files directly into it.

The repository is located in the following path:

There is a good write up on how to upgrade your software using the command line.

  1. the current version of the firmware.

Were the firmware uploaded via the APIC GUI? If this is true, the issue is related to defect CSCux40954. Please use ‘scp’ in APIC to copy the firmware into the controllers. Attached is the instructions:

Little Gotcha with APICs within ACI

This applies to versions of APIC controller software up to 1.1(4e)

Turns out the there is a bug that occurs when you connect an APIC to multiple leaves.  And yes that is stupid because you’re supposed to connect them to different leaves.  This bug manifests itself when integrating the VMM with the fabric.

Just be sure that you create an APIC policy in the policy groups. Fabric>Access Policies>Interface Policies>Policy Group

It’s a simple policy mine is:
CDP= Disabled
L2 Interface Policy= default

Then bind the policy to the leaves that the APIC’s are connected to.  Fabric>Access Policies>Interface Policies>Profiles>Leaf###.  The click the plus sign (+) and add in your newly created APIC Policy group for the interfaces that the APIC is connected to.

How to get code onto the APIC when the GUI upload fails

The following process worked when the GUI upload via http or scp failed for the 3.8GB APIC ISO file.

If SCP fails (or stalls), what you can do is use a program like Filezilla to connect to the APIC as admin and upload the image directly.

Once the image is in the admin’s home directory, you need to issue the command “firmware add <image_name>”. This adds the file to the firmware repository and should be seen in the GUI as well.

Can’t log into your APIC?

I ran into a split fabric issue setting up my test lab and got the following error trying to log into my 2nd APIC:

REST Endpoint user authorization datastore is not initialized – Check Fabric Membership Status of this fabric node

I was able to get logged into the APIC with the follow username and a blank password:


NOTE: as in the past physical access to a Cisco device equal total ownership.

Basically when installing the fabric for the 1st time you should only power on 1 APIC and discover the entire fabric, then add the other APICs 1 at a time.

How to find your ACI fabric serial numbers

Let’s say you need to open a TAC case and didn’t document all the serial numbers of your fabric upon installation.  You can get the membership information from the command line.

SSH to your APIC OOB management address and log in.

Then issue the following command:
acidiag fnvread

you will get output simular to the following:

For spines and Leaves

admin@apic1:~> acidiag fnvread
ID Name Serial Number IP Address Role State LastUpdMsgId
101 Leaf1 SALXXXXXXXX leaf active 0
102 Spine1 SALXXXXXXX spine active 0
103 Spine2 SALXXXXXXX spine active 0

For APIC’s

admin@apic1:~> acidiag verifyapic
openssl_check: certificate details
subject= CN=FCH1922V0L4,serialNumber=PID:APIC-SERVER-M1 SN:FXXXXXXXX
issuer= CN=Cisco Manufacturing CA,O=Cisco Systems
notBefore=Jul 14 14:52:07 2015 GMT
notAfter=Jul 14 15:02:07 2025 GMT
openssl_check: passed

Wierd 6500 interface output

So the question is what state is the following port in?

#sho run all | beg GigabitEthernet2/5
interface GigabitEthernet2/5
description LAB newtork
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 138
switchport mode trunk
no snmp trap link-status

The answer might not be what you think.

#sho int g2/5
GigabitEthernet2/5 is up, line protocol is up (connected)

#sho run int g2/5

Building configuration…

Current configuration : 230 bytes
interface GigabitEthernet2/5
description LAB newtork f
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 138
switchport mode trunk
no snmp trap link-status

#sho ver
Cisco IOS Software, s3223_rp Software (s3223_rp-IPBASEK9-M), Version 12.2(33)SXI14, RELEASE SOFTWARE (fc2)

The question is why?  I’m not sure other than it might be a code version bug.



So you really want to manually configure the 6800ia? It’s possible

Below is the command syntax to enable the fex configurations mode. First you will have to run “service internal” command, it will enable the hidden commands. However, do not forget to turn off the commands after making the changes using “no service internal”

6800-1#config t

6800-1(config)#service internal


6800-1#test platform software console fex 110 enable timeout 60  <<<<< This command will enable configuration mode on fex 110 for 60 minutes


Once done with the changes on the fex110, run the following command on the parent switch to turn off the service internal mode:

6800-1#config t

6800-1(config)#no service internal