Everyone who owns an Unifi product is familiar with SDN controller, the UI and how to configure it. When you need Jumbo Frames or MTU sizes above the default of 1500 you need to adjust your configuration. On the Unifi Switches this can easily be done from the controller UI itself. As you can see in the screenshot below (from my own Unifi SDN Controller), you only have to do one-click in order to enable it on the switch. This can be done on all the switches.
However, when you do this, it only enables jumbo frames on the same layer-2 segment. When we need jumbo frames across a layer-3 interface (Router), in this case, the USG, we need to enable a higher MTU size than 1500 on the specific interfaces.
Since there is no possibility in the UI we need to do this via the CLI. Login via SSH onto your USG and execute the following command:
admin@USG# set interfaces ethernet eth1 mtu 9000
In my case, I enabled Jumbo Frames on the physical eth1 interface (LAN1). Since I have a controller-based setup, the CLI input is not persistent. We need to convert these commands into json file configuration for the SDN controller. Refer to this blog post, to see how to do this.
My json configuration for the interfaces looks like this:
{“interfaces”: {“ethernet”: {“eth1”: {“mtu”: “9000”,“speed”: “auto”,“vif”: {“12”: {“address”: [“192.168.12.254/24”],“firewall”: {“in”: {“ipv6-name”: “LANv6_IN”,“name”: “LAN_IN”},“local”: {“ipv6-name”: “LANv6_LOCAL”,“name”: “LAN_LOCAL”},“out”: {“ipv6-name”: “LANv6_OUT”,“name”: “LAN_OUT”}},}},“13”: {“address”: [“192.168.13.254/24”],“firewall”: {“in”: {“ipv6-name”: “LANv6_IN”,“name”: “LAN_IN”},“local”: {“ipv6-name”: “LANv6_LOCAL”,“name”: “LAN_LOCAL”},“out”: {“ipv6-name”: “LANv6_OUT”,“name”: “LAN_OUT”}},}},
After I saved the configuration to my Unifi controller I was able to push the configuration successfully. I now have an Jumbo Frames enabled USG setup.
Thanks for reading, if you have any questions or remarks, please don’t hesitate to reach out!
Cheers!
-Wesley-
WesleyGeelhoed.nl 
I’m trying to get this to work, when I update the config.gateway.json file I get the error below:
configuration commit error. Error message: {“DELETE”:{“failure”:”0″,”success”:”1″},”SESSION_ID”:”8be99e781cd096ef2de08b32d1″,”SET”:{“error”:{“interfaces ethernet eth0 vif 100 mtu 9000″:”MTU must be least than or equal to parent interface MTU\n\nValue validation failed\n”,”interfaces ethernet eth0 vif 200 mtu 9000″:”MTU must be least than or equal to parent interface MTU\n\nValue validation failed\n”,”interfaces ethernet eth0 vif 99 mtu 9000″:”MTU must be least than or equal to parent interface MTU\n\nValue validation failed\n”},”failure”:”1″,”success”:”1″}}
LikeLike
Did you also set the MTU to 9000 on the physical interface first?
LikeLike
Yes on the PC i”m using it is. Do you mean on the USG itself? The switch has the jumbo frames enabled. Thanks for the quick reply!
LikeLike
Yeah the physical interface on the USG itself needs an 9000 MTU as well. Not only the VLAN (vif) interfaces.
LikeLike
Yes I do. Here is the text of my json file. I actually copied it from the config.boot file. Thanks for the help!!!
“interfaces”: {
“ethernet”: {
“eth0”: {
“address”: [
“192.168.10.1/24”
],
“description”: “LAN”,
“duplex”: “auto”,
“firewall”: {
“in”: {
“ipv6-name”: “LANv6_IN”,
“name”: “LAN_IN”
},
“local”: {
“ipv6-name”: “LANv6_LOCAL”,
“name”: “LAN_LOCAL”
},
“out”: {
“ipv6-name”: “LANv6_OUT”,
“name”: “LAN_OUT”
}
},
“mtu”: “9000”,
“speed”: “auto”,
“vif”: {
“100”: {
“address”: [
“192.168.100.1/24”
],
“firewall”: {
“in”: {
“ipv6-name”: “LANv6_IN”,
“name”: “LAN_IN”
},
“local”: {
“ipv6-name”: “LANv6_LOCAL”,
“name”: “LAN_LOCAL”
},
“out”: {
“ipv6-name”: “LANv6_OUT”,
“name”: “LAN_OUT”
}
},
“mtu”: “9000”
},
“200”: {
“address”: [
“192.168.200.1/24”
],
“firewall”: {
“in”: {
“ipv6-name”: “LANv6_IN”,
“name”: “LAN_IN”
},
“local”: {
“ipv6-name”: “LANv6_LOCAL”,
“name”: “LAN_LOCAL”
},
“out”: {
“ipv6-name”: “LANv6_OUT”,
“name”: “LAN_OUT”
}
},
“mtu”: “9000”
},
“99”: {
“address”: [
“172.16.0.1/24”
],
“firewall”: {
“in”: {
“ipv6-name”: “GUESTv6_IN”,
“name”: “GUEST_IN”
},
“local”: {
“ipv6-name”: “GUESTv6_LOCAL”,
“name”: “GUEST_LOCAL”
},
“out”: {
“ipv6-name”: “GUESTv6_OUT”,
“name”: “GUEST_OUT”
}
},
“mtu”: “9000”
}
}
},
“eth1”: {
“disable”: “””,
“duplex”: “auto”,
“speed”: “auto”
},
“eth2”: {
“address”: [
“dhcp”
],
“description”: “WAN”,
“dhcp-options”: {
“client-option”: [
“retry 60;”
],
“default-route”: “update”,
“default-route-distance”: “1”,
“name-server”: “no-update”
},
“duplex”: “auto”,
“firewall”: {
“in”: {
“ipv6-name”: “WANv6_IN”,
“name”: “WAN_IN”
},
“local”: {
“ipv6-name”: “WANv6_LOCAL”,
“name”: “WAN_LOCAL”
},
“out”: {
“ipv6-name”: “WANv6_OUT”,
“name”: “WAN_OUT”
}
},
“speed”: “auto”
},
“eth3”: {
“disable”: “””,
“duplex”: “auto”,
“speed”: “auto”
}
},
“loopback”: {
“lo”: “””
}
}
LikeLike
It should work when you only edit the ‘parent interface’ with the 9000 MTU as well. The VIF interfaces will automatically get the MTU of 9000, even when you don’t specifically configure it on the USG.
LikeLike
I too am receiving the same error mentioned above. I have the mtu set on the main eth, (eth0 in my case) along with a few vifs and when I Force Provision, I get the error described above. any ideas?
LikeLike
Only configure the MTU for the parent interface. The VIFs wil inherit the MTU settings.
LikeLike
thanks for your quick reply. i tried this as well but, it didn’t throw the error this time, but ping test with 8972 is not working through any of the vifs
LikeLike
just a little more on what I am testing, I am trying to setup NSX-T in my lab using Jumbo Frames with an USG, and when I try to vmkping out of the NSX-T host TEP interface (vmk10) to the NSX-T Edge TEP IP, I am unsuccessful. However, if I run the commands via the CLI on the USG to set the MTU on eth0 and the eth0 vif interface, the vmkping test works as expected.
LikeLike
This should work, since I have NSX-T running routed from Transport Node (ESXi) on one VLAN to the NSX-T Edge Node Transport Node on the other VLAN through my USG.
In my JSON configuration file I only have set the MTU9000 setting on my physical (eth1) interface and this works even on the underlying VIFs. Are you sure everything else in the chain supports jumbo frames?
Regarding NSX-T Edge TEP, as far as I know the NSX-T Edge Node TEP does not supports MTU sizes above 2000 (this was at least in previous NSX-T versions)
LikeLike
i send you a DM in the vexpert slack…let’s connect and discuss
LikeLike
Hi Alex,
Hope you’re doing well. I was curious, did you already make some progress with the USG MTU settings on the VIFs?
Thanks mate!
LikeLike
I did, I ended up writing a script that executes every hour and changes MTU settings for each VIF on the USG.
LikeLike
The other comments seem correct – I get the same issue, and I have manually set a different value of 1500 on a VIF under an interface that was set for 9000. With the JSON config set like the others, I get the same error, but manually on the USG it works until the config is removed. Theres something with setting it for the VIFs in the JSON file which is odd. If you only set the parent interface, the VIFs do not get set 9000, I have tested Jumbo pings with this and they fail, if you set the mtu for the vif in the CLI, it works.
LikeLike
Hello,
I ran across this blog post, which seems to address my issue. I have a very simple Unifi setup at a lab with a Unifi Security Gateway Pro and a single Unifi 24-port switch. WAN is eth0 and LAN is eth2 on the USG. I have jumbo frames enabled on the switch, which has a few ESXi hosts and a NAS attached to it. The NAS, the vmkernels, and the vDS are all set to 9000 MTU. My NFS and vMotion networks are VLANs only, and the jumbo frames traffic works as anticipated on those VLANs. Management, however, is on a routable network, but since the traffic between the hosts never leaves the switch, jumbo frames works fine there as well.
My issue arises when setting up a vSAN witness appliance for a 2-node vSAN. The two vSAN hosts are directly connected to each other with 10gig links, and management traffic between these two hosts and the other host that’s running the appliance VM are all humming along fine with jumbo frames. However, management traffic between the appliance VM and the other hosts is failing for any packet beyond 1472 bytes. I was guessing this might be caused by the VM’s traffic going through the router even though it’s all the same VLAN and subnet.
Before messing with my config.gateway.json file, I ran the command on the gateway itself:
set interfaces ethernet eth2 mtu 9000
However, this made no difference. Is there something else I need to do to make this “stick” long enough to test it? Or perhaps my issue is something else entirely? Thanks.
LikeLike
Hi Dade,
Thanks for reading my blogpost!
First of all, when you enter the the configuration via the CLI it ”should” stick long enough to test certain things. If you want to make sure, just bring the controller down.
For me; setting the MTU to 9000 on the physical interface of the USG was enough to get my overlay networking working with NSX-T. However, when I test a vmkping with 8972 bytes, it won’t work. This works only for me when I set the MTU of 9000 to the VLAN (VIF) interface as well.
So, you should do the same tests you did after you’ve set the MTU of 9000 to the specific VIF interfaces as well.
set interfaces ethernet eth2 vif (VLAN NUMBER) mtu 9000
If this works, you know where the problem lies. But then you will face the next issue; until now.. Me, and a lot of other people who are running USG with the Controller based json setup where not able to configure MTU on the vif interfaces via the json config file. The controller keeps throwing error messages. So if you want persistence, you should create some kind of script that configure the MTU for the VIFS on the CLI.
Let me know, if you came any further.
Thanks!
LikeLike
Hi Wesley,
The command on the VIF did not make a difference, so I think I have an issue somewhere else. However, I’m sure this information will help me at some point. Thanks for the reply, and I’ll update if I have any breakthrough.
LikeLike
I dug some more with just simple Windows 2016 VMs on different VLANs. Two are on the native VLAN (or VLAN 1 in my vDS’s DPG since the port profile is set to require VLAN tagging for all VLANs/no “default” VLAN), and the other VM is LAN 3. After setting jumbo frames within Windows on all three, I tested with normal packets just to make sure I could communicate.
Running
ping 192.168.x.x -f -l 1472
worked fine on all three VMs to the other two.
Next I went with just 1473. As expected, communication between the two VMs on the native VLAN is fine, but fails between VLANs 1 and 3.
Going from 192.168.1.x to 192.168.3.x (where 192.168.1.z is the gateway on VLAN 1)
ping 192.168.3.x -f -l 1472
Reply from 192.168.1.z: Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 192.168.3.x:
Packets: Sent = 4, Received = 1, Lost = 3 (75% loss)
On the USG, I ran all of these commands, one at a time, and retested after each change:
admin@zzzz:~$ configure
[edit]
admin@zzzz# set interfaces ethernet eth2 mtu 9000
[edit]
admin@zzzz# set interfaces ethernet eth2 vif 3 mtu 9000
[edit]
admin@zzzz# set interfaces ethernet eth2 vif 1 mtu 9000
[edit]
admin@zzzz# set interfaces ethernet eth2 vif 0 mtu 9000
[edit]
admin@zzzz#
(That vif 0 attempt was just me taking a shot in the dark)
Anyway, each subsequent test failed in the exact same way as it did before I had done anything at all on the USG.
I have verified on the controller that the switch has jumbo frames enabled, and I have verified that the vDS and all the vmkernel ports on all the hosts are set to MTU 9000.
According to my controller (v 5.12.72), firmware on my USG and the switch are all current.
Any thoughts on what I should try next? Thanks.
LikeLike
Sorry, correction to above (copy/paste fail):
Next I went with just 1473. As expected, communication between the two VMs on the native VLAN is fine, but fails between VLANs 1 and 3.
Going from 192.168.1.x to 192.168.3.x (where 192.168.1.z is the gateway on VLAN 1)
ping 192.168.3.x -f -l 1473
Reply from 192.168.1.z: Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Ping statistics for 192.168.3.x:
Packets: Sent = 4, Received = 1, Lost = 3 (75% loss)
LikeLike
Thank you so much, I was having issues where my wired connection to my NAS was traversing the router (using external domain name to connect) and causing connection issues. Setting the LAN interface to Jumbo Frames fixed it.
LikeLike
Thank you so much, I was having issues where my wired connection to my NAS was traversing the router (using external domain name to connect) and causing connection issues. Setting the LAN interface to Jumbo Frames fixed it.
LikeLike