{"id":3993,"date":"2018-10-08T16:59:36","date_gmt":"2018-10-08T21:59:36","guid":{"rendered":"http:\/\/www.jasemccarty.com\/blog\/?p=3993"},"modified":"2020-02-16T17:00:15","modified_gmt":"2020-02-16T23:00:15","slug":"sc2n-wts-mixed-mtu-67u1","status":"publish","type":"post","link":"https:\/\/www.jasemccarty.com\/blog\/sc2n-wts-mixed-mtu-67u1\/","title":{"rendered":"Understanding Mixed MTU support in Stretched &amp; 2 Node vSAN 6.7 U1 clusters with Witness Traffic Separation"},"content":{"rendered":"<p>Witness Traffic Separation introduced the ability to tag &#8220;witness&#8221; traffic on an alternate VMkernel interface on vSAN data nodes.<\/p>\n<p>Witness Traffic Separation is the feature that enabled 2 Node &#8220;Direct Connect&#8221; in vSAN 6.5, and was also added to later editions of vSAN 6.2 (vSphere 6.0 Update 3 &amp; higher).<\/p>\n<p>Some additional logic was added in vSAN 6.7 to support Witness Traffic Separation in Stretched Clusters as well.<\/p>\n<h2>MTU Size<\/h2>\n<p>Traditional vSAN Clusters require the MTU size to be uniform across all vSAN VMkernel interfaces. vSAN Stretched Clusters have had the same requirement, due to the vSAN data network connecting to the vSAN Witness Host. When VMkernel ports communicate with each other and have mixed MTU sizes, some potential communication challenges include dropped packets, retransmits, and the like.<\/p>\n<p>But what about configurations using Witness Traffic Separation?<\/p>\n<p>The &#8220;backend&#8221; vSAN data network doesn&#8217;t communicate with the vSAN Witness. Communication with the vSAN Witness is over a completely different VMkernel interface that has been tagged for &#8220;witness&#8221; traffic.<\/p>\n<p>It only makes sense that completely different MTU sizes on the vSAN data network and the network used to communicate with the vSAN Witness would work without issue.<\/p>\n<h2>The Need for Mixed MTU<\/h2>\n<p>One common ask for &#8220;Mixed MTU&#8221; when using Witness Traffic Separation, is when connectivity to the vSAN Witness has a limited MTU, or one that is less than desired for the backend vSAN data network.<\/p>\n<p>Examples of this, could be:<\/p>\n<ul>\n<li>vSAN Stretched Cluster\n<ul>\n<li>Backend vSAN data network supports Jumbo Frames &#8211; Desired by the customer<\/li>\n<li>Frontend vSAN witness network does not support Jumbo Frames\n<ul>\n<li>Low speed connection to an alternate datacenter, <a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/2016\/11\/16\/vsan-witness-in-vca-p1\/\" target=\"_blank\" rel=\"noopener noreferrer\">OVH<\/a>, or <a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/2018\/09\/27\/vsan-witness-appliance-in-vmware-cloud-on-aws\/\" target=\"_blank\" rel=\"noopener noreferrer\">VMC<\/a> running the vSAN Witness.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>2 Node vSAN\n<ul>\n<li>Backend Direct Connect vSAN with Jumbo Frames<\/li>\n<li>Frontend vSAN connectivity to vSAN Witness in a central DC, on OVH, or on VMC, connected over an IPSEC VPN w\/max MTU of 1500\n<ul>\n<li>May have to use an MTU of something like 1372<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>With versions of vSAN prior to 6.7 Update 1, the MTU must match on all vSAN VMkernel interfaces, regardless of being tagged for vSAN or Witness traffic types to be supported.<\/p>\n<h2>Support for Mixed MTU with 6.7 Update 1<\/h2>\n<p><a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/Mixed-MTU-67U1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-13849\" src=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/Mixed-MTU-67U1.png\" alt=\"Mixed MTU\" width=\"300\" height=\"208\" srcset=\"https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/Mixed-MTU-67U1.png 800w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/Mixed-MTU-67U1-220x153.png 220w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/Mixed-MTU-67U1-768x533.png 768w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/Mixed-MTU-67U1-274x190.png 274w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>VMware isn&#8217;t supporting mixed MTUs for interfaces that communicate directly with each other, but rather mixed for the different vSAN traffic types.<\/p>\n<p>The vSAN Health Check is updated in vSAN 6.7 Update 1 to recognize Witness Traffic Separation deployments and allow for a different MTU for the vSAN data and vSAN witness networks.<\/p>\n<p>Here is an illustration showing the vSAN data network (VMkernel vmk2) with Jumbo Frames configured and vSAN Traffic enabled on a host.<\/p>\n<p><a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-13842 size-large\" src=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02-1024x442.png\" alt=\"Mixed MTU\" width=\"580\" height=\"250\" srcset=\"https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02-1024x442.png 1024w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02-220x95.png 220w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02-768x331.png 768w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02-290x125.png 290w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU02.png 1402w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a>On this same host, the Management VMkernel (vmk0) is tagged for &#8220;witness&#8221; traffic with an MTU of 1500.<\/p>\n<p><a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-13843 size-large\" src=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03-1024x457.png\" alt=\"Mixed MTU\" width=\"580\" height=\"259\" srcset=\"https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03-1024x457.png 1024w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03-220x98.png 220w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03-768x343.png 768w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03-290x129.png 290w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU03.png 1398w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p>On the vSAN Witness, the Management VMkernel (vmk0) is tagged for vSAN Traffic with an MTU of 1500.<\/p>\n<p><a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-13845 size-large\" src=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04-1024x458.png\" alt=\"Mixed MTU\" width=\"580\" height=\"259\" srcset=\"https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04-1024x458.png 1024w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04-220x98.png 220w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04-768x343.png 768w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04-290x130.png 290w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU04.png 1360w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a><\/p>\n<p><i><\/i>Remember that the vSAN Witness has &#8220;vSAN Traffic&#8221; tagged, though it communicates with the &#8220;witness&#8221; tagged interfaces on the data nodes when using Witness Traffic Separation. <em>In this installation, vmk1 is not being used, this is also a valid configuration.<\/em><\/p>\n<p>Looking at the vSAN Health Check, it can be seen that the hosts are properly communicating with the vSAN Witness, even though they have different MTU settings.<\/p>\n<p><a href=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-13844 size-large\" src=\"http:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01-1024x480.png\" alt=\"Mixed MTU\" width=\"580\" height=\"272\" srcset=\"https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01-1024x480.png 1024w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01-220x103.png 220w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01-768x360.png 768w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01-290x136.png 290w, https:\/\/blogs.vmware.com\/virtualblocks\/files\/2018\/10\/MTU01.png 1353w\" sizes=\"auto, (max-width: 580px) 100vw, 580px\" \/><\/a>Notice in the vSAN Health Check, that communications to or from the vSAN Witness only occur on vmk0 in this illustration. Vmk0 on each of the data nodes, as well as the vSAN Witness, make up the &#8220;frontend&#8221; vSAN (witness) network. The data nodes are using vmk2 for &#8220;backend&#8221; vSAN Traffic.<\/p>\n<p>To summarize in an effort to avoid confusion:<\/p>\n<table cellspacing=\"5\" cellpadding=\"5\" align=\"center\">\n<tbody>\n<tr>\n<th bgcolor=\"#8c8c8c\"><span style=\"color: #ffffff;\">Scenario<\/span><\/th>\n<th bgcolor=\"#8c8c8c\"><span style=\"color: #ffffff;\">Mixed MTU Supported<\/span><\/th>\n<\/tr>\n<tr>\n<td>vSAN Stretched Clusters pre 6.7 Update 1<\/td>\n<td align=\"center\">No<\/td>\n<\/tr>\n<tr>\n<td bgcolor=\"#cccccc\">vSAN Stretched Clusters 6.7 Update 1 or higher <em>with Witness Traffic Separation<\/em><\/td>\n<td align=\"center\" bgcolor=\"#cccccc\">Yes<\/td>\n<\/tr>\n<tr>\n<td>vSAN 2 Node Clusters 6.2 or higher<\/td>\n<td align=\"center\">No<\/td>\n<\/tr>\n<tr>\n<td bgcolor=\"#cccccc\">vSAN 2 Node Clusters with Witness Traffic Separation 6.2-6.7<\/td>\n<td align=\"center\" bgcolor=\"#cccccc\">No<\/td>\n<\/tr>\n<tr>\n<td>vSAN 2 Node Clusters <em>with Witness Traffic Separation<\/em> 6.7 Update 1 or higher<\/td>\n<td align=\"center\">Yes<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Better flexibility<\/h2>\n<p>Customers, if they choose, can use Jumbo Frames on the vSAN data network, with no requirement to extend Jumbo Frames to the vSAN Witness. By allowing the ability to configure a different MTU for the &#8220;frontend&#8221; and &#8220;backend&#8221; vSAN networks, vSAN connectivity requirements can better align with the capabilities of the network and their requirements.<\/p>\n<p>&nbsp;<\/p>\n<p>This was originally posted on the VMware Virtual Blocks site: https:\/\/blogs.vmware.com\/virtualblocks\/2018\/10\/08\/sc2n-wts-mixed-mtu-67u1\/<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Witness Traffic Separation introduced the ability to tag &#8220;witness&#8221; traffic on an alternate VMkernel interface on vSAN data nodes. Witness Traffic Separation is the feature &hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3993","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts\/3993","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/comments?post=3993"}],"version-history":[{"count":2,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts\/3993\/revisions"}],"predecessor-version":[{"id":3995,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts\/3993\/revisions\/3995"}],"wp:attachment":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/media?parent=3993"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/categories?post=3993"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/tags?post=3993"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}