{"id":515,"date":"2010-01-31T18:22:43","date_gmt":"2010-01-31T23:22:43","guid":{"rendered":"http:\/\/www.jasemccarty.com\/blog\/?p=515"},"modified":"2010-01-31T18:22:43","modified_gmt":"2010-01-31T23:22:43","slug":"nfs-and-iscsi-performance-gotcha-vmnic-autonegotiation-rx-tx","status":"publish","type":"post","link":"https:\/\/www.jasemccarty.com\/blog\/nfs-and-iscsi-performance-gotcha-vmnic-autonegotiation-rx-tx\/","title":{"rendered":"NFS performance gotcha: vmnic Autonegotiation, RX, & TX"},"content":{"rendered":"

I recently migrated a production environment off of Fibre Channel over to NFS. For anyone looking to implement either NFS or iSCSI in a vSphere or VI3 environment, I would definitely recommend reading the post A \u201cMultivendor Post\u201d to help our mutual NFS customers using VMware<\/a><\/strong> hosted on Chad Sakac’s blog<\/a><\/strong> as well as on here<\/a><\/strong> Vaughn Stewart’s blog<\/a><\/strong>.\u00a0 It is a very good read, and stresses the point that the “storage network” should be configured appropriately for storage traffic.<\/p>\n

Best Practices<\/strong>
\nThe second bullet point in the section Performance consideration #2: Design a \u201cBet the Business\u201d Ethernet Network<\/strong> talks about enabling flow control.\u00a0 In the NetApp environment, typically switches are set to receive on and NFS targets are set to transmit on.<\/p>\n

I don’t have any EMC equipment, but I do have a NetApp filer.\u00a0 So I looked at the NetApp Technical Report:\u00a0 TR3749 NetApp and VMware vSphere Storage Best Practices<\/a><\/strong> as a reference in configuring my environment.<\/p>\n

Scott Lowe posted an article today on some EMC Celerra Optimizations for VMware on NFS<\/a><\/strong>, and is a good read, but I could not find anything related to flow control at the ESX level.\u00a0 I have an open question with Chad Sakac about recommended flow control settings with EMC storage.<\/p>\n

On page 46 of TR-3749 (Section 9.3) the first paragraph reads: Flow control is the process of managing the rate of data transmission between two nodes to prevent a fast sender from over running a slow receiver. Flow control can be configured ESX servers, FAS storage arrays, and network switches. It is recommended to configure the end points, ESX servers<\/strong> and NetApp arrays with flow control set to “send on” and “receive off.”<\/em><\/p>\n

Configuring Storage Nics<\/strong>
\nTo configure flow control from the Service Console it is pretty straightforward.\u00a0 Use ethtool<\/strong> to adjust the flow settings of a physical nic.<\/p>\n

The basic syntax to view the current configuration of a vmnic is:<\/p>\n

ethtool -a ethX<\/p><\/blockquote>\n

The syntax to change the configuration of a vmnic is different:<\/p>\n

ethtool -A ethX [autoneg on|off] [rx on|off] [tx on|off]<\/p><\/blockquote>\n

So to change the settings of vmnic2, the syntax would be:<\/p>\n

ethtool -A vmnic2 autoneg off rx off tx on<\/p><\/blockquote>\n

Upon initial setup I configured each of the storage nics with autonegotiation off, receive off, and transmit on.\u00a0 So my hosts and my NetApp were set to transmit, and my switches were set to receive, per TR-3749.\u00a0 Performance was awesome, and NetApp filer CPU performance was low as well.\u00a0 Things looked good.<\/p>\n

The Gotcha<\/strong>
\nIt is not unheard of to keep an ESX host up for months or years at a time, so the “gotcha” wasn’t apparent until several months after migrating VM’s from our older FC SAN to the NFS datastores presented by the NetApp.\u00a0 With about 300 guests at the time of initial setup, watching CPU rise (somewhat) on my filer did not seem strange as I migrated more guests from FC to NFS.<\/p>\n

One of my hosts indicated a hardware issue, so I evacuated guests from it, and took it offline.\u00a0 After careful investigation, and a replacement part, the host was brought back online.\u00a0 I still didn’t notice my issue at this point.\u00a0 But the CPU utilization of this host was a little more than it had been in the past, when loaded with the same number of VM’s with about the same workload.<\/p>\n

Another a couple hosts needed to be moved from the temporary location they were in to a more permanent location.\u00a0 Again, I evacuated the VM’s, powered them down, moved them, reran connections, and put the hosts back into service.\u00a0 Again, the hosts behaved about the same as before, but I still didn’t notice the gain in CPU utilization.<\/p>\n

I was looking at my filer, and did notice that the CPU utilization had jumped by about 10% on average.\u00a0 I did notice that the guests were restarting a little more slowly during the most recent boot storm after a patch window.\u00a0 Now, no additional VM’s were\u00a0 added, and no other changes were made to the environment.\u00a0 The only changes, were that hosts had been rebooted.\u00a0 Keep in mind it is not uncommon to run 70-80 guests per host for me.<\/p>\n

The “gotcha” was that the flow control settings are not persistent<\/em><\/strong> after reboots of an ESX host.<\/p>\n

Running ethtool -a against all of the vmnics on the moved and rebooted hosts, showed that autonegotiate was not set to autonegotiate off\/receive off\/transmit on.<\/p>\n

The Fix<\/strong>
\nTo ensure that all of my storage vmnics (4 per host) are properly configured, I modified \/etc\/rc.local<\/strong> to include the appropriate commands upon startup after an ESX reboot.<\/p>\n

ethtool -A vmnicW autoneg off rx off tx on
\nethtool -A vmnicX autoneg off rx off tx on
\nethtool -A vmnicY autoneg off rx off tx on
\nethtool -A vmnicZ autoneg off rx off tx on<\/p><\/blockquote>\n

Now every time a host is booted, the transmit configuration (per TR-3749) is restored.<\/p>\n

Note: This also works on ESXi, but will require modifying rc.local using the unsupported “Tech Support Mode.<\/a><\/strong>”<\/p>\n

The Conclusion<\/strong>
\nAfter correcting all hosts to reconfigure the “send on” settings on boot, VM’s are much more responsive during boot storms, overall host CPU is lower during normal operation, and the NetApp filer’s CPU utilization is lower as well.<\/p>\n

The point to the story, is that initial configurations can be lost on reboot depending on the “stickiness” of the configuration.<\/p>\n","protected":false},"excerpt":{"rendered":"

I recently migrated a production environment off of Fibre Channel over to NFS. For anyone looking to implement either NFS or iSCSI in a vSphere … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[37,38,56,58],"class_list":["post-515","post","type-post","status-publish","format-standard","hentry","category-virtualization","tag-esx","tag-esxi","tag-netapp","tag-nfs"],"_links":{"self":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts\/515","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/comments?post=515"}],"version-history":[{"count":0,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/posts\/515\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/media?parent=515"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/categories?post=515"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.jasemccarty.com\/blog\/wp-json\/wp\/v2\/tags?post=515"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}