Jase's Place

Over the past year or so, I have started using PowerShell more and more.

I was really exited to see the release of the NetApp Data ONTAP PowerShell Toolkit 1.0. I say this, because I have been using the “unsupported” PoshOnTap PowerShell extensions for some NetApp management. I have to give credit where credit is due, and Glenn Sizemore has done a great job with PoshOnTap.

PowerShell is really powerful, and because more and more vendors are providing more and more modules for the management of different aspects of the enterprise. With one script, I can create a NetApp volume, export it as an NFS export, and then connect it to all of my VMware vSphere hosts.

Note: This script is meant to be run from the vSphere PowerCLI.

##########################################################
# Create-And-Add-Storage-example.ps1
# Jase McCarty 6/7/2010
# Posh Script to add a volume on a NetApp Filer
# and present it to all vSphere hosts
##########################################################

# Add the vSphere PowerCLI SnapIn and the DATA ONTAP Module
Import-module DataONTAP

# Set my variables
$vCenter = vcenter.jasemccarty.com
$Filer = netapp1.jasemccarty.com
$aggr = aggr1
$newvol = volx

# Connect to vCenter and our NetApp Filer
Connect-VIServer $vCenter
Connect-NaController $Filer

# Create a new volume
New-NaVol $newvol $aggr 500g

# Set some options for the new volume
Set-NaVolOption $newvol no_atime_update yes
Set-NaVolOption $newvol fractional_reserve 0

# Set the SnapShot Reserve to 0
Set-NaSnapshotreserve  $newvol 0
Set-NaSnapshotschedule $newvol -Weeks 0 -Days 0 -Hours 0

# Add an NFS export
Add-NaNfsExport /vol/$newvol -Persistent -ReadWrite all-hosts -NoSuid -SecurityFlavors sys,krb5

# Get all the vSphere Hosts and add the NFS export
$Hosts = Get-VMHost
ForEach ($H in $Hosts)
{
 New-Datastore -Nfs -VMHost $_.Name -NAME $newvol -Path /vol/$newvol -NfsHost $Filer;
}

**Note, this is a pretty incomplete script.  Security for the export will have to be addressed, and the Filer management IP (netapp1.jasemccarty.com) is probably not the same as the IP used for the NFS export.  This is just an example of how this can be used.

Pretty cool.  I can only imagine the Poshibilities.

I was looking at some tweets, and noticed one from @ScriptingGuys about how to add voice to a PowerShell script.  Here is the actual tweet: http://twitter.com/ScriptingGuys/statuses/9343448757

Here’s the syntax in an example:
(New-Object -ComObject sapi.spvoice).speak(“Greetings Professor Falken”)
(New-Object -ComObject sapi.spvoice).speak(“Shall we play a game”)

So I took one of my PowerShell scripts, and modified it.  I chose my PowerCLI: NFS Settings for vSphere (NetApp NFS Recommendations) script, because it is a short script, and it was easy to update.

#***************************************************************************
#
# Update TCP and NFS Advanced Configuration Settings for vSphere w/Voice
# Author: Jase McCarty
# Date: 2/19/2010
#
#***************************************************************************

<span style="color: #0000ff;">(New-Object -ComObject sapi.spvoice).speak("What is the ESX Host Name or IP Address")</span>
$ESXHOST = Read-Host “Enter ESX Host Name or IP”
Connect-VIServer $ESXHOST

<span style="color: #0000ff;"> (New-Object -ComObject sapi.spvoice).speak("Now updating TCP and NFS Advanced Configuration Settings on host " + $ESXHOST)</span>
Write-Host “Updating TCP and NFS Advanced Configuration Settings on host ” + $ESXHOST

# Update TCP Settings
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name Net.TcpipHeapSize  -Value 30
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name Net.TcpipHeapMax  -Value 120

# Update NFS Settings (For ESX 3.5, MaxVolumes should be 32)
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name NFS.MaxVolumes  -Value 64
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name  NFS.HeartbeatMaxFailures -Value 10
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name  NFS.HeartbeatFrequency -Value 12
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name  NFS.HeartbeatTimeout -Value 5

(New-Object -ComObject sapi.spvoice).speak("NetApp Settings Complete")

Pretty cool addition.  A good use of this, (as well as text prompts) would be when running commands against vCenter, “vocally” indicating changes to objects (hosts, clusters, guests, etc) as they are being modified.

This might be a little much, but when running a script against many objects, it would not require having to watch the screen for output while the script is running.  Attention could then be focused on other things.

I don’t know if anyone is going to use this little addition, as it does take extra code.  Administrators who approach scripts like I do (with many status messages and the like) might find this as a cool addition.

Figured I’d share.

I have rebuilt several systems lately, all of which are connected to a NetApp filer.

Reference Material:
Because I’m connecting to a NetApp filer, I made sure to reference TR-3749 NetApp and VMware vSphere Storage Best Practices for the appropriate settings.  I am specifically referencing the settings in section 6 NFS Storage Recommendations on page 35.

After manually configuring the Advanced Settings for my first host, I decided I didn’t want to have to manually go through the process of updating these settings for each host.

Working Smarter:
So I put together a quick script to update the settings via the PowerCLI.

#***************************************************************************
#
# Update TCP and NFS Advanced Configuration Settings for vSphere
# Author: Jase McCarty
# Date: 1/15/2010
#
#***************************************************************************

$ESXHOST = Read-Host "Enter ESX Host Name or IP"
Connect-VIServer $ESXHOST

Write-Host "Updating TCP and NFS Advanced Configuration Settings"

# Update TCP Settings
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name Net.TcpipHeapSize -Value 30
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name Net.TcpipHeapMax -Value 120

# Update NFS Settings (For ESX 3.5, MaxVolumes should be 32)
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name NFS.MaxVolumes -Value 64
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name NFS.HeartbeatMaxFailures -Value 10
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name NFS.HeartbeatFrequency -Value 12
Set-VMHostAdvancedConfiguration -VMHost $ESXHOST -Name NFS.HeartbeatTimeout -Value 5

Verification:
Once hosts are added into vCenter, the NetApp VSC confirms that all settings are correct.

Method to the madness:
This script is run against individual hosts, whether they are connected to vCenter or not.  I chose to go this route, because I did not want to run a risk of updating a host that was already configured, and added to my vCenter installation.

I recently migrated a production environment off of Fibre Channel over to NFS. For anyone looking to implement either NFS or iSCSI in a vSphere or VI3 environment, I would definitely recommend reading the post A “Multivendor Post” to help our mutual NFS customers using VMware hosted on Chad Sakac’s blog as well as on here Vaughn Stewart’s blog.  It is a very good read, and stresses the point that the “storage network” should be configured appropriately for storage traffic.

Best Practices
The second bullet point in the section Performance consideration #2: Design a “Bet the Business” Ethernet Network talks about enabling flow control.  In the NetApp environment, typically switches are set to receive on and NFS targets are set to transmit on.

I don’t have any EMC equipment, but I do have a NetApp filer.  So I looked at the NetApp Technical Report:  TR3749 NetApp and VMware vSphere Storage Best Practices as a reference in configuring my environment.

Scott Lowe posted an article today on some EMC Celerra Optimizations for VMware on NFS, and is a good read, but I could not find anything related to flow control at the ESX level.  I have an open question with Chad Sakac about recommended flow control settings with EMC storage.

On page 46 of TR-3749 (Section 9.3) the first paragraph reads: Flow control is the process of managing the rate of data transmission between two nodes to prevent a fast sender from over running a slow receiver. Flow control can be configured ESX servers, FAS storage arrays, and network switches. It is recommended to configure the end points, ESX servers and NetApp arrays with flow control set to “send on” and “receive off.”

Configuring Storage Nics
To configure flow control from the Service Console it is pretty straightforward.  Use ethtool to adjust the flow settings of a physical nic.

The basic syntax to view the current configuration of a vmnic is:

ethtool -a ethX

The syntax to change the configuration of a vmnic is different:

ethtool -A ethX [autoneg on|off] [rx on|off] [tx on|off]

So to change the settings of vmnic2, the syntax would be:

ethtool -A vmnic2 autoneg off rx off tx on

Upon initial setup I configured each of the storage nics with autonegotiation off, receive off, and transmit on.  So my hosts and my NetApp were set to transmit, and my switches were set to receive, per TR-3749.  Performance was awesome, and NetApp filer CPU performance was low as well.  Things looked good.

The Gotcha
It is not unheard of to keep an ESX host up for months or years at a time, so the “gotcha” wasn’t apparent until several months after migrating VM’s from our older FC SAN to the NFS datastores presented by the NetApp.  With about 300 guests at the time of initial setup, watching CPU rise (somewhat) on my filer did not seem strange as I migrated more guests from FC to NFS.

One of my hosts indicated a hardware issue, so I evacuated guests from it, and took it offline.  After careful investigation, and a replacement part, the host was brought back online.  I still didn’t notice my issue at this point.  But the CPU utilization of this host was a little more than it had been in the past, when loaded with the same number of VM’s with about the same workload.

Another a couple hosts needed to be moved from the temporary location they were in to a more permanent location.  Again, I evacuated the VM’s, powered them down, moved them, reran connections, and put the hosts back into service.  Again, the hosts behaved about the same as before, but I still didn’t notice the gain in CPU utilization.

I was looking at my filer, and did notice that the CPU utilization had jumped by about 10% on average.  I did notice that the guests were restarting a little more slowly during the most recent boot storm after a patch window.  Now, no additional VM’s were  added, and no other changes were made to the environment.  The only changes, were that hosts had been rebooted.  Keep in mind it is not uncommon to run 70-80 guests per host for me.

The “gotcha” was that the flow control settings are not persistent after reboots of an ESX host.

Running ethtool -a against all of the vmnics on the moved and rebooted hosts, showed that autonegotiate was not set to autonegotiate off/receive off/transmit on.

The Fix
To ensure that all of my storage vmnics (4 per host) are properly configured, I modified /etc/rc.local to include the appropriate commands upon startup after an ESX reboot.

ethtool -A vmnicW autoneg off rx off tx on
ethtool -A vmnicX autoneg off rx off tx on
ethtool -A vmnicY autoneg off rx off tx on
ethtool -A vmnicZ autoneg off rx off tx on

Now every time a host is booted, the transmit configuration (per TR-3749) is restored.

Note: This also works on ESXi, but will require modifying rc.local using the unsupported “Tech Support Mode.”

The Conclusion
After correcting all hosts to reconfigure the “send on” settings on boot, VM’s are much more responsive during boot storms, overall host CPU is lower during normal operation, and the NetApp filer’s CPU utilization is lower as well.

The point to the story, is that initial configurations can be lost on reboot depending on the “stickiness” of the configuration.