HDT Pool Creation

A little video I created to show how easy it is to create HDT pool on HDS VSP array.

VMware SRM and Data Center Migration

Looking back a few months ago I posted about our Primary Data Center (PDC) migration project, and how we planned to migrate our fairly large VMware environment between datacenters. My initial thought was to extend our SAN's across the 10GbE Metro net (MAN) we have in place and do VMotions and SVmotions to the new hardware in the PDC. While this did seem like a good idea to me in the beginning the powers that be chose to use VMware Site Recovery Manager (SRM).

Let me just state now that SRM has worked great. The ease of use to SVMotion VM's to new "staging area" LUNs and then replicate (HDS HUR and NetApp Snapmirror) those to the PDC was so simple. Then the SRM migrations have been simple as well. The testplan have shown to be a great way to verify everything will work before the actual maintenance window.

Let me just say that I was wrong in thinking the SRM migrations would never work for our needs, but it does. Just another piece of VMware that just works and is very clean in its implementation.

Brocade 7500 SAN extension

I've been working on setting up SAN extension between 2 of my DataCenters in OKC. To extend the fabrics across the 10Gb/s link I am using the Brocade 7500 with FCIP license.

My initial configuration thought was to keep the 4 switches (2x48K's and 2x7500's) that I have in the mix as separate un-merged (using EX and VEX ports) fabrics. Well that won't work as you can't have EX ports on Edge fabrics and on the Backbone (the FCIP link). Below are the commands to setup the configuration I used.

 

(creating FCIP tunnel VEX-to-VE port link)

lfcrouter1# portcfgvexport -a 1 -f 80 24

lfcrouter1# portcfg fciptunnel ge1 create 0 remote-ip local-ip 700000 -c (fciptunnel 0 on ge1 with 700Mb/s commited bandwidth and compression enabled)

lfcrouter2# portcfgvexport -a 1 -f 81 24 

lfcrouter2# portcfg fciptunnel ge1 create 0 remote-ip local-ip 700000 -c (fciptunnel 0 on ge1 with 700Mb/s commited bandwidth and compression enabled)
rfcrouter1# portcfg fciptunnel ge1 create 0 remote-ip local-ip 700000 -c (fciptunnel 0 on ge1 with 700Mb/s commited bandwidth and compression enabled)
rfcrouter2# portcfg fciptunnel ge1 create 0 remote-ip local-ip 700000 -c (fciptunnel 0 on ge1 with 700Mb/s commited bandwidth and compression enabled)

lfcrouter1# portpersistentenable 24
lfcrouter2# portpersistentenable 24
rfcrouter1# portpersistentenable 24
rfcrouter2# portpersistentenable 24

Create ISL's from fcrouters to directors
Attach Fiber cable from lfcrouter1 to director1
Attach Fiber cable from lfcrouter2 to director2
Attach Fiber cable from rfcrouter1 to rdirector1
Attach Fiber cable from rfcrouter2 to rdirector2

On the directors I then create 2 LSAN zones with the Storage Array port WWN and the Host port WWN, map a couple of luns and BOOM the host in the remote datacenter sees the LUNs from the Primary DataCenter.

I am attaching a 7500 Supported configuration document as well that may help you with choosing your supported configuration for FC routing.

Vxplex DISABLED RECOVER state

After a NetApp 6080 hosting FCP LUNs failed this weekend we came into the office to notice many of the servers using those LUNs had offline volumes and disk groups.

Here was the state of the volume in question

v szdbor006du02 - DISABLED ACTIVE 2727606272 SELECT - fsgen

pl szdbor006du02-01 szdbor006du02 DISABLED RECOVER 2727606272 CONCAT - RW

sd szdbor006ddg01-01 szdbor006du02-01 szdbor006ddg01 0 209646560 0 c1t500A098187197B34d10 ENA

sd szdbor006ddg02-02 szdbor006du02-01 szdbor006ddg02 209648096 943459616 209646560 c1t500A098187197B34d11 ENA

sd szdbor006ddg03-01 szdbor006du02-01 szdbor006ddg03 0 1153107712 1153106176 c3t500A098287197B34d15 ENA

sd szdbor006ddg04-01 szdbor006du02-01 szdbor006ddg04 0 421392384 2306213888 c1t500A09828759382Fd50 ENA

Issued vxrecover on the volume and plex but the state never changed and I didn't find a vxrecovery task with ps or vxtask list. The recovery task was somehow confused I am guessing so to fix here is what I needed to do.

vxplex -g diskgroup det szdbor006du02-01

This put the plex into a DETACHED STALE state

vxmend -g diskgroup fix clean szdbor006du02-01

This put the plex back into a DETACHED CLEAN state at which point I could do a

vxvol -g diskgroup startall (I could have just put the volume name as well)

This enabled and started the volume. FSCK'd and remounted the FS.

Now to figure out why exactly the FAS6080 crashed just because of an HBA hiccup.

Hope this may be useful if you ever run into the same scenario.

NetApp fun for 24 hours

I've been working at our ORC Datacenter (Off-Site Records Center)installing 2 NetApp filers that I moved from our downtown DC. WOW...it was all going to so well until i booted up the new FAS3050 filers to replace the older 960 and 980 heads.

1st the 3050's complained about not seeing any disks it could own. Fixed that by booting maint mode and assigning them to the new 3050's.

2nd the 3050's complained that the disks had a mismatched OnTap version on them. 7.3.2 on the disks and the 3050's had 7.2.# on them.

3rd a netboot of the 3050's blew up every time. The nic would just go offline and hence would kill the netboot. I tried a netboot from downtown, from hou and directly connected to my laptop. None of which worked!

4th decided to just reuse the 900's. the first 960 boots up and and complains it cant grab ownership of the disks (because the 3050 grabbed them before). so now i have to re-rack the 3050 plug in the disk and remove_ownership.

5th the network ports locations weren't completely communicated correctly so they were on the correct VLAN's

24 hours later (930AM today) the 2 replication filers are back online in the ORC datacenter.

This freed up 8Kw of power in the downtown DC, so the DC manager is happy again.

(sorry for the bad grammar and capitalization)

Veritas SF patches....

I've been troubleshooting an issue on a Solaris 10 host with Veritas SF 4.1MP2.RP3 installed. The issue is when you reuse a LUN # vxvm and vxdmp don't realize that the LUN is actually new so DMP tries to use it as an alternate path for another disk. In my case we had migrated some LUNs to another storage port and HBA on the host for performance reasons. When we later mapped another LUN with the same # as used prior it freaked out and dropped the volume which in-turn blew up oracle.

In digging through this to find a release that had a fix I remember this site http://vias.symantec.com/labs/vpcs. It lists all the Veritas Storage products and the Maintenance Packs, Rolling Patches and Hot-fixes.

Hopefully someone else gets some good use out of it.

My first white paper...

I wrote a white paper that discussed HDS Storage Virtualization and VMware a couple years ago. I lost track of that white paper and found it this morning.

It was my first try at a publication so don't be too critical of me. I thought I would share it as well as plug my storage Vendor/Partner.

Lumenate VMware and Storage Virtualization

Lumenate