实战给装了 VMWare ESXi 的 DELL PERC H700 添加硬盘

公司有台带有 PERC H700 RAID 控制器的 DELL 服务器,上面安装了 VMWare ESXi 4.1。
由于业务扩张,硬盘空间不足,我们买了块新硬盘要加进去。
原来是3块300G的硬盘构成了RAID5,现在要通过重新配置阵列加一块进去。

中间费尽周折,会过头来再看,其实也不是那么难。相信下次就会顺利多了。

缘由
公司有台带有 PERC H700 RAID 控制器的 DELL 服务器,上面安装了 VMWare ESXi 4.1。
由于业务扩张,硬盘空间不足,我们买了块新硬盘要加进去。
原来是3块300G的硬盘构成了RAID5,现在要通过重新配置阵列加一块进去。

步骤0 备份数据

步骤1 扩展阵列

在 LSI 的主页上下载 MegaRAID VMWare Release
然后解开把文件通过 vSphare Client 上传到服务器上。然后在控制台上执行下面的命令:

/tmp/MegaCLI # ./MegaCli -LDRecon start r5 [Add PhysDrv[32:3]] L0 -a0*
 
Start Reconstruction of Virtual Drive Success.
 
Exit Code: 0x00
 
/tmp/MegaCLI # ./MegaCli -LDRecon ShowProg L0 -a0*      
 
Reconstruction on VD #0 (target id #0) Completed 0% in 0 Minutes.
 
Exit Code: 0x00

第二个命令是查看进展的。我们这个300G的硬盘大约花了不到三个小时的时间。

/tmp/MegaCLI # ./MegaCli -cfgdsply -aAll
 
==============================================================================
Adapter: 0
Product Name: PERC H700 Integrated
Memory: 512MB
BBU: Present
Serial No: 11B05V7
==============================================================================
Number of DISK GROUPS: 1
 
DISK GROUP: 0
Number of Spans: 1
SPAN: 0
Span Reference: 0x00
Number of PDs: 4   << 从3变成4了。
Number of VDs: 1
Number of dedicated Hotspares: 0
Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :Virtual Disk 0
RAID Level          : Primary-5, Secondary-0, RAID Level Qualifier-3
Size                : 836.625 GB  << 从557.75 GB变成836.625 GB了。
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 4
Span Depth          : 1

步骤2 扩展VMWare的存储区
先调查好系统状况:

~ # esxcfg-scsidevs -l
...
naa.6782bcb00f7de00014fe794f0593cfee
   Device Type: Direct-Access
   Size: 856704 MB
   Display Name: Local DELL Disk (naa.6782bcb00f7de00014fe794f0593cfee)
   Multipath Plugin: NMP
   Console Device: /vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfee
   Devfs Path: /vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfee
   Vendor: DELL      Model: PERC H700         Revis: 2.10
   SCSI Level: 5  Is Pseudo: false Status: on
   Is RDM Capable: false Is Removable: false
   Is Local: true
...
 
~ # fdisk -l
 
Disk /dev/disks/naa.6782bcb00f7de00014fe794f0593cfee: 898.3 GB, 898319253504 bytes
64 heads, 32 sectors/track, 856704 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
 
                                           Device Boot      Start         End      Blocks  Id System
/dev/disks/naa.6782bcb00f7de00014fe794f0593cfeep1             5       900    917504    5  Extended
/dev/disks/naa.6782bcb00f7de00014fe794f0593cfeep2           901      4995   4193280    6  FAT16
/dev/disks/naa.6782bcb00f7de00014fe794f0593cfeep3          4996    571136 579728384   fb  VMFS
/dev/disks/naa.6782bcb00f7de00014fe794f0593cfeep4   *         1         4      4080    4  FAT16

开始按照最下面的参考文献删除VMFS分区然后重建。

~ # fdisk /vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfee
 
The number of cylinders for this disk is set to 856704.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
 
Command (m for help): p
 
Disk /vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfee: 898.3 GB, 898319253504 bytes
64 heads, 32 sectors/track, 856704 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
 
                                                    Device Boot      Start         End      Blocks  Id System
/vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfeep1             5       900    917504    5  Extended
/vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfeep2           901      4995   4193280    6  FAT16
/vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfeep3          4996    571136 579728384   fb  VMFS
/vmfs/devices/disks/naa.6782bcb00f7de00014fe794f0593cfeep4   *         1         4      4080    4  FAT16

然后去 vSphere Client 看,可用空间已经增长了。

最初的状态:

加入硬盘后:

改好分区表之后的最终结果:

参考文献
MegaCli常用参数介绍
http://www.mysqlsupport.cn/megacli-study/
MegaCli -adpCount 【显示适配器个数】
MegaCli -AdpGetTime –aALL 【显示适配器时间】
MegaCli -AdpAllInfo -aAll 【显示所有适配器信息】
MegaCli -LDInfo -LALL -aAll 【显示所有逻辑磁盘组信息】
MegaCli -PDList -aAll 【显示所有的物理信息】
MegaCli -AdpBbuCmd -GetBbuStatus -aALL |grep ‘Charger Status’ 【查看充电状态】
MegaCli -AdpBbuCmd -GetBbuStatus -aALL【显示BBU状态信息】
MegaCli -AdpBbuCmd -GetBbuCapacityInfo -aALL【显示BBU容量信息】
MegaCli -AdpBbuCmd -GetBbuDesignInfo -aALL 【显示BBU设计参数】
MegaCli -AdpBbuCmd -GetBbuProperties -aALL 【显示当前BBU属性】
MegaCli -cfgdsply -aALL 【显示Raid卡型号,Raid设置,Disk相关信息】
MegaCli -cfgdsply -aALL |grep Policy 【查看Cache 策略设置】
MegaCli -AdpBbuCmd -GetBbuStatus -aALL |grep ‘Relative State of Charge’【查看充电进度百分比】
磁带状态的变化,从拔盘,到插盘的过程中。
Device |Normal|Damage|Rebuild|Normal
Virtual Drive |Optimal|Degraded|Degraded|Optimal
Physical Drive |Online|Failed –> Unconfigured|Rebuild|Online
Use of LSI MegaCli tool in ESX 4
http://communities.vmware.com/thread/228615

PERC H700 Online Capacity Expansion or RAID Level Migration
http://lists.us.dell.com/pipermail/linux-poweredge/2011-March/044451.html

MegaCli -LDRecon {-Start -rX [{-Add | -Rmv} -Physdrv[E0:S0,…>}|-ShowProg|-ProgDsply -Lx -aN
>

I was able to successfully reconfigure the VD from a 3 drive RAID5 set
to a 4 drive RAID5 set using the following command:
*# /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon start r5 [Add PhysDrv[32:3]]
L0 -a0*
Start Reconstruction of Virtual Drive Success.

And could check the status using:
*# /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon ShowProg L0 -a0*
Reconstruction on VD #0 (target id #0) Completed 76% in 200 Minutes.

Four hours later I’ve gone from from:
> Virtual Drive: 0 (Target Id: 0)
> Name :
> RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
> Size : 837.25 GB
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 3
To:
> Virtual Drive: 0 (Target Id: 0)
> Name :
> RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
> Size : 1.225 TB
> State : Optimal
> Strip Size : 64 KB
> Number Of Drives : 4
/opt/MegaRAID/MegaCli/MegaCli64 -getLdExpansionInfo
>>> -Lall -aALL
>>>
>>>
>>> Virtual Disk: 0 (Target Id: 0)
>>> Expansion : Not
>>> Possible
>>>
>>> Exit Code: 0x00

使用MegaCli和Smartctl获取普通磁盘和SSD磁盘信息_1
http://hi.baidu.com/stealth_space/blog/item/416e91acde14e8a4cb130c97.html

Adding a VMFS Extent using vmkfstools
http://www.zealkabi.com/2009/03/adding-vmfs-extent-using-vmkfstools.html

Extending an Existing VMFS-3 Volume

vmkfstools -Z –extendfs

This option adds another extent to a previously created VMFS volume
. You must specify the full path name, for example
/vmfs/devices/disks/vmhba0:1:2:1, not just the short name vmhba0:1:2:1. Each
time you use this option, you extend a VMFS-3 volume with a new extent so that the
volume spans multiple partitions. At most, a logical VMFS-3 volume can have 32
physical extents.

Example for Extending a VMFS-3 Volume:

vmkfstools -Z /vmfs/devices/disks/vmhba0:1:2:1 /vmfs/devices/disks/vmhba1:3:0:1

This example extends the logical file system by allowing it to span to a new partition.
The extended file system spans two partitions.vmhba1:3:0:1 and vmhba0:1:2:1. In
this example, vmhba1:3:0:1 is the name of the head partition.

Listing Attributes of a VMFS Volume

vmkfstools -P –queryfs [-h –human-readable]

When you use this option on any file or directory that resides on a VMFS volume, the
option lists the attributes of the specified volume. The listed attributes include the
VMFS version number (VMFS-2 or VMFS-3), the number of extents comprising the
specified VMFS volume, the volume label if any, the UUID, and a listing of the device
names where each extent resides.

You can specify the -h suboption with the -P option. If you do so, vmkfstools lists the capacity of the volume in a
more readable form, for example, 5k, 12.1M, or 2.1G.

For example:

# vmkfstools -P /vmfs/volumes/MSALUN12/AM_W2k3/AM_W2k3-flat.vmdk
VMFS-3.31 file system spanning 1 partitions.
File system label (if any): MSALUN12
Mode: public
Capacity 83214991360 (79360 file blocks * 1048576), 8964276224 (8549 blocks) avail
UUID: 47669f06-beeb8164-ed10-000e7fb4371c
Partitions spanned (on “lvm”):
vmhba1:0:12:1
(One or more partitions spanned by this volume may be offline)

How to increase the size of a local datastore … on an ESXi4?

fdisk /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0

As you can see on the picture, the local VMFS datastore is only 651MB. Also if you add up all the partitions you end up with a bit more than 1500MB. I’m definitely +500MB short.

Press d then Enter 2 to delete /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0p2
Press n and Enter to start creating a new partition.
Press p and Enter to identify that you are creating a primary partition.
Press 2 and Enter to identify that you are creating a second partition. A primary partition already exists.
Press Enter to accept the default ending value.
Press t and Enter to identify that you want to change the type of a partition. We want to change partition# 2.
Type fb and press Enter. This sets the volume to type VMFS.
Press w and press Enter to save the changes and exit fdisk.

Now that we have set properly the partition table, we need to grow the file system on it. The :2 in the command identifies that this operation is performed on the second partition. Run the following command:
vmkfstools –growfs /vmfs/devices/disks/vml.0000000000766d686261313a303a30:2 /vmfs/devices/disks/vml.0000000000766d686261313a303a30:2
Now in the Configuration tab of my ESXi4.0 host, I go to Storage, select the local datastore (datastore1), right click and select Refresh. Now the local VMFS datastore is 1120MB (1.12GB) and uses all remaining disk space.

[UPDATE] In certain occasion, you cannot grow the VMFS disk because it doesn’t exist anymore. In that case you will have to create a VMFS datastore. By default it will take all space available. The command is: vmkfstools -C vmfs3 /vmfs/devices/disks/vml.0000000000766d686261313a303a30:2
Note that you will have to add that new VMFS datastore to your host using the vCenter Client. You will be able to give it a name, for instance ‘datastore1′, and the set the block size (1MB by default).

Growing a datastore from the Service Console in ESX 4.0

实战 X40 升级 SSD(顺便加内存升 Windows 7)

昨天是 SSD 硬盘送货的日子。指定上午9点到12点。本来以为9点就会到的,没想到等啊等,等到11点半,肚子都饿了才来到。好大一个箱子。里面装满的慢慢的泡沫塑料。日本这边的包装真是没得说。硬盘,显卡先捞出来,然后捞了半天才捞到内存条。先上硬盘的图:

PhotoFast G-Monster-1.8"IDEPhotoFast G-Monster-1.8"IDE

注意到这里标的是 3.3V,而不是 3.3V 或 5V ,和台湾标的不一样。我特意写信问了日本的 PhotoFast,回信告诉我就是3.3V的,“如果你用了5V的电源我们可能不会提供保修”。这个问题以后再调查。

PhotoFast G-Monster-1.8"IDE

先打开后盖,把内存条换上去。这个过程很顺利。下面的照片上面是新的 1G 现代条,下面是以前的 512M BUFFALO(看上去也是现代的颗粒)。

现代1G 和 BUFFALO 512M 对比图现代1G 和 BUFFALO 512M 对比图(背面)

装好内存以后,开机验证,没问题。加上板载的256M,变成了1.25G内存。

下面开始搞硬盘。先让新旧2代硬盘来几个合影。

PhotoFast 64G 和 HITACHI 40G的合影(正面)PhotoFast 64G 和 HITACHI 40G的合影(背面)

再来两张单身照。

PhotoFast G-MONSTER 1.8“ 64G IDE

我对 PhotoFast G-MONSTER 1.8“ 64G IDE 的评价:就是山寨嘛……

除了整片有张图片之外,整个就跟软盘似的,连个说明都没有。

外包装盒倒还像是那么回事儿。

闲话少说,开装。我很认真的把原来的硬盘从盒子里拆下来,把这个黑漆漆的家伙装进去,然后开始往X40里面塞。结果塞了半天塞不进去,汗……

warrenwang 说的果然没错:“SSD直接插入X40 即可,不要再把原來HD的機架插上,不然不會WORK。” 把两个硬盘比了一下,针的位置不太一样,大概有一两毫米的差距。遂把盒子拆掉,直接插进去,结果更汗:开机之后硬盘灯常亮,看不到这块盘…… 赶紧拔出来。插进去容易拔出来难,在2个螺丝刀的帮助下终于拔了出来。难道是跳线不对?再次抱怨这个山寨货,也不说条线是怎么跳的。看了看我原配的硬盘上面是直接没有跳线帽的,于是把跳线帽拔掉,再插进去。结果仍然是硬盘灯常亮。肚子咕咕叫了,给PhotoFast 和代理店写信,做饭吃先。

叠叠乐

代理店很快回信了,他们认为很可能是盘是坏的,让我寄给他们看看。我打算收拾东西亲自去他们店里跑一趟,出门之前我又突然想起来,既然它的针有偏差,我摸黑插进去的时候会不会插歪掉呢,抱着试试看的心情,我开始把我的机器大卸八块了:

p1000062

硬盘就插在左下角这个地方,这下插准了。

p1000061

谢天谢地,总算能找到它了。

p1000057

在盖上盖子之前,还有一个惊喜的发现:怪不得我的喇叭最近一直不响,原来是上次那个粗心的维修点给我少插了一个插头!

p1000065

盒盖,上螺丝,丢了一个螺丝钉,丢了就丢了吧。

上电,没反应,汗…… 拆掉键盘,检查了一遍,没看到那个螺丝钉,把飞线的位置调整了一下,重新盖上,上螺丝。谢天谢地,终于启动了。接下来装 Windows 7。

(出差没带光驱,手头能启动的只有这个安装 Windows 7 的移动硬盘了。)

一切都好,就是显卡没有驱动程序。先测试一下硬盘。

CrystalDiskMark,左侧为原装的日立40G。右侧为新的PhotoFast 64G:

newc

HD Tune基准测试:(下面的 XP 风格为原装的日立40G,下同)

newc-1oldc-1

对照,这是一个5400转的3.5吋USB外置硬盘:

usb-1

文件基准测试:

newc-2

oldc-2

随机存取:

newc-3

oldc-3

测试完毕,下面开始搞 Intel 82855 的 Windows 7 驱动,暂时以失败告终,所以只是想简单知道答案的就不用看下去了。我一共试了4个版本:

6.14.10.3722: 这个不记得是从哪儿弄来的,有人说成功了,我这装了没反应。

6.14.10.4497: 这个是从Intel 官方网站下载的最新版本。装了之后看似正常,重启之后安装,最后形成两个带黄感叹号的设备。然后就进入了安装-重启-安装-重启的死循环。强装 915的驱动也无效。

6.14.10.4656:  这个是 水际天成 网友提供的。jerrycrystal 也推荐过这个版本。

然而我装这个之后直接蓝屏。按两位说的办法反复尝试,都是以失败告终。

6.14.10.4764: 这个是从Intel 官方网站下载的915GM最新版本。里面没有855的驱动,直接强装 915。这个是最“成功”的一个:黄感叹号没了,只是画面变成了640X480@16色(不是16位高彩,是16种颜色,我自上大学以来就没见过这样的)。怎么折腾他都坚持这个分辨率。

到这里已经不早了,睡觉。喇叭也响了,硬盘的啪啦啪啦声也彻底没了,听着优雅的轻音乐,欣然入眠。

今天早上开始想办法装 XP 试试看。下载 tm12345789 提供的 U 盘 DOS 启动工具。用我的 SanDisk U3盘成功启动,安装,然后装的XP拒绝启动。我以为是Windows 7 破坏了启动分区,于是再进 Windows 7 的PE,

bootsect /nt52 C: /mbr

修复很成功,但它依然不肯启动。到这时候,已经是今天中午了,我还得靠这机器和家里联系呢,遂装回Windows 7。现在就是这个状态:
干什么都很爽,只是由于没有显卡驱动不能看视频。以前用XP的时候是能顺利看 720P的,现在连DIVX也没法看,一愣一愣的。暂时不能看电影也好,这样才有时间写blog (不然依我的懒劲,您就看不到这一篇了),听音乐看书。

补张图,该盘在Windows 7 里面的评分达到了恐怖的 5.9:

windows7-score