13 replies to this topic

[fieldhouse]

The story:
I have a RAID volume on my main WHS2011 box that was getting pretty full - five 2TB drives in a RAID5 array with a sixth drive as a hot spare. I was down to around 300GB free space and debated whether I should move some stuff off or expand the array. I'm using a RockeRaid 2680 and have used OCE several times before to add drives as I moved chunks of stuff off of my EX490. So I thought, why not? I'll add another drive. I still have two unused ports on the 2680, might as well.

I had another 2TB drive that I had pulled out of the TR5M that used to be plugged into the EX490 but I wanted to be sure it was clean so I fired up Seatools for DOS on an old beater box and ran diagnostics and performed a full erase on the disk just to be on the safe side. Then I popped the drive into an open bay in the Icy Dock MB974SP-B (gotta love those catchy names Icy Dock comes up with). I started the OCE process through the GUI and after seeing it settle around 16 hours left to complete the expansion I left it to its own devices and hit the hay.

The next day I didn't remember to check the expansion progress before I took off for work. So when I got home I was surprised to find that, not only was the process not complete but that I couldn't connect to the 2680 controller through the GUI. WHS was ticked off because one of its volumes wasn't where it should be (obviously) but was still up and running since the OS is on a standalone drive using the controller on the motherboard. I could still access the two arrays on the other RAID controller in the system - a RocketRaid 622 - but I couldn't access the 622 through the GUI either. So, thinking I was taking the safe approach, I left things as is overnight to make sure the rebuild had time to finish on the off chance the 2680 was still working on the expansion.

Unfortunately, things weren't any better the next day and I decided I should go ahead and shut down / restart. When I brought things back up the 2680 claimed it was still in the process of rebuilding the array. I had a pretty good idea where things were going at this point as Windows after the restart claimed I had an 8GB partition of type RAW with a total disk size of 10TB. I went ahead and let the HighPoint finish doing what it thought needed to be done and noticed that the system seemed glitchy - becoming unresponsive at times to keyboard or mouse. After the rebuild of my 10 terrabytes of nothing finished, I checked the drives. The Seagate green drive I had added was in a funky state. The 2680 claimed the status was "normal" but when I tried to view SMART info through the RAID Management Console it returned an error.

E 5/9/2012 7:31:33 PM An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.

Gee, thanks. How helpful.
For the record, of the 28 drives currently on my spreadsheet (500 SATA or larger) only two have failed and both were 2TB Seagate Barracuda LP's. In fact, now that I think about it, this may be the warranty replacement for the other failed Barracuda LP. hmmm.

Anyway, the only thing I thought I could do that made any sense was to pull the "new" drive and have the controller use the existing hot spare drive to rebuild the array. After waiting another eternity for that to complete I have been left with a very large, and apparently unformatted, RAID volume. Which brings me to where I am now.

For context, this is all running virtual on a Server 2008 R2 Hyper-v host, Core i5-750 on a Gigabyte P55-USB3 motherboard with 16 GB RAM. The RAID volume was being used as a pass-through physical disk inside the WHS 2011 VM just for data. The VM OSes are hosted on a RAID5 array on the RocketRaid 622 along with another RAID5 array that I have set up as an iSCSI target (long story). The WHS 2011 is (was) the main purpose of the hardware but I was running it virtual so I could bring up other OSes when needed and not install a bunch of crap on the WHS OS. The system itself has been stable and behaving well for quite a while (since WHS 2011 was beta) and I generally have had no complaints about the hardware. I'll also mention that all of our digital photos and documents are safe on another drive so what I'm missing is primarily my movies, tv shows, kids videos, anime, ebooks. It sucks that it's not on the drives but it's not the end of the world (we still have half a year before December 21st rolls around.)

Is there anything I could have (should have) done differently? and, any suggestions on what to do now? i'm tempted to call the 7.5 TB of stuff gone and take the hardware out in the woods for target practise with an assault rifle...

[Greg Welch]

Is there anything I could have (should have) done differently? and, any suggestions on what to do now? i'm tempted to call the 7.5 TB of stuff gone and take the hardware out in the woods for target practise with an assault rifle...

I would pull the green drive and see if its possible to a consistency check in the raid software

I would try and do without Green,LP,and mixed mfg in raid arrays, try and use them for array backups. much better suited for this task.

[fieldhouse]

Thanks for the feedback. What really bugs me is that I was just in the process of planning out a move of the drives/arrays to a Norco 4220 case and doing away with the TR5M eSATA enclosures because I was considering them to be the "weak link" in my storage. I guess on the positive side, I don't need to worry about the data on the large array when I move the drives from their current locations to the new chassis.

I would pull the green drive and see if its possible to a consistency check in the raid software

The evil LP drive is out of the array. The rebuild after its removal supposedly completed successfully but on the off chance that it will help I've kicked off a "verify" of the array as you recommended. It ticks me off that SeaTools claims the drive in question is still good even though it very obviously is not. Placing it back in the EX490 causes the 490 to pause / hang, similar to the problems it appears the 2680 had with the drive. It's disappointing that the drive manufacturers are producing diagnostic tools that won't detect severe issues with the basic functionality of their products.
It is equally disappointing that HighPoint would lack whatever logic is needed to detect a non-responsive drive and work around the suspect drive. But I guess you get what you pay for and the 2680 definitely isn't the Cadillac or Mercedes of the RAID controller market.

I would try and do without Green,LP,and mixed mfg in raid arrays, try and use them for array backups. much better suited for this task.

Hindsight is always 20/20 and I would have preferred not mixing drive manufacturers. I am grouping relatively similar disks in my arrays - 7200 RPM 1.5 TB drives ini a different RAID array from the Green / LP drives but I'm trying to work with what I have. My philosophy with my v1 rig was to purchase as many different drives as possible to minimize multiple failures due to common issues across batches of drives. This worked really well with DE in v1 but it obviously doesn't lend itself well to a hardware RAID.
I've been burned in the past with arrays built with the same model/manufacturer date where a common flaw caused a cascading failure of drives and the added stress of rebuilding the array as each drive failed eventually caused catastrophic loss of the entire array so there are risks to any single volume RAID solution.

At this point I'm planning on moving back to a solely mirrored model since that allows for recovery of data even in the event of multiple failures as opposed to this whole array corruption BS. This should also have the added benefit of being able to spin drives down when they aren't in use - saving power costs.

WHS 2011 has been a huge disappointment from a reliability perspective. On v1 with DE I only ever experienced data loss once early on (when it was still beta) and that was due to some logic problem in the algorithm MS was using to rebalance files during disk removal from the drive pool.

Given how fragile RAID volumes apparently are when performing a rebuild or RAID level migration / online expansion, if I do use RAID in future builds I will stick with fixed-size volumes and take the longer route of copying/verifying data from one RAID volume to another when space becomes an issue. It takes a lot longer and requires more disks but generally should eliminate the possibility of waking up to a massive whole-volume corruption and complete data loss.

Does anyone have a suggestion for a JBOD controller to use with a SAS expander card? I was planning on using an Intel RES2SV240 with the Norco 4220 chassis but I'm doubting the RocketRaid 2680 would be up to the task considering how it's been handling five or six drives.

[Greg Welch]

First let me say ,you have a gift for writing

I'm on my phone so I will reply more later,

[fieldhouse]

lol. does that make me "special"?

thanks again. Not in any hurry (obviously, since this drive's been out of commission for several weeks now).

[ikon]

I do prefer somewhat higher end RAID cards: LSI, 3ware (now owned by LSI), Mylex. That said, many members are using RR cards very successfully - pcdoc in particular. The RR 2720 seems to be very popular. It's unfortunate that the higher end cards can be a bit expensive.

[Technogod]

Ditch RAID and use StableBit DrivePool. Even if you have a problem with DrivePool you still won't loose your data. All data is in NTFS format and can be read on any system. I've been using DrivePool since early beta and haven't lost any data. I've found multiple bugs that were fixed almost immediately by the developer. It's a very impressive piece of sftware. I also use StableBit Scanner to scan my drives. I think you would also have better luck with SeaTools checking your drives. You'll now be able to run it on the drives without having to remove them from the RAID ARRAY.

[HyperVG]

Have you thought about one or several Nas boxes? I have Three Dlink 323 mapped to my Hyper-V server. I have them set up for miroring. I am in the planing process to build a 4bay larger raid 5 aray but that is a huge expence.

Its just some thoughts and it is easyer to manage small amounts of data than a large amount.

http://www.qnap.com/...88&t=695&n=3888

[fieldhouse]

I do prefer somewhat higher end RAID cards: LSI, 3ware (now owned by LSI),...The RR 2720 seems to be very popular. It's unfortunate that the higher end cards can be a bit expensive.

I looked at the 2720 a bit this morning. I think if I'm going to shell out more money for a new controller it would be wise to pick up a full hardware RAID controller with onboard cache, etc.

Ditch RAID and use StableBit DrivePool. Even if you have a problem with DrivePool you still won't loose your data. All data is in NTFS format and can be read on any system. I've been using DrivePool since early beta and haven't lost any data.

I'm really tempted to head this direction now. The downside is that it uses a lot more disk capacity than a RAID5 array. If HD prices were lower I'd grab StabeBit in a heartbeat but with current prices it might actually end up costing less to get a LSI or even an Adaptec / Areca controller. Still, given the complete lack of recoverability with RAID5 this may be the best option.

I also use StableBit Scanner to scan my drives. I think you would also have better luck with SeaTools checking your drives. You'll now be able to run it on the drives without having to remove them from the RAID ARRAY.

This is actually a really attractive feature of the software solutions and something I miss from my days running v1.

Have you thought about one or several Nas boxes?

I have a couple smaller NAS boxes that I've been playing with (PogoPlug, Zyxel NSA210 w/ FFP, and recently a Synology DS-212j). From my experiences with them so far you end up paying more for the NAS than you'd pay for equivalent components in a PC. You do have the advantage of possibly lower power utilization depending on your setup. On the other hand, you're going to be using a lot more network bandwidth moving files to and fro. The QNAP looks like a nice little box and might work well with an external USB3 RAID array like the CineRAID CR-H458. But I'm still leaning towards some form of mirroring where I can pull the data off the drives even if half of my disks went up in smoke.

Has anyone messed around with Storage Spaces on Server 2012 much? I wonder how that compares with StableBit as far as the flexibility of configuration. I've been messing around with some different linux distros recently and have been getting spoiled with the ability to quickly throw together a directory structure made up of shares spread across multiple systems. It does't completely solve the drive pooling problem but it sure makes finding things easier. I've used DFS in the past but it seems like MS has gone out of their way over the years to make it more difficult to implement. Is Storage Spaces limited to local drives? Or can it include network locations as well?

[pcdoc]

Not to sound defensive but using like drives has always been deemed as a mandatory step in any raid setup. Using different drives on "any" raid controller can cause issues no matter the price, though some are more forgiving than others. Both the 2680 and the 2720 will do fine if a drive goes bad, will survive a reboot numerous time during a rebuild, and just about anything related to the card itself. That said, it did not obviously handle your green drive very well. In the end, is it really the cards fault if we deviate from MFG recommendations... Just my two cents.

[fieldhouse]

Not to sound defensive but using like drives has always been deemed as a mandatory step in any raid setup.

Apologies if something I said ruffled your feathers. That wasn't my intent. And I agree. This wasn't the most "kosher" RAID array. But it had been working fine up until recently.
My whole reason for posting was to solicit input and suggestions on data recovery (if there are any, which there haven't been so far) and failing that, some suggestions for planning around future failure (which have been plentiful and appreciated).

Using different drives on "any" raid controller can cause issues no matter the price, though some are more forgiving than others.

Absolutely no argument here. And the 2680 had been very tolerant of mixed drives up to that point. In fact, I don't believe that the results of this last array expansion would have been any different if all of the other drives had been identical. I'm not sure what exactly is wrong with this particular drive. It isn't "bad" in a way that Seagate or HighPoint recognize but it sure doesn't play well with others.

Both the 2680 and the 2720 will do fine if a drive goes bad, will survive a reboot numerous time during a rebuild, and just about anything related to the card itself. That said, it did not obviously handle your green drive very well. In the end, is it really the cards fault if we deviate from MFG recommendations... Just my two cents.

All the more reason why the results in this particular case are so puzzling.
Maybe I should have been a bit clearer on what drives are in the array that failed since there's been at least a couple comments around the fact that the drive that went bad was a green drive. All of the drives in the array were low-power drives.
Prior to expansion the array had been functioning for 6+ months of heavy utilization with the following drives:
Disk 1:WDC WD20EARS-00MVWB0, Controller 1, Channel 1 Port 1
Disk 2:ST32000542AS, Controller 1, Channel 1 Port 2
Disk 3:ST2000DL003-9VT166, Controller 1, Channel 1 Port 3
Disk 4:ST2000DL003-9VT166, Controller 1, Channel 2 Port 1
Disk 5:Hitachi HDS5C3020ALA632, Controller 1, Channel 2 Port 2

The last thing I would expect is for the array to fail by adding another ST32000542AS.

This is the full log (sans the user login messages)

I  5/20/2012 4:23:30 AM    Array '2TB_RAID5' verifying completed.  
I  5/19/2012 9:45:54 AM    Array '2TB_RAID5' verifying started.  
I  5/11/2012 2:47:59 PM    Array '2TB_RAID5' rebuilding completed.  
I  5/10/2012 7:11:54 PM    Array '2TB_RAID5' rebuilding started.  
W  5/10/2012 7:11:54 PM    Array '2TB_RAID5' rebuilding aborted.  
I  5/10/2012 7:11:54 PM    Array '2TB_RAID5' rebuilding started.  
W  5/10/2012 7:11:54 PM    Spare disk 'ST32000542AS-5XW18WW3' at Controller1-Channel4 took over.(Array: 2TB_RAID5)  
E  5/10/2012 8:14:51 AM    An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
E  5/10/2012 8:11:52 AM    An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
E  5/10/2012 1:49:46 AM    An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
E  5/9/2012 7:31:33 PM    An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
I  5/7/2012 4:30:13 PM    Array '2TB_RAID5' expanding/migrating completed.  
I  5/4/2012 8:49:11 PM    Array '2TB_RAID5' expanding/migrating started.  
I  5/4/2012 8:20:42 PM    Array '2TB_RAID5' expanding/migrating paused.  
I  5/4/2012 8:08:04 PM    Array '2TB_RAID5' expanding/migrating started.  
I  5/2/2012 10:48:51 PM    Array '2TB_RAID5' expanding/migrating started.  
I  5/2/2012 10:48:50 PM    OCE/ORLM destination RAID 5 Array '2TB_RAID5' has been created successfully (Disk 1:WDC WD20EARS-00MVWB0-WD-WMAZ20276744, Controller 1, Channel 1; Disk 2:ST32000542AS-9XW0ADE5, Controller 1, Channel 2; Disk 3:ST32000542AS-5XW0W62M, Controller 1, Channel 6; Disk 4:ST2000DL003-9VT166-5YD2YGJC, Controller 1, Channel 7; Disk 5:ST2000DL003-9VT166-5YD2R66D, Controller 1, Channel 8; Disk 6:Hitachi HDS5C3020ALA632-ML0220F30A54XD, Controller 1, Channel 5).  
E  5/2/2012 10:46:42 PM    Device 'ST32000542AS-9XW0ADE5' (Controller 1, Channel 2) has been initialized.  
W  5/2/2012 10:00:51 PM    Plugging device detected.('ST32000542AS-9XW0ADE5' at Controller1-Channel2)

[no-control]

Sorry no help on the recovery front. I agree with Doc's comment about mixing drives. Given the size of the array and the nature of its construction (mixed), and in the absence of an alternative, I would have opted for a RAID10 for a more robust solution. Or perhaps a smaller arrays to compartmentalize the loss and potential issues. The RAID5 with spare looks good on paper but given the matrix of drives, potential issues would increase exponentially. Thus threatening the stability of the array. It is possible that a higher end card with cache might have been more tolerant of a problem during expansion, but this point may be moot if it wasn't tolerant of the mix of drives.

While it blows you lost data, its still good info thank you for sharing.

[Greg Welch]

Sorry no help on the recovery front. I agree with Doc's comment about mixing drives. Given the size of the array and the nature of its construction (mixed), and in the absence of an alternative, I would have opted for a RAID10 for a more robust solution. Or perhaps a smaller arrays to compartmentalize the loss and potential issues. The RAID5 with spare looks good on paper but given the matrix of drives, potential issues would increase exponentially. Thus threatening the stability of the array. It is possible that a higher end card with cache might have been more tolerant of a problem during expansion, but this point may be moot if it wasn't tolerant of the mix of drives.

While it blows you lost data, its still good info thank you for sharing.

I have moved to an 8 drive Raid 10 in pairs, and I'm pretty happy with the results.

[pcdoc]

Apologies if something I said ruffled your feathers. That wasn't my intent. And I agree. This wasn't the most "kosher" RAID array. But it had been working fine up until recently.
My whole reason for posting was to solicit input and suggestions on data recovery (if there are any, which there haven't been so far) and failing that, some suggestions for planning around future failure (which have been plentiful and appreciated).
Absolutely no argument here. And the 2680 had been very tolerant of mixed drives up to that point. In fact, I don't believe that the results of this last array expansion would have been any different if all of the other drives had been identical. I'm not sure what exactly is wrong with this particular drive. It isn't "bad" in a way that Seagate or HighPoint recognize but it sure doesn't play well with others.
All the more reason why the results in this particular case are so puzzling.
Maybe I should have been a bit clearer on what drives are in the array that failed since there's been at least a couple comments around the fact that the drive that went bad was a green drive. All of the drives in the array were low-power drives.
Prior to expansion the array had been functioning for 6+ months of heavy utilization with the following drives:
Disk 1:WDC WD20EARS-00MVWB0, Controller 1, Channel 1 Port 1
Disk 2:ST32000542AS, Controller 1, Channel 1 Port 2
Disk 3:ST2000DL003-9VT166, Controller 1, Channel 1 Port 3
Disk 4:ST2000DL003-9VT166, Controller 1, Channel 2 Port 1
Disk 5:Hitachi HDS5C3020ALA632, Controller 1, Channel 2 Port 2

The last thing I would expect is for the array to fail by adding another ST32000542AS.

This is the full log (sans the user login messages)

I  5/20/2012 4:23:30 AM	Array '2TB_RAID5' verifying completed.  
I  5/19/2012 9:45:54 AM	Array '2TB_RAID5' verifying started.  
I  5/11/2012 2:47:59 PM	Array '2TB_RAID5' rebuilding completed.  
I  5/10/2012 7:11:54 PM	Array '2TB_RAID5' rebuilding started.  
W  5/10/2012 7:11:54 PM	Array '2TB_RAID5' rebuilding aborted.  
I  5/10/2012 7:11:54 PM	Array '2TB_RAID5' rebuilding started.  
W  5/10/2012 7:11:54 PM	Spare disk 'ST32000542AS-5XW18WW3' at Controller1-Channel4 took over.(Array: 2TB_RAID5)  
E  5/10/2012 8:14:51 AM	An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
E  5/10/2012 8:11:52 AM	An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
E  5/10/2012 1:49:46 AM	An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
E  5/9/2012 7:31:33 PM	An error occured on the disk at 'ST32000542AS-9XW0ADE5' at Controller1-Channel2.  
I  5/7/2012 4:30:13 PM	Array '2TB_RAID5' expanding/migrating completed.  
I  5/4/2012 8:49:11 PM	Array '2TB_RAID5' expanding/migrating started.  
I  5/4/2012 8:20:42 PM	Array '2TB_RAID5' expanding/migrating paused.  
I  5/4/2012 8:08:04 PM	Array '2TB_RAID5' expanding/migrating started.  
I  5/2/2012 10:48:51 PM	Array '2TB_RAID5' expanding/migrating started.  
I  5/2/2012 10:48:50 PM	OCE/ORLM destination RAID 5 Array '2TB_RAID5' has been created successfully (Disk 1:WDC WD20EARS-00MVWB0-WD-WMAZ20276744, Controller 1, Channel 1; Disk 2:ST32000542AS-9XW0ADE5, Controller 1, Channel 2; Disk 3:ST32000542AS-5XW0W62M, Controller 1, Channel 6; Disk 4:ST2000DL003-9VT166-5YD2YGJC, Controller 1, Channel 7; Disk 5:ST2000DL003-9VT166-5YD2R66D, Controller 1, Channel 8; Disk 6:Hitachi HDS5C3020ALA632-ML0220F30A54XD, Controller 1, Channel 5).  
E  5/2/2012 10:46:42 PM	Device 'ST32000542AS-9XW0ADE5' (Controller 1, Channel 2) has been initialized.  
W  5/2/2012 10:00:51 PM	Plugging device detected.('ST32000542AS-9XW0ADE5' at Controller1-Channel2)

No ruffled feathers just a point to all reading these posts. Several have played with fire and mixed and matched and have been successful (lucky). If done correctly RAID is far more reliable and faster than many solutions out there assuming you heed the warnings. In addition, RAID by itself is certainly not a backup solution so you still needs a plan "B". I point these things out as many have asked or tried the same things and I want to bring it as to caution anyone else from trying this out if they data is valuable to them. Thanks for sharing all your trials so hopefully if will help someone else.