I remember seeing alpha quality defrag software for linux back in ’96 or so, but nothing since. Defragging today on a thin provisioning volume can be dangerous actually, as it can cause the system to consume more space than it previously did depending on where the data is being moved to.

I too look forward to file system level integration, the nice thing about the 3PAR stuff is you can just write zeros to those blocks like the MS sdelete utility, while the API is nice(actually wasn’t aware there was a difference I thought the VXFS stuff just did the zeroing), I think file systems of the future will do automatic scrubbing of deleted data, now that there is hardware that can take advantage of it(I’m sure the concept will spread to other vendors like most things do).

Chris

Nate, we appreciate your patience and support. You shouldn’t have to wait much longer now.

The technology for zeroing out blocks on disk comes from security-oriented file system technology. The utility that Nate references for Windows is called SDelete, for secure delete. This link text is to a Microsoft Technet article on SDelete that gives some information on how it works.

The most efficient way to reclaim storage will be a storage reclamation API, like the one that Symantec and 3PAR developed and is part of our Thin Reclamation for Veritas Storage Foundation – and is in the process of becoming a standard.

A good example for this I think is MySQL, when you optimize/rebuild a table in MySQL(which happens if you add an index, or want to compact the table) it completely re-writes the table out to disk, essentially doubling it’s size on the SAN, when the rebuild is complete the original table is deleted and the new table is renamed to the same name as the old table, From the file system’s view the space is reclaimed but from the SAN’s view it only got worse.

In my early days of 3PAR usage(at another company) I spent a lot of time analyzing and optimizing my usage of thin provisioning, not knowing your data usage patterns can be dangerous if your over provisioning and don’t have much budget. But I figured it out for the most part and today I do use LVM on both linux and windows to logically restrict volumes from getting out of hand, knowing that I can do an online resize pretty easily if they *really* do need that extra space.

My current company deployed an Exanet NAS cluster almost a year ago with our 3PAR, I was told by the local support guy that they were thin provisioning friendly based on his experience though the company’s official line was they didn’t support it, well they weren’t as friendly as I was expecting. I over provisioned up front, not thinking about the implications at the time(tight time constraints etc), the result today is the file system has roughly 60TB in it, and on the raw disks it is using about 112TB. Early on in the process I ran an online conversion going from RAID 5 3+1 to RAID 5 5+1(took a while given the volume of data and activity on the system but there was no application impact), which saved roughly 15TB of raw space, and saved us from having to buy more spindles.

I’m hopeful to reclaim most of this space and get back to at least RAID 5 3+1 in the not too distant future.

I’m pushing to add another 100(x1TB SATA) disks early next year, more for I/O than for space, it will again take some time to re-stripe all of the data, and give us a ton more room for long running snapshots and online backups.

Chris

I don’t like the idea of agents; (a) they have to be maintained and are not popular in a lot of companies (b) they can fail or might not get installed, so wouldn’t reclaim data. I think back to IXFP and Iceberg (STK) of the 90′s which on the mainframe required a started task running somewhere to kick off garbage collection. Not an elegant solution.

Chris

As for you thought the arrays already had it, they already had the hardware, we’ve(as a 3PAR customer) been waiting on the software for a while now, so been unable to
use that technology. Even though it was announced today, it’s not available for use today unfortunately, apparently it will be “soon”, kind of frustrating to me as a customer I mean they semi announced it last year(with the T-class release), then they announce it again this year but it’s still not really available yet.

I read a year or two ago some special tool that NetApp used to reclaim space from NTFS file systems, I think it did something similar though the docs if I recall right mentioned it was very I/O intensive and to only run it during low activity periods. And it was only for NTFS, not any other FS.

MS has a more generic tool apparently that can zero out deleted data in a NTFS file system, I think 3PAR uses that for some of their demonstrations. I’m not aware of anything quite so “canned” that can work on other systems. Those systems you have to create the big files filled with zeros to free up space, no way(that I know of) to only zero out the data that has been marked for re-use by the file system, short of the new VXFS integration (aka Norton File System).

Given the concept seems so simple(in practice it may be complex I don’t know), I would expect other vendors to follow, looking for wider adoption of zeroing out deleted files by file systems automatically as time goes on.

You could have an agent on the host that monitors the filesystem as seen by the host and communicates that information to the storage device.

John F.