Special OS Provisioning and OS Configuration profiles are not required to use a MAC address for provisioning an operating system When you apply a plan that includes the OS Provisioning and OS Configuration profiles, you step through the plan to verify the configuration and provide final information before starting the job. Manual Net Boot. Over-provisioning an SSD - does it still hold? Albeit interesting from someone who would consider over-provisioning in the first place. – sourcejedi Sep 15 '16. Even being able to manually implement Overprovisioning through any partition software, as Echohce mention, I am also unsure if Over-Provisioning is still necessary. I mentioned in the link above that all references I found related to this subject in some websites are from 2010, 2013 and, in the best scenario, 2014.
Many of us fondly remember the traditional “15 squares puzzle.” With just one empty square, shifting pieces to new locations requires a fair amount of effort because there is so little free space. Having the puzzle fifteen-sixteenths full makes for challenging and entertaining gameplay, but it’s certainly not the kind of performance limitation you want in your solid state drive (SSD). Imagine if the same puzzle were only half-full with eight pieces. It could be solved almost instantly. More free space enables faster piece movement and task (game) completion.
SSDs work on a very similar principle. Visualize the NAND flash memory inside of an SSD as a puzzle, except that the amount of free space in a drive is not fixed. Manufacturers utilize various tactics to improve performance, and one of these is to allocate more free space, a process known as over-provisioning.
The minimum amount of over-provisioning for an SSD is set at the factory, but users can allocate more space for better performance. Either way, a moderate understanding of over-provisioning is necessary in order to make better SSD purchasing decisions and to configure drives in the most advantageous way possible for each unique environment and use case.
Background: The Nature of HDD vs. SSD Writes
Here’s another visualization exercise: Imagine having a 3,000-page encyclopedia that has been completely randomized. None of the entries are in order. The only feasible way of finding any given piece of information is through the table of contents, which keeps track of each entry’s location. Without the table of contents, the book becomes essentially unusable.
This model explains how information is generally written to hard disk drives (HDDs). Data gets placed wherever it will fit and often in a way that best assists read/write performance. The physical location of the data is not important because the master file table (or MFT, effectively the drive’s table of contents) keeps track of every chunk of data. Deleting a file from an HDD is unnecessary. Simply erase its entry in the MFT, and from the host’s point of view it’s gone. Only when new data physically overwrites the old is it truly gone, which is why forensic software can often recover “deleted” files from systems. The key point, though, is that the hard drive doesn’t care if there is data in sectors or not. The host only sees sectors in terms of occupied or available for writing.
SSDs work very differently. The fundamental unit of NAND flash memory is typically a 4 kilobyte (4KB) page, and there are usually 128 pages in a block. Writes can happen one page at a time, but only on blank (or erased) pages. Pages cannot be directly overwritten. Rather, they must first be erased. However, erasing a page is complicated by the fact that entire blocks of pages must be erased at one time. When the host wants to rewrite to an address, the SSD actually writes to a different, blank page and then updates the logical block address (LBA) table (much like the MFT of an HDD). Inside the LBA table, the original page is marked as “invalid” and the new page is marked as the current location for the new data.
Of course, SSDs must erase these invalid pages of data at some point, or the usable space on the SSD would eventually fill up. SSDs periodically go through a process called garbagecollection to clear out invalid pages of data. During this process, the SSD controller, or flash controller, that manages NAND flash memory in an SSD, reads all the good pages of a block
(skipping the invalid pages) and writes them to a new erased block. Then the original block is erased, thus preparing it for new data.
Amount of Over-provisioning
All SSDs reserve some amount of space for these extra write operations, as well as for the controller firmware, failed block replacements, and other unique features that vary by SSD controller manufacturer. The minimum reserve is simply the difference between binary and decimal naming conventions. Many people are blissfully unaware that one gigabyte (GB) is precisely 1,000,000,000 bytes, and one gibibyte (GiB) is precisely 2^30 = 1,073,741,824 bytes, or about 7.37% more than a GB. Many people are also blissfully unaware that storage is properly measured in gigabytes, whereas memory is properly measured in gibibytes. Even though SSDs are built from NAND flash memory chips, they are marketed as storage devices, and SSD manufacturers reserve the extra 7.37% of memory space as a provision for background activities such as garbage collection. For example, a 128GB SSD will inherently include 128 * 73,741,824 = 94.4 million bytes of built-in over-provisioning.
A “Billion” Bytes of Storage | A “Billion” Bytes of Memory | |
| Name | Gigabyte (GB) | Gibibyte (GiB) |
| Mathematical Equivalent | 10^9 | 2^30 |
| # of Bytes | 1,000,000,000 | 1,073,741,824 |
So even if an SSD appears to be full, it will still have 7.37% of available space with which to keep functioning and performing writes. Most likely, though, write performance will suffer at this level. (Think in terms of the 15 squares puzzle with just one free square.)
In practice, an SSD’s performance begins to decline after it reaches about 50% full. This is why some manufacturers reduce the amount of capacity available to the user and set it aside as additional over-provisioning. For example, a manufacturer might reserve 28 out of 128GB and market the resulting configuration as a 100GB SSD with 28% over-provisioning. In actuality, this 28% is in addition to the built-in 7.37%, so it’s good to be aware of how vendors toss these terms around. Users should also consider that an SSD in service is rarely completely full. SSDs take advantage of this unused capacity, dynamically using it as additional over-provisioning.
| Marketed OP* | 0% | 7% | 16% | 28% |
| True Physical OP* | 7% | 15% | 25% | 37% |
| SSD Physical Cap | Resulting SSD User Capacity | |||
| 64 | 64 | 60 | 55 | 50 |
| 128 | 128 | 120 | 110 | 100 |
| 256 | 256 | 240 | 220 | 200 |
*Rounded results
Some SSD manufacturers provide software tools to allow for over-provisioning of drives by the user. Actually, even without special software, any user can set aside a portion of the SSD when first setting it up in the system by creating a partition that does not use the drive’s full capacity. This unclaimed space will automatically be used by the controller as dynamic over-provisioning.
There is one obvious drawback to over-provisioning: the more unused capacity one reserves to increase writing speeds, the less capacity there is for storage. With hard drives, the somewhat similar practice of short stroking, which confines reads and writes to the fastest outer tracks of the drive platters, is less penalizing because the cost per gigabyte is lower with magnetic media. At $1 to $3 per gigabyte for enterprise-class SSDs, the decision to give up 25% or more of a drive’s capacity becomes more difficult. There had better be some real benefit when giving up those expensive bytes!
In fact, there are plenty of benefits—under the right circumstances.
Over-provisioning, Random vs. Sequential Writes, and Entropy
The above graph represents testing conducted by Seagate using an SSD based on Toshiba 24nm MLC NAND flash memory and a Seagate® SandForce® SF-2281 Flash Controller with DuraWrite™ data reduction technology. There is more to consider here than may be initially apparent. Let’s explain several key elements about sequential vs. random data patterns.
When an SSD arrives new from the factory, writes will gradually fill the drive in a progressive, linear pattern until the addressable storage space has been entirely written. Essentially, this reflects an ideal sequential writing condition. No garbage collection has swung into play at this point, and the little pockets of invalid data caused by deletions has yet to impact performance because there has been no need to write to those pockets with new data.
However, once garbage collection begins, the method by which the data is written – sequentially vs. randomly – begins to affect the performance. Sequentially written data from the host will constantly fill whole flash memory blocks, and when the data is replaced it generally replaces the entire block of pages. Then during garbage collection all pages in that block are invalid and nothing needs to be moved to another block. This is the fastest possible garbage collection – i.e., no garbage to collect. The horizontal lines in the “sequential write graph” show how sequential write performance stays relatively constant regardless of how much over-provisioning is applied.
What does affect performance is the entropy of the data, provided the SSD is using a flash controller that supports a data reduction technology, such as a SandForce Flash Controller. The entropy of data is the measure of the randomness of that data, not to be confused with the data being written randomly vs. sequentially. For example, a completely encrypted data file, an MPEG movie, or a compressed ZIP file will have the highest entropy, while database, executable, and other file types will have lower entropy. As the entropy of the data decreases, the write reduction-capable flash controller will take advantage of the lower entropy and provide higher performance. However, the performance remains constant with a given over-provisioning level when written sequentially.
In contrast, when data is written randomly to the SSD, the data that is marked invalid is scattered throughout the entire SSD creating many small holes in every block. Then when garbage collection acts on a block containing randomly written data, more data must be moved to new blocks before the block can be erased. The red line of the Random Writes graph (above) shows how most SSDs would operate. Note that in this case, as the amount of over-provisioning increases, the gain in performance is quite significant. Just moving from 0% over-provisioning (OP) to 7% OP improves performance by nearly 30%. With flash controllers that use a data reduction technology, the performance gains are not as significant, but the performance is already significantly higher for any given level of OP.
Is Over Provisioning Necessary
Over-provisioning and Write Amplification
As mentioned earlier, SSD writes generally involve writing data more than once: initially when saving the data the first time and later when moving valid data during multiple garbage collection cycles. As a result, it’s common for more data to be written to an SSD’s flash memory than was originally issued by the host system. This disparity is known as write amplification, and it is generally expressed as a multiple. For instance, if 2MB of data is written to flash while only 1MB was issued from the host, this would indicate a write amplification of 2.0. Obviously, write amplification is undesirable because it means that more data is being written to the media, increasing wear and negatively impacting performance by consuming precious bandwidth to the flash memory. Several factors can contribute to write amplification, chief among these being the percentage of data written randomly vs. sequentially.
Surprisingly, it is also possible to write less data to flash than was issued by the host. (This would be expressed as a write amplification of, say, 0.5 or 0.7.) DuraWrite data reduction technology is probably today’s best-known method of accomplishing this through real-time data manipulation. Only SSDs with a similar data reduction technology can create a write amplification of less than one. As the entropy of the data from the host goes down, DuraWrite technology results in less and less data being written to the flash memory, leaving more space for over-provisioning. Without a similar data reduction technology, an SSD would be stuck with higher write amplification.
Note that additional over-provisioning and a data reduction technique such as DuraWrite technology can achieve similar write amplification results with different trade-offs. Benchmarking reveals that only drives with DuraWrite data reduction technology or something similar are able to take advantage of entropy-related write amplification reduction and the resulting performance improvements. Conventional SSDs without a similar technology are limited to the write amplification from a given over-provisioning level. As an example, a conventional SSD with 28% over-provisioning will exhibit the same write amplification (3.0) as an SSD with DuraWrite technology writing a 75% entropy stream with 0% over-provisioning, all other factors being equal. In other words, this scenario shows how an SSD equipped with DuraWrite technology could display the same level of write amplification as a standard SSD while reclaiming 28% of the storage capacity.
The Next Efficiency Level
An SSD does not natively know which blocks of data are invalid and available for replacing with new data. Only when the operating system (OS) tries to store new data in a previously used location does the SSD know that a particular location contains invalid data. All free space not consumed by the user becomes available to hold whatever the SSD believes is valid data. This is why the storage industry created the TRIM command. TRIM enables the OS to alert the SSD about pages that now contain unneeded data so they can be tagged as invalid. When this is done, the pages do not need to be copied during garbage collection and wear leveling. This reduces write amplification and improves performance. The graphic below shows how much of a difference TRIM can make in allowing more capacity to be available for over-provisioning.
TRIM is yet another method that vendors can employ to boost over-provisioning, thereby increasing performance and drive longevity. It shows a more preferable way to reclaim SSD capacity for acceleration compared to forcing drives to permanently surrender large swaths of their capacity. Using TRIM with DuraWrite technology, or similar combinations of complementary technologies, can yield even more impressive results.
Conclusion
Buyers should take a close look at their workloads, assess the typical entropy levels of their data sets, and consider which SSD technologies will provide the greatest benefits for their invested dollars. By reducing write amplification and employing technologies that make SSD operation ever more efficient, buyers will not only get more storage for each dollar, but that storage will perform faster and last longer than other options could possibly provide.
This is a guest post by Kent Smith, senior director of marketing for LSI’s Flash Components Division,
overseeing all outbound marketing and performance analysis for the company.
It may sound crazy, but hard disk drives do not actually have a delete command. Now we all know HDDs have a fixed capacity, so over time the older data must somehow get removed, right? Actually it is not removed, but overwritten. The operating system uses a reference table to track the locations (addresses) of all data on the HDD. This table tells the OS which spots on the HDD are used and which are free. When the OS or a user deletes a file from the system, the OS simply marks the corresponding spot in the table as free, making it available to store new data. The HDD is told nothing about this change, and it does not need to know since it would not do anything with that information. When the OS is ready to store new data in that location, it just sends the data to the HDD and tells it to write to that spot, directly overwriting the prior data. It is simple and efficient, and no delete command is required.
Manual Over Provisioning Mac Os
However, with the advent of NAND flash-based solid state drives (SSDs) a new problem emerged. In a previous article I wrote, Gassing up your SSD, I explained how NAND flash memory pages cannot be directly overwritten with new data, but must first be erased at the block level through a process called garbage collection.
The SSD uses non-user space in the flash memory (over provisioning or OP) to improve performance and longevity of the SSD. In addition, any user space not consumed by the user becomes what we call dynamic over provisioning – dynamic because it changes as the amount of stored data changes.
Garbage collection starts when a flash block is full of data, usually a mix of valid (good) and invalid (older, replaced) data. The invalid data must be tossed out to make room for new data, so the Flash Storage Processor (FSP) copies the valid data of a flash block to a previously erased block, and skips copying the invalid data of that block. The final step is to erase the original whole block, preparing it for new data to be written.
When less data is stored by the user, the amount of dynamic OP increases, further improving performance and endurance. The problem I alluded to earlier is caused by the lack of a delete command. Without a delete command, every SSD will eventually fill up with data, both valid and invalid, eliminating any dynamic OP. The result would be the lowest possible performance at that factory OP level. So unlike HDDs, SSDs need to know what data is invalid in order to provide optimum performance and endurance.
Keeping your SSD TRIM
A number of years ago, the storage industry got together and developed a solution between the OS and the SSD by creating a new SATA command called TRIM. It is not a command that forces the SSD to immediately erase data like some people believe. Actually the TRIM command can be thought of as a message from the OS about what previously used addresses on the SSD are no longer holding valid data. The SSD takes those addresses and updates its own internal map of its flash memory to mark those locations as invalid. With this information, the SSD no longer moves that invalid data during the GC process, eliminating wasted time rewriting invalid data to new flash pages. It also reduces the number of write cycles on the flash, increasing the SSD’s endurance. Another benefit of the TRIM command is that more space is available for dynamic OP.
Army Provisioning Manual
Today, most current operating systems and SSDs support TRIM, and all SandForce-based SSDs have always supported TRIM. Note that most RAID environments do not support TRIM, although some RAID 0 configurations have claimed to support it.
Can data reduction technology substitute TRIM and drain the invalid data away?
Imagine a bathtub full of water and asking someone to empty the tub while you turn your back for a moment. When you look again and see the water is gone, do you just assume someone pulled the drain plug? I think most people would, but what about the other methods of removing the water like with a siphon, or using buckets to bail out the water? In a typical bathroom you are not likely to see these other methods used, but that does not mean they do not exist. The point is that just because you see a certain result does not necessarily mean the obvious solution was used.
I see a lot of confusion in forum posts from SandForce-based SSD users and reviewers over how LSI's DuraWrite data reduction and advanced garbage collection technology relates to the SATA TRIM command. As before, without the TRIM command the SSD assumes all of the user capacity is valid data, and creating more free space through over provisioning or using less of the total capacity enables the SSD to operate more efficiently by reducing the write amplification, which leads to increased performance and flash memory endurance. So without TRIM the SSD operates at its lowest level of efficiency for a particular level of over provisioning.
Will you drown in invalid data without TRIM?
TRIM is a way to increase the free space on an SSD – what we call “dynamic over provisioning” – and DuraWrite technology is another method to increase the free space. Since DuraWrite technology is dependent upon the entropy (randomness) of the data, some users will get more free space than others depending on what data they store. Since the technology works on the basis of the aggregate of all data stored, boot SSDs with operating systems can still achieve some level of dynamic over provisioning even when all other files are at the highest entropy, e.g., encrypted or compressed files.
With an older operating system or in an environment that does not support TRIM (most RAID configurations), DuraWrite technology can provide enough free space to offer the same benefits as having TRIM fully operational. In cases where both TRIM and DuraWrite technology are operating, the combined result may not be as noticeable as when they’re working independently since there are diminishing returns when the free space grows to greater than half of the SSD storage capacity.
So the next time you fill your bathtub, think about all the ways you can get the water out of the tub without using the drain. That will help you remember that both TRIM and DuraWrite technology can improve SSD performance using different approaches to the same problem. If that analogy does not work for you, consider the different ways to produce a furless feline, and think about what opening graphic image I might have used for a more jolting effect. I presented on this topic in detail at the Flash Memory Summit in 2011. You can read it in full here.
Kent Smithis senior director of marketing for LSI’s Flash Components Division, overseeing all outbound marketing and performance analysis.Prior to LSI, Kent was senior director of corporate marketing at SandForce, acquired by LSI in 2012. His more than 25 years of marketing and management experience in computer storage and high-technology includes senior management positions at companies including Adaptec, Acer, Polycom, Quantum and SiliconStor.
Republished with permission.
