NVML 1.0, DAX and the NVM Programming Model
posted on April 12, 2016 by Amit Golander
As a veteran researcher, I’m always interested in brand new approaches for doing things. One great example is the new NVM programming model proposed by SNIA (NPM). An approach that took a big step forward this week, as version 1.0 of the associated NVM library (NVML) was released.
For the few people who understood this first paragraph – congratulation on the major milestone.
If you haven’t really understood it however – don’t go away. I’ll start from the beginning…
New families of storage medias and device types have been introduced and announced recently. These persistent memory (PM) technologies have amazing hardware properties, and there seems to be a growing consensus that they are going to revolutionize Compute as we know it. Here are two example articles about it from last week:
- “Memory and storage boundary changes” (The Register, Apr 4th)
- “The non-volatile memory revolution: Bigger than SSDs” (ZDNet, Apr 8th)
The billion dollar question then is which software approach to choose in order to leverage the unique PM hardware.
Software Approach Survey
There are three main software approaches for using PM. This is not going to be a Right Vs. Wrong discussion, as there are tradeoffs. In sake of clarity, I enumerated the approaches: I, II and III. This should help us stay at the technical, not the marketing, level.
Figure 1 illustrates that the key parameter for classifying software approaches is the amount of re-used Vs. re-written software. Approach I is tuned for maximal software reuse. Approach II rewrites system software, but leaves the application untouched. Approach III rewrites all layers and moves the application to a new programming model.
Approaches I and II have the benefit of supporting applications “out of the box” with no modification. But Approach II requires additional development work from the system software vendor. Work that has to be justified in extra performance levels. Figure 2 shows that the commonly adopted FIO benchmark was able to achieve 2-3 orders of magnitude better throughput AND latency just by switching the software approach. Moving from Approach I (ZFS or XFS on NVDIMMs) to Approach II (Plexistor SDM) has brought tremendous value.
Further comparing the three approaches, performance, software reuse and other technical properties are summarized in Figure 3. Most properties are self-explanatory, but two are worth elaborating on: “Data services model” and “$/GB”. IT administrators use data services to comply with organizational requirements (e.g. use the storage array snapshot to create a backup). Approaches I and II follow the same model, while Approach III moves a lot of that responsibility to the application developer.
The second property “$/GB” is mainly a function of supporting multiple tiers and being able to automatically move data between them according to usage patterns. This is important because PM cost per capacity unit ($/GB) is high. PM cost is expected to drop with volume and time, but it will not catch up with SSD and HDD in coming years. Auto tiering is a vendor dependent feature. Sticking to the examples used in Figure 2: ZFS supports it, XFS and XFS-DAX do not and Plexistor SDM does.
Now that we’ve established a clean technical background, let’s apply it back to the market.
Mapping Industry Efforts
Figure 4 maps the theoretical software approaches mentioned above, to actual implementations and players. It is not meant as a comprehensive industry-wide survey of all software solutions, just the leading ones. Also, with PM being a hot topic, I’m sure many other solutions will join down the road.
Incumbent storage vendors, such as EMC, Netapp and Nutanix have based their rich code base on the block layer. This means that their only viable option to use PM as a storage tier is to wrap it up with an abstraction layer that will present it as a block device. i.e. Approach I.
In contrast, SDM or file system vendors that deeply believed in PM for the past couple of years, such as Microsoft and Plexistor, have built file-level solutions that use PM as a storage tier. Both companies added support for DAX, which enables zero copy of memory mapped files (MMF), and can support approaches II and III.
As a side note, it will be interesting to compare the performance and features provided by Microsoft ReFS and Plexistor SDM. But, it will have to wait until Windows Server 2016 is released. Also, as interesting as the comparison may be for us techies, I doubt if many Windows users will move to Linux because of it. People do not switch religions overnight.
The DAX support in Microsoft ReFS and Plexistor SDM, as well as the ones being built for EXT4-DAX and XFS-DAX are the foundation for approach III. Intel and HPE have been leading the standardization of this approach through the SNIA NVM Programming Model documents and the complementary NVML library and tests. The same library that just reached its major milestone – the stable 1.0 release.
We at Plexistor are continuously running the 1,779 NVML tests on our SDM software. We’re doing that for a year now, because we believe that customers should be able to run traditional and NVM-aware applications side by side. Customers should be able to transparently share data between these applications and enforce the same policies on that data, regardless of how it was created.
If you’re developing an application on top of NVML – we invite you to run it Plexistor SDM, which can be downloaded at http://www.plexistor.com/download/