It stands out, because it didn't sell. Which is weird because there were some pretty big pros about using them. The latency for updating 1 byte was crazy good. Some databases or journals for something like zfs really benefited from this.
Intel did a spectacularly poor job with the ecosystem around the memory cells. They made two plays, and both were flops.
1. “Optane” in DIMM form factor. This targeted (I think) two markets. First, use as slower but cheaper and higher density volatile RAM. There was actual demand — various caching workloads, for example, wanted hundreds of GB or even multiple TB in one server, and Optane was a route to get there. But the machines and DIMMs never really became available. Then there was the idea of using Optane DIMMs as persistent storage. This was always tricky because the DDR interface wasn’t meant for this, and Intel also seems to have a lot of legacy tech in the way (their caching system and memory controller) and, for whatever reason, they seem to be barely capable of improving their own technology. They had multiple serious false starts in the space (a power-supply-early-warning scheme using NMI or MCE to idle the system, a horrible platform-specific register to poke to ask the memory controller to kindly flush itself, and the stillborn PCOMMIT instruction).
2. Very nice NVMe devices. I think this was more of a failure of marketing. If they had marketed a line of SSDs that, coupled with an appropriate filesystem, could give 99% fsync latency of 5 microseconds and they had marketed this, I bet people would have paid. But they did nothing of the sort — instead they just threw around the term “Optane” inconsistently.
These days one could build a PCM-backed CXL-connected memory mapped drive, and the performance might be awesome. Heck, I bet it wouldn’t be too hard to get a GPU to stream weights directly off such a device at NVLink-like speeds. Maybe Intel should try it.
One of the many problems was trying to limit the use of Optane to Intel devices. They should have manufactured and sold Optane memory and let other players build on top of it at a low level.
Which “Optane memory”? The NVMe product always worked on non-Intel. The NVDIMM products that I played with only ever worked on a very small set of rather specialized Intel platforms. I bet AMD could have supported them about as easily as Intel, and Intel barely ever managed to support them.
The consumer "Optane memory" products were a combination of NVMe and Intel's proprietary caching software, the latter of which was locked to Intel's platforms. They also did two generations of hybrid Optane+QLC drives that only worked on certain Intel platforms, because they ran a PCIe x2+x2 pair of links over a slot normally used for a single X2 or x4 link.
Yes, the pure-Optane consumer "Optane memory" products were at a hardware level just small, fast NVMe drives that could be use anywhere, but they were never marketed that way.
Exactly. I happen to have all AMD sitting around here, and buying my first Optane devices was a gamble, because I had no idea if they'd work. Only reason I ever did, is they got cheap at one point and I could afford the gamble.
That uncertainty couldn't have done the market any favors.
I feel like this is proving my point. You can’t read “Optane” and have any real idea of what you’re buying.
Also… were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges in the “Rapid Storage” family where some secret sauce in the PCIe host lied to the OS about what was actually connected so an Intel driver could replace the OS’s native storage driver (NVMe, AHCI, or perhaps something worse depending on generation) to implement all the actual logic in software?
It didn’t help Intel that some major storage companies started selling very, very nice flash SSDs in the mean time.
> were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges
They were definitely part of the series of massive kludges. But aside from the Intel platforms they were marketed for, I never found a PCIe host that could see both of the NVMe devices on the drive. Some hosts would bring up the x2 link to the Optane half of the drive, some hosts would bring up the x2 link to the QLC half of the drive, but I couldn't find any way to get both links active even when the drive was connected downstream of a PCIe switch that definitely had hardware support for bifurcation down to x2 links. I suspect that with appropriate firmware hacking on the host side, it may have been possible to get those drives fully operational on a non-Intel host.
Why on Earth did Intel implement this as a 2x2 device? They could have implemented multiple functions or they could have used a PCIe switch or they could have exposed their device as an NVMe device with multiple namespaces, etc. (I won’t swear that all of these would have worked nicely. But all of them would have performed better than arbitrarily splitting the link in half.)
Maybe they didn’t own any of the IP for the conventional SSD part and couldn’t make it play ball?
The Optane side of the drive used the same x2 controller as the pure-Optane cache drives. The NAND side used a Silicon Motion controller, same as their consumer QLC drives of the era. They almost literally just crammed their two existing consumer products onto one PCB and shipped it. Intel was never interested enough in the consumer applications of Optane to design a good, useful SSD controller around it, and they weren't going to let a third-party like Silicon Motion make an Optane-compatible controller.
It isn't weird at all. I would be surprised if it ever succeed in the first place.
Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted another Intel only tech lock in. And despite the high price, it barely made any profits per unit compared to NAND and DRAM which was at the time making historic high profits. Once the NAND and DRAM cycle went down again cost / performance on Optane wasn't as attractive. Samsung even made some form of SLC NAND that performs similar to Optane but cheaper, and even they end up stopped developing for it due to lack of interest.
A ways back, I wrote a sort of database that was memory-mapped-file backed (a mistake, but I didn’t know that at the time), and I would have paid top dollar for even a few GB of NVDIMMs that could be put in an ordinary server and could be somewhat straightforwardly mounted as a DAX filesystem. I even tried to do some of the kernel work. But the hardware and firmware was such a mess that it was basically a lost cause. And none of the tech ever seemed to turn into an actual purchasable product. I’m a bit suspicious that Intel never found product-market fit in part because they never had a credible product on the NVDIMM side.
Somewhere I still have some actual battery-backed DIMMs (DRAM plus FPGA interposer plus awkward little supercapacitor bundle) in a drawer. They were not made by Intel, but Intel was clearly using them as a stepping stone toward the broader NVDIMM ecosystem. They worked on exactly one SuperMicro board, kind of, and not at all if you booted using UEFI. Rebooting without doing the magic handshake over SMBUS [0] first took something like 15 minutes, which was not good for those nines of availability.
[0] You can find my SMBUS host driver for exactly this purpose on the LKML archives. It was never merged, in part, because no one could ever get all the teams involved in the Xeon memory controller to reach any sort of agreement as to who owned the bus or how the OS was supposed to communicate without, say, defeating platform thermal management or causing the refresh interval to get out of sync with the DIMM temperature, thus causing corruption.
I’m suspicious that everything involved in Optane development was like this.
I worked at Micron in the SSD division when Optane (originally called crosspoint “Xpoint”) was being made. In my mind, there was never a real serious push to productize it. But it’s not clear to me whether that was due to unattractive terms of the joint venture or lack of clear product fit.
There was certainly a time when it seemed they were shopping for engineers opinions of what to do with it, but I think they quickly determined it would be a much smaller market anyway from ssds and didn’t end up pushing on it too hard. I could be wrong though, it’s a big company and my corner was manufacturing and not product development.
Cost was fantastically cheap, if you take into account that Optane is going to live >>10x longer than a SSD.
For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.
When most people are running databases on AWS RDS, or on ridiculous EBS drives with insanely low throughput and latency, it makes sense to me.
There are very few applications that benefit from such low latency, and if one has to go off the standard path of easy, but slow and expensive and automatically backup up, people will pick the ease.
Having the best technology performance is not enough to have product market fit. The execution required from the side of executives at Intel is far far beyond their capability. They developed a platform and wanted others to do the work of building all the applications. Without that starting killer app, there's not enough adoption to build an ecosystem.
I never understood what they're meant to do. Intel seemed to picture some future where RAM is persistent; but they were never close to fast enough to replace RAM, and the option to reboot in order to fix some weird state your system has gotten itself into is a feature of computers, not a problem to work around.
IMO, the reason they didn't sell is the ideal usage for them is pairing them with some slow spinning disks. The issue Optane had is that SSD capacity grew dramatically while the price plummeted. The difference between Optane and SSDs was too small. Especially since the M.2 standard proliferated and SSDs took advantage of PCI-E performance.
I believe Optane retained a performance advantage (and I think even today it's still faster than the best SSDs) but SSDs remain good enough and fast enough while being a lot cheaper.
Optane was a victim of its own hype, such as “entirely new physics”, or “as fast as RAM, but persistent”. The reality felt like a failure afterwards even though it was still revolutionary, objectively speaking.
In "databases and journals" you rarely update just one byte, you do a transaction that updates data, several indexes and metadata. All of that needs to be atomic.
Power failure can happen in between any of "1 byte updates with crazy latencies." However small latency is, power failure is still faster. Usually, there is a write ahead or some other log that alleviates the problem, this log is usually written in streaming fashion.
What is good, though, is that "blast radius" [1] of failure is smaller than usual - failed one byte write rarely corrupts more that one byte or cache line. SQLite has to deal with 512 (and even more) bytes long possible corruptions on most disks, with Optane it is not necessarily so. So, less data to copy, scan, etc.
Optane didn't sell because they focused on their weird persistent DIMM sticks, which are a nightmare for enterprise where for many ordinary purposes you want ephemeral data that disappears as soon as you cut power. Thet should have focused on making ordinary storage and solving the interconnect bandwidth and latency problems differently, such as with more up-to-date PCIe standards.
My understanding is Optane is still unbeaten when it comes to latency. Has anyone examined its use as an OS volume, compared to today's leading SSD's? I know the throughput won't be as high, but in my experience that's not as important to how responsive your machine feels as latency.
Optane was crazy good tech, it way just too expensive at the time for mass adoption, but the benefits were so good.
Looking at those charts, besides the DWPD it feels like normal NVMe has mostly caught up. I occassionally wonder where a gen 7/8(?) optane would be today if it caught on, it'd probably be nuts.
One potential application I briefly had hope for was really good power loss protection in front of a conventional Flash SSD. You only need a little compared to the overall SSD capacity to be able to correctly report the write was persisted, and it's always running, so there's less of a 'will PLP work when we really need it?' question. (Maybe there's some use as a read cache too? Host RAM's probably better for that, though.) It's going to be rewritten lots of times, but it's supposed to be ready for that.
It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficiently, and a lot is already built around dealing with the tens-of-microseconds latencies (or worse--networked block storage!). Read latencies you can cache your way out of, and writers can either accept commit latency or play it a little fast and loose (count a replicated write as safe enough or...just not be safe). You have to improve on Flash by enough to make it worth the leap while remaining cheaper than other approaches to the same problem, and you have to be confident enough in pulling it off to invest a ton up front. Not easy!
I feel sorry about the situation. From my perspective Optane was a godsend for databases. I was contemplating building a system. Could've been a pinnacle of vertical scalability for cheap.
Around the time of Optane's discontinuation, the rumor mill was saying that the real reason it got the axe was that it couldn't be shrunk any, so its costs would never go down. Does anyone know if that's true? I never heard anything solid, but it made a lot of sense given what we know about Optane's fab process.
And if no shrink was possible, is that because it was (a) possible but too hard; (b) known blocks to a die shrink; or (c) execs didn't want to pay to find out?
I think a factor in Optane's demise was dishonesty on the part of low-end laptop manufacturers.
You were a poor student looking for an entry-level cheap laptop and saw a lot of models with 4GB [RAM] "memory" and suddenly there was this one model with "20 GB memory" for the same price. Seemed attractive to the regular guy, but this in fact was 4GB RAM + 16 GB Optane non-volatile storage (and maybe a paltry 32 GB SSD for the rest). Optane would be treated as a drive for storing the Windows OS.
That conflation of Optane storage as "memory", hinting that this was equal to RAM, turned many people against it once they fell victims to that bait-and-switch.
Is there any alternative today? I benched my brand new pcie5 ssd today and while it does over 10GB/s sequential data for random small chunks it barely exceeds 70MB/s. Meanwhile Optane was 5 times faster than that.
I wish it had made it to thumb drives for its superior data retention. It doesn't work by holding a charge; it's phase-change, so it retains data longer.
Did it ever have real input costs which justified its pricing or was it totally a synthetic price to cover an imagined value above base, and some IPR?
I may be wrong, but I tend to think the fab cost has very little to do with the complexity, but not nothing, and the outcome cost is very unrelated to price at a viable yield.
If optane failed, it was probably because Intel didn't like the economics of pricing it to succeed.
Sure, they were expensive but they have great endurance and sustained read and write speeds. I use one in my car for camera recordings. I had gone through several other drives but this one has been going on 3 or 4 years now without issue. I have a couple more in use too. It's a shame this tech is going away because it's excellent.
Fabs are expensive and all, but maybe running a right-sized fab could have still been profitable at making optane for low-latency work that it was so good at. Even moreso with RAM prices as they are.
the tech had so much potential, but it was during the era when Intel dominated. other than the pcie add-in cards that can act has fast cache, all the major use cases were down to their whims. platform support (or the lack thereof) killed it before it could be realized.
too bad there was no apple moment (e.g. with thunderbolt) where a third-party was allowed to make it more widespread.
157 comments
1. “Optane” in DIMM form factor. This targeted (I think) two markets. First, use as slower but cheaper and higher density volatile RAM. There was actual demand — various caching workloads, for example, wanted hundreds of GB or even multiple TB in one server, and Optane was a route to get there. But the machines and DIMMs never really became available. Then there was the idea of using Optane DIMMs as persistent storage. This was always tricky because the DDR interface wasn’t meant for this, and Intel also seems to have a lot of legacy tech in the way (their caching system and memory controller) and, for whatever reason, they seem to be barely capable of improving their own technology. They had multiple serious false starts in the space (a power-supply-early-warning scheme using NMI or MCE to idle the system, a horrible platform-specific register to poke to ask the memory controller to kindly flush itself, and the stillborn PCOMMIT instruction).
2. Very nice NVMe devices. I think this was more of a failure of marketing. If they had marketed a line of SSDs that, coupled with an appropriate filesystem, could give 99% fsync latency of 5 microseconds and they had marketed this, I bet people would have paid. But they did nothing of the sort — instead they just threw around the term “Optane” inconsistently.
These days one could build a PCM-backed CXL-connected memory mapped drive, and the performance might be awesome. Heck, I bet it wouldn’t be too hard to get a GPU to stream weights directly off such a device at NVLink-like speeds. Maybe Intel should try it.
> Optane memory
Which “Optane memory”? The NVMe product always worked on non-Intel. The NVDIMM products that I played with only ever worked on a very small set of rather specialized Intel platforms. I bet AMD could have supported them about as easily as Intel, and Intel barely ever managed to support them.
Yes, the pure-Optane consumer "Optane memory" products were at a hardware level just small, fast NVMe drives that could be use anywhere, but they were never marketed that way.
That uncertainty couldn't have done the market any favors.
Also… were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges in the “Rapid Storage” family where some secret sauce in the PCIe host lied to the OS about what was actually connected so an Intel driver could replace the OS’s native storage driver (NVMe, AHCI, or perhaps something worse depending on generation) to implement all the actual logic in software?
It didn’t help Intel that some major storage companies started selling very, very nice flash SSDs in the mean time.
> were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges
They were definitely part of the series of massive kludges. But aside from the Intel platforms they were marketed for, I never found a PCIe host that could see both of the NVMe devices on the drive. Some hosts would bring up the x2 link to the Optane half of the drive, some hosts would bring up the x2 link to the QLC half of the drive, but I couldn't find any way to get both links active even when the drive was connected downstream of a PCIe switch that definitely had hardware support for bifurcation down to x2 links. I suspect that with appropriate firmware hacking on the host side, it may have been possible to get those drives fully operational on a non-Intel host.
Maybe they didn’t own any of the IP for the conventional SSD part and couldn’t make it play ball?
Optane died because Intel wanted to kill AMD and ARM with it. Killed Intel and a great technology along with it.
>Which is weird....
It isn't weird at all. I would be surprised if it ever succeed in the first place.
Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted another Intel only tech lock in. And despite the high price, it barely made any profits per unit compared to NAND and DRAM which was at the time making historic high profits. Once the NAND and DRAM cycle went down again cost / performance on Optane wasn't as attractive. Samsung even made some form of SLC NAND that performs similar to Optane but cheaper, and even they end up stopped developing for it due to lack of interest.
Somewhere I still have some actual battery-backed DIMMs (DRAM plus FPGA interposer plus awkward little supercapacitor bundle) in a drawer. They were not made by Intel, but Intel was clearly using them as a stepping stone toward the broader NVDIMM ecosystem. They worked on exactly one SuperMicro board, kind of, and not at all if you booted using UEFI. Rebooting without doing the magic handshake over SMBUS [0] first took something like 15 minutes, which was not good for those nines of availability.
[0] You can find my SMBUS host driver for exactly this purpose on the LKML archives. It was never merged, in part, because no one could ever get all the teams involved in the Xeon memory controller to reach any sort of agreement as to who owned the bus or how the OS was supposed to communicate without, say, defeating platform thermal management or causing the refresh interval to get out of sync with the DIMM temperature, thus causing corruption.
I’m suspicious that everything involved in Optane development was like this.
There was certainly a time when it seemed they were shopping for engineers opinions of what to do with it, but I think they quickly determined it would be a much smaller market anyway from ssds and didn’t end up pushing on it too hard. I could be wrong though, it’s a big company and my corner was manufacturing and not product development.
For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.
There are very few applications that benefit from such low latency, and if one has to go off the standard path of easy, but slow and expensive and automatically backup up, people will pick the ease.
Having the best technology performance is not enough to have product market fit. The execution required from the side of executives at Intel is far far beyond their capability. They developed a platform and wanted others to do the work of building all the applications. Without that starting killer app, there's not enough adoption to build an ecosystem.
I believe Optane retained a performance advantage (and I think even today it's still faster than the best SSDs) but SSDs remain good enough and fast enough while being a lot cheaper.
The ideal usage of optane was as a ZIL in ZFS.
Power failure can happen in between any of "1 byte updates with crazy latencies." However small latency is, power failure is still faster. Usually, there is a write ahead or some other log that alleviates the problem, this log is usually written in streaming fashion.
What is good, though, is that "blast radius" [1] of failure is smaller than usual - failed one byte write rarely corrupts more that one byte or cache line. SQLite has to deal with 512 (and even more) bytes long possible corruptions on most disks, with Optane it is not necessarily so. So, less data to copy, scan, etc.
[1] https://sqlite.org/psow.html
https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-...
In an era of RAM shortages and quarterly price increases, Optane remains viable for swap and CPU/GPU cache.
They suck for large sequential file access, but incredible for small random access: databases.
Looking at those charts, besides the DWPD it feels like normal NVMe has mostly caught up. I occassionally wonder where a gen 7/8(?) optane would be today if it caught on, it'd probably be nuts.
It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficiently, and a lot is already built around dealing with the tens-of-microseconds latencies (or worse--networked block storage!). Read latencies you can cache your way out of, and writers can either accept commit latency or play it a little fast and loose (count a replicated write as safe enough or...just not be safe). You have to improve on Flash by enough to make it worth the leap while remaining cheaper than other approaches to the same problem, and you have to be confident enough in pulling it off to invest a ton up front. Not easy!
And if no shrink was possible, is that because it was (a) possible but too hard; (b) known blocks to a die shrink; or (c) execs didn't want to pay to find out?
You were a poor student looking for an entry-level cheap laptop and saw a lot of models with 4GB [RAM] "memory" and suddenly there was this one model with "20 GB memory" for the same price. Seemed attractive to the regular guy, but this in fact was 4GB RAM + 16 GB Optane non-volatile storage (and maybe a paltry 32 GB SSD for the rest). Optane would be treated as a drive for storing the Windows OS.
That conflation of Optane storage as "memory", hinting that this was equal to RAM, turned many people against it once they fell victims to that bait-and-switch.
I may be wrong, but I tend to think the fab cost has very little to do with the complexity, but not nothing, and the outcome cost is very unrelated to price at a viable yield.
If optane failed, it was probably because Intel didn't like the economics of pricing it to succeed.
https://goughlui.com/2024/07/28/tech-flashback-intel-optane-...
too bad there was no apple moment (e.g. with thunderbolt) where a third-party was allowed to make it more widespread.