I have a giant storage RAID for my home server, with a bunch of 16TB drives. I bought each of the drives used about three years ago, and they cost about $120 each. They have been working fine until last night.
One of them appears to be broken [1]. No big deal, this is what RAIDs are for, I go and try to find one and now they're going anywhere between 2-4x that price, for a used one! It's not going to bankrupt me (and having a home server is a privilege in the first place, that's not lost on me), but I really hope that the others survive, at least until this storage crunch is over. If it ever does end...sigh.
I guess I didn't realize that even relatively slow storage like spinner drives was going to be affected too.
[1] I think, I am really hoping it's just a bad connection or something but I haven't fully diagnosed it yet.
ETA: Looks like at least in my case it was actually just a bad SATA cable. The drive is reading properly and resilvering now. Phew.
I believe they intentionally did this, causing people huge import fees in some cases, in order to not remove the “26” sold on their listings that are now astronomically priced: https://ebay.us/m/mGRdiT
Edit: They also lied on their customs declarations (!)
Yeah, totally agree. I believe that the axing of several anti-monopoly enforcement departments and regulations in the largest market in the world (the US) is effectively a very big wink wink, nudge nudge to market participants; Trump basically got to play as Oprah for Big Businesses everywhere: "You get to be a cartel! You get to be a cartel! Everybody gets to be a cartel!"
He's basically created a sort of one-sided economic "Ferry Ordeal" (like the Joker on The Dark Knight [1]), basically leaving us consumers to not be exploited only if there are decent men at the helm of big businesses. It could be asymmetrical instead of one-sided if you consider that the people can only tolerate so much squeezing before they start clamoring for guillotines [2].
I have a home array too, though based on 4Tb drives (6x, two lots of RAID10, one 3x4T one 3x6T, linked as an LVM volume as I extended it by adding the second array some time ago).
I was planning to downsize anyway as most of the media I don't need to keep and I plan to replace that server with a much lower power one with a bunch of smaller SSDs. Luckily I bought the SSDs (and the other parts) before the recent price hikes, I just haven't got around to building the machine! Hopefully they all work when I do finally get around to it…
As I said, they were used, so I knew that a drive breaking was kind of an inevitability. As far as I'm aware there's no warranty, I certainly didn't pay for an extended one.
Good news though, since writing this I just started playing with dmesg and smartctl, it actually might be something with the SATA connector. At least those are still pretty cheap.
It makes sense, at the time you bought them there was no supply crunch plus being run in RAID. Would have been a decent deal.
Nowadays I feel like an underworld scrap goblin, all the old PVRs from family and friends are being cracked open for the HDDs. Time to slink off to my cave of spinning platters.
I've recently bought 3x32TB (new) and 2x28TB (recertified) drives for a new NAS as my old one started running out of space on some drives (local LLMs and datasets or media for dataset preparations are huge these days) as I expect the prices to go up considerably due to most HDD manufacturers being booked years in advance already. Some drives don't even make it to retail at all (44TB Seagate).
If I need storage now and I can't get flash disks, I will buy mechanical. Production will not ramp up overnight (or at all - there is no point investing in scaling up if I think the situation is only temporary), so an increase in demand inevitably will result in prices going up.
Btw gas goes up when oil goes up not just because it can replace it in some applications, but also because it is often produced and transported in the same areas as oil, by the same companies involved in oil, so it is typically affected in similar ways.
Customer demands (or hoarding) sentiments kept us busy on the production lines of racks with 1K+ drives and I think we've already shipped out more than 5K of them since early last year. No supply constraints on cpu and memory parts due to the way this sku was designed but damn did we have so many defective hdd's; I'd ballpark about 20% from the vendor and 10% from mishandling.
Yeah I have no idea the direct cause. I didn't think that the SATA controllers for a hard drive took that much.
It could be a secondary effect; SSDs have gotten so expensive that people are willing to put up with spinners and thus there's an increased demand. No idea, I'm sure an economist or something will do a write up of the downstream effects of the RAM crunch causes eventually.
2. Building factories requires capital to be invested now.
3. The return-on-capital will only be obtained in the next n years.
4. But if demand goes down, we'll have much more supply than demand, leading to a cutthroat price competition, which could prevent the factory costs from ever being recouped.
5. There are only limited number of suppliers and they’re making a killing from all of this.
Didn’t these companies collude before? I’m sorry but I don’t believe that they’re also using this opportunity to make as much money as possible as well.
And to add to that 4th point, there is a very real chance that many the “AI” companies that are buying all this hardware could go bankrupt at anytime and in rapid succession, leaving drive/memory manufactures with a bunch of unpaid orders and a second hand market now saturated with liquidated equipment.
Much like the GPU/crypto boom/bust some years ago, but on an even larger scale.
> 4. But if demand goes down, we'll have much more supply than demand, leading to a cutthroat price competition, which could prevent the factory costs from ever being recouped.
Why not normalize that, if things don't work out, the government steps in, buys the business, and offers a better regulated version of the business?
Capitalism is generally touted as useful for innovation; but haven't we paid collectively for the initial investment for many businesses many times over now that there's little innovation done? By definition there's no innovation once the only thing happening in a business sector is consolidation.
So, I'd conclude that capitalism isn't useful to us at that point, and actually harmfully inefficient (too costly for consumers), so we'd be wise to do away with it by that point in favor of a more efficiently run operational/economic system... That would actually be innovative! What that system looks like, I'm not sure, but I imagine it's something like a cooperative.
Production was close to maximum anyway because of existing demand and how expensive new fabs are to bring up. The boom in AI use wasn't sufficiently planned for as it wasn't expected at the scale we are seeing, so to scale up means building more fabs - a long and expensive task. The situation is not helped by one of the AI companies successfully negotiating huge deals for a year worth of production with the two biggest providers at the same time and keeping it secret enough that neither bumped up prices in the deal as a result of knowing what the play was.
Building these factories goes way beyond "nontrivial". We may see a few come online in a couple of years though, but time will tell if the extra capacity alleviates the crunch.
Most players are rational actors and this isn't their first time in this sort of market. Getting more production online if you do not have slack like downtime takes quite a while. And there is no guarantee that they can sell the new production even at previous rates if they end up in over supply.
Generally people just don't understand how long ramping up new production facilities can take and what is the realistic pay off period for them.
Adding to what others have said, several hdd, ssd and ram manufactures have said that they don't believe demand will be sustained in time for long enough to warrant extra investments in increasing production.
Because the manufacturers of storage (and RAM) consider it to be an AI bubble, too. The surge in demand is a short burst, not a sustained one. Hence it makes no sense to throw a whole lot of ressources into scaling up production, when the demand won't be there anymore in 1-2 years when the factories will be ready.
Shouldn't countries wanting sovereign infrastructure create subsidies for creation of factories/job creation and also selling first/primarily within the region if it might cost on just a few million dollars (preferably a new competitor)
I think one flaw in my thinking could be that there might be a lack of experience within the people for something like this, do you consider it to be a factor and would it be difficult to hire people relevant to such fab?
Because TI has a ton of microcontrollers, power management ICs, opamps and so on that doesn't need or is even desirable to produce on smaller processes.
Yes, but that isn't where the increase in demand is. Those things are affected by fabs that could be producing them producing something else, but there will be some stock floating in the system so some resistance to increased prices needed to justify new fab resource, and if things correct a bit in the coming year the maths for a new build might look more dubious. Those with the money to fund a new fab right now are more likely to fund something capable of producing the newer part types. I could be wrong, but the fact that significant new fabs like that are not in progress right now would suggest not.
Because it operates exactly as a drug dealer. It gives you first shot for free (reasonable opportunity to move up) and after you are hooked it makes sure that it extracts all your money (subscription and inability to own anything).
I bought 8x 4TB NAS RED 5400RPM HDD and paid in total AUD$1.4k, I am not even joking. My pool is only 8TB mirrorer and I am keeping the other 16TB as spare because things will get worse.
I bought them brand new because used ones cannot be trusted but even that brings risks.
WD has been selling 7200RPM labelled as 5400RPM disks.
WD was selling SMR disks instead of CMR disks.
Seagate disks used into crypto farms are being sold as brand new, so folks are finding our their brand new disks have 65k hours.
Some years ago, Modi announced that he was going to make India go all-in on semiconductors. When I read that the first facility to begin commercial production was going to be Micron with memory chips, I did an eyeroll. Memory chips? And just an assembly and testing facility? To me that seemed like an easy cop-out. To me (my admittedly naive self), wafer fabs and CPUs seemed like the real game.
Now, with what has happened with memory chip prices, it almost seems like they got lucky (the Micron facility is doing commercial shipments now).
Obama used to talk about having "spooky" good luck. I think Modi has some of that too.
69 comments
One of them appears to be broken [1]. No big deal, this is what RAIDs are for, I go and try to find one and now they're going anywhere between 2-4x that price, for a used one! It's not going to bankrupt me (and having a home server is a privilege in the first place, that's not lost on me), but I really hope that the others survive, at least until this storage crunch is over. If it ever does end...sigh.
I guess I didn't realize that even relatively slow storage like spinner drives was going to be affected too.
[1] I think, I am really hoping it's just a bad connection or something but I haven't fully diagnosed it yet.
ETA: Looks like at least in my case it was actually just a bad SATA cable. The drive is reading properly and resilvering now. Phew.
For example this seller: https://www.ebay.com/str/disctechllc
“Accidentally miss-priced” a bunch of drives, and then instead of canceling the orders, refunded everyone, but still shipped packages: https://forums.servethehome.com/index.php?threads/enterprise...
I believe they intentionally did this, causing people huge import fees in some cases, in order to not remove the “26” sold on their listings that are now astronomically priced: https://ebay.us/m/mGRdiT
Edit: They also lied on their customs declarations (!)
He's basically created a sort of one-sided economic "Ferry Ordeal" (like the Joker on The Dark Knight [1]), basically leaving us consumers to not be exploited only if there are decent men at the helm of big businesses. It could be asymmetrical instead of one-sided if you consider that the people can only tolerate so much squeezing before they start clamoring for guillotines [2].
[1]: https://batman.fandom.com/wiki/Ferry_Ordeal_and_Skyscraper_B...
[2]: https://youtu.be/TMHCw3RqulY
I was planning to downsize anyway as most of the media I don't need to keep and I plan to replace that server with a much lower power one with a bunch of smaller SSDs. Luckily I bought the SSDs (and the other parts) before the recent price hikes, I just haven't got around to building the machine! Hopefully they all work when I do finally get around to it…
Good news though, since writing this I just started playing with dmesg and smartctl, it actually might be something with the SATA connector. At least those are still pretty cheap.
Nowadays I feel like an underworld scrap goblin, all the old PVRs from family and friends are being cracked open for the HDDs. Time to slink off to my cave of spinning platters.
It's odd mechanical disks also surged, I thought it was only transistor based memory that are becoming rarity.
Or does it work like with fuel, gas and electricity goes up when oil spikes ?
Btw gas goes up when oil goes up not just because it can replace it in some applications, but also because it is often produced and transported in the same areas as oil, by the same companies involved in oil, so it is typically affected in similar ways.
It could be a secondary effect; SSDs have gotten so expensive that people are willing to put up with spinners and thus there's an increased demand. No idea, I'm sure an economist or something will do a write up of the downstream effects of the RAM crunch causes eventually.
AI runs on data above all else. Gotta feed the compute.
1. Factories take time to build.
2. Building factories requires capital to be invested now.
3. The return-on-capital will only be obtained in the next n years.
4. But if demand goes down, we'll have much more supply than demand, leading to a cutthroat price competition, which could prevent the factory costs from ever being recouped.
China is doing it but that was for merely creating tech independence.
Didn’t these companies collude before? I’m sorry but I don’t believe that they’re also using this opportunity to make as much money as possible as well.
Much like the GPU/crypto boom/bust some years ago, but on an even larger scale.
> 4. But if demand goes down, we'll have much more supply than demand, leading to a cutthroat price competition, which could prevent the factory costs from ever being recouped.
Why not normalize that, if things don't work out, the government steps in, buys the business, and offers a better regulated version of the business?
Capitalism is generally touted as useful for innovation; but haven't we paid collectively for the initial investment for many businesses many times over now that there's little innovation done? By definition there's no innovation once the only thing happening in a business sector is consolidation.
So, I'd conclude that capitalism isn't useful to us at that point, and actually harmfully inefficient (too costly for consumers), so we'd be wise to do away with it by that point in favor of a more efficiently run operational/economic system... That would actually be innovative! What that system looks like, I'm not sure, but I imagine it's something like a cooperative.
Generally people just don't understand how long ramping up new production facilities can take and what is the realistic pay off period for them.
Production is actually falling this year compared to last for Samsung and Hynix (there was an article on here a few days ago?).
I think one flaw in my thinking could be that there might be a lack of experience within the people for something like this, do you consider it to be a factor and would it be difficult to hire people relevant to such fab?
> just a few million dollars
TSMC Arizona projected investment is $165 billion. Not millions. And yes apparently hiring the right staff has been one of the issues.
People really underestimate the work of Maurice Chang.
I bought 8x 4TB NAS RED 5400RPM HDD and paid in total AUD$1.4k, I am not even joking. My pool is only 8TB mirrorer and I am keeping the other 16TB as spare because things will get worse.
I bought them brand new because used ones cannot be trusted but even that brings risks.
WD has been selling 7200RPM labelled as 5400RPM disks.
WD was selling SMR disks instead of CMR disks.
Seagate disks used into crypto farms are being sold as brand new, so folks are finding our their brand new disks have 65k hours.
Now, with what has happened with memory chip prices, it almost seems like they got lucky (the Micron facility is doing commercial shipments now).
Obama used to talk about having "spooky" good luck. I think Modi has some of that too.
We are trully doomed.