When you create an archive, do not skip the step of reading it back. I'm not talking about just having the system confirm it is readable. I mean actually restoring the data from the tape. I've had tapes fail to retrieve data off of the tape even though the writing app confirmed it was readable.
Drives can have issues reading data written by another drive. Drives can have issues reading a tape it wrote itself.
Why write one when you can write two at twice the time/cost? In all of my experience with tape, I've never written just one tape. In the early days of DVDs, the images were written to LTO to send to have the discs pressed. Two were always sent on the chance one wasn't readable and schedules were so tight that the loss of a couple of days to send a new tape would wreck delivery dates. Another facility wrote three tapes: one kept on site, one sent to IronMountain, one sent to the client directly. Plenty of occasions saw the local copy unreadable and the IronMountain copy was retrieved. These were managed by large robotic tape libraries so each tape was written in a different drive at the same time so it was 3x cost while only 1x time.
Use the drive cleaning cassette as recommended. Your dirty heads can cause the data to not be written/read correctly depending on which head is dirty. If you're having issues reading, clean the heads before giving up on the tape. If you're having issues reading a tape just written, do not think someone else will be able to read it.
Ensure whoever might be using a tape you've written understands what was used to write the tape. Not all archivers are compatible. If you use the tar command to write to tape, people using a GUI app won't be able to read it.
Perhaps one of you is living in a much more humid and/or hot environment that causes issues with the writing process.
Debugging this was really fun, especially due to the randomness factor. :)
I suspect that the number of problems you encounter from tapes scales superlinearly because they physically move. This leads to a lot of unpredictable failure modes. HDD racks also fail a lot, but in more predictable ways because all the data is managed in software.
There wasn't any value to it, but I wish I still had it.
Data was only 3-4 years old at most, but there was never a recovery failure - at least one that was visible during the restore process.
Oh - important data-point - we never re-used tapes. It was one-and-done-and-off-to-iron-mountain. (That eventually would have changed - but we were only 4 years into our tape-rotation when I left)
Anecdote of 1, but it was over probably 10+ restores of data. Not certain how many tapes were actually read - we would just jam them in 24 at a time and let the index figure out which ones to restore from.
The caution to validate backups by doing constant restore + validation is a good one though. Nobody cares about backups, it's the restores they want.
* https://www.blocksandfiles.com/tape/2025/07/29/why-lto-10-fe...
> The tape drives use a strong error correction algorithm that makes data recovery possible when lost data is within one track. Also, the head assembly includes separate read heads positioned just "behind" each write head, to verify the data just written without making a separate verification pass. This allows the drive to write a second copy of any data that fails the verify without the help of the host system.
* https://en.wikipedia.org/wiki/Linear_Tape-Open#Logical_struc...
Been a long time since I've dealt with audio gear, but I'm pretty sure they also had separate playback/record heads in the tape path as well.
But when you try to read it back with a different, correctly aligned machine, it is unreadable.
In addition the need for cloud providers to comply with userdata erasure means they have to frequently test their ability to read and write their backups - which should increase your confidence in their correctness. Yes problems can and do still happen, but IME far less often than the more traditional tape setups GP describes.
* Don't rely on vendor-specific compression and/or encryption which may vary between different tape drives.
* If you do use encryption, try to ensure you have a way to recover from corruption in the middle of the bitstream (see parent's point on two copies, or CTR/GCM mode, or FEC/erasure coding).
* Index the contents (file names, sizes, at least), or use an archive format that puts the index at the start of the tape (note: tar does not).
[1]: https://rustic.cli.rs/docs/commands/init/cold_storage.html
if you actually do the math, that's expensive
I have several thousand video files totaling several TB that I store as a hobby.
This is purely for backup purposes, so I almost never need to retrieve the files. That said, I'd hesitate to use a service that charges something like $100 to retrieve a 1 GB file.
My understanding is that with AWS S3, you can retrieve up to 1 TB of data almost for free by using CloudFront.
I'm currently storing my files in the us-east-1 region.
50GB? Is that a joke?
About 20 years ago, we used to keep track of this stuff. Even batches from normally excellent brands would be bad. We considered this short term, volatile storage.
At the end, even the best brands outsourced and we quit using them. Now, M-Disk from Verbatim is decent, although rarely used.
Plus, having your backups on used hardware _instills_ the importance of proper backup hygiene like nothing else!
Sure, if you ignore my last line saying it's easy to have 2-3 copies.
> For proper archival, each project gets its own backup device not shared with other projects
"its own backup device" implies only 1 copy? That's so much worse than 3 copies on shared drives.
And who isn't using shared drives? Companies use shared drives. Amazon uses shared drives. And you could double it to 6 copies for ridiculous levels of reliability and it would still be under $10 per shoot.
And if that's twisting intent, then you twisted mine much harder with "500 shoots die with it", because I explicitly suggested multiple copies instead of just treating it as assumed. What's with this one-sided accusation?
And if you're buying three drives per client that's an extra $200 per photoshoot in this scenario if we're talking hard drives. It's a massive cost. And it would need to be hard drives, since the client certainly can't read tapes, probably can't read a bluray, and a USB stick wouldn't be reliable unplugged. Also the idea of handing over a whole drive depends on them having a full license to every file you're archiving. For a photoshoot it's just not worth it in comparison to copying their deliverables folder out of the shared vault.
Edit: I guess you could go with stacks and stacks of blurays and be ready to hand out a drive every time you hand a disc out.
It's cheaper for consumers to have a drive bundled with the data storage, such as a hard drive.
Tape drives will always be needed for those who need fewer IOPS but more capacity. After all, tape drives allow you to build tape libraries with custom capacity vs drives (aka IOPS and bandwidth).
Consumers are more predictable and want IOPS more than capacity. So SSDs already are better, let alone HDDs.
There are problems with cloud backups, but they are solvable. One is confidentiality/privacy, that's what encryption is for. The other is that you can lose your data because the could provider doesn't like you, it is a failure mode, in the same way that your tapes can burn. Use redundancy, like with any unreliable storage solution.
With an onsite NAS, local copies and cloud backup, you meet the "3 2 1" rule (3 copies, 2 storage types, 1 offsite) without the hassle of dealing with tapes, and it will probably be cheaper too, unless you are particularly data-hungry.
As a nerd that formerly had a DDS4 drive it's tempting to get an older LTO drive, but sadly I can't expect it to survive for long in a house full of cat hair.
And it's kinda maddening because it can't be all that hard to have some filters.
They're not only used in such controlled environments, but yes, they are pickier.
LTO-10 30TB: $259–$300 per cartridge (~$8.63–$10/TB) LTO-10 40TB: $491–$555 per cartridge (~$12.28–$13.88/TB) LTO-9 18TB: $92–$129 per cartridge (~$5.14–$7.17/TB)
https://tapebackup.org/lto-tape-price-trend/lto10-price
One thing worth noting is that the tape cartridges themselves aren't necessarily the biggest expense. Standalone LTO-10 drives are currently listed at around $25,000.
The drives are a fairly fixed CapEx: once they're purchased they don't cost must/anything to run. You may replace them every few years with newer tech for higher capacities of course.
Libraries/robotics last even longer, and can stick around for a decade or more once purchased.
But the fact that -ALL- LTO drive-mechanisms are build by IBM maybe also doesn't help the pricing...
And for long term archive storage, the main problem is not the media, it is how are you going to read the media. With the drives being considerably more fragile that the tapes. A good plan is to store an entire spare restoration kit with the media. With the understanding it probably will not work when it needs to. So store the really critical stuff printed out so the mark one eyeball can parse it.
Just a note: do NOT buy LTO-10 right now. The capacity increase is not worth the price bump, and the drives are still very new. There are still no half-height drives for tape libraries available, for example.
[1] https://docs.aws.amazon.com/snowball/latest/developer-guide/...
They often cost billions to build and they take loads of data, but most of it ends up heavily distilled by the time it goes to the final analysis pipeline, so keeping the raw data on tape is pretty common.
There was a time when tape was more specialized and we were worried about loosing our limited suppliers. The explosion in the ML that powers search and more recently the AI boom makes the technology look a lot more stable these days, which is a relief for us.
Alas, M-Disks are hugely more expensive than magnetic tapes, even when not taking into account their much greater volume.
The biggest M-Disks are of 100 GB and the cheapest that I see on Amazon (in 25-piece sets) are at about $9 without taxes and shipping. At that price, I can buy not 0.1 TB, but 2 TB of magnetic tapes, which also have a much lower volume, at 18 TB per tape for the older and cheaper LTO-9 cartridges. I can easily carry with me a suitcase with 360 TB of data (20 LTO-9 cartridges), or store it in a safe place. Carrying 3600 M-Disks or storing them in my home would be impossible.
M-Disks may be used only to archive some small amounts of more valuable data, e.g. documents or programs that you have written yourself, a few scanned legal documents, a few more important books, a small amount of music that you prefer, a few family photographs.
For bulk data, if you cannot afford the cost of tape drives, the only alternative is HDDs, but with HDDs you must take care to always use both duplicate HDDs and archive files with redundancy, which allow the regeneration of their content when any not too large part of the file becomes corrupted. You must also take care to migrate the data periodically to new HDDs, after a few years.
Due to their small size I have not tried to use M-Disks until now, so I do not know if my older Blu-ray recorder can write them. There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
I'm a noob when it comes to tape backups. Is copying the same stuff to multiple tapes standard practice?
> There is also the problem that the availability in the future of Blu-ray recorders and readers is uncertain.
LTO drives used to read tapes from 2 generations back, then just 1. LTO 10 isn't backwards compatible. Future availability of compatible tape drives is also uncertain.
Therefore with HDDs I have always written the same data on 2 HDDs, and this saved me a few times, when after several years of storage both HDDs contained a few corrupted sectors, but not in the same places.
With tapes, I also always write at least 2, but frequently even 3, because their low cost makes this affordable. With tapes, the greatest danger is that if a drive malfunctions mechanically while a tape cartridge is inserted, it might damage that tape.
Having at least 2 copies of each tape or HDD also allows their storage in different locations.
Like I have said, I always combine this redundancy of having duplicate storage media with intra-archive-file redundancy, so I typically add 10% redundancy to each compressed and encrypted archive, so that for any corruption that affects less than 10% of the file the content is still recoverable.
I am still using the older LTO-7. While I might upgrade to LTO-8 or even LTO-9, an upgrade to LTO-10 is unlikely, unless great price reductions will happen in the future, which I doubt.
So I am very concerned by the lack of a good archival solution that is affordable by ordinary people and by small businesses, not only by very rich companies.
Microsoft has demonstrated an ideal archival medium (in glass), but for now the cost of their laser system is too great for commercialization. That appears to be mostly a chicken and egg problem, the lasers and the other optical components are too expensive because they currently are produced only in very small quantities, but they cannot be incorporated into a high-volume application, because they are too expensive.
If you are a business of course tape is cheap. Even then, a few usb harddrives are cheap as well and much faster.
I mean, it's plausible, yet I find it hard to believe this solely drives demand 5 times. Because if it does then what, companies were not using cold storage backups before?
As someone in HPC: people never delete stuff. It can stick around forever, and so if more bits are generated, there are more bits that people don't want to delete "just in case". Depending on the organization, quota enforcement can be tighter or looser.
HSM is fairly popular in the HPC space for this reason:
* https://en.wikipedia.org/wiki/Hierarchical_storage_managemen...
Your newest/hottest bits are on NVMe, anything older than (e.g.) 3 months gets move to SSD, anything older than that to spinning rust HDDs, and then to tape.
There are several file systems where this happens in the background: the file still shows up at /path/for/stuff/foo.txt but the contents are shuffled about behind the scenes.
This is a catastrophic event that you can't mitigate except with cold storage offline.
What I observed in the rising cybersecurity attacks happening around me:
(i) There are organizations thinking that they have [complete] backups and being aware they don't once the cyberattack if effective.
(ii) Even in organizations with complete backups they don't have a [good] process to recover them, and the operations suffered a lot.
Corollary: backups are, in general, not battle tested.
Even within the same company. Every division has backup processes, only few actually try to recover the data once in a while...
I've ran drills with companies which concluded that the incremental backups couldn't be restored due to some data corruption issue (no readback test was ever done).
Luckily the company also did a full backup every 6 months, so a disaster would have only thrown them back 6 months at max! /s ;)
But yes, some issues were even more simple: I had a case where performing a restore required twice the storage size of the uncompressed data, and the company actually didn't have the infrastructure ready to do that in case of a disaster...
Don't get me wrong, drills are important! You should do them. However backups without a restore drill better than no backups at all. Even if you only get the last full backup, the data from the last 6 months will be the easiest to recreate from scratch if you must.
I'm suspecting the AI companies simply save everything (even if you said "no"), just in case they need it later for training.
Why? Because there is no moral anymore. They blatantly ignored copyright after all.
The millions of pictures I never got to sort, the multiple versions of a file that I wrote 2 years ago, etc.
But in the view of an AI-company that's probably a paradox. All this data is charged and could be used to learn more about me, it should never be deleted! Instead, some hole should be carved out of a mountain to store it, billing me every month!
Thinking...
User has 15 pictures of his dead mother, deleting 14.
Saved 20 MB! :)
For the pictures, I set up a batch script the one time that used exiftool to read all of my photo's exif data (for creation date), rename the file to that date / time, and put it in a year/month folder structure. Once that's normalized it was pretty straightforward to identify and delete duplicates (assuming the exif data on duplicates has the same date/time in it).
Duplicates are easy, that gets sorted quite fast. Picking 1-2 out of those 5 slightly different pictures of the same occasion is another task.
Shipping your drives to Sam Altman. If everybody did that, he could then deduplicate the pictures of the same thing that multiple people took.
As time goes by, people are sending more and more. It started with just prompts, then the local source code, then calendar, emails, local files...
At some point in the future the amount of data being stored indiscriminately must consume such a huge amount of resources and cost, that it becomes a hidden fixed cost within a country's GDP, without any tangible benefit...
That continues to be the assumption, yes. Disregarding all the waste being created in the process.
What's fascinating is the fact that the production of TAPE sees a 5x boost. That's the format with the slowest retrieval rate, only making sense for long-term storage, usually 1+ years.
So is there really 5x more valuable data now, or is it just not economic to actually judge the need for preserving data...?
Cloud backup is good until a new world war erupts since data centers are gonna be bombarded first.
What is going on with cerabyte?
That's what they keep telling everyone else for years now...
That's where all my random pictures, duplicate bank statements and savegames will go. For the benefit of mankind! (and a small monthly fee)
Point.de
Point archival gateway.
Beautiful. I love it.