> recently we had a particularly bad case: a customer sent us an approximately 1 GB project file which was corrupt, for a game already published to Steam, which they'd spent months working on. They told us all their backups were corrupt too... Predictably, WinRAR's repair tool produced a 1 GB ZIP file that contained nothing.
Thirty years of creating and storing zip files on every major and some minor OSes, filesystems, sketchy transfer protocols, and unreliable media make me think that the huge number of issues this company has with corrupted ZIP files has nothing to do with the format or with its customers' bad practices. I think their software might just produce bad zip files.
The more popular your software is, the more likely you'll get bug reports from people with just bad hardware. I know a few applications now that perform a simple memory/CPU test when sending a bug report and reject those that are basically impossible.
Firefox was the only program that was crashing on my "Vmin Shift Instability" bugged CPU. Before I even knew what the culprit was, I filed a bug report with Firefox and the first question they asked me was if my CPU was a Raptor Lake (it was).
It's mind blowing for me whenever the hardware abstraction leaks like this. You (ie Firefox) had to call the instructions in a specific order to cause the hardware to fail.
It does make sense that a browser, being the thing most people would be running all day, basically fuzzes CPUs for free... they generate their own CPU instructions at runtime from the JIT engine(s), and different styles of webpages could cause every possible combination of instructions to appear throughout the day on every machine.
This is right. There's a reason people squat domain names that differ from well-known ones by a single bit flip. Think of how many old, shitty phones there are in the world, built using the cheapest available components and running in someone's crumb-filled, humid pocket right next to their magnetic key fob thing.
Author here: we write zip files with the widely-used library zip.js [1], so if that's writing bad zip files, the problem is going to be widespread across many web apps. But I'd argue that zip.js is thoroughly battle-tested (current open issue count: 0) and not the cause of the problem.
I think you misunderstood: the point was sometimes project corruption is due to a defect in our software, in which case it is definitely our fault and we race to fix it. But most reports of corruption are bad zip files, which aren't our software (unless you think the widely used and battle-tested library zip.js is writing bad zip files, which I don't think is the case). My best guess is it's faulty hardware or system-level issues.
I just tested. 7-Zip is already doing exactly what the author's recovery tool does: if I delete the metadata at the end of the file, it still can open and extract files, while complaining about the harmless "Unexepcet end of data". Further, even if I delete the actual content of the last file, it still can show the directory structure, it just can't extract data of the last file.
> Perhaps surprisingly, the ZIP file puts the central directory - which lists the content of the ZIP file - at the end of the file
This is very common for archive files. It lets you easily append a file to the end of the archive (overwriting the directory) followed by the updated directory. If it were at the start, you’d have to rewrite the entire contents of the archive to grow the directory.
Maybe. But Unix types have always been all-in on tar. They love tar. They continue to use some form of tar to this day.
This was actually a 'cultural issue' back in the day, because Sun Java JAR files were just ZIPs. For good reason. But that wasn't the blessed holy unix archive format.
I always assumed you could work around this by pre-allocating a large enough header space to hold a reasonable sized directory tree. You could even then append a supplementary directory if you run out of space and just stick a pointer in the first one. I'm pretty sure that's just reinventing an actual filesystem, but if it works...
Losing the back half of the achieve doesn't lose your directory list. (Truth be told, I prefer the approach of sticking a copy on the front and back for redundancy, but I appreciate that that's a very specific tradeoff)
Do you really need a central directory? The tar and cpio file formats basically append files serially with a small header preceeding each one, and for most use cases this is sufficient.
I'm curious, is there a reason why you can't incrementally prepend to files? I haven't seen a filesystem that allows prepending to a file without rewriting the entire file. Is it just because it's not traditional, or is there any particular technical reason why it would be too expensive, or etc?
In a world long accustomed to "append only" files, use cases would be relatively few.
Vs. the folks creating and testing filesystems would have to do a whole lot of work to add that feature. Followed by the folks doing higher-level software, most of which which need to handle files being prepended to.
In short - interesting idea, not worth the effort.
the size of the prepended data would rarely be an exact number of blocks such that it became purely a metadata update (see also: inode small file optimisation), and thus you would have a small "hole" in the file between the new data and the existing data, which would have to be bookkept safely.
Note that bsdtar, which, incidentally, is included in modern Windows versions as tar.exe, can extract as many files as are fully present in truncated ZIP files, because it doesn't read the central header (to detect trunctation) until after extracting or listing files.
It's an interesting tale of software spelunking and edge cases, but I was wondering about this:
> Things like the uncompressed size and CRC-32 can be disregarded.
If you compute a running crc32 then you could check for that value in addition to the uncompressed size, giving you 8 bytes of precision. And if you decompress data as you go, you could check for the uncompressed size as well, giving you 12 bytes of precision. Surely that's enough to make sure you always find the real boundary of a file in the zip.
Zip files are an example of something that works, are used extensively, and have a number of weird design decisions. Some of this just history, other not so much. But good luck replacing them for customers or friends.
They have some cool advantages. Like being able to read directly from the archive without decompressing the whole thing, per file checksum, and skipping compression on parts which won’t benefit. You can also wedge large amounts of arbitrary data into the file without changing it which… is kinda weird and sometimes useful?
Nasty bits are they used 32 bit ints all over the place, so rely on hacks to support larger files. Can’t support true streaming decompression. Implementations can vary quite a lot, as can compatibility. They also have a number of old and weird features that people don’t really use. Like being able to split a zip file into multiple parts, some ancient compression techniques etc.
It seems technically possible to create an append-only ZIP writer that would only add files at the end of a ZIP archive behind the existing central directory, and then write a new central directory including the new files (and excluding the deleted files). That might make it more difficult to irrecoverably damage the archive as most of it contents will likely be preserved after e.g. an abrupt loss of power.
Grow (append to) the specified zip archive, instead of creating a new one. If this operation fails, zip attempts to restore the archive to its original state. If the restoration fails, the archive might become corrupted. This option is ignored when there's no existing archive or when at least one archive member must be updated or deleted.
The more popular your software is, the more likely you'll get bug reports from people with just bad hardware. I know a few applications now that perform a simple memory/CPU test when sending a bug report and reject those that are basically impossible.
E.g.: 10% of Firefox crashes are caused by bitflips : https://news.ycombinator.com/item?id=47252971
Firefox was the only program that was crashing on my "Vmin Shift Instability" bugged CPU. Before I even knew what the culprit was, I filed a bug report with Firefox and the first question they asked me was if my CPU was a Raptor Lake (it was).
It's mind blowing for me whenever the hardware abstraction leaks like this. You (ie Firefox) had to call the instructions in a specific order to cause the hardware to fail.
It does make sense that a browser, being the thing most people would be running all day, basically fuzzes CPUs for free... they generate their own CPU instructions at runtime from the JIT engine(s), and different styles of webpages could cause every possible combination of instructions to appear throughout the day on every machine.
This is right. There's a reason people squat domain names that differ from well-known ones by a single bit flip. Think of how many old, shitty phones there are in the world, built using the cheapest available components and running in someone's crumb-filled, humid pocket right next to their magnetic key fob thing.
Author here: we write zip files with the widely-used library zip.js [1], so if that's writing bad zip files, the problem is going to be widespread across many web apps. But I'd argue that zip.js is thoroughly battle-tested (current open issue count: 0) and not the cause of the problem.
[1] https://gildas-lormeau.github.io/zip.js/
Thank you for replying back with relevant details.
Are you perhaps zipping something that other zip.js apps are not? Perhaps you have stumbled upon some edge case.
There's tons of marginal hardware out there, and game developers sometimes push hardware pretty hard and may run early drivers, etc.
As the old saying goes, an untested backup is a non backup.
Only time .zip failed me was like in early 90's because floppy discs failed, not the .zip part.
They said as much in their intro.
Due to a rare defect in our software, sometimes the zip is corrupted.
We blame the user for not backing up, or backing up corrupted files.
That is when I stopped reading.
I think you misunderstood: the point was sometimes project corruption is due to a defect in our software, in which case it is definitely our fault and we race to fix it. But most reports of corruption are bad zip files, which aren't our software (unless you think the widely used and battle-tested library zip.js is writing bad zip files, which I don't think is the case). My best guess is it's faulty hardware or system-level issues.
I just tested. 7-Zip is already doing exactly what the author's recovery tool does: if I delete the metadata at the end of the file, it still can open and extract files, while complaining about the harmless "Unexepcet end of data". Further, even if I delete the actual content of the last file, it still can show the directory structure, it just can't extract data of the last file.
> Perhaps surprisingly, the ZIP file puts the central directory - which lists the content of the ZIP file - at the end of the file
This is very common for archive files. It lets you easily append a file to the end of the archive (overwriting the directory) followed by the updated directory. If it were at the start, you’d have to rewrite the entire contents of the archive to grow the directory.
> This is very common for archive files.
Maybe. But Unix types have always been all-in on tar. They love tar. They continue to use some form of tar to this day.
This was actually a 'cultural issue' back in the day, because Sun Java JAR files were just ZIPs. For good reason. But that wasn't the blessed holy unix archive format.
I always assumed you could work around this by pre-allocating a large enough header space to hold a reasonable sized directory tree. You could even then append a supplementary directory if you run out of space and just stick a pointer in the first one. I'm pretty sure that's just reinventing an actual filesystem, but if it works...
> ... pre-allocating a large enough header space to hold ...
Isn't the point of compression is to eliminate such wastes of space?
It's a trade between size and speed. If you want fast lookup, it's not a waste.
Sure, this would also work. But it seems more complicated -- what would be the benefit?
Losing the back half of the achieve doesn't lose your directory list. (Truth be told, I prefer the approach of sticking a copy on the front and back for redundancy, but I appreciate that that's a very specific tradeoff)
Isn't this basically FAT without the actual allocation table?
Do you really need a central directory? The tar and cpio file formats basically append files serially with a small header preceeding each one, and for most use cases this is sufficient.
You don't need one, but it's convenient for listing the files in the archive without having to scan through it.
I'm curious, is there a reason why you can't incrementally prepend to files? I haven't seen a filesystem that allows prepending to a file without rewriting the entire file. Is it just because it's not traditional, or is there any particular technical reason why it would be too expensive, or etc?
In a world long accustomed to "append only" files, use cases would be relatively few.
Vs. the folks creating and testing filesystems would have to do a whole lot of work to add that feature. Followed by the folks doing higher-level software, most of which which need to handle files being prepended to.
In short - interesting idea, not worth the effort.
its a legacy from the tape-storage times
the size of the prepended data would rarely be an exact number of blocks such that it became purely a metadata update (see also: inode small file optimisation), and thus you would have a small "hole" in the file between the new data and the existing data, which would have to be bookkept safely.
Note that bsdtar, which, incidentally, is included in modern Windows versions as tar.exe, can extract as many files as are fully present in truncated ZIP files, because it doesn't read the central header (to detect trunctation) until after extracting or listing files.
It's an interesting tale of software spelunking and edge cases, but I was wondering about this:
> Things like the uncompressed size and CRC-32 can be disregarded.
If you compute a running crc32 then you could check for that value in addition to the uncompressed size, giving you 8 bytes of precision. And if you decompress data as you go, you could check for the uncompressed size as well, giving you 12 bytes of precision. Surely that's enough to make sure you always find the real boundary of a file in the zip.
Why do they use zip at all with all these design flaws? Wouldn't there be better alternatives?
Zip files are an example of something that works, are used extensively, and have a number of weird design decisions. Some of this just history, other not so much. But good luck replacing them for customers or friends.
They have some cool advantages. Like being able to read directly from the archive without decompressing the whole thing, per file checksum, and skipping compression on parts which won’t benefit. You can also wedge large amounts of arbitrary data into the file without changing it which… is kinda weird and sometimes useful?
Nasty bits are they used 32 bit ints all over the place, so rely on hacks to support larger files. Can’t support true streaming decompression. Implementations can vary quite a lot, as can compatibility. They also have a number of old and weird features that people don’t really use. Like being able to split a zip file into multiple parts, some ancient compression techniques etc.
It seems technically possible to create an append-only ZIP writer that would only add files at the end of a ZIP archive behind the existing central directory, and then write a new central directory including the new files (and excluding the deleted files). That might make it more difficult to irrecoverably damage the archive as most of it contents will likely be preserved after e.g. an abrupt loss of power.
Not the same. Grow overwrites the old central directory in place. My approach would be to keep it where it was and appending a new central directory.