1 yr. ago

if you could standardise a file format for a specific task what would you pick and why

if you could pick a standard format for a purpose what would it be and why?

e.g. flac for lossless audio because...

(yes you can add new categories)

summary:

photos .jxl
open domain image data .exr
videos .av1
lossless audio .flac
lossy audio .opus
subtitles srt/ass
fonts .otf
container mkv (doesnt contain .jxl)
plain text utf-8 (many also say markup but disagree on the implementation)
documents .odt
archive files (this one is causing a bloodbath so i picked randomly) .tar.zst
configuration files toml
typesetting typst
interchange format .ora
models .gltf / .glb
daw session files .dawproject
otdr measurement results .xml

283

You're viewing a single thread.

283 comments

This is the kind of thing i think about all the time so i have a few.

Archive files: .tar.zst
- Produces better compression ratios than the DEFLATE compression algorithm (used by .zip and gzip/.gz) and does so faster.
- By separating the jobs of archiving (.tar), compressing (.zst), and (if you so choose) encrypting (.gpg), .tar.zst follows the Unix philosophy of "Make each program do one thing well.".
- .tar.xz is also very good and seems more popular (probably since it was released 6 years earlier in 2009), but, when tuned to it's maximum compression level, .tar.zst can achieve a compression ratio pretty close to LZMA (used by .tar.xz and .7z) and do it faster^1.
  
  zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup.
Image files: JPEG XL/.jxl
- "Why JPEG XL"
- Free and open format.
- Can handle lossy images, lossless images, images with transparency, images with layers, and animated images, giving it the potential of being a universal image format.
- Much better quality and compression efficiency than current lossy and lossless image formats (.jpeg, .png, .gif).
- Produces much smaller files for lossless images than AVIF ^2
- Supports much larger resolutions than AVIF's 9-megapixel limit (important for lossless images).
- Supports up to 24-bit color depth, much more than AVIF's 12-bit color depth limit (which, to be fair, is probably good enough).
Videos (Codec): AV1
- Free and open format.
- Much more efficient than x264 (used by .mp4) and VP9^3.
Documents: OpenDocument / ODF / .odt
- @raubarno@lemmy.ml says it best here. .odt is simply a better standard than .docx.
it’s already a NATO standard for documents Because the Microsoft Word ones (.doc, .docx) are unusable outside the Microsoft Office ecosystem. I feel outraged every time I need to edit .docx file because it breaks the layout easily. And some older .doc files cannot even work with Microsoft Word.

Permanently Deleted
- .tar is pretty bad as it lacks in index, making it impossible to quickly seek around in the file.
  
  .tar.pixz/.tpxz has an index and uses LZMA and permits for parallel compression/decompression (increasingly-important on modern processors).
  
  https://github.com/vasi/pixz
  
  It's packaged in Debian, and I assume other Linux distros.
  
  Only downside is that GNU tar doesn't have a single-letter shortcut to use pixz as a compressor, the way it does "z" for gzip, "j" for bzip2, or "J" for xz (LZMA); gotta use the more-verbose "-Ipixz".
  
  Also, while I don't recommend it, IIRC gzip has a limited range that the effects of compression can propagate, and so even if you aren't intentionally trying to provide random access, there is software that leverages this to hack in random access as well. I don't recall whether someone has rigged it up with tar and indexing, but I suppose if someone were specifically determined to use gzip, one could go that route.
By separating the jobs of archiving (.tar), compressing (.zst), and (if you so choose) encrypting (.gpg), .tar.zst follows the Unix philosophy of "Make each program do one thing well.".

wait so does it do all of those things?
- So there's a tool called tar that creates an archive (a .tar file. Then theres a tool called zstd that can be used to compress files, including .tar files, which then becomes a .tar.zst file. And then you can encrypt your .tar.zst file using a tool called gpg, which would leave you with an encrypted, compressed .tar.zst.gpg archive.
  
  Now, most people aren't doing everything in the terminal, so the process for most people would be pretty much the same as creating a ZIP archive.
By separating the jobs of archiving (.tar), compressing (.zst), and (if you so choose) encrypting (.gpg), .tar.zst follows the Unix philosophy of “Make each program do one thing well.”.

The problem here being that GnuPG does nothing really well.

Videos (Codec): AV1

Much more efficient than x264 (used by .mp4) and VP9[3].

AV1 is also much younger than H264 (AV1 is a specification, x264 is an implementation), and only recently have software-encoders become somewhat viable; a more apt comparison would have been AV1 to HEVC, though the latter is also somewhat old nowadays but still a competitive codec. Unfortunately currently there aren't many options to use AV1 in a very meaningful way; you can encode your own media with it, but that's about it; you can stream to YouTube, but YouTube will recode to another codec.
- The problem here being that GnuPG does nothing really well.
  
  Could you elaborate? I've never had any issues with gpg before and curious what people are having issues with.
  
  Unfortunately currently there aren’t many options to use AV1 in a very meaningful way; you can encode your own media with it, but that’s about it; you can stream to YouTube, but YouTube will recode to another codec.
  
  AV1 has almost full browser support (iirc) and companies like YouTube, Netflix, and Meta have started moving over to AV1 from VP9 (since AV1 is the successor to VP9). But you're right, it's still working on adoption, but this is moreso just my dreamworld than it is a prediction for future standardization.
  
  Could you elaborate? I’ve never had any issues with gpg before and curious what people are having issues with.
  
  This article and the blog post linked within it summarize it very well.
  
  Encrypting Email
  
  Don’t. Email is insecure . Even with PGP, it’s default-plaintext, which means that even if you do everything right, some totally reasonable person you mail, doing totally reasonable things, will invariably CC the quoted plaintext of your encrypted message to someone else
  
  Okay, provide me with an open standard that is widely-used that provides similar functionality.
  
  It isn't there. There are parties who would like to move email users into their own little proprietary walled gardens, but not a replacement for email.
  
  The guy is literally saying that encrypting email is unacceptable because it hasn't been built from the ground up to support encryption.
  
  I mean, the PGP guys added PGP to an existing system because otherwise nobody would use their nifty new system. Hell, it's hard enough to get people to use PGP as it is. Saying "well, if everyone in the world just adopted a similar-but-new system that is more-amenable to encryption, that would be helpful", sure, but people aren't going to do that.
  
  The message to be taken from here is rather "don't bother", if you need secure communication use something else, if you're just using it so that Google can't read your mail it might be ok but don't expect this solution to be secure or anything. It's security theater for the reasons listed, but the threat model for some people is a powerful adversary who can spend millions on software to find something against you in your communication and controls at least a significant portion of the infrastructure your data travels through. Think about whistleblowers in oppressive regimes, it's absolutely crucial there that no information at all leaks. There's just no way to safely rely on mail + PGP for secure communication there, and if you're fine with your secrets leaking at one point or another, you didn't really need that felt security in the first place. But then again, you're just doing what the blog calls LARPing in the first place.
  
  Super interesting stuff! Thank you for sharing.
.odt is simply a better standard than .docx.

No surprise, since OOXML is barely even a standard.
is av1 lossy
- AV1 can do lossy video as well as lossless video.
I get better compression ratio with xz than zstd, both at highest. When building an Ubuntu squashFS

Zstd is way faster though
Damn didn't realize that JXL was such a big deal. That whole JPEG recompression actually seems pretty damn cool as well. There was some noise about GNOME starting to make use of JXL in their ecosystem too...

wait im confusrd whats the differenc ebetween .tar.zst and .tar.xz

Different ways of compressing the initial .tar archive.

Permanently Deleted

But it's not a tarxz, it's an xz containing a tar, and you perform operations from right to left until you arrive back at the original files with whatever extensions they use.

If I compress an exe into a zip, would you expect that to be an exezip? No, you expect it to be file.exe.zip, informing you(and your system) that this file should first be unzipped, and then should be executed.
- it's an xz containing a tar
  
  So what? When you zip 5 documents together do you name it .zip or .config.lib.sh.deb.zip?
  
  No, you expect it to be a file.exe.zip
  
  Double extensions are not conventional on Windows, so no, I do not.
  
  Dots in filenames are commonly used in any operating system like name_version.2.4.5.exe or similar... So I don't see a problem.
  
  Dots yes, nested extensions no.
  
  The expected behavior is: you have a .exe binary called Example.exe. This is an executable.
  
  Now you zip it. It's no longer an executable binary, it's a zip archive. Yes, the data can be reconstructed into the original file - but it is not the original file. It should now be called Example.zip, as it is a zip file.
  
  This is important both for user mental models, but also because operating systems that use extensions as the primary indicator of file type often will hide known extensions by default, and the nested extensions in the name can create trouble.
  
  Hiding part of a file name might be the real problem. A IMG.jpg.exe - would result in a harmless looking JPG, but it isn't.
  
  A IMG.jpg.exe - would result in a harmless looking JPG, but it isn’t.
  
  True. Which is why getting people used to seeing .extension.extension2 is a terrible convention. Call a .zip a .zip, not a .exe.zip.
  
  which will not stop a bad guy form doing so. Therefore dont hide part of a filename and get people used to seeing tar.gz
use a real operative system then

Sounds like a Windows problem

Cool. So it means it's a problem for over 70% of all active desktop and laptop computers.

I get the frustration, but Windows is the one that strayed from convention/standard.

Also, i should've asked this earlier, but doesn't Windows also only look at the characters following the last dot in the filename when determining the file type? If so, then this should be fine for Windows, since there's only one canonical file extension at a time, right?

You're absolutely correct when it comes to how Windows will interpret the file - it will ignore all the preceding "extensions" and will use the last one as the filetype and as the hook for whatever default action or application should handle it. However, getting people used to double extensions is one quick way of increasing the success rate of attacks such as the infamous ".pdf.exe" invoice from an email attachment. It also creates issues with renaming files and, though admittedly not many, some Windows application's own file pickers.

Still - from just a theoretical point of view, I can't see how Windows' convention is worse, in fact, it makes significantly more sense. If I zip a file, it doesn't matter what it was in a previous life, it's now a zip - this is also how Unix deals with many filetypes, I've never seen a .h264.mp4 file, even though the .mp4 container can actually represent different types of encoding. Why have one filetype use the Windows convention and another, for no reason, a different one?

However, getting people used to double extensions is one quick way of increasing the success rate of attacks such as the infamous “.pdf.exe” invoice from an email attachment.

Very good point. Though, i would argue that this would be much less of a problem if Windows stopped sometimes hiding file extensions.

I can’t see how Windows’ convention is worse

I don't believe what you're referring to is really a Windows versus Linux/Unix thing.

If I zip a file, it doesn’t matter what it was in a previous life, it’s now a zip - this is also how Unix deals with many filetypes, I’ve never seen a .h264.mp4 file, even though the .mp4 container can actually represent different types of encoding.

I disagree, but i do get what you're saying here. I don't think that example really works though, because a .mp4 file isn't derived from a .h264 file. A .mp4 is a container that may include h264-encoded video, but it may also have a channel with Opus-encoded audio or something. It's apples and oranges.

Also, even though there shouldn't be any technical issues with this on Windows, you can still use a typical short filename suffix if you wish, though i would argue that using the long filename suffix is more expressive. From "tar (computing)" on Wikipedia:

Compressor	Long	Short
bzip2	.tar.bz2	.tb2, .tbz, .tbz2, .tz2
gzip	.tar.gz	.taz, .tgz
lzip	.tar.lz
lzma	.tar.lzma	.tlz
lzop	.tar.lzo
xz	.tar.xz	.tx
compress	.tar.Z	.tZ, .taZ
zstd	.tar.zst	.tzst

Though, i would argue that this would be much less of a problem if Windows stopped sometimes hiding file extensions.

You and I are in strong agreement here, I think hidden file extensions is a super bad move.

And as per the table you kindly included, if there's a short single extension version of the file, I'm happy. I still don't understand the logic behind using the long version for user-facing files, like file sharing or software distribution, but different conventions coexisting isn't anything new in computing so it is what it is.
- I think part of the reason why the long extension is often preferred is because it's much clearer and it's guaranteed to be supported and decompressed by the respective tools. Even when they don't suppot tar archives, they'll just give you the uncompressed tar in that case.
  
  It's also very common to do that with other extensions (not just .tar) when compressing big files. For example, when archiving logs they'll often be stored as .log.gz, which makes it automatically clear that it's a log file directly compressed with gzip and meant to be examined with tools like zcat and zless to view it.
  
  And in cases like that you really need it to be clear on what data does the gzip stores, since it does not keep metadata about the file so you might not be able to get back the original name/extension of the file if you rename the gz file.

I get your point. Since a .tar.zst file can be handled natively by tar, using .tzst instead does make sense.
There already are conventional abbreviations: see Section 2.1. I doubt they will be better supported by tools though.
- That's much better. Thanks for actually answering the comment, rather than the usual "Windows bad, Linux good, upvotes please"
  
  In this case it really seems this windows convention is bad though. It is uninformative. And abbreviations mandate understanding more file extensions for no good reason. And I say this as primarily a windows user. Hiding file extensions was always a bad idea. It tries to make a simple reduced UI in a place where simple UI is not desirable. If you want a lean UI you should not be handling files directly in the first place.
  
  Example.zip from the other comment is not a compressed .exe file, it's a compressed archive containing the exe file and some metadata. Windows standard tools would be in real trouble trying to understand unarchived compressed files many programs might want to use for logging or other data dumps. And that means a lot of software use their own custom extensions that neither the system nor the user knows what to do with without the original software. Using standard system tools and conventions is generally preferable.
I would argue what windows does with the extensions is a bad idea. Why do you think engineers should do things in favour of these horrible decisions the most insecure OS is designed with?

You've viewed 283 comments.