command line util to encode/decode framed packets?
command line util to encode/decode framed packets?
I'm currently trying to set up a homebrew cassette tape storage format, but trying to use existing tech where possible. I was excited to see that minimodem already exists for converting an audio stream to a byte stream, and is even available in termux for android, so I could decode cassettes with my phone! However, I'd like some sort of higher-level tool to encode and decode "packets" or "slices" so that I can add error correction. I'm sure this sort of thing must exist for amature radio purposes.
I could write a script that cuts a file into slices, with checksums and redundancy for each slice, and then pads them with null bytes so I can isolate each frame when decoding. What I want is to find out if that's already been done. I've heard of AX.25 packets but I can't find a tool that does that with stdio.
The command you're looking for is tape archiver, cunningly called
tar.Holy f… I thought you’re joking but yes tar is indeed a tape archiver
That is what tar stands for. Interestingly enough, tar is based on an older util called ar, which was just a generic archive.
Interesting. Is that some new project?
No, but to learn bring computing to 2024 and to pad my github I'm currently working on a modern replacement written in React that can also use Electron on the desktop. It's pretty light at around 150mb. I don't have plans to natively port it to Linux, but it'll work in Windows Linux and MacOS with a little tweaking.
I do have plans to release the Docker image I've used for testing later this year along with a Snap and Flatpak so I guess it will natively work on Ubuntu.
Oh, and longterm goals are to have an Android and iPhone port. I have an ad provider set up so I can offer the mobile apps for free (gotta hustle) but I'll have a patreon and kofi set up by then for donations and a mostly ad-free version that aggregates usage info of your device and apps but doesn't give away any personally-identifiable details if you opt-out.
I do almost nightly coding streams on Twitch and Youtube but I blur out the code itself because I don't want anyone borrowing my idea and implementation.
You can find me on Insta, TikTok, Facebook and I'll have links to my X (formally known as Twitter), Telegram, Discord, Riot, and Mastodon accts. I also have curated some playlists on Apple Music, Spotify, Tidal, and Amazon Music that include some chillwave tracks as well as some binaural beats, white noise, pink noise, and brown noise (to name a few).
I also have an affiliate link for a microdose provider that gives me a cut everytime you make a purchase which helps me out.
I tried that first! But
tarcomplains if it can't find the file header! So I still need to do some sort of packets. Unless you know some sort of workaround?You're probably decoding noise or in the middle of the bit stream.
What you're looking for is called "preamble." That's a sequence of bits used to synchronize the decoder (marks the start of data, useful in modulation schemes for clock recovery, and a few other things).
Looking at minimodem's manual, try using the sync-byte option. Prepend your tar stream with a string of bytes, like 0x01, before sending to minimodem for encoding. Then use the sync code option to mark the start of the tar bit stream. This is as simple as cat preamble.bin myfiles.tar | minimodem --tx ....
Other things to consider: start small with 300 baud BFSK before speeding up. Test with wav files before attempting physical tape or speakers and a microphone.
Hmm... I think looking at this from a radio perspective isn't helpful. I found more resources when ignoring the media. Perhaps par2 or RAR would be useful? Generate error correction media first, then write to media.
Generally in radio, you could just request a retransmission, so I didn't find much from that angle.
You might also find something useful when looking at tape backup programs. You're not using LTO, but the principles are the same, so maybe there's some tooling that would be compatible.
I did use
par2andtarto generate redundancy, but I still need a way to locate it in the bytestream. Tar doesn't seem to reliably mark the start or end of files :/Tar doesn't natively have an index to immediately seek to files in the tar archive, though I know that it's possible to extend it with an index somehow, because pixz will do a parallel LZMA compression that involves generating and using an index for tar archives.
EDIT: Oh, I think I see what you mean. You're saying that you want to use tar to store the redundancy files, not generate redundancy data for the file as a whole? Like, a tar of PAR2 files? I don't think that that'll work, because you'll want redundancy for tar's metadata too.
EDIT2: So what you want is a single bytestream with forward error correction, not a set of files that provide it.
kagis
It looks like this guy has an implementation, and says that he's using Reed-Solomon, but that it's also just his weekend project, so...shrugs
You may be interested in section 23.4 (non-rewinding tape device)