I've been looking around for a scripting language that:
has a cli interpreter
is a "general purpose" language (yes, awk is touring complete but no way I'm using that except for manipulating text)
allows to write in a functional style (ie. it has functions like map, fold, etc and allows to pass functions around as arguments)
has a small disk footprint
has decent documentation (doesn't need to be great: I can figure out most things, but I don't want to have to look at the interpter source code to do so)
has a simple/straightforward setup (ideally, it should be a single executable that I can just copy to a remote system, use to run a script and then delete)
Do you know of something that would fit the bill?
Here's a use case (the one I run into today, but this is a recurring thing for me).
For my homelab I need (well, want) to generate a luhn mod n check digit (it's for my provisioning scripts to generate synchting device ids from their certificates).
I couldn't find ready-made utilities for this and I might actually need might a variation of the "official" algorithm (IIUC syncthing had a bug in their initial implementation and decided to run with it).
I don't have python (or even bash) available in all my systems, and so my goto language for script is usually sh (yes, posix sh), which in all honestly is quite frustrating for manipulating data.
I've looked into this a lot actually. There see many options. I'll highlight the pros and cons of each option.
Lua: extremely lightweight, but standard library is lacking, and doesn't include stuff like map or fold. But that would be easy to fix.
Python: thicc standard library, but is not lightweight by any means. There are modifications made to be more shell like, such as xonsh
Rash: based on scheme, very much functional but if you're not used to lisp style, might take a bit to get used to it. This is actually my favorite option. It has a cli interpreter, and really pleasant to use. Cons is... Well it's not very common
You can honestly use any language. Even most compiled languages have a way to run immediately.
Not quite a scripting language, but I highly recommend you check out cosmo for your usecase. Cosmopolitan, and/or Actually Portable Executable (APE for short) is a project to compile a single binary in such a way that is is extremely portable, and that single binary can be copied across multiple operating systems and it will still just run. It supports, windows, linux, mac, and a few BSD's.
another vote for Lua – lua5.4 is available for all 8 Alpine architectures, tiny installed size (120–200 kB) (and Alpine package only installs two files)
Perl is already installed on most linux machines and unless you start delving into module usage, you won't need to install anything else.
Python is more fashionable, but needs installing on the host and environments can get complicated. I don't think it scales as well as Perl, if that's a concern of yours.
Technically, you could bundle a Perl script with the interpreter on another system using pp and run the packed version on systems with no installed Perl, but at that point you might as well just use a compiled language.
Why aren't python and bash be available in all your systems? Which languages would be?
I would've recommended python, otherwise perl or Haskell (maybe Haskell's too big) or something, but now I'm worried that whatever reason makes python undoable also makes perl etc. undoable
I don't have python (or even bash) available in all my systems, and so my goto language for script is usually sh (yes, posix sh), which in all honestly is quite frustrating for manipulating data.
Why are you making it hard on yourself? apt/dnf install a language to use and use it.
luajit is small, fast(well, it can jit), and has a small but complete standard library and can do FFI pretty easily, should be ideal for most homelab usecase
There's always nushell. It's fairly new, not quite to 1.0 yet (0.96.1 at time of writing), but the constant breaking changes seemed to have stopped. It hits all your points and it's quite fun to use when writing scripts. Bonus that it's also pretty much tailor-made to manipulate data.
Why does it need to be a scripting (by this I assume interpreted) language? For your requirements - particularly lightweight distribution - a precompiled binary seems more appropriate. Maybe look into Go, which is a pretty simple language that can be easily compiled to native binaries.
I would go with Guile, because it is built-in to the Guix Package Manager which is a really good general-purpose package manager.
It ticks several of your boxes:
has a CLI interpreter
is a general purpose language, Scheme, amd compliant with revisions 5, 6, and 7 of the language standard
allows writing in a functional style (it is one of the original functional programming languages)
small disk footprint, but still large enough to be "batteries included"
decent documentation, especially if you use Emacs
simple setup: not so much, unless you are using Guix to begin with. The standard distribution ships with lots of pre-built bytecode files, you need an installer script to install everything.
It also has pretty good libraries for system maintenance and reporting:
I don't know if it matches your desire for easy install of small disk space, but it might make up for it in other arenas - Ruby is my new-found love when making simple scripts. Being able to mostly emulate the shell integration that bash has by just using backticks to call a shell command is the killer feature in my book.
The smallest footprint for an actual scripting probably will be posix sh - since you already have it ready.
A slightly bigger footprint would be Python or Lua.
If you can drop your requirement for actual scripting and are willing to add a compile step, Go and it's ecosystem is pretty dang powerful and it's really easy to learn for small automation tasks.
Personally, with the requirement of not adding too much space for runtimes, I'd write it in go. You don't need a runtime, you can compile it to a really small zero dependency lib and you have clean and readable code that you can extend, test and maintain easily.
In most scripting languages you have the interpreter binary and the (standard) libraries as separate files. But creating self-extracting executables, that clean up after themselves can easily be done by wrapping them in a shell script.
IMO, if low dependencies and small size is really important, you could also just write your script in a low level compiled language (C, Rust, Zig, ...), link it statically (e.g. with musl) and execute that.
Realistically whatever problems you see in python will be there for any other language. Python is the most ubiquitously available thing after bash for a reason.
Also you mentioned provisioning scripts, is that Ansible? If so python is already there, if you mean really just bash scripts I can tell you that does not scale well. Also if you already have some scriptsz what language are they on? Why not write the function there?
Also you're running syncthing on these machines, I don't think python is larger than that (but I might be wrong).
I can’t really think of anything that’s less frustrating than sh and ticks all your boxes. You can try TCL but it’s bound to be a shit show.
It was painful to use two decades ago.
Perl is a step up in terms of developer comfort, but it’s at the same time too big and too awkward to use.
A scripting language written in Rust would certainly fulfill you requirement of only needing to copy one file since they are always statically linked and you can even statically compile against musl so it will work on any Linux system without needing a correct libc. Maybe check out rhai.
Maybe something like Elvish or Nushell could be worth a look. They have a lot of similarities to classic shells like bash, but an improved syntax and more powerful features. Basically something in between bash and Python. Not sure about disk footprint or general availability/portability though
If you are interested in tiny lisp like languages, this gitlab could be of interest to you.
Full disclaimer, I came across it a few years back as I am the maintainer of arkscript (which will get faster and better in the v4, so that data about it there is accurate as a baseline of « it should be at least this good but can be even better now ».
Elixir checks most of those boxes. If you want a good functional scriptibg language, Elixir soynds like the go to. Some lisp language like guile should also be sufficient, and probably have a lighter footprint.
This requirement stands out though:
has a simple/straightforward setup (ideally, it should be a single executable that I can just copy to a remote system, use to run a script and then delete)
Thats basically what ansible does. If you plan on doing this to multiple machines you should just use ansible. Also how do you plan on ensuring the scripting interpreter is installed on the machines?
Tried bash, Make, and awk/sed. All hit brick walls. Finally landed on pyinvoke. Two dependencies to install on any new machine. Never had problems. Also, easy to debug and modify as projects evolve.
As for languages intended to be run as a script first and foremost, consider powershell. It's object oriented (and not, like other script languages, just serialising/deserialising everything to JSON under the hood). The syntax takes some time getting used to, but it's cross-platform, quite powerful, and has very good editor support.
Something I haven't seen here yet, but may be worth considering: several programming languages support invoking the compiler to run a source file rather than compile it into a binary.
Go may be worth trying as a Python replacement. It's strictly a programming language, but the language has been written to make it fast to compile. For simple scripts, there's not much difference between the startup time for Python and a compiler invocation of Go.
If you want a more functional programming, Kotlinscript may be to your liking (though that comes with a rather large JVM+compiler dependency that's not very portable). Kotlinscript is basically Kotlin (the programming language) executed like a script by putting the Kotlin compiler in the hashbang.
Similarly, Java can these days also be executed like a script if you invoke it as java some-script.java. Using Java like this doesn't allow for importing dependencies, though, which you may want if you ever need to process JSON.
These compilers aren't single binaries, but they are available as normal OS packages on most modern distros.
If you want to optimisme for "I want to copy a statically linked file to an alpine container", maybe look into fish or zsh.
For some types of data manipulation, PHP may be a good fit. It's not just for web development, although many of its more optimised features are designed for that. Useful features include clear iterations over maps/dictionaries, quite strong static types it you bother to put them in the script, and lots of examples of how to accomplish something online.
Typescript can also be invoked directly these days, and it's even better than PHP (and arguably most programming languages) when it comes to types. I find the time it takes for the JS VM to initialise frustrating, but maybe more modern interpreters such as deno do better in this regard.
Perl would be my candidate for more advanced text handling than what sh can do.
Never used Lua but I think it's fun.
If nothing else works, just learn C/Rust. There's plenty of that on Linux systems, I think you'll be able to manage. Yes, it doesn't meet a lot of your requirements.
I used Python for 15 years or so until they changed from v2 to v3. At that point I realised I couldn't understand my old code because it lacked types, so I got discouraged with that. So rather than learn v3 I stopped using it.
Perl is a disaster. sh is good for shell scripts but let's not stretch it.
TypeScript can use all the JS libraries and runs on node which is supported by all sorts of platforms. Yes there are a few holes in the type safety, so don't do that.
The internet is full of "how to do X in JS". You can read them and add the types you need.
JavaScript through Node.js, or TypeScript through Deno if you like typed languages. They both check all your boxes (just check the size of the executables to make sure that it's what you would consider "small footprint").
Both languages and runtimes are quite popular, so you will find any answers on StackOverflow.
They are both single-executable with little dependencies, and Deno can also compile your scripts to self-contained executables.
As a bonus, both support the vast and extensive NPM package repository where you can find all sort of libraries for even the most complex tasks.
And they work with your favourite IDE or editor, not just syntax highlighting, but also contextual suggestions.
It is possible to wrap something like python into a single file, which is extracted (using standard shell tools) into a tmpdir at runtime.
You might also consider languages that can compile to static binaries - something like nim (python like syntax), although you could also make use of nimscript. Imagine nimscript as your own extensible interpreter.
Similarly, golang has some extensible scripting languages like https://github.com/traefik/yaegi - go has the advantage of easy cross compiling if you need to support different machine architectures.
Bro seriously just slap pyenv + pyenv-virtualenv on your systems and you’re good to go. They’re absolutely trivial to install. Iirc the latter is not a thing in windows, but if you’re stuck on windows for some reason and doing any serious scripting, you should be using WSL anyways.
My go to for most of what you mention is Go, but that’s obviously a compiled language and not for scripting. Or is it - What do you think about https://github.com/traefik/yaegi, which provides an interpreter and REPL for Go? It would let you use a performant and well documented language in a more portable scripting way, but not preclude you from generating statically linked binaries if and when that’s convenient.
perl might be on all your systems. It’s kind-of a legacy, but still actively developed. It’s not a great language: it looks like bash scripting on steroids. But if you just need to write some small scripts with a language more powerful than awk or bash, it does the job. If perl isn’t on all of your systems already, then I would choose a better scripting language.