9mo ago

[solved] Is there a way to automatically import all .nix files in a directory?

My solution:

nix

    
let

  nixFilesInDirectory = directory:
    (
      map (file: "${directory}/${file}")
      (
        builtins.filter
          (
            nodeName:
              (builtins.isList (builtins.match ".+\.nix$" nodeName)) &&
              # checking that it is NOT a directory by seeing
              # if the node name forcefully used as a directory is an invalid path
              (!builtins.pathExists "${directory}/${nodeName}/.")
          )
          (builtins.attrNames (builtins.readDir directory))
      )
    );

  nixFilesInDirectories = directoryList:
    (
      builtins.concatMap
        (directory: nixFilesInDirectory directory)
        (directoryList)
    );
  # ...
in {
  imports = nixFilesInDirectories ([
      "${./programs}"
      "${./programs/terminal-niceties}"
  ]);
  # ...
}

snippet from the full source code: quazar-omega/home-manager-config (L5-L26)

credits:

base script: comment on "getting all configs from folder" (Reddit)
Started developing from that piece that implements the general idea with only builtin functions, so I tried as best I could to stick to the builtins
isDir: nixpkgs (GitHub)
Used to filter out directories from the items to be included

I'm trying out Nix Home Manager and learning its features little by little.
I've been trying to split my app configurations into their own files now and saw that many do the following:

Make a directory containing all the app specific configurations:

 undefined

    
programs/
└── helix.nix

Make a catch-all file default.nix that selectively imports the files inside:

 undefined

    
programs/
├── default.nix
└── helix.nix

Content:

nix

    
{
  imports = [
    ./helix.nix
  ];
}

Import the directory (picking up the default.nix) within the home-manager configuration:

nix

    
{
  # some stuff...
  imports = [
    ./programs
  ];
 # some other stuff...
}

I'd like to avoid having to write each and every file I'll create into the imports of default.nix, that kinda defeats the point of separating it if I'll have to specify everything anyway, so is there a way to do so? I haven't found different ways to do this in various Nix discussions.

Example I'm looking at: https://github.com/fufexan/dotfiles/blob/main/home/terminal/default.nix

My own repository: https://codeberg.org/quazar-omega/home-manager-config

14 comments

I do have a function that I cribbed from somewhere, unfortunately I didn't note down the source at the time so can't give them due credit here.
filterNixFiles = k: v: v == "regular" && lib.hasSuffix ".nix" k;
importNixFiles = path: (lib.lists.forEach (lib.mapAttrsToList (name: _: path + ("/" + name)) (lib.filterAttrs filterNixFiles (builtins.readDir path)))) import;
Then used as normal
modules = [ home-manager.nixosModules.home-manager vscode-server.nixosModule ] ++ (importNixFiles ./hosts/required) ++ (importNixFiles ./hosts/machines/${hostname})
Edit
Apologies for formatting on mobile
- Haha, I read this just after trying to figure it out on my own and it seems the solution is kind of similar to what I got, but this looks like it's probably more robust since it extracts it all into a function (though I have yet to understand the syntax 👀). So thanks, I'll be trying that as well!

https://gist.github.com/yunfachi/6c073cc6edc18f78cfc60bb9bb3f7143
- I've analyzed the script a bit (..ok for more than 2 hours + 2 of refactoring), because the first time I absolutely didn't understand it, now I've got it, but still, I won't ever understand, why make the syntax so confusing?
  My system definition shouldn't be a codegolfing competition (•ˋ _ ˊ•)
  TL;DR: I liked your script as a challenge to learn even more and I'm glad I did! Now I know a quite a bit more about the functions that Nix provides and how to use them, not a lot, but better than my previous almost 0, so thank you!
  
  There were multiple problems with this solution as I tried it:
  The missing baseName on line 39 which was needed to actually filter out the file path of the script that is being ran, because the paths I got out of readDir were relative (just for me? Did I change something in my environment? I'm not usre, the docs aren't very clear)
  It doesn't filter out files that are not .nix files
  It doesn't filter out directories (may be intentional though, I personally don't think that's a great idea as far as I got)
  I'll post later my own further improved solution starting from my own (tbh, by now more like our) script.

At scale, you'll appreciate explicitly spelling out your imports. I currently have 23 importable files, of which two are mutually incompatible (headless vs. Xorg). I don't want a glob over these files because no machine can have all of them; indeed, most machines only have like five imports from the list.
What might be more interesting to you is a common collection of modules which must be imported everywhere. To achieve this, I explicitly declare a commonModules at the top of my flake and reuse it in each machine definition. Another approach might be a common.nix module which recursively contains the common modules as its own imports.
Finally, it doesn't "defeat[] the point of separating" expressions into multiple files to avoid globbing over them. Because NixOS/HM modules are monoidal, they often factor nicely. When you have a dozen different services, you could stuff all of them into one file with one networking.firewall.allowedTCPPorts if you wanted, or you could put each service into its own file and let each module bring its own port to the combined configuration. The latter is easier at scale; I have nine modules declaring TCP ports and five machine-specific TCP ports as well, and it would be a pain to put all of them in one location.
- Thanks for the input! I figured there would be a reason why nobody seems to be doing it, but I still struggle to understand, at least for my current use case.
  What I'm trying to achieve for now is a solid configuration for my own user on any machine, I'm not trying to (and can't) manage my own system currently as I'm using Fedora Kinoite as the host with only the Nix package manager installed. For now I haven't had the chance to make machine specific configurations but I'm wondering, if on top of how it works now, we could write something like imports = [ ./programs/* ] and have all Nix files in that directory be imported, wouldn't that be a good feature to have? Maybe you do have multiple machines, but maybe you also have several directories from where you will want to import everything regardless of the machine, sure you could make just one file for those if you're not going to make distinctions, but I don't want to put everything in one file because it would just get huge, whereas several files that do one thing are just easier to reason about to me.
  common collection of modules which must be imported everywhere
  That sounds interesting, do you have any examples I can refer to to know how to do that?
  Because NixOS/HM modules are monoidal, they often factor nicely.
  What does that mean exactly? I'm not really knowledgeable about functional programming, though that plus the rest of paragraph makes me think of how definitions are "composable" (maybe not the right word) in the sense that you can append and override options that are defined in other imported files without nullifying what was defined in them, is that it?
  
  Oh, right, monoids! Yes, you understand correctly.
  A monoid is a collection of objects that has some sort of addition and zero. (Depending on your maths background, it might equivalently have some sort of multiplication and unit.) Addition must be associative, and addition with zero must not have any effect. Monoids let us think of a system as built from a sequence of operations; each operation adds to the system, preparing its state incrementally.
  Sometimes monoids are commutative, which means that the order of additions is irrelevant to the result. Commutative monoids let us think of a system as built from a collection of operations without worrying about the order in which those operations are applied.
  NixOS modules (and HM modules, etc.) are commutative monoids. The zero is {}. The module system lets options declare their own monoids which ride along, like my example of allowedTCPPorts. Because we can combine sets of port numbers (with set union) and get more sets, we can factor a set of ports into many smaller subsets and put each one in their own file. Here's my shortest module, for an internal Docker registry, docker-registry.nix:
  nix
  
  { networking.firewall.allowedTCPPorts = [ 5000 ]; services.dockerRegistry = { enable = true; enableGarbageCollect = true; }; }
  
  I'm adding some code snippets from my homelab's flake. Minor details are changed. Note how I have a core.nix and also separate files for adding Avahi (zeroconf) and SSH, and for fixing bufferbloat. I could have them as one file, but it's easier to come back to them after several years this way. (bufferbloat.nix was last changed in December 2021, for example.)
  I know that some of this code style probably seems weird. Think of it as heavily inspired by Puppet, Chef, Ansible, HCL, etc.; when we are configuring a system, it is very very nice to be able to comment out a single line at a time.

There's the readDir builtin, but I expect nix might complain if you use that
- Oh this is actually great! Searching it up a bit I came across this old Reddit thread that has a solution, so I applied it to my own config which first got me this:
  nix
  
  { # ... imports = builtins.map (file: "${./programs}/${file}") (builtins.attrNames (builtins.readDir ./programs)); # ... }
  Then I wanted to refine it to only import .nix files (very much trial and error generated code, I don't even know what I'm doing, but it somehow works lmao):
  nix
  
  { # ... imports = builtins.map (file: "${./programs}/${file}") (builtins.filter (file: (builtins.isList (builtins.match ".+\.nix$" file))) (builtins.attrNames (builtins.readDir ./programs))); # ... }
  Thanks for the tip, gotta say I'm very happy! Though I'm wondering how could there not be a simpler solution, it seems like such a common/basic thing to want, maybe what I'm doing is actually undesirable for some reason I'm too noob to understand?

Thanks again, everyone!!!
btw, @samae@lemmy.menf.in, just updated my post with my solution as promised :)

14 comments