Support MIX_TARGET

[mayl]

I've been using this and mostly it's been working great, thanks for sharing it!

I've hit kind of an edge case with path dependencies and MIX_TARGET. I've observed that mixRelease builds all the dependencies with the default MIX_TARGET=host even when the main project is building with a different MIX_TARGET. For most of my deps that inconsistency doesn't appear to create an issue but for some reason when I build the main project with the modified MIX_TARGET, my path dependency fails when it tries to find one of it's dependencies (elixir_make in this case). I get roughly the same behavior with or without --include-paths and elixir_make is in the deps.nix, and was built and used by other dependencies successfully so I'm pretty sure I've narrowed it to something about the MIX_TARGET not matching.

Have you seen issues like this before with MIX_TARGET? Is there a convenient way to add an environment variable to every buildMix dep without individually overriding each package? If there was a way to do that, I have a similar use case for building for ARM under emulation on Nixos...

Right now the problem is in a repo I can't share, but if I need to I think I can probably reproduce it.

[camelpunch]

Would global overrides to apply to every built package work here?

[camelpunch]

I guess built-in detection of MIX_TARGET would set that env var for you in all deps. I’ve never worked on an elixir project that uses it so might need some assistance!

[camelpunch]

I think you can set env vars on all packages like this:

pkgs.callPackages ./path/to/deps.nix {
  overrides = (
    final: prev:
    builtins.mapAttrs (
      appName: drv:
      drv.overrideAttrs {
        MIX_TARGET = "mypreferredtarget";
      }
    ) prev
  );
};

Does this suit your use case?

[mayl]

Sorry it's taken me a while to get back to this, things have been busy. I appreciate you looking at this issue.

Thanks for your suggested approach for setting the environment for all packages. I actually patched deps_nix to get MIX_TARGET in all my packages a couple weeks ago, it seemed to get me closer, but didn't fully fix my issue, and I haven't gotten back to it yet.

To be more specific, the situation that's giving me trouble is trying to use Circuits.UART and substitute out it's backend for certain targets with CircuitsSim (based on MIX_TARGET). While I can get it to work with interactively running mix in a devshell, I haven't gotten it to work in a nix build. I think there's some kind of undeclared/circular dependency these packages setup between each other that only becomes a problem when they all aren't compiled at the same time, but I'm still digging into it.

I haven't fully figured out how Mix compilation and configuration works, so maybe this doesn't make sense but I'm starting to suspect there might be a more general issue. deps_nix builds all dependencies separately and passes the config path to each package. But it seems to me that since what config.exs actually evaluates to is partly a function of the OS environment just passing the same config.exs to each dep may not be sufficient to ensure the deps are consistent with the top level mixRelease. It seems like ideally all the environment variables from the top level mixRelease would get also get passed into each dep so that all the configs were consistent.

[camelpunch]

Yeah our team uses a few dependencies that need e.g. access to the app's config at compile time. We tend to link their source code into the expected place in mixRelease's postUnpack phase in that case.

If you have a lot of co-dependent dependencies maybe a FOD would work better for you, or at least be less of a headache.

[mayl]

OK, I think that gives me something to look at. Do you have an (even a redacted) example postUnpack you can share? I'm following what you mean, but I'm not familiar enough with the Mix compilation to know where it is expecting the source code to be linked without some digging.

I've done an FOD before, and I had thought about pivoting back to that but I really like how deps_nix works! As far as I know, this CircuitsSim dependency is the only one which is going to need this special treatment so I'm interested in trying your postUnpack suggestion before I give up on deps_nix

[camelpunch]

Sure. In our case, the Avrora library has a customisable thing that you set up in your app, so it needs to be present when the app compiles.

    postUnpack = ''
      ln -sfv ${mixNixDeps.avrora.src} ${mixNixDeps.avrora.name}
    '';

[mayl]

I've been looking at this on and off, and I think I finally founds something closer to the real issue, which is that setting MIX_TARGET actually changes the directory under _build which mix uses. I've got a PR against Nixpkgs with more details: NixOS/nixpkgs#437325

I think that deps.nix should maybe take mixEnv and mixTarget (they can default to "prod" and "host") that way dependencies which compile differently for different targets/environments, but I still am kind of confused about the role of config.exs when building things with deps.nix. Does that capture those variables somehow?

[camelpunch]

If I'm understanding you correctly, config.exs doesn't by default capture anything - it's just a script that gets invoked. mix.exs lets you declare dependencies for different targets, as I'm sure you're aware.

Different apps and dependencies could do all sorts of nasty things with MIX_TARGET, though.

Have you successfully used different nix packages generated by deps_nix for different deployment targets?

I think I might need walking through this a bit slower, possibly with some concrete examples as it all seems a bit fuzzy.

[mayl]

Yea sorry I realize my communication here has been pretty confusing - there were multiple interacting issues I was contending with that I really didn't understand and was kind of thrashing as I figured them out. So sorry my reports on this are kind of scattered on confusing... it's not you it's definitely me!

So as a concrete example, my project is using circuits_i2c. My default build (MIX_TARGET=host) targets an embedded system where that library connects to the linux i2c driver via a NIF, and won't start properly on a system where that driver is not available. I also build a mixRelease for qemu (MIX_TARGET=qemu) and in that case the real driver is not available so I am setting up a simulated driver with circuits_sim. My project's config.exs has something like this in it, which swaps in the circuits_sim backend so I can run in that environment:

if config_target() == :qemu do
  config :circuits_i2c, default_backend: CircuitsSim.I2C.Backend
  config :circuits_sim,
    config: [
      {CircuitsSim.Device.MCP23008, bus_name: "i2c-0", address: 0x20},
    ]
end

One thing to note is that setting this config changes how circuits_i2c is compiled, it's not runtime only. For this to work properly, mixTarget when evaluating the config.exs inside deps.nix's buildMix call for circuits_i2c has to match mixTarget when I evaluate the overall release.

With the nixpkgs change I just linked which allows individual libraries to build correctly when MIX_TARGET is set, I've been successful building my project using the global override method you described above.

At this point, if I'm suggesting anything I think it's just to make it more ergonomic to at minimum get mixTarget (and mixEnv) into deps.nix, but probably some more generic way of adding extra arguments to mixBuild inside of deps.nix. Right now the only way I know of to get this into my dependencies is to use the override method you showed me above, which works but is not very discoverable/ergonomic.

I think there could be an additional issue, which is can deps_nix make a deps.nix that includes all possible dependencies, regardless of mixEnv and/or mixTarget? I guess including them all could incur more than an optimal amount of compilation, not including them all would mean you could only build certain configurations with a given deps.nix generated file...

[camelpunch]

The former feature, making mixTarget and mixEnv more accessible, sounds totally reasonable and doable.

The latter, of including all possible deps, sounds similar to #27 in that we want the same names to appear in the deps.nix but potentially belonging to different envs and targets. Sounds like the seed of a new feature to namespace packages.

[multivac61]

Getting mixEnv would be very useful, e.g., for getting dependencies needed to run mix format --check-formatted in CI.