Ability to use the dotenv key inside included Taskfiles

[apgrucza]

Currently the dotenv key cannot be used inside included Taskfiles. Doing so gives the following error:

Included Taskfiles can't have dotenv declarations. Please, move the dotenv declaration to the main Taskfile

Suppose I have the below structure, where the root Taskfile includes the Taskfiles in the subdirectories:

.
├── app
│   └── Taskfile.yaml
├── deployment
│   └── Taskfile.yaml
└── Taskfile.yaml

I find it useful to be able to have the choice to run Task from either:

1. the root directory, e.g. task app:build, or
2. the app subdirectory, e.g. task build

I can't do this and also make use of the dotenv feature. If I place dotenv in the root Taskfile, it will be ignored in case 2. If I place dotenv in the included Taskfile, I get an error in case 1. I'd like dotenv declarations in included files to be processed without error. If the same .env file is referenced in multiple included Taskfiles, this should also not produce an error.

This does raise a question about order of precedence when different .env files are specified across different Taskfiles. Perhaps we could have a rule whereby values in a .env file override values from an included .env file. Or if it's simpler, just return an error if different Taskfiles reference different .env files.

[pd93]

@apgrucza This doesn't solve your problem per se, but you could take a look at the problem/solution in #920 to see if the scenario described there could work for you. You may find that it helps to reduce the number of Taskfiles that you need to include in your project too. However, this probably isn't practical if the Taskfiles in your subdirectories are very specific (i.e. cannot be generalised).

[apgrucza]

Thanks for the suggestion @pd93. In my case the Taskfiles in the subdirectories are indeed very specific and so cannot be generalised.

I could of course move all tasks into the root and have just a single Taskfile, but I prefer the separation of concerns that my current structure provides.

[pd93]

Yep, that's understandable. Just thought I'd see if there was a solution that could work for you in the meantime.

Also, see this related PR and comment where this topic has come up before: #904 (comment)

[apgrucza]

Regarding this comment from #904:

In a future PR, we could allow dotenv on included Taskfiles, but that would require a refactor so these env would be scoped in the included Taskfile and not merged into the main Taskfile. In short, dotenv would only affect tasks on its own Taskfile and not all tasks. At least that's how I would expect it to work, but this is open to discussion, and we may want to open an issue about this.

I didn’t show it in my example above, but I also have a common Taskfile that’s included from all other Taskfiles. The common Taskfile defines shared variables, some of which need to be based on content from .env files. So scoping dotenv to individual Taskfiles would not work for me.

#1030 covers the topic of Taskfile-scoped variables, so I’ve added my thoughts in a comment over there.

[bayeslearnerold]

I am running into the same organizational question, see below:

.
├── .env
├── .taskfiles
│   ├── folder-a
│   │   └── Taskfile.yml
│   │   └── .env-2
│   └── folder-b
│       └── Taskfile.yml
├── subprj
│   ├── folder-c
│   │   └── Taskfile.yml
│   │   └── .env-3
│   └── folder-d
│       └── Taskfile.yml
└── Taskfile.yml

1. What are the recommended guidelines for including task files? I've slightly altered the example to clarify my question. Consider a directory named .taskfiles, designed to store common task routines for inclusion in other task files. According to this ticket, it seems these task files should not contain environment variable inclusions. Is that interpretation correct?

2. My goal is to execute tasks both from the project's root and within subdirectories, such as sub-prj/folder-c. Moreover, I want to ensure that a task file in folder-c can incorporate common task modules from the .taskfiles directory. This is permissible, right?

3. However, I encounter a problem when attempting to "include" a task file from folder-c into the root task file. Perhaps this action should not be an inclusion but rather a direct invocation?

[firstmustburn]

FWIW I found a workaround for this that works okay for my use case. I have some variables that I want to be common across all tasks, but I want each folder to have its own Taskfile, and I want them to be usable from a hierarchical include (for rollup tasks) as well as locally from the folder.

With the structure below, I can run the task from either location and it works fine. The secret is putting the tasks in the .actual files
and including them in the local Taskfile and the hierarchical one. This does require 3.40, I had 3.30 and it had not implemented the flatten: true that is used in the subfolder Taskfile.yml.

. $ task dep:deploymenta:tryout
foo
./deployments/deploymenta $ task tryout
foo

and a rollup task:

. $ task tryout-all
Do something big with foo
foo for a
foo for b

├ Taskfile.env <-- variables go here
├ Taskfile.yml <-- main folder taskfile
├─ deployments
│   ├─ Taskfile.yml <-- deployments taskfile - just includes the Taskfile for each deployment
│   ├─ deploymenta
│   │   ├─ Taskfile.yml.actual <-- this has the real tasks in it
│   │   ├─ Taskfile.yml <-- this includes the .actual with the dotenv for use in the folder
│   ├─ deploymentb
│   │   ├─ Taskfile.yml.actual <-- this has the real tasks in it
│   │   ├─ Taskfile.yml <-- this includes the .actual with the dotenv for use in the folder

taskfile.env

REMOTE_STUB_NAME=foo

main folder taskfile

version: '3'
dotenv: [ "Taskfile.env" ]
includes:
  dep: 
    taskfile: ./deployments/Taskfile.yml
    dir: ./deployments
tasks:
  #example of a rollup task 
  tryout-all:
    cmds:
      - echo "Do something big with {{.REMOTE_STUB_NAME}}"
      - task: dep:deployementa:tryout
      - task: dep:deployementb:tryout 

deployments taskfile

version: '3'
includes:
  deploymenta: 
    taskfile: ./deploymenta/Taskfile.yml.actual
    dir: ./deploymenta
  deploymentb: 
    taskfile: ./deploymentb/Taskfile.yml.actual
    dir: ./deploymentb

deploymentsa taskfile.actual

version: '3'
tasks:
  tryout:
    cmds:
      - echo "{{.REMOTE_NAME_STUB}} for a"

deploymentsa Taskfile.yml

version: '3'
includes:
  actual:
    taskfile: Taskfile.yml.actual
    flatten: true
dotenv: ["../../Taskfile.env"]

deploymentsb have the same structure as deployments a

Some notes:

• Any Taskfile that doesn't depend on the Taskfile.env variables can just be a normal taskfile.
  • The deployments taskfile is an example of this. Because it's only function is to do the hierarchical include, there's no Taskfile.yml.actual. If it had tasks of its own that depended on the Taskfile.env variables, they would need to be put into a Taskfile.yml.actual that was also included at that level.
  • The main taskfile is a second example -- nothing includes it, so no .actual needed

[gergely-pepper]

I'm curious, why is this error added like that? As far as I can see, without the error, the included envs would overwrite envs in root one by one... and that's all. Did I miss something? If not, why isn't it allowed?

[vendelin8]

I figured out a very simple workaround. Demo here: https://github.com/vendelin8/task-include-dotenv

[trulede]

I've created a PR which implements this. I don't understand why it was not supported, other than the behaviour is non-deterministic (the order that files are imported, depends on import sequence).

Please try the PR out and offer feedback. Perhaps, we figure out why is not supported?

#2636