[BUG] Jinja2 3.0 support breaking changes

[Oloremo]

Jinja2 3.0 was released a few days ago and seems like Salt can't work with it.

We have issues like this:

    Data failed to compile:'
----------'
    Rendering SLS ''base:formula.logstash.common/install'' failed: Jinja variable ''pillar'' is undefined'
/opt/behavox/salt/cache/minion/files/base/formula/kibana/vars.yaml(2):'
---'
---'
{% set third_party_packages_path = pillar.get(''third_party_packages_path'') %}    <======================'
[...]'

[myii]

As mentioned in Slack:

I'm wondering if this should actually be considered to be a bug rather than a feature request. We've started seeing failures across various formulas and I did a test run to determine that 32 of around 100 formulas are now failing: https://saltstack-formulas.zulipchat.com/#narrow/stream/239693-CI/topic/myii-ci.2F2021-W20d/search/fail.

Here's an example of a PR that fixes the issue: saltstack-formulas/postgres-formula#311. I'm not so sure this will work everywhere, though.

Should we patch our 32 formulas or should we wait for a resolution in the main Salt repo?

[baby-gnu]

I agree this is a bug.

The salt special variables are not passed in successive import.

I made a little test where direct-import is working but not indirect-import:

• /srv/salt/foo/init.sls

# -*- mode: salt; coding: utf-8 -*-
# vim: ft=sls

include:
  - .direct-import
  - .indirect-import

• /srv/salt/foo/direct-import.sls

# -*- mode: salt; coding: utf-8 -*-
# vim: ft=sls

{%- from "foo/libsaltcli.jinja" import cli %}

test-direct-import-of-cli:
  test.nop:
    - name: {{ cli }}

• /srv/salt/foo/indirect-import.sls

# -*- mode: salt; coding: utf-8 -*-
# vim: ft=sls

{%- from "foo/test.jinja" import test_cli %}

test-indirect-import-of-cli:
  test.nop:
    - name: {{ test_cli }}

• /srv/salt/foo/libsaltcli.jinja

# -*- coding: utf-8 -*-
# vim: ft=jinja

{#- Get the relevant values from the `opts` dict #}
{%- set opts_cli = opts.get('__cli', '') %}
{%- set opts_masteropts_cli = opts | traverse('__master_opts__:__cli', '') %}

{#- Determine the type of salt command being run #}
{%- if opts_cli == 'salt-minion' %}
{%-   set cli = 'minion' %}
{%- elif opts_cli == 'salt-call' %}
{%-   set cli = 'ssh' if opts_masteropts_cli in ('salt-ssh', 'salt-master') else 'local' %}
{%- else %}
{%-   set cli = 'unknown' %}
{%- endif %}
{%- do salt['log.debug']('[libsaltcli] the salt command type has been identified to be: ' ~ cli) %}

• /srv/salt/foo/test.jinja

# -*- coding: utf-8 -*-
# vim: ft=jinja

{%- from "foo/libsaltcli.jinja" import cli %}

{%- set test_cli = cli %}

DEBUG log

root@buster:~/foo-template# ./bin/kitchen converge default-debian-10-master-py3
-----> Starting Test Kitchen (v2.7.2)
-----> Creating <default-debian-10-master-py3>...
       Sending build context to Docker daemon  1.527MB
       Step 1/15 : FROM saltimages/salt-master-py3:debian-10
        ---> d7387439734b
       Step 2/15 : RUN [ ! -f "/sbin/initctl" ] || dpkg-divert --local --rename --add /sbin/initctl                 && ln -sf /bin/true /sbin/initctl
        ---> Using cache
        ---> c4a05d347404
       Step 3/15 : ENV DEBIAN_FRONTEND noninteractive
        ---> Using cache
        ---> 334170a53a0b
       Step 4/15 : ENV container docker
        ---> Using cache
        ---> 62cf5721ebcb
       Step 5/15 : RUN if ! getent passwd kitchen; then                   useradd -d /home/kitchen -m -s /bin/bash -p '*' kitchen;                 fi
        ---> Using cache
        ---> 50fabb644481
       Step 6/15 : RUN echo "kitchen ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/kitchen
        ---> Using cache
        ---> f431fed3317b
       Step 7/15 : RUN echo "Defaults !requiretty" >> /etc/sudoers.d/kitchen
        ---> Using cache
        ---> eab969a55238
       Step 8/15 : RUN mkdir -p /home/kitchen/.ssh
        ---> Using cache
        ---> c37241bc8100
       Step 9/15 : RUN chown -R kitchen /home/kitchen/.ssh
        ---> Using cache
        ---> 481d3f040742
       Step 10/15 : RUN chmod 0700 /home/kitchen/.ssh
        ---> Using cache
        ---> b43a294604bf
       Step 11/15 : RUN touch /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 92b0e899f03a
       Step 12/15 : RUN chown kitchen /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 47e6bc8e70f5
       Step 13/15 : RUN chmod 0600 /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 44a65bbab94f
       Step 14/15 : RUN mkdir -p /run/sshd
        ---> Using cache
        ---> 54055f04ed28
       Step 15/15 : RUN echo ssh-rsa\ AAAAB3NzaC1yc2EAAAADAQABAAABAQDtXRBQobr6N22\+mW\+ve6SDKyUJHIyrMYHYw0TxKw7C69E/JxZQuBHklfNHXAEVkvit0f7iptN3cZIOy7rBXd4iWtX0Je6TdTMrPfkxO/cBdaRrcsqdKQPTUSMLmozncNk0VXpI7xL8cfSfqFiggqNgQbuCh/RC/ziXDQPwqZzD9WQD0sF3r6TC\+jAEoP6YWnUqREiTKOSc5UBsPgPKza6vNp\+El8G8Z78pqG6gJ7NIDe4lbXxrSCF2b5m9Cm6QJyy6OFrSCNW2D4hu2/BvHWGT81JBMH3PgXefPbLYIwvK/kREu4jBROFEHMSBirQm1JKpfEubB/mFPXxNbvgMjM1B\ kitchen_docker_key >> /home/kitchen/.ssh/authorized_keys
        ---> Using cache
        ---> 02deffd10007
       Successfully built 02deffd10007
       2c23c82f968ca76ecedd9c1592888a233abbc0989643a5aac6210254895ce0c0
       0.0.0.0:36392
       Waiting for SSH service on localhost:36392, retrying in 3 seconds
       [SSH] Established
       Finished creating <default-debian-10-master-py3> (0m3.94s).
-----> Converging <default-debian-10-master-py3>...
       Preparing files for transfer
       Preparing salt-minion
       Preparing pillars into /srv/pillar
       Preparing formula: foo from /root/foo-template
       Preparing state_top
       Preparing scripts into /etc/salt/scripts
       You asked for latest and you have 3003+0na.82c373d installed, sweet!
       Transferring files to <default-debian-10-master-py3>
       Install External Dependencies
       Content of /tmp/kitchen//srv/salt :
       total 16
       drwxr-xr-x 3 kitchen kitchen 4096 May 18 09:58 .
       drwxr-xr-x 3 kitchen kitchen 4096 May 18 09:58 ..
       drwxr-xr-x 2 kitchen kitchen 4096 May 18 09:58 foo
       -rw-r--r-- 1 kitchen kitchen   25 May 18 09:58 top.sls
       [DEBUG   ] Reading configuration from /tmp/kitchen/etc/salt/minion
       [DEBUG   ] Guessing ID. The id can be explicitly set in /etc/salt/minion
       [DEBUG   ] Found minion id from generate_minion_id(): 2c23c82f968c
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Override  __grains__: <module 'salt.loaded.int.log_handlers.sentry_mod' from '/usr/local/lib/python3.7/dist-packages/salt/log/handlers/sentry_mod.py'>
       [DEBUG   ] Configuration file path: /tmp/kitchen/etc/salt/minion
       [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
       [DEBUG   ] Grains refresh requested. Refreshing grains.
       [DEBUG   ] Reading configuration from /tmp/kitchen/etc/salt/minion
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/local/lib/python3.7/dist-packages/salt/grains/zfs.py'>
       [DEBUG   ] Elapsed time getting FQDNs: 0.11402416229248047 seconds
       [DEBUG   ] LazyLoaded zfs.is_supported
       [DEBUG   ] Determining pillar cache
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded jinja.render
       [DEBUG   ] LazyLoaded yaml.render
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded jinja.render
       [DEBUG   ] LazyLoaded yaml.render
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded state.highstate
       [DEBUG   ] LazyLoaded direct_call.execute
       [DEBUG   ] Override  __grains__: <module 'salt.loaded.int.module.grains' from '/usr/local/lib/python3.7/dist-packages/salt/modules/grains.py'>
       [DEBUG   ] LazyLoaded grains.get
       [DEBUG   ] LazyLoaded saltutil.is_running
       [DEBUG   ] LazyLoaded config.get
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [DEBUG   ] Updating roots fileserver cache
       [DEBUG   ] Gathering pillar data for state run
       [DEBUG   ] Determining pillar cache
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded jinja.render
       [DEBUG   ] LazyLoaded yaml.render
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Finished gathering pillar data for state run
       [INFO    ] Loading fresh modules for state activity
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded jinja.render
       [DEBUG   ] LazyLoaded yaml.render
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] In saltenv 'base', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/top.sls' to resolve 'salt://top.sls'
       [DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
       [DEBUG   ] No dest file found
       [INFO    ] Fetching file from saltenv 'base', ** done ** 'top.sls'
       [DEBUG   ] compile template: /tmp/kitchen/var/cache/salt/minion/files/base/top.sls
       [DEBUG   ] Jinja search path: ['/tmp/kitchen/var/cache/salt/minion/files/base']
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [PROFILE ] Time (in seconds) to render '/tmp/kitchen/var/cache/salt/minion/files/base/top.sls' using 'jinja' renderer: 0.021555423736572266
       [DEBUG   ] Rendered data from file: /tmp/kitchen/var/cache/salt/minion/files/base/top.sls:
       ---
       base:
         "*":
         - foo
       
       [DEBUG   ] Results of YAML rendering: 
       OrderedDict([('base', OrderedDict([('*', ['foo'])]))])
       [PROFILE ] Time (in seconds) to render '/tmp/kitchen/var/cache/salt/minion/files/base/top.sls' using 'yaml' renderer: 0.00038504600524902344
       [DEBUG   ] LazyLoaded confirm_top.confirm_top
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded compound_match.match
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] compound_match: 2c23c82f968c ? *
       [DEBUG   ] LazyLoaded glob_match.match
       [DEBUG   ] compound_match 2c23c82f968c ? "*" => "True"
       [DEBUG   ] LazyLoaded saltutil.sync_all
       [DEBUG   ] Syncing all
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/clouds'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing clouds for environment 'base'
       [INFO    ] Loading cache from salt://_clouds, for base
       [INFO    ] Caching directory '_clouds/' for environment 'base'
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_clouds'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/beacons'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing beacons for environment 'base'
       [INFO    ] Loading cache from salt://_beacons, for base
       [INFO    ] Caching directory '_beacons/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=0 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_beacons'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/modules'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing modules for environment 'base'
       [INFO    ] Loading cache from salt://_modules, for base
       [INFO    ] Caching directory '_modules/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=0 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_modules'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/states'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing states for environment 'base'
       [INFO    ] Loading cache from salt://_states, for base
       [INFO    ] Caching directory '_states/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=0 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_states'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/sdb'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing sdb for environment 'base'
       [INFO    ] Loading cache from salt://_sdb, for base
       [INFO    ] Caching directory '_sdb/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=0 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_sdb'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/grains'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing grains for environment 'base'
       [INFO    ] Loading cache from salt://_grains, for base
       [INFO    ] Caching directory '_grains/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_grains'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/renderers'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing renderers for environment 'base'
       [INFO    ] Loading cache from salt://_renderers, for base
       [INFO    ] Caching directory '_renderers/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_renderers'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/returners'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing returners for environment 'base'
       [INFO    ] Loading cache from salt://_returners, for base
       [INFO    ] Caching directory '_returners/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_returners'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/output'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing output for environment 'base'
       [INFO    ] Loading cache from salt://_output, for base
       [INFO    ] Caching directory '_output/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_output'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/utils'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing utils for environment 'base'
       [INFO    ] Loading cache from salt://_utils, for base
       [INFO    ] Caching directory '_utils/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_utils'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/log_handlers'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing log_handlers for environment 'base'
       [INFO    ] Loading cache from salt://_log_handlers, for base
       [INFO    ] Caching directory '_log_handlers/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_log_handlers'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/executors'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing executors for environment 'base'
       [INFO    ] Loading cache from salt://_executors, for base
       [INFO    ] Caching directory '_executors/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_executors'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/proxy'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing proxy for environment 'base'
       [INFO    ] Loading cache from salt://_proxy, for base
       [INFO    ] Caching directory '_proxy/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_proxy'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/engines'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing engines for environment 'base'
       [INFO    ] Loading cache from salt://_engines, for base
       [INFO    ] Caching directory '_engines/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_engines'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/thorium'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing thorium for environment 'base'
       [INFO    ] Loading cache from salt://_thorium, for base
       [INFO    ] Caching directory '_thorium/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_thorium'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/serializers'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing serializers for environment 'base'
       [INFO    ] Loading cache from salt://_serializers, for base
       [INFO    ] Caching directory '_serializers/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_serializers'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/matchers'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing matchers for environment 'base'
       [INFO    ] Loading cache from salt://_matchers, for base
       [INFO    ] Caching directory '_matchers/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_matchers'
       [INFO    ] Creating module dir '/tmp/kitchen/var/cache/salt/minion/extmods/pillar'
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded roots.envs
       [DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
       [INFO    ] Syncing pillar for environment 'base'
       [INFO    ] Loading cache from salt://_pillar, for base
       [INFO    ] Caching directory '_pillar/' for environment 'base'
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Local cache dir: '/tmp/kitchen/var/cache/salt/minion/files/base/_pillar'
       [DEBUG   ] Refreshing modules...
       [INFO    ] Loading fresh modules for state activity
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded jinja.render
       [DEBUG   ] LazyLoaded yaml.render
       [DEBUG   ] Returning file list from cache: age=1 cache_time=20 /tmp/kitchen/var/cache/salt/minion/file_lists/roots/base.p
       [DEBUG   ] Could not find file 'salt://foo.sls' in saltenv 'base'
       [DEBUG   ] In saltenv 'base', looking at rel_path 'foo/init.sls' to resolve 'salt://foo/init.sls'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/foo/init.sls' to resolve 'salt://foo/init.sls'
       [DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://foo/init.sls'
       [DEBUG   ] No dest file found
       [INFO    ] Fetching file from saltenv 'base', ** done ** 'foo/init.sls'
       [DEBUG   ] compile template: /tmp/kitchen/var/cache/salt/minion/files/base/foo/init.sls
       [DEBUG   ] Jinja search path: ['/tmp/kitchen/var/cache/salt/minion/files/base']
       [PROFILE ] Time (in seconds) to render '/tmp/kitchen/var/cache/salt/minion/files/base/foo/init.sls' using 'jinja' renderer: 0.0010530948638916016
       [DEBUG   ] Rendered data from file: /tmp/kitchen/var/cache/salt/minion/files/base/foo/init.sls:
       # -*- mode: salt; coding: utf-8 -*-
       # vim: ft=sls
       
       include:
         - .direct-import
         - .indirect-import
       
       [DEBUG   ] Results of YAML rendering: 
       OrderedDict([('include', ['.direct-import', '.indirect-import'])])
       [PROFILE ] Time (in seconds) to render '/tmp/kitchen/var/cache/salt/minion/files/base/foo/init.sls' using 'yaml' renderer: 0.00033926963806152344
       [DEBUG   ] In saltenv 'base', looking at rel_path 'foo/direct-import.sls' to resolve 'salt://foo/direct-import.sls'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/foo/direct-import.sls' to resolve 'salt://foo/direct-import.sls'
       [DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://foo/direct-import.sls'
       [DEBUG   ] No dest file found
       [INFO    ] Fetching file from saltenv 'base', ** done ** 'foo/direct-import.sls'
       [DEBUG   ] compile template: /tmp/kitchen/var/cache/salt/minion/files/base/foo/direct-import.sls
       [DEBUG   ] Jinja search path: ['/tmp/kitchen/var/cache/salt/minion/files/base']
       [DEBUG   ] In saltenv 'base', looking at rel_path 'foo/libsaltcli.jinja' to resolve 'salt://foo/libsaltcli.jinja'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/foo/libsaltcli.jinja' to resolve 'salt://foo/libsaltcli.jinja'
       [DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://foo/libsaltcli.jinja'
       [DEBUG   ] No dest file found
       [INFO    ] Fetching file from saltenv 'base', ** done ** 'foo/libsaltcli.jinja'
       [DEBUG   ] LazyLoaded log.debug
       [DEBUG   ] [libsaltcli] the salt command type has been identified to be: local
       [PROFILE ] Time (in seconds) to render '/tmp/kitchen/var/cache/salt/minion/files/base/foo/direct-import.sls' using 'jinja' renderer: 0.012192964553833008
       [DEBUG   ] Rendered data from file: /tmp/kitchen/var/cache/salt/minion/files/base/foo/direct-import.sls:
       # -*- mode: salt; coding: utf-8 -*-
       # vim: ft=sls
       
       test-direct-import-of-cli:
         test.nop:
           - name: local
       
       [DEBUG   ] Results of YAML rendering: 
       OrderedDict([('test-direct-import-of-cli', OrderedDict([('test.nop', [OrderedDict([('name', 'local')])])]))])
       [PROFILE ] Time (in seconds) to render '/tmp/kitchen/var/cache/salt/minion/files/base/foo/direct-import.sls' using 'yaml' renderer: 0.00030684471130371094
       [DEBUG   ] In saltenv 'base', looking at rel_path 'foo/indirect-import.sls' to resolve 'salt://foo/indirect-import.sls'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/foo/indirect-import.sls' to resolve 'salt://foo/indirect-import.sls'
       [DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://foo/indirect-import.sls'
       [DEBUG   ] No dest file found
       [INFO    ] Fetching file from saltenv 'base', ** done ** 'foo/indirect-import.sls'
       [DEBUG   ] compile template: /tmp/kitchen/var/cache/salt/minion/files/base/foo/indirect-import.sls
       [DEBUG   ] Jinja search path: ['/tmp/kitchen/var/cache/salt/minion/files/base']
       [DEBUG   ] In saltenv 'base', looking at rel_path 'foo/test.jinja' to resolve 'salt://foo/test.jinja'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/foo/test.jinja' to resolve 'salt://foo/test.jinja'
       [DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://foo/test.jinja'
       [DEBUG   ] No dest file found
       [INFO    ] Fetching file from saltenv 'base', ** done ** 'foo/test.jinja'
       [DEBUG   ] In saltenv 'base', looking at rel_path 'foo/libsaltcli.jinja' to resolve 'salt://foo/libsaltcli.jinja'
       [DEBUG   ] In saltenv 'base', ** considering ** path '/tmp/kitchen/var/cache/salt/minion/files/base/foo/libsaltcli.jinja' to resolve 'salt://foo/libsaltcli.jinja'
       [ERROR   ] Rendering exception occurred
       Traceback (most recent call last):
         File "/usr/local/lib/python3.7/dist-packages/salt/utils/templates.py", line 497, in render_jinja_tmpl
           output = template.render(**decoded_context)
         File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 1289, in render
           self.environment.handle_exception()
         File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 924, in handle_exception
           raise rewrite_traceback_stack(source=source)
         File "<template>", line 4, in top-level template code
         File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 1394, in make_module
           return TemplateModule(self, ctx)
         File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 1524, in __init__
           body_stream = list(template.root_render_func(context))  # type: ignore
         File "/tmp/kitchen/var/cache/salt/minion/files/base/foo/test.jinja", line 4, in top-level template code
           {%- from "foo/libsaltcli.jinja" import cli %}
         File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 1394, in make_module
           return TemplateModule(self, ctx)
         File "/usr/local/lib/python3.7/dist-packages/jinja2/environment.py", line 1524, in __init__
           body_stream = list(template.root_render_func(context))  # type: ignore
         File "/tmp/kitchen/var/cache/salt/minion/files/base/foo/libsaltcli.jinja", line 5, in top-level template code
           {%- set opts_cli = opts.get('__cli', '') %}
         File "/usr/local/lib/python3.7/dist-packages/jinja2/sandbox.py", line 326, in getattr
           value = getattr(obj, attribute)
       jinja2.exceptions.UndefinedError: 'opts' is undefined
       
       During handling of the above exception, another exception occurred:
       
       Traceback (most recent call last):
         File "/usr/local/lib/python3.7/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
           output = render_str(tmplstr, context, tmplpath)
         File "/usr/local/lib/python3.7/dist-packages/salt/utils/templates.py", line 504, in render_jinja_tmpl
           raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
       salt.exceptions.SaltRenderError: Jinja variable 'opts' is undefined
       /tmp/kitchen/var/cache/salt/minion/files/base/foo/libsaltcli.jinja(5):
       ---
       # -*- coding: utf-8 -*-
       # vim: ft=jinja
       
       {#- Get the relevant values from the `opts` dict #}
       {%- set opts_cli = opts.get('__cli', '') %}    <======================
       {%- set opts_masteropts_cli = opts | traverse('__master_opts__:__cli', '') %}
       
       {#- Determine the type of salt command being run #}
       {%- if opts_cli == 'salt-minion' %}
       {%-   set cli = 'minion' %}
       [...]
       ---
       [CRITICAL] Rendering SLS 'base:foo.indirect-import' failed: Jinja variable 'opts' is undefined
       /tmp/kitchen/var/cache/salt/minion/files/base/foo/libsaltcli.jinja(5):
       ---
       # -*- coding: utf-8 -*-
       # vim: ft=jinja
       
       {#- Get the relevant values from the `opts` dict #}
       {%- set opts_cli = opts.get('__cli', '') %}    <======================
       {%- set opts_masteropts_cli = opts | traverse('__master_opts__:__cli', '') %}
       
       {#- Determine the type of salt command being run #}
       {%- if opts_cli == 'salt-minion' %}
       {%-   set cli = 'minion' %}
       [...]
       ---
       [DEBUG   ] Using pkg_resources to load entry points
       [DEBUG   ] LazyLoaded highstate.output
       local:
           Data failed to compile:
       ----------
           Rendering SLS 'base:foo.indirect-import' failed: Jinja variable 'opts' is undefined
       /tmp/kitchen/var/cache/salt/minion/files/base/foo/libsaltcli.jinja(5):
       ---
       # -*- coding: utf-8 -*-
       # vim: ft=jinja
       
       {#- Get the relevant values from the `opts` dict #}
       {%- set opts_cli = opts.get('__cli', '') %}    <======================
       {%- set opts_masteropts_cli = opts | traverse('__master_opts__:__cli', '') %}
       
       {#- Determine the type of salt command being run #}
       {%- if opts_cli == 'salt-minion' %}
       {%-   set cli = 'minion' %}
       [...]
       ---
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <default-debian-10-master-py3>.  Please see .kitchen/logs/default-debian-10-master-py3.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

[baby-gnu]

The salt special variables are not passed in successive import.

It works if I modify /srv/salt/foo/test.jinja like this:

--- foo/test.jinja.orig	2021-05-18 12:07:27.592339605 +0200
+++ foo/test.jinja	2021-05-18 12:06:25.054300769 +0200
@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # vim: ft=jinja
 
-{%- from "foo/libsaltcli.jinja" import cli %}
+{%- from "foo/libsaltcli.jinja" import cli with context %}
 
 {%- set test_cli = cli %}

[baby-gnu]

This is a major change since we may want to have the salt special variables even without context like in saltstack-formulas/template-formula#230 (comment)

[myii]

For the Formulas' team, we're rebuilding our CI images to force the downgrade of Jinja2 from 3.0.x to 2.11.3. That's got our CI working again. Obviously, this is only a workaround until the result of this feature request (bug report?) is clear.

[ggiesen]

If/when this gets addressed, could we get #35398 addressed at the same time?