2019.2.1: Jinja from import broken

[awerner]

Description of Issue

Starting with salt version 2019.2.1, the following statement is broken:

{% from "formula/map.jinja" import defaults with context %}

This is the error that occurs:

           Rendering SLS 'base:formula' failed: Jinja error: formula/map.jinja
       Traceback (most recent call last):
         File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
           output = template.render(**decoded_context)
         File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
           return self.environment.handle_exception(exc_info, True)
         File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
           reraise(exc_type, exc_value, tb)
         File "<template>", line 1, in top-level template code
         File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 189, in get_source
           raise TemplateNotFound(template)
       TemplateNotFound: formula/map.jinja
       
       ; line 1
       
       ---
       {% from "formula/map.jinja" import defaults with context %}    <======================

The same setup is working with 2019.2.0.

Setup

formula/map.jinja:

{% set defaults = {'foo': 'bar'} %}

formula/init.sls:

{% from "formula/map.jinja" import defaults with context %}

salt-call --local state.apply formula

Steps to Reproduce Issue

This happens for us using our kitchen-salt based setup. So only salt-call and salt-minion are installed in the vms.

Versions Report

Salt Version:
           Salt: 2019.2.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Also happens with Debian 8, Debian 9, Ubuntu 16.04 LTS, Ubuntu 18.04 LTS.

[lomeroe]

seems to be from the changes in #53563

If I revert the "file_client" function in salt/utils/jinja.py back to the state it was in in 2019.2.0, things start working again for me.

It seems that in my case, when file caching fails, the wrong fileclient is being returned.

Changing the function to the following to add more logging:

    def file_client_new(self):
        '''
        Return a file client. Instantiates on first call.
        '''
        # If there was no file_client passed to the class, create a cache_client
        # and use that. This avoids opening a new file_client every time this
        # class is instantiated
        log.debug('ELM - self._file_client == {}'.format(self._file_client))
        if self._file_client is None:
            log.debug('ELM - SaltCacheLoader._cached_client == {}'.format(SaltCacheLoader._cached_client))
            if not SaltCacheLoader._cached_client:
                SaltCacheLoader._cached_client = salt.fileclient.get_file_client(
                    self.opts, self.pillar_rend)
            self._file_client = SaltCacheLoader._cached_client
        log.debug('ELM - returning self._file_client == {}'.format(self._file_client))
        return self._file_client

Then returns

[DEBUG   ] ELM - self._file_client == None
[DEBUG   ] ELM - SaltCacheLoader._cached_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM - returning self._file_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM self.cached == []
[DEBUG   ] ELM - caching template path/to/defaults.yml
[DEBUG   ] ELM - saltpath == salt://path/to/defaults.yml
[DEBUG   ] ELM - self._file_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM - returning self._file_client == <salt.fileclient.PillarClient object at 0x7f80f8a01d10>
[DEBUG   ] ELM - file should be cached
[DEBUG   ] ELM - filepath == /var/cache/salt/minion/files/base/path/to/defaults.yml
[DEBUG   ] Jinja Error

Adding similar debug logging to the "old" function retuns a file client of <salt.fileclient.FSClient object at 0x7f02ee4a5c50>

Seems the PillarClient object is being loaded when the pillar data is rendered (my example is a masterless minion) and then is not being overwritten with a FSClient object to get the necessary files from the fileserver...that's just a guess though

ping @twangboy

[kiemlicz]

Did someone found some workaround?
It disables the whole release for me... and it's the only release for Debian 10

[awerner]

@kiemlicz For other distri we pin to 2019.2.0 right now..
Have you tried to use the Debian 9 repo for 2019.2.0 with Debian 10?

[kiemlicz]

Based on #52590
It is impossible to do so.

I remember trying before 2019.2.1 release without any success. Why should it change now?

[3issamben]

I can also confirm that I'm seeing the exact same issue (Salt 2019.2.1, CentOS 7.5 with Python 2.7)

[OrangeDog]

In contrast, I'm not seeing this issue.

Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: 0.32.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.8 (default, Oct  7 2019, 12:59:55)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ISO-8859-1
        machine: x86_64
        release: 4.15.0-65-generic
         system: Linux
        version: Ubuntu 18.04 bionic

[cachedout]

Should this be closed now that #54780 is closed?

[KChandrashekhar]

Closing this as #54780 fixes the issue.