Extension modules not available through the salt.cmd runner

[mirceaulinic]

Description of Issue/Question

Using the salt.cmd runner, one is able to execute commands that do not depend necessarily on a certain minion, e.g. http.query etc.:

$ sudo salt-run salt.cmd test.ping
True

But with an extension module, it doesn't:

$ sudo salt-run salt.cmd transit.ping
Exception occurred in runner salt.cmd: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 395, in _low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/runners/salt.py", line 67, in cmd
    utils=utils(__opts__)).get(fun)(*args, **kws)
TypeError: 'NoneType' object is not callable

Although the execution module is available:

$ sudo salt edge01.bjm01 transit.test
edge01.bjm01:
    True

(edge01.bjm01 is a proxy minion).

I am running 2016.11.4:

Salt Version:
           Salt: 2016.11.4

Dependency Versions:
           cffi: 1.9.1
       cherrypy: unknown
       dateutil: 2.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.6
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 14.4.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.4
        machine: x86_64
        release: 4.4.17-cloudflare
         system: Linux
        version: debian 8.4

[gtmanfred]

Have you run salt-run saltutil.sync_modules on the master?

I can't think of a reason that it wouldn't have loaded unless they aren't synced. The minion_mods is used to load them in the salt.cmd runner module, which does use _module_dirs, and loads all the extmod locations on the master.

Thanks,
Daniel

[mirceaulinic]

Yes, @gtmanfred I did run:

$ sudo salt-run saltutil.sync_modules
- modules.capirca_acl
- modules.napalm_acl
- modules.napalm_bgp
- modules.napalm_cfnet

However, the request still fails:

$ sudo salt-run salt.cmd transit.ping
Exception occurred in runner salt.cmd: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 395, in _low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/runners/salt.py", line 67, in cmd
    utils=utils(__opts__)).get(fun)(*args, **kws)
TypeError: 'NoneType' object is not callable

[gtmanfred]

Hrm, that is odd, gimme a bit and I will try and replicate this on my end.

Thanks,
Daniel

[gtmanfred]

I am able to replicate the error, but as soon as I sync modules it works.

[root@salt ~]# tree /var/cache/salt/master/extmods/
/var/cache/salt/master/extmods/

0 directories, 0 files
[root@salt ~]# tree /srv/salt/_modules/
/srv/salt/_modules/
└── testrunner.py

0 directories, 1 file
[root@salt ~]# salt-run salt.cmd testrunner.ping 2>/dev/null
Exception occurred in runner salt.cmd: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/client/mixins.py", line 395, in _low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/runners/salt.py", line 67, in cmd
    utils=utils(__opts__)).get(fun)(*args, **kws)
TypeError: 'NoneType' object is not callable
[root@salt ~]# salt-run saltutil.sync_modules
- modules.testrunner
[root@salt ~]# salt-run salt.cmd testrunner.ping
True

[mirceaulinic]

Did you try with a proxy minion by any chance @gtmanfred? It may be related -- the proxy minions seem to have a very bizarre behaviour when it comes to extension modules caching.
For example, sometimes the modules simply disappear: #35443

[gtmanfred]

salt-run is meant to be run on the master? Also, i don't see modules.transit transferred in your sync_modules, does one of those modules have the __virtualname__ transit? Or is this module still in /etc/salt/proxy-modules?

If you move it to salt://_modules and sync it down does the error go away?

I am wondering now if this is just a problem with the module_dirs setting?

Thanks,
Daniel

[mirceaulinic]

Also, i don't see modules.transit transferred in your sync_modules

I think it was already cached. For example, if I run it right now, the output is:

$ sudo salt-run --out=raw saltutil.sync_modules
[]

It is under salt://_modules, synced and available to run on the CLI.

Applying a change inside the module and resync:

$ sudo salt-run --out=raw saltutil.sync_modules
['modules.napalm_transit']

This is the virtualname:

__virtualname__ = 'transit'

However, the __virtual__ fun was:

def __virtual__():
    if HAS_NAPALM and 'proxy' in __opts__:
        return __virtualname__
    else:
        return False

Which made the module unavailable as the proxy key does not exist on the master, but only on the proxy minion level.

After changing the __virtual__ function to:

def __virtual__():
    if HAS_NAPALM:
        return __virtualname__
    else:
        return False

I am able to run $ sudo salt-run salt.cmd transit.ping without any issues.

I am happy to close this ticket, but do you think there's any workaround, as there are many modules testing against the proxy key inside the __opts__? As a side note, the documentation recommends this: https://docs.saltstack.com/en/latest/topics/proxyminion/index.html#the-proxyenabled-directive. In other words I am just wondering if there would be possible to gate a way (hack?) for modules having this test, rather than updating each and every module.

Thanks,
-Mircea

[gtmanfred]

So, the __proxyenabled__ key is only used by the proxy minion, and it is in modules like test.ping So if we see the proxyenabled string, we cannot be certain that we shouldn't load those modules on regular minions.

I think we could possibly add another dunder variable that says to only load modules if the minion is a proxy minion.

@cro do you have an opinion about this?

Thanks,
Daniel

[cro]

My current opinion is to deprecate the __proxyenabled__ directive, as it has caused no end of trouble--at one point it prevented all state modules from loading for proxy minions.

I think I'd prefer to get rid of __proxyenabled__ and replace it with the variable gtmanfred suggests (__proxyonly__? __load_on_proxy_only__?)--that variable would prevent certain modules from loading on regular minions and only load them on proxies.

[mirceaulinic]

In principle any sounds equally good to me, perhaps __proxyonly__ may be more appropriate.

The problem in that case was that my module was not available on the master. What would be the behaviour in that case? When __proxyonly__ = True, would that module be available only on the proxies and also on the master?

[gtmanfred]

I think that the problem is that proxyonly would only load if salt.utils.is_proxy returns True which would be false for the master.

What we could do is create something that uses the raw_mod from the loader, and ignores __virtual__ and runs the module no matter what virtual returns.

https://docs.saltstack.com/en/latest/ref/clients/#salt.loader.raw_mod

And expose that in the salt runner.

[mirceaulinic]

Forgot to reply here.
This sounds good to me, but it may not work well with modules requiring a third party lib.
There aren't so many modules though and we can maybe just change the code to 'proxy' in __opts__ or __opts__['__role'] == 'master' for those that make sense (for some modules it wouldn't make sense).
What do you think?

[gtmanfred]

Yeah, i don't think that is too crazy, @cro do you have any opinion about this?

…

On Mon, May 22, 2017 at 1:47 AM, Mircea Ulinic ***@***.***> wrote: Forgot to reply here. This sounds good to me, but it may not work well with modules requiring a third party lib. There aren't so many modules though and check 'proxy' in __opts__ or __opts__['__role'] == 'master' for those that make sense (for some of them it wouldn't make sense). What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#41024 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAssoXC38P9dx9q56ilqbdrvA67sDBWTks5r8T2OgaJpZM4NPUoo> .

[mirceaulinic]

As this makes sense to everyone involved in this thread, I am going to close this issue. Thanks, @gtmanfred @cro