Skip to content
Snippets Groups Projects
  • Toshio Kuratomi's avatar
    52449cc0
    AnsiballZ improvements · 52449cc0
    Toshio Kuratomi authored
    Now that we don't need to worry about python-2.4 and 2.5, we can make
    some improvements to the way AnsiballZ handles modules.
    
    * Change AnsiballZ wrapper to use import to invoke the module
      We need the module to think of itself as a script because it could be
      coded as:
    
          main()
    
      or as:
    
          if __name__ == '__main__':
              main()
    
      Or even as:
    
          if __name__ == '__main__':
              random_function_name()
    
      A script will invoke all of those.  Prior to this change, we invoked
      a second Python interpreter on the module so that it really was
      a script.  However, this means that we have to run python twice (once
      for the AnsiballZ wrapper and once for the module).  This change makes
      the module think that it is a script (because __name__ in the module ==
      '__main__') but it's actually being invoked by us importing the module
      code.
    
      There's three ways we've come up to do this.
      * The most elegant is to use zipimporter and tell the import mechanism
        that the module being loaded is __main__:
        * https://github.com/abadger/ansible/blob/5959f11c9ddb7b6eaa9c3214560bd85e631d4055/lib/ansible/executor/module_common.py#L175
        * zipimporter is nice because we do not have to extract the module from
          the zip file and save it to the disk when we do that.  The import
          machinery does it all for us.
        * The drawback is that modules do not have a __file__ which points
          to a real file when they do this.  Modules could be using __file__
          to for a variety of reasons, most of those probably have
          replacements (the most common one is to find a writable directory
          for temporary files.  AnsibleModule.tmpdir should be used instead)
          We can monkeypatch __file__ in fom AnsibleModule initialization
          but that's kind of gross.  There's no way I can see to do this
          from the wrapper.
    
      * Next, there's imp.load_module():
        * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
        * imp has the nice property of allowing us to set __name__ to
          __main__ without changing the name of the file itself
        * We also don't have to do anything special to set __file__ for
          backwards compatibility (although the reason for that is the
          drawback):
        * Its drawback is that it requires the file to exist on disk so we
          have to explicitly extract it from the zipfile and save it to
          a temporary file
    
      * The last choice is to use exec to execute the module:
        * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
        * The code we would have to maintain for this looks pretty clean.
          In the wrapper we create a ModuleType, set __file__ on it, read
          the module's contents in from the zip file and then exec it.
        * Drawbacks: We still have to explicitly extract the file's contents
          from the zip archive instead of letting python's import mechanism
          handle it.
        * Exec also has hidden performance issues and breaks certain
          assumptions that modules could be making about their own code:
          http://lucumr.pocoo.org/2011/2/1/exec-in-python/
    
      Our plan is to use imp.load_module() for now, deprecate the use of
      __file__ in modules, and switch to zipimport once the deprecation
      period for __file__ is over (without monkeypatching a fake __file__ in
      via AnsibleModule).
    
    * Rename the name of the AnsiBallZ wrapped module
      This makes it obvious that the wrapped module isn't the module file that
      we distribute.  It's part of trying to mitigate the fact that the module
      is now named __main)).py in tracebacks.
    
    * Shield all wrapper symbols inside of a function
      With the new import code, all symbols in the wrapper become visible in
      the module.  To mitigate the chance of collisions, move most symbols
      into a toplevel function.  The only symbols left in the global namespace
      are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
    
    revised porting guide entry
    
    Integrate code coverage collection into AnsiballZ.
    
    ci_coverage
    ci_complete
    52449cc0
    History
    AnsiballZ improvements
    Toshio Kuratomi authored
    Now that we don't need to worry about python-2.4 and 2.5, we can make
    some improvements to the way AnsiballZ handles modules.
    
    * Change AnsiballZ wrapper to use import to invoke the module
      We need the module to think of itself as a script because it could be
      coded as:
    
          main()
    
      or as:
    
          if __name__ == '__main__':
              main()
    
      Or even as:
    
          if __name__ == '__main__':
              random_function_name()
    
      A script will invoke all of those.  Prior to this change, we invoked
      a second Python interpreter on the module so that it really was
      a script.  However, this means that we have to run python twice (once
      for the AnsiballZ wrapper and once for the module).  This change makes
      the module think that it is a script (because __name__ in the module ==
      '__main__') but it's actually being invoked by us importing the module
      code.
    
      There's three ways we've come up to do this.
      * The most elegant is to use zipimporter and tell the import mechanism
        that the module being loaded is __main__:
        * https://github.com/abadger/ansible/blob/5959f11c9ddb7b6eaa9c3214560bd85e631d4055/lib/ansible/executor/module_common.py#L175
        * zipimporter is nice because we do not have to extract the module from
          the zip file and save it to the disk when we do that.  The import
          machinery does it all for us.
        * The drawback is that modules do not have a __file__ which points
          to a real file when they do this.  Modules could be using __file__
          to for a variety of reasons, most of those probably have
          replacements (the most common one is to find a writable directory
          for temporary files.  AnsibleModule.tmpdir should be used instead)
          We can monkeypatch __file__ in fom AnsibleModule initialization
          but that's kind of gross.  There's no way I can see to do this
          from the wrapper.
    
      * Next, there's imp.load_module():
        * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
        * imp has the nice property of allowing us to set __name__ to
          __main__ without changing the name of the file itself
        * We also don't have to do anything special to set __file__ for
          backwards compatibility (although the reason for that is the
          drawback):
        * Its drawback is that it requires the file to exist on disk so we
          have to explicitly extract it from the zipfile and save it to
          a temporary file
    
      * The last choice is to use exec to execute the module:
        * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
        * The code we would have to maintain for this looks pretty clean.
          In the wrapper we create a ModuleType, set __file__ on it, read
          the module's contents in from the zip file and then exec it.
        * Drawbacks: We still have to explicitly extract the file's contents
          from the zip archive instead of letting python's import mechanism
          handle it.
        * Exec also has hidden performance issues and breaks certain
          assumptions that modules could be making about their own code:
          http://lucumr.pocoo.org/2011/2/1/exec-in-python/
    
      Our plan is to use imp.load_module() for now, deprecate the use of
      __file__ in modules, and switch to zipimport once the deprecation
      period for __file__ is over (without monkeypatching a fake __file__ in
      via AnsibleModule).
    
    * Rename the name of the AnsiBallZ wrapped module
      This makes it obvious that the wrapped module isn't the module file that
      we distribute.  It's part of trying to mitigate the fact that the module
      is now named __main)).py in tracebacks.
    
    * Shield all wrapper symbols inside of a function
      With the new import code, all symbols in the wrapper become visible in
      the module.  To mitigate the chance of collisions, move most symbols
      into a toplevel function.  The only symbols left in the global namespace
      are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
    
    revised porting guide entry
    
    Integrate code coverage collection into AnsiballZ.
    
    ci_coverage
    ci_complete
Code owners
Assign users and groups as approvers for specific file changes. Learn more.