Script and interpreter generation
=================================

This recipe is very similar to zc.recipe.egg, and if you are familiar with its
options, you will be able to use this one easily.

The script and interpreter generation in this recipe are improved from
those provided by zc.recipe.egg in two basic ways.

- The interpreter generated by the script supports all interpreter
  options, as opposed to the subset provided by zc.recipe.egg.

- Both scripts and interpreters from this recipe can optionally choose
  to include site-packages, and even sitecustomize.

The recipe takes several options.  First, here's the list of the options
that overlap from the standard zc.recipe.eggs scripts recipe.  After
this, we'll list the new options and describe them.

* eggs
* find-links
* index
* python
* extra-paths
* entry-points
* scripts
* dependent-scripts
* interpreter
* arguments
* initialization
* relative-paths

In addition to these, the recipe offers these new options.  They are
introduced here, and described more in depth below.

include-site-packages
    You can choose to have the site-packages of the underlying Python
    available to your script or interpreter, in addition to the packages
    from your eggs.  See the section on this option for motivations and
    warnings.  As of zc.buildout 1.5.2, this defaults to true, for increased
    backwards compatibility with pre 1.5 recipes and buildouts.

allowed-eggs-from-site-packages
    Sometimes you need or want to control what eggs from site-packages are
    used. The allowed-eggs-from-site-packages option allows you to specify a
    whitelist of project names that may be included from site-packages.  You
    can use globs to specify the value.  It defaults to a single value of '*',
    indicating that any package may come from site-packages.

    Here's a usage example::

        [buildout]
        ...

        allowed-eggs-from-site-packages =
            demo
            bigdemo
            zope.*

    This option interacts with the ``include-site-packages`` option in the
    following ways.

    If ``include-site-packages`` is true, then
    ``allowed-eggs-from-site-packages`` filters what eggs from site-packages
    may be chosen.  Therefore, if ``allowed-eggs-from-site-packages`` is an
    empty list, then no eggs from site-packages are chosen, but site-packages
    will still be included at the end of path lists.

    If ``include-site-packages`` is false, the value of
    ``allowed-eggs-from-site-packages`` is irrelevant.

extends
    You can extend another section using this value.  It is intended to be
    used by extending a section that uses this package's scripts recipe.
    In this manner, you can avoid repeating yourself.

exec-sitecustomize
    Normally the Python's real sitecustomize module is not processed.
    If you want it to be processed, set this value to 'true'.  This will
    be honored irrespective of the setting for include-site-packages.

script-initialization
    The standard initialization code affects both an interpreter and scripts.
    The code in script-initialization is used only for the generated scripts.

Finally, the "interpreter" entry point ignores ``script-initialization``,
``scripts``, and ``arguments``, and provides yet another additional option.

name
    While, by default, the interpreter recipe takes the name of the
    section to be the desired interpreter name, you can specify the
    interpreter name here instead.

Script generation
-----------------

Generating a basic script looks virtually identical to using zc.recipe.egg.

(Note that the find-links and index values are typically not needed; they
are included to help make this document run as a test successfully.)

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = demo
    ...
    ... [demo]
    ... recipe = z3c.recipe.scripts
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server))

    >>> print system(buildout),
    Installing demo.
    Getting distribution for 'demo<0.3'.
    Got demo 0.2.
    Getting distribution for 'demoneeded'.
    Got demoneeded 1.2c1.
    Generated script '/sample-buildout/bin/demo'.

    >>> print system(join(sample_buildout, 'bin', 'demo')),
    2 2

Interpreter generation
----------------------

As with zc.recipe.egg, you can generate an interpreter with the default
script recipe shown above by supplying the "interpreter" option.
This example will create both an entry point script and an interpreter.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = demo
    ...
    ... [demo]
    ... recipe = z3c.recipe.scripts
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... interpreter = py
    ... """ % dict(server=link_server))

    >>> print system(buildout),
    Uninstalling demo.
    Installing demo.
    Generated script '/sample-buildout/bin/demo'.
    Generated interpreter '/sample-buildout/bin/py'.

You can also generate an interpreter alone with the ``interpreter`` recipe.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = py
    ... include-site-packages = false
    ...
    ... [py]
    ... recipe = z3c.recipe.scripts:interpreter
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server))

    >>> print system(buildout),
    Uninstalling demo.
    Installing py.
    Generated interpreter '/sample-buildout/bin/py'.

In both cases, the bin/py script works by restarting Python after
specifying a special path in PYTHONPATH.  This example shows the UNIX version;
the Windows version actually uses subprocess instead.

    >>> cat(sample_buildout, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
    #!/usr/bin/python2.4 -S
    <BLANKLINE>
    import os
    import sys
    <BLANKLINE>
    argv = [sys.executable] + sys.argv[1:]
    environ = os.environ.copy()
    path = '/sample-buildout/parts/py'
    if environ.get('PYTHONPATH'):
        path = os.pathsep.join([path, environ['PYTHONPATH']])
    environ['PYTHONPATH'] = path
    os.execve(sys.executable, argv, environ)

The path is a directory that contains two files: our own site.py and
sitecustomize.py.  The site.py is modified from the underlying Python's
site.py, and is responsible for setting up our paths. The
sitecustomize.py is responsible for running the initialization code
provided.

    >>> ls(sample_buildout, 'parts', 'py')
    -  site.py
    -  sitecustomize.py

Here's an example of using the generated interpreter.

    >>> print system(join(sample_buildout, 'bin', 'py') +
    ...              ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
    ['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
     '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
    <BLANKLINE>

Including site-packages and sitecustomize
-----------------------------------------

As introduced above, this recipe supports including site packages.  This has
some advantages and some serious dangers.

A typical reason to include site-packages is that it is easier to
install one or more dependencies in your Python than it is with
buildout.  Some packages, such as lxml or Python PostgreSQL integration,
have dependencies that can be much easier to build and/or install using
other mechanisms, such as your operating system's package manager.  By
installing some core packages into your Python's site-packages, this can
significantly simplify some application installations.

However, doing this has a significant danger.  One of the primary goals
of buildout is to provide repeatability.  Some packages (one of the
better known Python openid packages, for instance) change their behavior
depending on what packages are available.  If Python curl bindings are
available, these may be preferred by the library.  If a certain XML
package is installed, it may be preferred by the library.  These hidden
choices may cause small or large behavior differences.  The fact that
they can be rarely encountered can actually make it worse: you forget
that this might be a problem, and debugging the differences can be
difficult.  If you allow site-packages to be included in your buildout,
and the Python you use is not managed precisely by your application (for
instance, it is a system Python), you open yourself up to these
possibilities.  Don't be unaware of the dangers.

To show off these features, we need to use buildout with a Python
executable with some extra paths to show ``include-site-packages``; and one
guaranteed to have a sitecustomize module to show
``exec-sitecustomize``.  We'll make one using a test fixture called
``make_py``. The os.environ change below will go into the sitecustomize,
and the site_packages_path will be in the Python's path.

    >>> py_path, site_packages_path = make_py(initialization='''\
    ... import os
    ... os.environ['zc.buildout'] = 'foo bar baz shazam'
    ... ''')
    >>> print site_packages_path
    /executable_buildout/site-packages

Now let's take a look at include-site-packages.

The default is value true (as of zc.buildout 1.5.2).

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = py
    ... executable = %(py_path)s
    ... exec-sitecustomize = false
    ...
    ... [py]
    ... recipe = z3c.recipe.scripts:interpreter
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server, py_path=py_path))

    >>> print system(buildout),
    Uninstalling py.
    Installing py.
    Generated interpreter '/sample-buildout/bin/py'.

Now executable_buildout/site-packages is included in sys.path.

    >>> print system(join(sample_buildout, 'bin', 'py') +
    ...              ''' -c "import sys, pprint; pprint.pprint(sys.path)"''')
    ... # doctest: +ELLIPSIS
    ['',
     '/sample-buildout/parts/py',
     ...,
     '/sample-buildout/eggs/demo-0.2-pyN.N.egg',
     '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg',
     '/executable_buildout/eggs/setuptools-X-pyN.N.egg',
     '/executable_buildout/site-packages']
    <BLANKLINE>

If you set it to false, they are excluded.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = py
    ... executable = %(py_path)s
    ... exec-sitecustomize = false
    ...
    ... [py]
    ... recipe = z3c.recipe.scripts:interpreter
    ... include-site-packages = false
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server, py_path=py_path))

    >>> print system(buildout),
    Uninstalling py.
    Installing py.
    Generated interpreter '/sample-buildout/bin/py'.

    >>> print system(join(sample_buildout, 'bin', 'py') +
    ...              ''' -c "import sys, pprint; pprint.pprint(sys.path)"''')
    ... # doctest: +ELLIPSIS
    ['',
     '/sample-buildout/parts/py',
     ...,
     '/sample-buildout/eggs/demo-0.2-pyN.N.egg',
     '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
    <BLANKLINE>

As described above, the allowed-eggs-from-site-packages option lets us
control what site-packages eggs zc.buildout will allow to fulfill
dependencies.  The behavior was described above with an example (and the
implementation is tested elsewhere), so we'll only look at some simple and
common use cases here.

Sometimes you may want to allow site-packages to be available but you don't
want your package to depend on it using setup.py.  For instance, perhaps you
are writing an application, and you want to depend on your system's packaging
of the PostgreSQL code, but the system Python does not use eggs to
package it, so you need to manage the two separately.  In this case, you
might not want to use any eggs from site-packages, but you want it available.
In this case, you can use allowed-eggs-from-site-packages with an empty value
to keep any egg from being used from site-packages.

Here's an example.  Let's say we have a Python with demo and demoneeded
installed as eggs in the system Python.  Normally, they will be used to
fulfill dependencies, because allowed-eggs-from-site-packages defaults to
the value "*" (allow any package).  (We use an empty find-links value to say
that buildout may not look elsewhere for the package. We use a different
eggs-directory for isolation, so that eggs obtained other parts of the
document do not affect this example.)

    >>> from zc.buildout.tests import create_sample_sys_install
    >>> create_sample_sys_install(site_packages_path)
    >>> import zc.buildout.easy_install
    >>> zc.buildout.easy_install.clear_index_cache()

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = eggs
    ... eggs-directory = tmpeggs
    ... find-links =
    ...
    ... [primed_python]
    ... executable = %(py_path)s
    ...
    ... [eggs]
    ... recipe = z3c.recipe.scripts
    ... python = primed_python
    ... eggs = demoneeded
    ... ''' % globals())

    >>> print system(buildout),
    Creating directory '/sample-buildout/tmpeggs'.
    Uninstalling py.
    Installing eggs.

That succeeds fine, getting demoneeded from the Python site-packages.

However, when allowed-eggs-from-site-packages is an empty value, demoneeded
is not allowed to come from site-packages, and the buildout fails.

    >>> zc.buildout.easy_install.clear_index_cache()
    >>> rmdir('tmpeggs')
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = eggs
    ... eggs-directory = tmpeggs
    ... find-links =
    ...
    ... [primed_python]
    ... executable = %(py_path)s
    ...
    ... [eggs]
    ... recipe = z3c.recipe.scripts
    ... python = primed_python
    ... allowed-eggs-from-site-packages =
    ... eggs = demoneeded
    ... ''' % globals())
    >>> print system(buildout),
    Creating directory '/sample-buildout/tmpeggs'.
    Uninstalling eggs.
    Installing eggs.
    Couldn't find index page for 'demoneeded' (maybe misspelled?)
    Getting distribution for 'demoneeded'.
    While:
      Installing eggs.
      Getting distribution for 'demoneeded'.
    Error: Couldn't find a distribution for 'demoneeded'.

The include-sitepackages and allowed-eggs-from-site-packages options both
can be obtained from the buildout section if they are not set locally.

.. ReST comment: PyPI readers don't need the demonstration, but here it is.

   This succeeds:

    >>> from zc.buildout.tests import create_sample_sys_install
    >>> create_sample_sys_install(site_packages_path)
    >>> import zc.buildout.easy_install
    >>> zc.buildout.easy_install.clear_index_cache()

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = eggs
    ... eggs-directory = tmpeggs
    ... find-links =
    ...
    ... [primed_python]
    ... executable = %(py_path)s
    ...
    ... [eggs]
    ... recipe = z3c.recipe.scripts
    ... python = primed_python
    ... eggs = demoneeded
    ... ''' % globals())

    >>> print system(buildout),
    Installing eggs.

   This fails:

    >>> zc.buildout.easy_install.clear_index_cache()
    >>> rmdir('tmpeggs')
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = eggs
    ... eggs-directory = tmpeggs
    ... allowed-eggs-from-site-packages =
    ... find-links =
    ...
    ... [primed_python]
    ... executable = %(py_path)s
    ...
    ... [eggs]
    ... recipe = z3c.recipe.scripts
    ... python = primed_python
    ... eggs = demoneeded
    ... ''' % globals())
    >>> print system(buildout),
    Creating directory '/sample-buildout/tmpeggs'.
    Uninstalling eggs.
    Installing eggs.
    Couldn't find index page for 'demoneeded' (maybe misspelled?)
    Getting distribution for 'demoneeded'.
    While:
      Installing eggs.
      Getting distribution for 'demoneeded'.
    Error: Couldn't find a distribution for 'demoneeded'.

Remember that you can provide multiple lines to the
allowed-eggs-from-site-packages option, each specifying a whitelist of
allowed packages.  Globs (* and ?) are allowed.

Next we will use the exec-sitecustomize option.  It simply copies
Python's underlying sitecustomize module, if it exists, to the local
version.  The os.environ change shown above in the make_py call will go
into the sitecustomize.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = py
    ... executable = %(py_path)s
    ...
    ... [py]
    ... recipe = z3c.recipe.scripts:interpreter
    ... exec-sitecustomize = true
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server, py_path=py_path))

    >>> print system(buildout),
    Installing py.
    Generated interpreter '/sample-buildout/bin/py'.

    >>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
    ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
    <BLANKLINE>
    # The following is from
    # /executable_buildout/parts/py/sitecustomize.py
    ...
    import os
    os.environ['zc.buildout'] = 'foo bar baz shazam'

    >>> print system(join(sample_buildout, 'bin', 'py') +
    ...              ''' -c "import os; print os.environ['zc.buildout']"''')
    foo bar baz shazam
    <BLANKLINE>

It also will be honored in the buildout section if it is not set locally.

.. ReST comment: PyPI users don't need to see this test.  This verifies that
   exec-sitecustomize is honored if it is in the buildout section.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = py
    ... executable = %(py_path)s
    ... exec-sitecustomize = true
    ...
    ... [py]
    ... recipe = z3c.recipe.scripts:interpreter
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server, py_path=py_path))

    >>> print system(buildout),
    Updating py.

    >>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
    ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
    <BLANKLINE>
    # The following is from
    # /executable_buildout/parts/py/sitecustomize.py
    ...
    import os
    os.environ['zc.buildout'] = 'foo bar baz shazam'

    >>> print system(join(sample_buildout, 'bin', 'py') +
    ...              ''' -c "import os; print os.environ['zc.buildout']"''')
    foo bar baz shazam
    <BLANKLINE>

Options
-------

We'll focus now on the remaining options that are different than
zc.recipe.egg.

Let's look at the ``extends`` option first.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = demo python
    ...
    ... [demo]
    ... recipe = z3c.recipe.scripts
    ... eggs = demo<0.3
    ... include-site-packages = false
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... initialization =
    ...     import os
    ...     os.environ['zc.buildout'] = 'sha boo bop bazoodle'
    ...
    ... [python]
    ... recipe = z3c.recipe.scripts:interpreter
    ... extends = demo
    ... initialization =
    ...     import os
    ...     os.environ['zc.buildout'] = 'foo bar baz shazam'
    ... """ % dict(server=link_server))

That makes it easier to specify some initialization for the interpreter
that is different than a script, while duplicating other configuration.

Now let's put it in action.

    >>> print system(buildout),
    Uninstalling py.
    Installing demo.
    Generated script '/sample-buildout/bin/demo'.
    Installing python.
    Generated interpreter '/sample-buildout/bin/python'.

    >>> print system(join(sample_buildout, 'bin', 'python') +
    ...              ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
    ['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
     '/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
    <BLANKLINE>
    >>> print system(join(sample_buildout, 'bin', 'python') +
    ...              ''' -c "import os; print os.environ['zc.buildout']"'''),
    foo bar baz shazam

Note that the parts/py directory has been cleaned up, and parts/python has
been created.

    >>> ls(sample_buildout, 'parts')
    d  buildout
    d  demo
    d  python

If you want to have initialization that only affects scripts, not the
interpreter, you can use script-initialization.  Here's a demonstration.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = demo
    ...
    ... [demo]
    ... recipe = z3c.recipe.scripts
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... interpreter = py
    ... script-initialization =
    ...     print "Hi from the script"
    ... """ % dict(server=link_server))

    >>> print system(buildout),
    Uninstalling python.
    Uninstalling demo.
    Installing demo.
    Generated script '/sample-buildout/bin/demo'.
    Generated interpreter '/sample-buildout/bin/py'.

    >>> print system(join(sample_buildout, 'bin', 'py') +
    ...              ''' -c "print 'Hi from the interpreter'"'''),
    Hi from the interpreter

    >>> print system(join(sample_buildout, 'bin', 'demo')),
    Hi from the script
    2 2

The last new option is ``name``.  This simply changes the name of the
interpreter, so that you are not forced to use the name of the section.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = interpreter
    ...
    ... [interpreter]
    ... name = python2
    ... recipe = z3c.recipe.scripts:interpreter
    ... eggs = demo<0.3
    ... find-links = %(server)s
    ... index = %(server)s/index
    ... """ % dict(server=link_server))

    >>> print system(buildout),
    Uninstalling demo.
    Installing interpreter.
    Generated interpreter '/sample-buildout/bin/python2'.

    >>> print system(join(sample_buildout, 'bin', 'python2') +
    ...              ' -c "print 42"')
    42
    <BLANKLINE>

The other options all identical to zc.recipe.egg.
