shepherd package

Subpackages

• shepherd.actions package
  • Submodules
  • shepherd.actions.ansible module
  • shepherd.actions.create module
  • shepherd.actions.destroy module
  • Module contents
• shepherd.common package
  • Submodules
  • shepherd.common.exceptions module
  • shepherd.common.plugins module
  • shepherd.common.utils module
  • Module contents
• shepherd.resources package
  • Subpackages
    • shepherd.resources.aws package
      • Submodules
      • shepherd.resources.aws.instance module
      • shepherd.resources.aws.key module
        • shepherd.resources.aws.iamkeys
      • shepherd.resources.aws.securitygroup module
        • shepherd.resources.aws.securitygroup
      • shepherd.resources.aws.securityingress module
      • shepherd.resources.aws.user module
      • shepherd.resources.aws.volume module
      • Module contents
  • Module contents
• shepherd.storage package
  • Submodules
  • shepherd.storage.dynamo module
  • Module contents

Submodules

shepherd.config module

shepherd.manager

This modules contains the Config object itself, which is the primary outward facing object for the package (kind of like app in Flask).

Notes:

• Currently the builtin paths and plugins are hardcoded, we may want to

inspect the plugins module and dirnames to create those variables. * See the common/config.schema for the schema of the settings dict. * the default settings dict is:

{
    'manifest_path': '',
    'verbosity': 2,
    'retries': 120,
    'delay': 5,
    'vars': {},
    'storage': {
        'name': 'DynamoStorage',
        'settings': {}
    },
}

class shepherd.config.Config(settings, name)

Bases: object

The Config is responsible for managing the plugins and executing given tasks.

classmethod get(name='')

Use this to access your desired Config.

Args:

name (str, optional): the unique name of the config you

want returned.

Returns: the config obj

Raises:

KeyError: if a config by that name does’t exist.

get_plugins(category_name=None, plugin_name=None)

get_plugins returns a deepcopy of all the plugins fitting the search criteria. While this isn’t very memory efficient our plugins should be small and few between enough that it’ll be worth getting independent copies of them. For example we will likely want to work with multiple copies of the Same Resource plugin.

Args:

category_name (str, optional): a category to search for plugins in. plugin_name (str, optional): the name of the plugin to look for.

Returns:

list: of the plugins that match the criteria.

logging_verbosity = 0

classmethod make(settings=None, name='')

When first setting up the Config you should call this class method.

Args:

settings (dict, optional): desire settings values overriding the defaults. name (str, optional): the name of the config

Returns: the created config obj

classmethod make_from_file(filename, name='')

Loads the settings dict from a file and passes it to Config.make.

Args:

filename (str): name of the file to load name (str, optional): the name of the config

Returns:

Config: the created config obj

name

settings

class shepherd.config.PluginFileAnalyzerInspection(name, paths)

Bases: yapsy.PluginFileLocator.IPluginFileAnalyzer

This PluginFileAnalyzer determines the plugins via inspection. If the module contains a class that subclasses

getInfosDictFromPlugin(dirpath, filename)

Returns the extracted plugin informations as a dictionary. This function ensures that “name” and “path” are provided.

Args:

dirpath (str): the directory of the file filename (str): the filename

getModulePaths(paths)

Extracts the module_paths from the provided lis of paths.

Args:

paths (list): list of paths to walk

getPluginClass(filename)

Extracts the plugin class from the given file.

Args:

filename (str): name of file to inspect

Returns:

str: name of the plugin class

isValidPlugin(filename)

Checks if the given filename is a valid plugin for this Strategy

Args:

filename (str): name of the file to inspect.

shepherd.environment module

An Environment is a stack that can maintain other stacks.

The idea of an environment is that it maintains a set of resources that are used by multiple stacks. For example if we have a set of stacks that are related to each other we may want them share some resources. Be careful not to overuse environments though. If 2 stacks are coupled enough to have a lot of shared resources you probably want to merge them into one stack.

EX) Assume we are using AWS for our example, if we want to promote a stack from staging to production the Environment would maintain the elasticips outside of an individual stack. Alternatively, we might want one stack to be able to communicate with another (minimally), so we might have an Environment that creates a globally accessible security group with which individual stacks can register to communicate.

class shepherd.environment.Environment(name, config_name)

Bases: shepherd.stack.Stack

The Environment is a subclass of a Stack object.

It provides a set useful methods for working with groups of stacks at an environment level.

add(name, create=False)

Adds a stack with the given name to the environment.

If the stack cannot be found and create is true a new stack will be created and added to the environment.

remove(name, delete=False)

Removes a stack from the environment.

If delete is set to true the stack will attempt to be deprovisioned.

restore_stacks()

Restores all stack object identified by our stack_names.

stack_names

shepherd.manifest module

Handles loading manifest files into a list of valid resource dicts. Possibly from multiple files.

class shepherd.manifest.Loader(filename)

Bases: object

The Loader handles packaging multiple template files together.

Template files can be included in each other like so. If the value in a list or dict is a dict with 1 entry, where the key is include and the value is the path to the file.

List:

MyList: [
    ...
    {"include" : "../path/to/file"},
    ...
]

Dict:

MyDict: {
    ...
    "include foo" : "../path/to/foo.json"},
    ...
}

NOTE: in the dict case the key value pair will be ignore like the value in the list. Also, the file being imported must be a dict at the top level.

load(filename)

Handles the actual loading and preporcessing of files.

Args:

filename (str): the file to load.

Returns:

collection: returns load collection and all recusively loaded ones.

run()

Runs the loader returning the completed

Returns:

collection: Returns the fully loaded and unified template.

class shepherd.manifest.Manifest(config)

Bases: object

This class is responsible for loading and parsing the stack template files. It can then be used for building stack objects.

In order,

NOTE: the expected layout of the resources list should look something like this.

[
    {
        'Name': 'MyInstance',
        'Type': 'Instance',
        'AvailabilityZone' : 'some-availablity-zone',
        'ImageId' :          'i-foo42',
        'InstanceType' :     't1-micro',
        'KeyName' :          'mykey.pem',
        'SpotPrice' :        '2.50',
        'SecurityGroups' :   [
            'StackWebserverSecurityGroup',
            'sg-bar24',
            ...
        ],

        'UserData' : '#!/bin/bash\necho "initializing stack"\n'
        'Tags' : [
            {
                'Key' : 'Name',
                'Value' : 'Dashboard'
            },
            {
                'Key" : "StackLevel',
                'Value' : 'Prod'
            },
            ...
        ],

    },
    ...
]

VS:

{
    "MyInstance" :
    {
        "Type" : "AWS::EC2::Instance",
        "Properties": {
            "Tags" : [
                {
                    "Key" : "Name",
                    "Value" : "Webserver"
                },
                {
                    "Key" : "StackLevel",
                    "Value" : { "Ref" : "StackLevel" }
                }
            ],
            "AvailabilityZone" : { "Ref" : "StackAvailabilityZone" },
            "ImageId" :          { "Ref" : "WebserverAMI" },
            "InstanceType" :     { "Ref" : "WebserverInstanceType" },
            "SecurityGroups" : [
                { "Ref" : "StackWebserverSecurityGroup" },
                sg-bar24
            ],
            "KeyName" :          { "Ref" : "CloudKeyName" },
            "SpotPrice" :        { "Ref" : "T1MicroSpotPrice" },

            "UserData" :
            { "Fn::Base64" : { "Fn::Join" : [ "", [
            "#!/bin/bash\n "initializing stack"\n"
            ]]}}
        }
    },
    ...
}

Notice how only stack resource level variables like ‘StackDashboardSecurityGroup’ remain. Also, how the the name has been moved inside the dict and the properties have been moved up a level.

clear()

Cleans up the working directory

load()

Loaders are responsible for taking the list of files and building a single python dict from them. The default loader is used if none is specified, which simply reads either the json or yaml files with the name params or resources.

map()

Maps values in the params dict to the values in the resources dict using jinja2 variable referencing syntax

parse()

Loads the settings(s) into the template and params dicts, which can then be validated.

resources

shepherd.stack module

user calls a stack, passing a filepath to the template to parse. The returned stack object has all params and resources for that stack, Stacks will support higher level operations like “create”,”wake”,”sleep”.

class shepherd.stack.Stack(name, config)

Bases: object

The Stack object maintains and manipulates the list of cloud resources that need to be provisioned.

deprovision_resources(resources=None)

Handles building a list of destroy tasks and running them with dynamic dependency handling via run_tasks.

NOTE: We are also responsible for inverting the dependency cases to the standard creation dependencies.

Args:

resources (list, optional): a list of the subset of resources in the stack

to deprovision. Defaults to all of them.

Raises:

StackError: if not all resources are successfully deprovisioned.

classmethod deserialize(data)

Builds a stack from the data dictionary.

Details:

• Uses the settings and config_name values to create a Config.
• Uses the local_name and the config to create a new Stack instance.
• sets the global_name, tags and resource list (by deserializing the resources)

Args:

data (dict): a dictionary holding the state of a stack

Returns:

TYPE: Description

deserialize_resources(resource_list)

Given a list or resource dicts we:

1. loop over them
2. extract the resource plugin name from the ‘Type’
3. get the plugin and deserialize it from the dict
4. add the deserialized resource to the list of resources.

Args:

resource_list (list): list of dictionaries which are to be

loaded into resource attributes

Raises:

PluginError: if no Resources have been loaded or if a particular

Resource type specified in each dict doesn’t exist.

get_global_resource_name(name)

Generates the global name for a resources, which by defaults is

{resource_name}_{stack_name}_{stack_creation}

Note:

This global name is truncated to be only 64 characters long due some restrictions with AWS.

Args:

name (str): local name of the resource

Returns:

str: the global name

get_resource_by_name(local_name)

Takes a string for the unique local resource name and returns the resource that matches.

NOTE: if you somehow manage to change the object name to no longer be unique this method will only return the first one it finds.

Args:

local_name (str): local name of the resource to search by

Returns:

the resource or None

get_resource_by_tags(tags)

Returns a list of resources where the resource tags contains the same key values as the tags provided.

Args:

tags (dict): of tags to search by.

Returns:

list: of resource with the specified tags.

get_resource_by_type(resource_type)

Takes a string representing the resource type you want to retrieve and returns a list of all resources that match that type. The type is determined by either the name of the resources class or by the resource type attribute if one exists.

Args:

resource_type (str): the name of the resource type you like to get

eg: all Instances

Returns:

list: of the resource of that type.

global_name

local_name

classmethod make(name, config)

Handles creating a stack instance.

Args:

name (str): the stack name config (Config): the Config object

provision_resources(resources=None)

Handles building a list of create tasks and running them with dynamic dependency handling via run_tasks. Roles back changes in case of failure. ke: if provisioning fails any already provisioned resource are deprovision automatically.

Args:

resources (list, optional): a list of the subset of resources in the stack

to provision. Defaults to all of them.

Raises:

StackError: if not all resources are successfully provisioned.

classmethod restore(name, config)

Handles restoring a stack with the given name from the storage plugin specified in config.settings

Args:

name (str): the stack name to look for in the storage plugin config (the): the config object used to find the storage

plugin to load the stack from.

Raises:

PluginError: if the storage plugin listed in the config.settings

can’t be found.

save()

Saves this stack to the storage plugin specified in config.settings.

Raises:

PluginError: if the storage plugin listed in the config.settings

can’t be found.

serialize()

Serializes a stack by writing the following attributes to a dict.

1. local_name
2. global_name
3. config_name
4. settings
5. tags
6. resources (serialized)

Returns:

dict: containaining the stack stack state

settings

tags

task_function_wrapper(function)

Wraps a resource function with

• saving the stack
• logging errors
• etc

Args:

function (function): the function to wrap

Module contents

shepherd cloud management framework

shepherd is a modular framework for deploying and modifying cloud deployments, written in Python.