-
Notifications
You must be signed in to change notification settings - Fork 1
Design
Umpire has gone through a lot of iterations, and currently exists somewhere between a beta release and v1. While it's not perfect by any means, it does do its job admirably well. It prominently uses the multiprocessing library to create asynchronous tasks, and is able to
At the core of Umpire is a concept of a "Module". The Module classes themselves exist in the MaestroOps repository, though they are currently only used in the Umpire project. Aside from the repository having a terrible name, Modules were incredibly useful in making this project. Modules were created to have a single interface for both synchronous (Module) and asynchronous (AsyncModule) jobs.
Here is the current definition of the base class, Module.
class Module(object)
def __init__(ioc=None)
def getObject(name)
def getObjectInstance(name)
def run(kwargs={})
def start(kwargs={})
def help()
Where the methods provide the following functionality:
| Method | Description |
|---|---|
| __init__ | Takes an ObjectContainer (defined in MaestroOps) as a parameter, which is an semi-IoC container type instance and should contain all the Module types used within the application. |
| getObject | Takes a Module's id, and will return the instance of that object. Useful for singletons. (Not implemented) |
| getObjectInstance | Takes a Module's id, and will return a new instance of that object. |
| help | Prints the HELPTEXT (deprecated) |
| run | Takes a set of kwargs as parameters, and runs the task. This is the main method of a Module. Do not call this method directly, call start(). |
| start | Calls run(). |
The AsyncModule class has an interface very similar to Module, with a few key differences:
| Method | Description |
|---|---|
| finish | The callback method of start(). Extracts the result from the "run" method, and will set the "result" member. If "run" produced an exception, it will set "result" to None and "exception" to the instance of the exception. This method can be overridden to add or change functionality. |
| log | Takes a string, and by default logs to stderr. It uses a thread-safe logger. |
| start | Takes a set of kwargs, like it's superclass method. Creates a non-daemonized task, and begins the "run" method. It also sets the AsyncModule's state to RUNNING. |
It also features a few internal methods that set the state and format the exceptions to allow them to be returned to the calling process. These methods are lightweight and documented within the source code.
The ModuleExector class extends Module, and provides an entrypoint for Python. Additionally, it exposes a method to allow registration of modules to the underlying ObjectContainer.
| Method | Description |
|---|---|
| entry | Provides an entrypoint to an application. It parses arguments to a kwargs dictionary, and will call the "start" method with these kwargs. |
| register | Exposes the "register" method of an underlying ObjectContainer. It initializes the ObjectContainer that will be passed to all other instances of Modules. |
Umpire.py is the entrypoint for the application and contains the Umpire class, which is a ModuleExecutor type. It determines where the Umpire cache should exist, and then calls the Deploy module synchronously.
Notes:
- It parses the system argument vector (sys.argv) itself, and does not use the kwargs passed in. It should be modified to use the kwargs.
Deploy contains the coordinating logic of umpire, and is responsible for parsing the JSON deployment file and kicking off the modules which fetch and unpack the dependencies. The options that the JSON file can contain are documented in the README file at the root of the project.
Deploy will read the data from the JSON file and start a "Fetch" module for each dependency listed.
(TODO: extract reading logic and create an adapter, so other data types are supported)
Notes:
- deploy.py also contains Windows specific logic. This logic appears at the beginning of the file and is documented with comments.
- deploy.py is the file that requires the most refactoring. It contains a large block of code that should be broken down into methods, or other modules where applicable.
Fetch contains the logic which queries the cache for a dependency. If the dependency doesn't exist within the cache, the Fetch module will download it from the source repository (currently only S3 is supported).
Fetch will convert the data obtained from Deploy for a particular dependency into an S3 url (ex: s3://thirdpartydependencies/win64/poco/1.1.1). Using this URL, it will download to a temporary directory in the cache and call the unpack function if requested to do so before placing the dependency in the correct location in the cache. For more info on the cache, view the Cache section.
Notes:
- Fetch is also responsible for md5 verification between the cache and the remote server, if keep_update is specified as True for a given dependency. If this is not specified, umpire will not query the remote server to see if a package has been updated.
s3.py contains S3 helper functions the AsyncS3Downloader module. It takes an S3 URL, and can query AWS S3 case-insensitively to find the corresponding package and download it to a specific location.
Notes:
- AsyncS3Downloader will download ALL files matching a prefix. That said, each "folder" in S3 should only contain a single package. Multiple packages will lead to undefined behaviour.
- It has some hooks for progress reporting, though it is unclear how to accomplish this in the higher levels of the application.
- S3.py uses boto3, and will use the same authentication schemes that boto3 offers.
- All authenticated calls to s3 will fallback to anonymous credentials, and will throw an exception if that fails.
A very simple module which will determine which unpacking method to use based on a file type. Currently only supports .tgz .tar.gz and .zip extentions.
The beginnings of an self-auto-updating process which would keep umpire updated automatically. This was scrapped, but the code still exists here and is referenced from deploy.py (though it will never reach there, unless the flag is changed manually).
The cache is a simple representation of dependencies that have been downloaded from the remote repository. The root of cache contains separate folders for different repositories. Files are copied or linked directly from this location.
The cache root contains a "downloading" folder, which houses incomplete files while they are downloading. Additionally, it contains caches for each repository. A repository cache is prefixed with the identifier followed by the type (currently only "s3").
For example: thirdpartydependencies.s3
Within each repository cache is a copy of the file structure that exists on the remote cache. For umpire this is currently:
$PLATFORM/$NAME/$VERSION/
For example: win64/dds_client/v1
Within this folder contains the dependency files, which are either linked or copied from this location.
Since Umpire is multi-processed, and has the ability to share caches across systems, lockfiles are a necessity on a per-dependency basis. As of the latest version, lockfiles have a specific format and name. A .umplock file looks like this:
host_id::pid
Where the host_id is a unique identifier for the host, and the pid is the process ID. These two value determine how/if/when the Umpire process reading the lockfile can unlock so it may re-lock and modify the dependency. For example, if the host_id matches the host_id of the calling Umpire, and the pid no longer exists, Umpire will detect this and remove the lockfile. More situations are defined within the "lock" method of the cache.
Notes:
- The "unique identifier for the lock file is currently the hostname -- not unique!
- The "worst-case" situation has umpire wait 300s for the cache to unlock. This is when the host_id does not match the host_id of the calling process. This timeout is configurable.
Info files contain information regarding the current level of the cache. These files currently only exist in the folder containing the dependency, and contain info like the MD5, name, platform, version and the version of the .umpire file for backwards compatibility.
- You cannot have the same dependency in the same repository using
"unpack": Trueand"unpack": False-- this will cause umpire to deploy whichever version was added to the cache first. - There is currently no clean up logic for dependencies that have new versions. Once a dependency version has been cached, it will exist indefinitely.
Configuration values are set in the config.py file. The application currently always uses the default values as this file is overwritten every upgrade. Effort will need to be made to migrate this to a proper configuration file.
Included in the repository is a sanity testing suite which tests basic functionality of umpire using a test repository which will need to be configured on a per user basis. Among the things tested are:
- Asynchronous downloads
- tgz and zip unpacking
- Contention on a single dependency (i.e. lockfiles)
- Deployment using copy
- Deployment using symlink
You can run the entire suite of tests with two commands:
cd ./umpire/test
python -m unittest discover -s . -p 'tc*.py'
Currently these tests only run on UNIX based systems.
It has quite a few interdependencies, and a switch to using dependency injection is planned, and is evident in some parts of the code, though far from complete and would require some rethinking around the base classes.
Due to the design choices in umpire, this can result in long methods for a Module. Since Modules are procedural in nature, this isn't that large of a problem -- though some effort into refactoring should be made.