Development Guide¶
Starting the services locally¶
Please refer to the local install guide to start the development.
Dependencies¶
All AM services use the following libraries:
Backend and workers only:
- minio - as an Amazon S3 compatible object store
UI only:
Adding new AM Backend functionality¶
The code for the AM Backend can be found in the am module:
- init.py: is used to configure the app and setup all the app routes
- main.py: a startup script that can be used for development and testing
- controller.py: the backend controller which connects the mangers to the API routes
- managers.py: the worker manager, keeping the state of all registered workers
- routes.py: all the REST routes, in falcon format
Adding new UI functionality¶
The code for the AM Backend can be found in the ui module:
- init.py: is used to configure the app and setup all the app routes
- main.py: a startup script that can be used for development and testing
- backend.py: the UI controller which connects the mangers to the API routes
- *_utils.py: utils classes for alerts or Jinja2 integration
- backend.py: the client used to make requests to the backend
- controller.py: the UI controller which connects the mangers to the API routes
- routes.py: all the REST routes, in falcon format (falcon docs on routing)
- middleware.py: a content type validator
The Jinja2 templates written in HTML and are located in the ui/templates.
The frontend uses a few JavaScript libraries. All these must be added to package.json and the assets (js, css and font files exposed by the library) must be added to move-assets.js.
Adding a new Worker¶
Creating a worker requires changes in several places:
- a Markdown documentation file in
docs/workers
- a Dockerfile in a new subdirectory of
docker/
- the worker itself, in a new subdirectory of
workers/
- an addition to
docker-compose.yml
You will also need to modify the docker-compose.am.yml
template used by the
OVE Installer.
All new workers can extend the BaseWorker
class. The methods that require an implementation
are marked as abstract in the base class:
from typing import List, Dict
from common.entities import OveMeta
from workers.base import BaseWorker
class NewWorker(BaseWorker):
def worker_type(self) -> str:
"""
:return: the worker type as a string. This value can be a valid WorkerType or anything else
"""
return ""
def extensions(self) -> List:
"""
:return: the extensions handled by this worker
"""
return []
def description(self) -> str:
"""
:return: description in human-readable format
"""
return ""
def docs(self) -> str:
"""
:return: the worker documentation document url, in markdown format
see https://github.com/trentm/python-markdown2 for more details
"""
return ""
def parameters(self) -> Dict:
"""
:return: the worker parameter description, in json-form format:
see http://www.alpacajs.org/ for more details
"""
return {}
def process(self, project_name: str, meta: OveMeta, options: Dict):
"""
Override this to start processing
:param project_name: name of the project to process
:param meta: the object to process
:param options: task options, passed by the asset manager. Can be empty
:return: None
:raises: Any exception is treated properly and logged by the safe_process method
"""
pass
# this is where the worker processing happens
To test a worker using Docker, you will need to expose the correct port (e.g, with -p 6090:6090
), and set the
environment variables SERVICE_AM_HOSTNAME
, SERVICE_AM_PORT
, and SERVICE_HOSTNAME
. If testing with an instance of
the Asset Manager on another machine, that machine must be able to resolve the SERVICE_HOSTNAME
to the machine running
the worker.