Development Guide¶
Starting the services locally¶
Please refer to the local install guide to start the development.
Dependencies¶
All AM services use the following libraries:
Backend and workers only:
- minio - as an Amazon S3 compatible object store
UI only:
Adding new AM Backend functionality¶
The code for the AM Backend can be found in the am module:
- init.py: is used to configure the app and setup all the app routes
- main.py: a startup script that can be used for development and testing
- controller.py: the backend controller which connects the mangers to the API routes
- managers.py: the worker manager, keeping the state of all registered workers
- routes.py: all the REST routes, in falcon format
Adding new UI functionality¶
The code for the AM Backend can be found in the ui module:
- init.py: is used to configure the app and setup all the app routes
- main.py: a startup script that can be used for development and testing
- backend.py: the UI controller which connects the mangers to the API routes
- *_utils.py: utils classes for alerts or Jinja2 integration
- backend.py: the client used to make requests to the backend
- controller.py: the UI controller which connects the mangers to the API routes
- routes.py: all the REST routes, in falcon format (falcon docs on routing)
- middleware.py: a content type validator
The Jinja2 templates written in HTML and are located in the ui/templates.
The frontend uses a few JavaScript libraries. If you add new dependencies, then these must be added to package.json and the assets (js, css and font files exposed by the library) must be added to move-assets.js.
You may find it convenient to set the SERVICE_AM_HOSTNAME
environment variable to point at an instance of the AM backend running on a different machine during development and testing.
Adding a new Worker¶
Creating a worker requires changes in several places:
- a Markdown documentation file in
docs/workers
- a Dockerfile in a new subdirectory of
docker/
- the worker itself, in a new subdirectory of
workers/
- an addition to
docker-compose.yml
You will also need to modify the docker-compose.am.yml
template used by the
OVE Installer.
All new workers can extend the BaseWorker
class. The methods that require an implementation
are marked as abstract in the base class:
from typing import List, Dict
from common.entities import OveMeta
from workers.base import BaseWorker
class NewWorker(BaseWorker):
def worker_type(self) -> str:
"""
:return: the worker type as a string. This value can be a valid WorkerType or anything else
"""
return ""
def extensions(self) -> List:
"""
:return: the extensions handled by this worker
"""
return []
def description(self) -> str:
"""
:return: description in human-readable format
"""
return ""
def docs(self) -> str:
"""
:return: the worker documentation document url, in markdown format
see https://github.com/trentm/python-markdown2 for more details
"""
return ""
def parameters(self) -> Dict:
"""
:return: the worker parameter description, in json-form format:
see http://www.alpacajs.org/ for more details
"""
return {}
def process(self, project_id: str, meta: OveMeta, options: Dict):
"""
Override this to start processing
:param project_id: name of the project to process
:param meta: the object to process
:param options: task options, passed by the asset manager. Can be empty
:return: None
:raises: Any exception is treated properly and logged by the safe_process method
"""
pass
# this is where the worker processing happens
To test a worker using Docker, you will need to expose the correct port (e.g, with -p 6090:6090
), and set the
environment variables SERVICE_AM_HOSTNAME
, SERVICE_AM_PORT
, and SERVICE_HOSTNAME
. If testing with an instance of
the Asset Manager on another machine, that machine must be able to resolve the SERVICE_HOSTNAME
to the machine running
the worker.