Skip to main content

​
App Container

The app container handles all user interactions and data analytics tasks. It comes with a built-in queue to process tasks asynchronously. For long-running tasks, it sends a job to the queue so that the request can be returned right away, and the background workers handle the tasks. We use the Celery library for managing workers. We use Redis as the broker and the result backend for Celery workers.
There are three types of long-running tasks, each type has a dedicated task queue as they require different types of resources and have different runtimes. We use a worker to handle each type of task. They can be configured usingsupervisord Number of Workers: You launch multiple workers by starting multiple instances of the Celery worker command managed by supervisord. Number of Concurrent Jobs per Worker (-c option): This controls how many tasks a single worker can process simultaneously.

​
Annotation worker

The annotation worker handles document annotation tasks, e.g., creating review batches, creating datasets, updating annotations, etc. These tasks are more I/O-bound than CPU-intensive. This means that the performance bottleneck for these tasks typically lies in disk access or network communication rather than the computation power of the CPU. Each worker is mostly waiting on database I/O, therefore you could potentially configure each Celery worker with a much higher concurrency setting. For example, with 32 CPUs, you can use the following command:

​
Data processing worker

The data processing worker handles data ingestion and processing. It sends jobs to the Data processing container which runs the data processing pipelines. The data processing jobs require substantial computational resources and use Ray for parallel processing. You can set the number of CPUs used by each Ray task by setting the RAY_NUM_CPUS environment varaible. Each Celery worker handles one Ray task at a time and each Ray task uses multiple CPUs. You can start multiple Celery workers (each handling one task at a time) if there are sufficient CPUs available.

​
ML model worker

The ML model worker handles model training/inference tasks. It sends jobs to the ML backend container which runs the machine learning models. Training machine models are CPU-intensive and have high memory requirements. Using multiple workers each with -c 1 might provide better performance and reliability, ensuring that each task can utilize its designated resources without interference.

​
Data Processing Container

The data processing container runs data processing jobs and uses Celery and Redis to handle long-running background tasks. It communicates with the App container through HTTP requests at http://host.docker.internal:5007 if it is running on the same host machine or an internal IP address if running on a different machine.

​
ML Backend Container

The ML backend hosts machine learning models and handles background training jobs with Celery and Redis. The ML container handles the training and prediction jobs sent by App container. It communicates with the App container through HTTP requests at http://host.docker.internal:5006 if it is running on the same host machine or an internal IP address if running on a different machine.

​
ML GPU Container

Some ML models are served using GPU and running inside an NVIDIA GPU container. It requires installing Docker with NVIDIA Container Toolkit.
  • NVIDIA GPU Drivers 535.161.07
  • CUDA Version: 12.2
  • NVIDIA Container Toolkit 1.14.5-1
The ML GPU container communicates with the App container through HTTP requests at http://host.docker.internal:5008 if it is running on the same host machine or an internal IP address if running on a different machine.