Docker Images for Optimization

Short guide how to create a new optimization docker image

Background

Rembrandt uses predefined Docker images for executing the allocation algorithms. As described in the section Resource Assignment, these images need to be registered to Rembrandt before they can be used in recipes, including the incoming set of resources types, as well as the output resource type. In the allocation process, the following steps are performed in order to execute the dockerized allocation algorithm:

1. A temporary folder within the dataExchangeDirectory (see Back-end Configuration) is created using a random, unique identifier.

2. For each list of resources (each corresponding to one of the configured input resource types), a file is created in this new directory. The file is named after the respective resource type and contains the serialized list (JSON) of the resources.

3. The configured Docker image is instantiated and the path to the just created directory is mounted into the container at /mnt/rembrandt/ . This path is also available in the container under the environment variable folderPath.

4. The container can now read and parse the files in the directory to retrieve the resources needed for computing.

5. As soon as the algorithm has finished, the container must return the optimization result: This is done by writing the output back to the directory in serialized form (JSON). The file must be named result.txt.

6. Once the Docker container has terminated, Rembrandt reads the result file and tries to convert the content to resource instances of the type configured for the algorithm.

For a new optimization Docker image it will be necessary to have some kind of a wrapper script within the image, in order to comply to the steps above. The wrapper must handle the file-based communication, parse the objects and start the actual optimization algorithm in an appropriate way.

Data-Structure

The input files provided to the container contain the lists of serialized resource instances. Such a file could look like this:

Parcel.txt

[
    {
        "attributes": {
            "Sender": "Sven",
            "Receiver": "Christian",
            "startTime": "16:51:00",
            "distance": "13"
        }
    },
    {
        "attributes": {
            "Sender": "Sven",
            "Receiver": "Max",
            "startTime": "16:44:00",
            "distance": "75"
        }
    }
]

Obviously, two parcels are provided to the optimization algorithm by Rembrandt after evaluating all previous steps in the recipe.

For the result.txt, Rembrandt expects the same JSON-data structure, which looks like this in general:

result.txt

[
    {
        "attributes": {
            "attributeKey": "attributeValue",
            "attributeKey2": "attributeValue2"
        }
    },
    {
        "attributes": {
            "attributeKey": "attributeValueForSecondResourceInstance",
            "attributeKey2": "attributeValue2ForSecondResourceInstance"
        }
    }
]

The used resource type must not be named in the result file as the output resource type of the algorithm was already provided in the algorithm-definition in Rembrandt. The attribute keys that must be used in the result.txt correspond to the name of the attributes defined for this resource type in Rembrandt.

Example using Node / JavaScript

This section gives an overview how the structure for a new Docker image could look like. It consists of a npm package definition, the wrapper-script and the Dockerfile. The optimization algorithm is omitted and must be added for each specific use-case. In general, it should be possible to create a working image by copying the following files and implementing a custom optimization algorithm (implement myOptimizationMethod in index.js or add another file which contains the definition).

After all required adjustments have been made, simple run the following command in the directory:

docker build -t rembrandt/testOptimization .

The new Docker image named rembrandt/testOptimization is build and added to the local Docker image repository. In Rembrandt, this image can now be added as an algorithm and used in recipes. Below, the three files mentioned above are displayed and described in more detail.

The files can also be found in the official Rembrandt repository on GitHub. Click here

package.json - NPM Package Definition

package.json

{
  "name": "rembrandt-test-optimization",
  "version": "1.0.0",
  "description": "This is an example for creating a new optimization Docker image for Rembrandt.",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "author": "",
  "license": "MIT"
}

index.js - Wrapper-Script

The following script is an example for a possible wrapper script. It reads the two lists of input resources of the types Parcel and Driver. Then, the optimization algorithm is called and provided with the inputs. At the end, the output of the optimization is written back to the directory.

index.js

const folderPath = process.env.folderPath;
// fs is a module for handling file operations.
const fs = require('fs');

// Use Resource Type Names as configured in Rembrandt
const inputResourceTypes = ["Parcel", "Driver"];

/* Iterates over all configured resource types, reads the respective file and parses it to objects.
 * The returned object could look like this:
 *   { "Parcel": [ ParcelObject1, ParcelObject2, ParcelObject3 ],
 *     "Driver": [ DriverObject1, DriverObject2 ] }
 * Where ParcelObjectX and DriverObjectX are objects themselves, following the structure as defined in Rembrandt.
 */
function readInputFiles() {
  const readInputs = {};
  inputResourceTypes.forEach(resourceType => {
    readInputs[resourceType] = JSON.parse(fs.readFileSync(`${folderPath}/${resourceType}.txt`));
  });
  return readInputs;
}

function writeResult(result) {
  fs.writeFileSync(folderPath + '/result.txt', JSON.stringify(result));
}

const inputResources = readInputFiles();

// Do some object transformation here if needed.

// Call the optimization method here and pass the required arguments. E.g.:
const optimizedResult = myOptimizationMethod(inputResources["Parcel"], inputResources["Driver"]);

writeResult(optimizedResult);

Dockerfile - Instructions How to Build the Image

Dockerfile

FROM node:alpine
WORKDIR /home/node/app
COPY . .
RUN npm install
CMD npm start