Procedural Content Generation

Reinforcement learning can be prone to over-fitting in environments where the initial conditions are limited and the environment dynamics are deterministic. Procedural content generation is an important tool in Reinforcement learning, as it allows level maps to be created on-the-fly. This gives the agent a much more complex challenge, and stops it from being able to overfit on a small dataset of levels.

Level Maps

Levels in Griddly environments are defined by strings of characters. The MapCharacter used are defined in the GDY files of the game. These MapCharacter can be found in the GDY files or in the game’s documentation.

Basic Map

W  W  W  W  W  W
W  A  .  .  .  W
W  .  .  .  .  W
W  .  .  .  .  W
W  .  .  .  g  W
W  W  W  W  W  W

How the above Doggo level is rendered.

You can see in this map example above that the A character defines the Dog and the g character defines the goal. W defines the walls and . is reserved for empty space.

This is a basic example and generating levels for this environment might not be too interesting…

Clusters Level Generator

A much more complicated example would be to use the Clusters game and generate new levels. The aim of the Clusters game is for the agent to push coloured blocks together to form “clusters”, whilst avoiding spikes. The game is fully deterministic and there are only 5 levels supplied in the original GDY file. This makes it a perfect candidate for building new levels and testing if Reinforcement Learning can still solve these levels!

Level Generator Class

Here’s an example of a level generator for the cluster’s game. Levels are generated with simple configurable heuristics such as maximum number of each coloured boxes and maximum numbers of spikes. The boxes and spikes are randomly placed in the grid to create the initial game layout. The agent is also added to the grid in a random position.

The LevelGenerator class can be used as a base class. Only the generate function needs to be implemented.

class ClustersLevelGenerator(LevelGenerator):
    BLUE_BLOCK = 'a'
    BLUE_BOX = '1'
    RED_BLOCK = 'b'
    RED_BOX = '2'
    GREEN_BLOCK = 'c'
    GREEN_BOX = '3'

    AGENT = 'A'

    WALL = 'w'
    SPIKES = 'h'

    def __init__(self, config):
        self._width = config.get('width', 10)
        self._height = config.get('height', 10)
        self._p_red = config.get('p_red', 1.0)
        self._p_green = config.get('p_green', 1.0)
        self._p_blue = config.get('p_blue', 1.0)
        self._m_red = config.get('m_red', 5)
        self._m_blue = config.get('m_blue', 5)
        self._m_green = config.get('m_green', 5)
        self._m_spike = config.get('m_spike', 5)

    def _place_walls(self, map):

        # top/bottom wall
        wall_y = np.array([0, self._height - 1])
        map[:, wall_y] = ClustersLevelGenerator.WALL

        # left/right wall
        wall_x = np.array([0, self._width - 1])
        map[wall_x, :] = ClustersLevelGenerator.WALL

        return map

    def _place_blocks_and_boxes(self, map, possible_locations, p, block_char, box_char, max_boxes):
        if np.random.random() < p:
            block_location_idx = np.random.choice(len(possible_locations))
            block_location = possible_locations[block_location_idx]
            del possible_locations[block_location_idx]
            map[block_location[0], block_location[1]] = block_char

            num_boxes = 1 + np.random.choice(max_boxes - 1)
            for k in range(num_boxes):
                box_location_idx = np.random.choice(len(possible_locations))
                box_location = possible_locations[box_location_idx]
                del possible_locations[box_location_idx]
                map[box_location[0], box_location[1]] = box_char

        return map, possible_locations

    def generate(self):
        map = np.chararray((self._width, self._height), itemsize=2)
        map[:] = '.'

        # Generate walls
        map = self._place_walls(map)

        # all possible locations
        possible_locations = []
        for w in range(1, self._width - 1):
            for h in range(1, self._height - 1):
                possible_locations.append([w, h])

        # Place Red
        map, possible_locations = self._place_blocks_and_boxes(

        # Place Blue
        map, possible_locations = self._place_blocks_and_boxes(

        # Place Green
        map, possible_locations = self._place_blocks_and_boxes(

        # Place Spikes
        num_spikes = np.random.choice(self._m_spike)
        for k in range(num_spikes):
            spike_location_idx = np.random.choice(len(possible_locations))
            spike_location = possible_locations[spike_location_idx]
            del possible_locations[spike_location_idx]
            map[spike_location[0], spike_location[1]] = ClustersLevelGenerator.SPIKES

        # Place Agent
        agent_location_idx = np.random.choice(len(possible_locations))
        agent_location = possible_locations[agent_location_idx]
        map[agent_location[0], agent_location[1]] = ClustersLevelGenerator.AGENT

        level_string = ''
        for h in range(0, self._height):
            for w in range(0, self._width):
                level_string += map[w, h].decode().ljust(4)
            level_string += '\n'

        return level_string

This generates levels like the following:


A 10x10 map generated by the above code.

Using LevelGenerator

In the most simple case, the level generator can be used just before the level resets and the generated string can be passed to env.reset(level_string=...)

if __name__ == '__main__':

    config = {
        'width': 10,
        'height': 10

    renderer = RenderToFile()

    level_generator = ClustersLevelGenerator(config)

    env = gym.make('GDY-Clusters-v0')


Using LevelGenerators with RLLib

The LevelGenerator base class is compatible with RLLib and can be used and configured through the standard RLLib configuration.

For example, the level generator and its parameters can be set up in the env_config in the following way:

'config': {


    'env_config': {
        'generate_valid_action_trees': True,
        'level_generator': {
            'class': ClustersLevelGenerator,
            'config': {
                'width': 6,
                'height': 6,
                'p_red': 0.7,
                'p_green': 0.7,
                'p_blue': 0.7,
                'm_red': 4,
                'm_blue': 4,
                'm_green': 4,
                'm_spike': 4