Skip to main content

Landscapes - World Partition and Tiling

This discusses making large Unreal landscapes using data from a real-world location using World Partition to support larger maps.

width90

Background​

This is part of a series of articles about landscapes, other articles are:

Landscapes - Building GDAL
Landscapes - Material Concepts
Landscapes - Nanite Tessellation
Landscapes - Using Real Location Data Landscapes - World Partition and Tiling

Approach​

Following on from the previous article which showed how to download DEM data for real locations and how to make a heightmap to create a landscape in Unreal, this article looks at World Partition works and how we can import larger heightmaps.

The process described here uses tools from the GDAL library, instructions on building them can be found at Nanite Landscape Materials - GDAL

Data​

We will be reusing the demdata.tif file we downloaded from USGS in the previous article. This covers an area approximately 58km by 78km.

Conversion to meters​

This command creates a file demdata30m.tif which is the DEM data converted from degrees to meters, with one pixel approximately every 38m.

gdalwarp -t_srs EPSG:3857 -dstnodata -9999 -ot Float32 -r cubic demdata.tif demdata30m.tif -wo NUM_THREADS=ALL_CPUS -overwrite

World Partition​

World Partition is described here. It breaks the level into regions and:

  • dynamically loads content into memory for the regions which the player can see
  • dynamically loads HLOD lower resolution actors for content which is distant from the player

As the player moves actors are streamed in and out to try and reduce memory use.

A landscape created with World Partition is divided into regions and each region has some number of LandscapeStreamingProxy objects which manage streaming of that part of the landscape.

The World Partition Editor is a window which displays a 2d view minimap so you can load and unload regions at will as you work on them.

The minimap is created using the Build | Build Minimap menu option or the Build | Minimap button on the World Partition Editor window.

Hierarchical Levels of Detail (HLODs) enabled Unreal to display less-detailed versions of distant objects. They are built using the Build | Build HLODS menu or the Build | HLODs button on the World Partition Editor window.

Stretching a Heightmap​

From the previous article we have the file heightmap1m_8192.png which has samples every 1m and a resolution of 8192x8192, so covering 8km x 8km.

If we don't want to retain fidelity to the real-world terrain, we can create a larger terrain by importing it with large scale values.

The values we used before were:

width90

If we double the X and Y scale values the terrain will be twice as large, i.e. 16km x 16km.

This does the effect that when compared to a person, all terrain features have double in size, but it's a simple way of creating a large terrain.

If we do want to create a map from real world data with stretching it, we can use tiles.

Tiles​

A large heightmap can be broken down into tiles, each represented by a separate file. Some larger heightmaps can only be loaded if they are broken into tiles (see the table below).

Using the demdata.tif file from the previous article, in which the measurements are degrees, we can create the equivalent file in ~30m resolution using this command:

gdalwarp -t_srs EPSG:3857 -dstnodata -9999 -ot Float32 -r cubic demdata.tif demdata30m.tif -wo NUM_THREADS=ALL_CPUS -overwrite

We can get the min and max heights from demdata30.tif using this command:

gdalinfo -mm demdata30m.tif

Computed Min/Max=1131.897,3204.311

We can make a heightmap, still with ~30m resolution, with this command:

gdal_translate -ot UInt16 -scale 1131.897 3204.311 0 65535 -of PNG demdata30m.tif heightmap30m.png

One way to split the heightmap at 1m measurements is to convert it from 30m to 1m resolution using this command:

gdalwarp -t_srs EPSG:3857 -dstnodata -9999 -tr 1 1 -ot Uint16 -r bilinear heightmap30m.png heightmap1m.png -wo NUM_THREADS=ALL_CPUS -overwrite

The bilinear parameter determines how gdalwarp will position the new points it inserts every meter between the existing 30m points. There are many other options on how to do this, they are listed on this page

Tile Sizes​

We need have a way of working out what tile size we should use. Only certain tile sizes work with world partition.

For example, if we decide to have a tile size of 2017m, and a landscape of 5 x 5 tiles, so a total of 25 tiles.

The total heightmap resolution is 5 x 2017 = 10085 x 10085

The landscape import UI will suggest these values:

  • Section size 255x255 quads
  • Sections per component 2x2 sections
  • Number of components 20 x 20
  • Overall resolution 10201 x 10201
  • Total components 400

The overall resolution is 10201, but the heightmap is 10085 so the landscape will be bigger than the heightmap. Pixels on two edges of the heightmap between the edge of the tiles and the end of the landscape will be filled with zeros, producing an edge like this:

To make the tile size match the landscape size we need to use a heightmap size which both:

  • has resolution = ( 511 * N ) + ( N - 1 )
  • can be divided into a specified number of tiles, i.e. is not a prime number
  • can be divided into a small number of tiles, for example the resolution 45901 can be divided into 38809 tiles each 233x233 resolution, but we don't want to be making 38809 tiles, it will be too slow to both create and load them.

If we use a python script to generate some possibilities and set a max tile count of 200, we get this table:

Landscape SizeNum Tiles per sideTile Width (pixels)Tile CountVertex CountNeeds TilesLoadable
511x51177349261121FalseTrue
2041x2041131571694165681FalseTrue
4081x408175834916654561FalseTrue
4081x40811137112116654561FalseTrue
5611x56113118196131483321FalseTrue
6631x66311934936143970161FalseTrue
7141x714137193136950993881FalseTrue
7651x7651710934958537801FalseTrue
8671x86711366716975186241FalseTrue
8671x86712337752975186241FalseTrue
8671x86712929984175186241FalseTrue
9691x96911188112193915481FalseTrue
11221x112217160349125910841FalseTrue
12751x12751413111681162588001FalseTrue
14791x147917211349218773681FalseTrue
15301x15301111391121234120601FalseTrue
15301x15301131177169234120601FalseTrue
16321x1632119859361266375041FalseTrue
18361x183617262349337126321FalseTrue
18361x18361434271849337126321FalseTrue
20401x2040123887529416200801FalseTrue
20911x20911111901121437269921FalseTrue
21421x2142131691961458859241FalseTrue
21931x219317313349480968761FalseTrue
21931x21931131687169480968761FalseTrue
23461x2346129809841550418521FalseTrue
25501x255017364349650301001FalseTrue
26011x26011377031369676572121FalseTrue
26011x26011191369361676572121FalseTrue
26521x26521112411121703363441FalseTrue
28561x28561132197169815730721FalseTrue
29071x290717415349845123041FalseTrue
32131x321311129211211032401161FalseTrue
32131x321312313975291032401161FalseTrue
32641x3264174663491065434881FalseTrue
33661x336614182116811133062921TrueTrue
35191x351911327071691238406481TrueTrue
35701x357011918793611274561401TrueTrue
36211x3621175173491311236521TrueTrue
37231x372313112019611386147361TrueTrue
37741x377411134311211424383081TrueTrue
38251x382512913198411463139001TrueTrue
39781x3978175683491582527961TrueTrue
40291x402914393718491623364681TrueTrue
41821x418211332171691748996041TrueTrue
43351x4335176193491879309201TrueTrue
43351x433511139411211879309201TrueTrue
43861x438612319075291923787321TrueTrue
44881x4488137121313692014304161TrueTrue
45391x453911923893612060342881TrueTrue
46921x4692176703492201580241TrueFalse

The "Needs Tiles" column shows whether a heightmap of this size can be loaded only by breaking it in to tiles. With Unreal Engine 5.5.3 tiles become necessary when the vextex count * 2 (for 16bits per pixel PNG format) exceeds the limit of an int32 variable (2147483647).

The "Loadable" column shows whether a heightmap of this size can be loaded at all. With Unreal Engine 5.5.3, even when using tiles, heightmaps with a vertex count which exceeds the limit of an int32 variable (2147483647) cannot be loaded.

Making tiles​

This python script will make and run a set of GDAL commands which will create separate PNG files for each tile.

import sys
import os
import subprocess
from concurrent.futures import ThreadPoolExecutor, as_completed
from tqdm import tqdm
import argparse

def run_command(cmd):
    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    return cmd, process.returncode, stdout.decode(), stderr.decode()

def make_tiles( input_file:str, tile_dir:str, tile_size:int, tile_count:int ):

    tile_prefix:str = "tile"
    min_height:float = 0
    max_height:float = 65535
    format:str = "-ot UInt16 -of PNG -scale " + str(min_height) +  " " + str(max_height) + " 0 65535"

    os.makedirs( tile_dir, exist_ok=True)

    commands: str = []

    for x in range(0, tile_count):
        for y in range(0, tile_count):
            cutting_frame = "-srcwin " + str(x*tile_size) + " " + str(y*tile_size) + " " + str(tile_size) + " " + str(tile_size)
            output_path = tile_dir + "/" + tile_prefix + "_x" + str(x) + "_y" + str(y) + ".png"
            full_command = "gdal_translate " + format + " " + cutting_frame + " " + input_file + " " + output_path
            commands.append( full_command )

    with ThreadPoolExecutor(max_workers=10) as executor:
        futures = {executor.submit(run_command, cmd): cmd for cmd in commands}

        with tqdm(total=len(futures), desc="Processing", unit="task") as pbar:
            for future in as_completed(futures):
                cmd, returncode, stdout, stderr = future.result()
                pbar.update(1)  

parser = argparse.ArgumentParser()
parser.add_argument("--input", type=str, default="", help="input height map")
parser.add_argument("--tile-dir", type=str, default="", help="output tile directory")
parser.add_argument("--tile-size", type=int,  default="-1", help="tile size in pixels")
parser.add_argument("--tile-count", type=int, default="-1", help="number of tiles per side of outer square")

args = parser.parse_args()

if args.input == "" or args.tile_dir == "":
    parser.print_usage()
    sys.exit()

if args.tile_size < 0:
    parser.print_usage()
    sys.exit()

if args.tile_count < 0:
    parser.print_usage()
    sys.exit()

make_tiles( tile_dir=args.tile_dir, tile_count=args.tile_count, tile_size=args.tile_size, input_file=args.input )

20911 x 20911​

If we choose a landscape size of 20911 x 20911, from the above table:

  • each side is 11 tiles
  • each tile is 1901 pixels
  • there are 121 tiles.

The following command will create the tiles in a directory called ".\tiles20911":

python maketiles.py --input heightmap1m.png --tile-dir .\tiles20911 --tile-count 11 --tile-size 1901

To import the tile set, set the Heightmap File property on the Import from Tile tab to the first tile, i.e. tile_x0_y0.png

This imports into Unreal with:

  • Section Size 255x255
  • Sections per Component 1x1
  • Number of Components 82x82
  • Overall resolution 20911x20911
  • Region count 36

41821 x 41821​

The following command will create the tiles in a directory called ".\tiles41821":

python maketiles.py --input heightmap1m.png --tile-dir .\tiles41821 --tile-count 13 --tile-size 3217

This imports into Unreal with:

  • Section Size 255x255
  • Sections per Component 1x1
  • Number of Components 164x164
  • Overall resolution 41821x41821
  • Total components 26896
  • Region count 36

It takes significantly longer to import than the 20911 x 20911 resolution.

If we save the map with the name "41821" the created actors are in this directory

..\Content_ExternalActors_\Maps\41821

The are 4.07 GB

Limits​

The largest file (i.e. the highest pixel resolution) Unreal can Import is 43351 x 43351. Internally (in FLandscapeTiledImage::ReadRegion) Unreal uses int32 values to index into heightmap data - the max value for an int32 is 2,147,483,647, so the resolution needs to be less than sqrt(2,147,483,647) which is 46340.

Bugs​

The FLandscapeImageFileCache caches image data such as resolutions. If you set the Heightmap File property of the landscape dialog to a heightmap file it will read the file resolution and save it in the cache. If you subsequently recreate the file at a different resolution and reuse the import file dialog it will populate the import dialog with old data from the cache, not what is in the recreated file.

To stop this happening change to some random PNG file and then change back to the file you want to use, this clears the cache.

Higher resolutions​

Do we get a better landscape if we take the original DEM data sampled at degree resolution, convert it to meters, then convert it to a higher resolution?

Specifically, if we compare a heightmap where each pixel is ~34m, to a heightmap where each pixel is 5m, is there a noticeable difference?

Logically we cannot add detail, but depending on how we do the sampling we will both:

  • add new points between the original DEM points
  • adjust the position of the original DEM points

If we start with the demdata.tif which has this elevation data:

Computed Min/Max=3595.000,6180.000

we can convert it to heightmap measured in meters:

gdalwarp -t_srs EPSG:3857 -dstnodata -9999 -ot Float32 demdata.tif demdata_meters.tif -wo NUM_THREADS=ALL_CPUS -overwrite 
gdal_translate -scale 3595 6180 0 65535 -ot UInt16 -of PNG demdata_meters.tif heightmap_meters.png
# trim to a 511 x 511 pixel size
gdal_translate -srcwin 0 0 511 511 -ot UInt16 -of PNG heightmap_meters.png heightmap_meters511.png

The Pixel Size in heightmap_meters511.png is (33.785552205353980,-33.785552205353980)

Therefore the Scale property of the Unreal landscape import screen should be

3378.5522, 3378.5522, 6180.000 * 100 / 512 = 1202.33

Then we make a heightmap which is a 5m resolution by sampling the heightmap_meters.png file:

gdalwarp -tr 5 5 -ot Uint16 -r cubic heightmap_meters.png heightmap_5meters.png -wo NUM_THREADS=ALL_CPUS -overwrite

The heightmap_meters511.png is 511 pixels by 33.7855m = 17,264.39m so truncate the 5m file to 3,453 x 3,453 pixels:

gdal_translate -srcwin 0 0 3453 3453 -ot UInt16 -of PNG heightmap_5meters.png heightmap_5meters511.png

The Pixel Size in heightmap_5meters511.png is = (5.000000000000000,-5.000000000000000)

Therefore the Scale property of the Unreal landscape import screen should be

500, 500, 1202.33

Comparing the lit wireframe views of the two landscapes, firstly the 33.78 meters per pixel:

And the 5m per pixel:

This shows that resampling the heightmap to a higher resolution both smooths the landscape and adds more detail.

Another advantage of higher resolutions is that the triangles are smaller so textures used in materials will not be stretched so much.

References​