TurokEX Map Format Specifications

//
// Turok EX Map Format Specifications
//
// By BehemothProgrammer
// Revised: 07/03/19
//

//-----------------------------------------------------------------------------
//
// What is this document for?
//
//-----------------------------------------------------------------------------

This technical document explains the Turok Dinosaur Hunter Remastered .map file
format and is targeted towards users with programming experience.

//-----------------------------------------------------------------------------
//
// Map Format
//
//-----------------------------------------------------------------------------

Map files (.MAP) are binary files that are read with offset addresses to data structures.
It uses a kexArchive format which is also based directly on Iguana's archive format for their file structure in Turok 1 and 2.
Having a good idea of how the archive format works will allow you to be able to parse the file,
including data from the Turok 1 and 2 roms which applies to CARTDATA.DAT in Turok 1. And for N64 roms,
same format except that the dwords are in big endian format.

There are two types of archive formats: indexed and data sets. I'll start off with indexed:

In an indexed archive the header always begins with 4 bytes(int) which is the number of offsets. Then for N offsets you got 4 bytes(int).
And there is 1 offset added to the end of the offsets that points to the end of the archive.
Each offset is relative to the beginning of the archives header.
These offsets can lead to 1 of 3 things; another indexed archive, a data set archive, or arbitrary data.
If you want to determine the size of the data it's offsetting to, you can subtract the next offset with the current one.
The archives have padded the data in rounds of 8/16 bytes.

Data sets are similar, except that they begin with 8 bytes. first 4 bytes(int) being the stride size, and the second 4 bytes(int) is the count.
Following that is a glob of bytes which is stride size * count (the data structure)

//-----------------------------------------------------------------------------
//
// Step by step walkthrough of reading a map
//
//-----------------------------------------------------------------------------

So let's start by opening a .map file in a hex editor of your choice.
The first thing you'll see is a indexed archive so that means the very first 4 bytes(int) in the file is the root archive headers number of offsets.

You'll see that the value should be 7, for 7 offsets. (It'll say 8 if there's visibility tables for the map)
Now let's read the first offset(the maps version number) for this root indexed archive(the next 4 bytes(int)).
Go to that byte in your hex editor at 0 (start of header) + the first offset value.
Which will take you to arbitrary data, 4 bytes(int), which we know is the maps version number and it's value should be 1.

Now let's go back to our root indexed archive and follow the second offset which will take us to another indexed archive which we already know has
offsets related to the maps World Properties. This world indexed archive has 3 offsets.

The first offset in the world archive leads to arbitrary data:      Sunlight Direction  - 12 bytes (3 floats, xyz)
The second offset in the world archive leads to arbitrary data:     Sunlight Color      - 16 bytes (4 floats, rgba)
The third offset in the world archive leads to arbitrary data:      Ambient Color       - 16 bytes (4 floats, rgba)

Now let's go back to our root indexed archive and follow the third offset which will take us to a dataset archive which we know
is just a string for the World Sky Material. Remember this is a dataset archive. We read the archive header;
the first 4 bytes(int) being the size of each object in bytes, in this example is 29 (for: skies/skyMaterials/sky_brown and a 0 null byte at the end).
the second 4 bytes(int) being the number of objects which is 1 (for 1 string).
We then read the next 29 bytes as chars for our string of the World Sky Material (and you probably want to remove or not read the last 0 null character from the string).

Now let's go back to our root indexed archive and follow the fourth offset which will take us to a indexed archive (Collision) which has 3 offsets.
The first offset leads to a dataset archive that is all the vertices in the map used for our sectors.
Again, read it's header like you did with the World Sky Material string.
the first 4 bytes(int) is the size in bytes that each vertice uses (which is 16 bytes)
the second 4 bytes(int) being the number of vertices in the map.

name                bytes              notes
---------------------------------------------------------
Position            12 (3 floats)      xyz
Height              4 (float)          The ceiling height

Let's follow the second offset in the Collision indexed archive which takes us to a dataset archive (SectorSets).
Read the header to get the size(which is 64) and count.

name                bytes              notes
---------------------------------------------------------
flags               4 (uint)
fog color           4 (4 bytes)        rgba
water color         4 (4 bytes)        rgba
fog zfar            4 (float)
water zfar          4 (float)
water height        4 (float)
sky height          4 (float)
sky speed           4 (float)
blend length        4 (float)
args                12 (6 int16)
floor impact id     1 (byte)
wall impact id      1 (byte)
ambience            1 (byte)
automap color ID    1 (byte)
music               1 (byte)
cull bits           1 (byte)           Kex editor assigns a default value of 255
unused              2 (2 bytes)        unused bytes. both are always 0

Let's follow the third offset in the Collision indexed archive which takes us to a dataset archive (Sectors).
Read the header to get the size(which is 18) and count. IMPORTANT NOTE: The kex editor saves sectors of size 18 bytes, however the original maps only stored 16 bytes of data that did not include draw order, which was added in TurokEX. So the size might be 16 or 18 depending on if it's from the original turok map editor or the kex editor.

name                bytes              notes
---------------------------------------------------------
SectorSet Index     2 (uint16)
flags               2 (uint16)          This is exactly the same as the first 16 bits in the SectorSet flags
indices             6 (3 uint16)        The 3 vertice indexes this sector uses
edge links          6 (3 uint16)        The 3 edges of this sector. The index to the other sector that shares an edge with the current sector. If it's 0xffff then nothing is linked and the edge is a solid wall.
draw order          2 (int16)          Prioritizes automap colors (in kex editor saved maps only)

Now let's go back to our root indexed archive and follow the fifth offset which will take us to a indexed archive (Grid Bounds) which has 3 offsets.
The first offset leads to a dataset archive that is the grid bounds size width and height. Read the header. Size and count should both be 2.

name                bytes              notes
---------------------------------------------------------
width               2 (int16)          number of grid bounds on the x axis
height              2 (int16)          number of grid bounds on the y axis

The second offset in the Grid Bounds indexed archive leads to a dataset archive that is all the grid bounds min x and y position.
Read the header. Size(which is 8) and count (which should be width*height).

name                bytes              notes
---------------------------------------------------------
min x               4 (float)
min y               4 (float)

The third offset in the Grid Bounds indexed archive leads to a dataset archive that is all the grid bounds max x and y position.
Read the header. Size(which is 8) and count (which should be width*height).

name                bytes              notes
---------------------------------------------------------
max x               4 (float)
max y               4 (float)

Now let's go back to our root indexed archive and follow the sixth offset which will take us to a indexed archive (Grid Sections).
The number of offsets is the number of grid sections.
Each offset leads to a indexed archive which is the static meshes that the grid section contains and it has 2 offsets.
The static meshes archives first offset leads to a dataset archive that is all the static meshes properties in this grid section.
Read the header. Size(which is 88) and count.

name                bytes              notes
---------------------------------------------------------
origin              12 (3 floats)      xyz
scale               12 (3 floats)      xyz
rotation            16 (4 floats)      quaternion xyzw
bounding box min    12 (3 floats)      xyz
bounding box max    12 (3 floats)      xyz
texture index       4 (int)
flags               4 (uint)
sector index        4 (int)
collision radius    4 (float)
collision height    4 (float)
cull bits           4 (int)            cull bits was used for vis culling and its only used in Hub 5. It's just a arbitrary flag. If the flag matches the bits set for the sector you're viewing in, then they're culled out. cull bits were manually set by hand by the developers. Kex Editor always sets this to a value of -1

The second offset in the static meshes archive leads to a indexed archive (static meshes model files)
The number of offsets is the number of static meshes in the grid section.
Each offset leads to a dataset archive that is the string used for the static meshes model path.

Now let's go back to our root indexed archive and follow the seventh offset which will take us to a indexed archive (Actors) which has 3 offsets.
The first offset leads to a dataset archive that is all the actors in the map properties. Read the header. Size(which is 140) and count.

name                   bytes              notes
---------------------------------------------------------
type                   4 (int)
origin                 12 (3 floats)      xyz
scale                  12 (3 floats)      xyz
yaw                    4 (float)          in rads
sector index           4 (int)
bounding box min       12 (3 floats)      xyz
bounding box max       12 (3 floats)      xyz
sight range            4 (float)          squared
sight fov              4 (float)          in rads
loud range             4 (float)          squared
quiet range            4 (float)          squared
attack range           4 (float)          squared
fly height             4 (float)          squared
collision width        4 (float)
collision wall width   4 (float)
collision height       4 (float)
collision dead height  4 (float)          also known as Step Height
melee range            4 (float)          squared
leash radius           4 (float)          squared
trigger delay          4 (float)
species mask           4 (int)            unused in Turok EX. Kex editor sets this value to 255.
Spawn Flags 1          4 (uint)
Spawn Flags 2          4 (uint)
Spawn Flags 3          2 (uint16)
Health                 2 (int16)
TID                    2 (int16)
Target TID             2 (int16)
Max Regenerations      2 (int16)
Attack Chance          1 (byte)
Trigger Anim           1 (byte)
Variant                1 (sbyte)
Texture                1 (sbyte)
Params 1               1 (sbyte)
Params 2               1 (sbyte)

The second offset in the actors archive leads to a indexed archive (actors model files)
The number of offsets is the number of actors in the map.
Each offset leads to a dataset archive that is the string used for the actors model path.

The third offset in the actors archive leads to a indexed archive (actors anim files)
The number of offsets is the number of actors in the map.
Each offset leads to a dataset archive that is the string used for the actors anim path.


If there was only 7 offsets in the root indexed archive then you've read through the entire file! If you have an eighth offset then that will lead
to the maps Visibility tables dataset archive which answers the question: which staticmeshes are visible from this sector.
Size 328 (example. number of static meshes / 8 (for each bit) then ceil it plus padding)
Count 4207 (example. This is number of sectors in the map)


Before you start writing your code to read/write the map you pretty much have to have
a class/system for handling indexed/data set archives in place before hand.

Working C# code I've made for reading and writing Turok 1 and 2 PC Remastered map files can be found in this Unity package.
I highly suggest looking at T1MapMono.cs and Archive.cs if you're having trouble understanding.
https://www.dropbox.com/s/0v95pis1skotk62/TurokEditors.unitypackage?dl=0

//-----------------------------------------------------------------------------
//
// Enums
//
//-----------------------------------------------------------------------------

enum ActorSpawnFlags1
{
    Solid                 = (1 << 0),
    ProjectileAttack1     = (1 << 1),
    Leader                = (1 << 2),
    SnapToFloor           = (1 << 3),
    ExplosionDeath        = (1 << 4),
    ClimbWalls            = (1 << 5),
    ProjectileAttack2     = (1 << 6),
    NoRepeatExplosion     = (1 << 7),
    DieOnExplosion        = (1 << 8),
    Flocker               = (1 << 9),
    SlowEnemy             = (1 << 10),
    RandomResurrection    = (1 << 11),
    RandomFeignDeath      = (1 << 12),
    Kamikaze              = (1 << 13),
    AvoidPlayers          = (1 << 14),
    FloatInWaterOnDeath   = (1 << 15),
    Teleport              = (1 << 16),
    CastShadow            = (1 << 17),
    TeleportWait          = (1 << 18),
    UseStrongAttacks      = (1 << 19),
    UseWeakAttacks        = (1 << 20),
    Sniper                = (1 << 21),
    MeltOnDeath           = (1 << 22),
    AvoidWater            = (1 << 23),
    Flying                = (1 << 24),
    TeleportAvoidWater    = (1 << 25),
    TeleportAvoidCliffs   = (1 << 26),
    TriggerStuff          = (1 << 27),
    CannotCauseAFight     = (1 << 28),
    NoWallCollision       = (1 << 29),
    ScreenShake           = (1 << 30),
    RespawnAnimation      = (1 << 31)
}

enum ActorSpawnFlags2
{
    DropItemMask1           = (1 << 0), //Explosive Shells
    DropItemMask2           = (1 << 1), //Grenade
    DropItemMask3           = (1 << 2), //Medium Health
    DropItemMask4           = (1 << 3), //Full Health
    DropItemMask5           = (1 << 4), //Ultra Health
    DropItemMask6           = (1 << 5), //Small Health
    DropItemMask7           = (1 << 6), //Large Health
    DropItemMask8           = (1 << 7), //Minigun ammo
    DropItemMask9           = (1 << 8), //Mortal Wound
    DropItemMask10          = (1 << 9), //Rockets
    DropItemMask11          = (1 << 10), //Shotgun Shells
    DropItemMask12          = (1 << 11), //Energy Cell
    DropItemMask13          = (1 << 12), //Large Energy Cell
    DropItemMask14          = (1 << 13), //Clip
    RemoveOnCompletion      = (1 << 14),
    NoBlood                 = (1 << 15),
    HoldTriggerAnimation    = (1 << 16),
    ProjectileAttack3       = (1 << 17),
    ProjectileAttack4       = (1 << 18),
    DropItemOnDamage        = (1 << 19),
    NoAutomapDraw           = (1 << 20),
    AlternateMoves          = (1 << 21),
    AntiAliasingReduced     = (1 << 22),
    AntiAliasingFull        = (1 << 23),
    ProjectileAttack5       = (1 << 24),
    ProjectileAttack6       = (1 << 25),
    MortalWoundImpact       = (1 << 26),
    StayInWater             = (1 << 27),
    DeviceWarpDeath         = (1 << 28),
    StoreWarpReturn         = (1 << 29),
    ProjectileAttack7       = (1 << 30),
    ProjectileAttack8       = (1 << 31)
}

enum ActorSpawnFlags3
{
    ReturnWarp              = (1 << 0),
    PlayTriggerAnimOnce     = (1 << 1),
    RegenerateFromStart     = (1 << 2),
    WalkInStraightLine      = (1 << 3),
    KillOutsideOfView       = (1 << 4),
    NoThinker               = (1 << 5),
    AvoidPlayers2           = (1 << 6),
    NoViolentDeath          = (1 << 7),
    ProjectileAttack9       = (1 << 8),
    ProjectileAttack10      = (1 << 9),
    MakeSpawnAnimVisible    = (1 << 10),
    NoDrawOnCamera          = (1 << 11)
}

enum SpeciesMask //Unused. Here for completion.
{
    Raptor          = (1 << 0),
    Human1          = (1 << 1),
    SaberTiger      = (1 << 2),
    Dimetrodon      = (1 << 3),
    Triceratops     = (1 << 4),
    Moschops        = (1 << 5),
    Pteranodon      = (1 << 6),
    Subterranean    = (1 << 7),
    Leaper          = (1 << 8),
    Alien           = (1 << 9),
    Hulk            = (1 << 10),
    Robot           = (1 << 11),
    MantisBoss      = (1 << 12)
}

enum SectorSetFlags
{
    Water                   = (1 << 0),
    Block                   = (1 << 1),
    Toggle                  = (1 << 2),
    Cliff                   = (1 << 3),
    Climb                   = (1 << 4),
    OneSided                = (1 << 5),
    Ceiling                 = (1 << 6),
    Crawl                   = (1 << 7),
    EnterCrawl              = (1 << 8),
    Hidden                  = (1 << 9),
    Entered                 = (1 << 10),
    Secret                  = (1 << 11),
    Restricted              = (1 << 12),
    SlopeTest               = (1 << 13),
    DeathPit                = (1 << 14),
    Mapped                  = (1 << 15),
    Event                   = (1 << 16),
    Reapeatable             = (1 << 17),
    Teleport                = (1 << 18),
    Damage                  = (1 << 19),
    DrawSky                 = (1 << 20),
    TeleportAir             = (1 << 21),
    Lava                    = (1 << 22),
    EventSound              = (1 << 23),
    AntiGravity             = (1 << 24),
    Ladder                  = (1 << 25),
    Checkpoint              = (1 << 26),
    SaveGame                = (1 << 27),
    WarpReturn              = (1 << 28),
    ShallowWater            = (1 << 29),
    DrawSun                 = (1 << 30),
    StoreWarpReturn         = (1 << 31)
}

enum StaticMeshFlags
{
    Collide             = (1 << 0),
    NoPolyCollision     = (1 << 1),
    NoDrawCamera        = (1 << 2)
}