Ghost Data (File Format)

From MK8
Revision as of 15:16, 15 October 2022 by Scutlet (talk | contribs) (driver data, race settings, yaz0/replay data (WIP))
Jump to navigation Jump to search
Under Construction
This article is not finished. Help improve it by adding accurate information or correcting grammar and spelling.
Platform Notice
This article is about Mario Kart 8 for Wii U. Mario Kart 8 Deluxe uses a similar file format, but that is still a WIP and the offsets are different.

In Mario Kart 8, Ghost Data is stored in DAT-files. This includes player ghosts, staff ghosts, and downloaded ghosts. MKTV replays have a similar file format. Each Ghost is stored in a separate DAT-file.

Player Ghosts, Downloaded Ghosts, and MKTV Replays are stored in the user's save folder. Staff Ghosts are stored among the game files.

Differences with Mario Kart Wii

In Mario Kart Wii, as well as earlier titles, ghost files most notably store controller inputs. When a ghost is viewed, the game replays these controller inputs. If the game is perfectly deterministic, the same controller inputs should always yield the same finishing time. In practice however, Mario Kart Wii is not quite able to recreate the circumstances in which the ghost was recorded leading to the rare Wiggler Glitch. This causes the ghost to descynchronise from the original Time Trial, often failing to finish.

In Mario Kart 8, this is different. Ghost files must also contain kart positions in some form, possibly in addition to controller inputs (unknown). In practise, this means that a ghost is always able to finish a race in the exact same way as it was recorded in. This can be verified by replacing the file contents of a ghost file with that of another. When doing so, the ghost will ignore any collisions of the track (e.g. drive through walls) and follow the layout of the course it was initially recorded in.

As a result, this means that the Wiggler Glitch cannot occur in Mario Kart 8.

Filename

Some ghost data information is encoded into the filename of the ghost file. The first two characters indicate what type of ghost data is inside the file. The remaining characters should be interpreted as hexadecimal values.

Ghost Data Filename
Offset (#characters) Size (#characters) Description
0 2 Ghost Type. "sg" for Staff Ghost, "gs" for Player Ghost (possibly a shorthand for "ghost save"), "dg" for Downloaded Ghost, "rp" for MKTV Replay.
2 2 Ghost Number. For staff ghosts and player ghosts this is simply the track ID minus 16.

For downloaded ghosts it is a value from 0x00-0x0f, representing one of the 16 available downloaded ghost slots. If the game detects multiple downloaded ghosts with the same ghost number (even if they're not for the same track), it will keep the ghost for the track with the lowest trackID and discard all others. This number also allows the game to differentiate between ghosts which have otherwise identical data in the filename.

4 2 Track ID. Each track has a unique value here. This does not correspond to the order in which the apear in the game's cups. E.g. Mario Circuit has a value of 16.
6 2 Character ID. E.g. Yoshi has a value of 4.
8 2 Character Variant ID. Only used fro Yoshi, Shy Guy, and Mii (standard and Amiibo suits)
10 2 Mii Weight Class. Only used for Mii to signify whether it is a light (0x00), medium (0x01), or heavy (0x02) character. Defaults to 0x00 for all other characters.
12 2 Kart ID. E.g. Buggybud has a value of 9.
14 2 Wheels ID. E.g. Button Wheels have a value of 6.
16 2 Glider ID. E.g. Cloud Glider has a value of 1.
18 1 Minutes of total finishing time.
19 2 Seconds of total finishing time.
21 3 Milliseconds of total finishing time.
24 1 Minutes of lap 1 finishing time.
25 2 Seconds of lap 1 finishing time.
27 3 Milliseconds of lap 1 finishing time.
30 1 Minutes of lap 2 finishing time.
31 2 Seconds of lap 2 finishing time.
33 3 Milliseconds of lap 2 finishing time.
36 1 Minutes of lap 3 finishing time.
37 2 Seconds of lap 3 finishing time.
39 3 Milliseconds of lap 3 finishing time.
42 1 Minutes of lap 4 finishing time. "9" if there is no 4th lap.
43 2 Seconds of lap 4 finishing time. 3b (i.e., "59") if there is no 4th lap.
45 3 Milliseconds of lap 4 finishing time. 3e7 (i.e., "999") if there is no 4th lap.
48 1 Minutes of lap 5 finishing time.
49 2 Seconds of lap 5 finishing time.
51 3 Milliseconds of lap 5 finishing time.
54 40 Player Name in UTF-16 (big Endian).
94 2 Flag ID. E.g. The Dutch flag has a value of 5e.
96 2 Motion Control Flag. 0x01 if motion controls are used. 0x00 otherwise.
98 2/4 Unknown. Probably padding (Always 0). Prior to v4 of the game, this could be 2 characters instead if the ghost was a player ghost. The cause for this is unknown.
102 1 Minutes of lap 6 finishing time.
103 2 Seconds of lap 6 finishing time.
105 3 Milliseconds of lap 6 finishing time.
108 1 Minutes of lap 7 finishing time.
109 2 Seconds of lap 7 finishing time.
111 3 Milliseconds of lap 7 finishing time.
114 End of filename; ".dat" extension here.

Game Versions

Lap 4 and 5 finishing times are unused before version 4 of the game. It's unknown if these values are properly read or written to if modifying the lap count of an existing track. Lap 6 and 7 finishing times were only introduced in version 4 of the game. Before that, these values were absent from the filename entirely. Attempting to load Ghost Data from this version of the game in an earlier version will make the game unable to recognise the file. However, renaming the file by removing these extra values will make the Ghost Data load properly. Furthermore, Ghost Data created before v4 of the game is recognised in v4 (and beyond), even though the extra lap finishing times are missing.

This change is a direct result of the introduction of GCN Baby Park in the second DLC pack, as it is the only track that has more than three laps.

Effect In-Game

Changing these values by renaming the file affects how a ghost is displayed. Changing any of the values will result in the game displaying the modified information on the ghost summary (the additional info accessed by pressing "+" on the controller). Furthermore, changing the Character ID, Kart ID, Wheels ID, or Glider ID will result in the game using these during the ghost race itself, regardless of the values for these inside the file itself.

Furthermore, a character's animations can break if the Character ID in the filename does not match the one inside the file contents.

If the Track Number and Track ID do not correspond to the same track, then the game will be unable to recognise the file.

If multiple Ghost files for the same track are present in the game's files, then the game will only show one. It's unverified if it picks the fastest time, the one recorded earliest, or based on something else.

Player Ghost File Header

The following header is absent for Staff Ghosts. However, a Player or Downloaded Ghost can easily be converted into a Staff Ghost by removing this header and renaming the first two characters of the filename from "gs"/"dg" to "sg". This header is identical to the header of a save file. Ghosts downloaded through the Nintendo Clients package do not have this header.

Extra Ghost Header
Offset Size Description
0x00 4 "CTG0" in ASCII. Unknown abbreviation.
0x04 4 Unknown. Likely game version.
0x08 4 File Size (in bytes).
0x0c 44 Probably Padding (always 0)
0x38 4 CRC-32 Checksum over all bytes after this header until the end of the file.
0x3c 12 Probably Padding (always 0)
0x48 End of extra ghost header

Data

The remaining offset values will assume that the additional 0x48 bytes of header data are not present in the ghost file, so if you're using ghost data that was either set in-game or downloaded in-game, then add an additional 0x48 to each of the offsets below.

General Information
Offset Size Description
0x00 6 Likely file magic. The values are always 00 00 04 00 03 A0
0x06 2 Unknown. Possibly game version.
0x08 4 Size of payload (everything after the 0x48-byte header, if present).

Timestamp

When a new ghost is created, the timestamp from the user's local time is used and included in the ghost data.

Timestamp
Offset Size Description
0x0c 4 Year.
0x10 4 Month.
0x14 4 Day.
0x18 4 Day of Week, with 0x00 being Sunday.
0x1c 4 Hour.
0x20 4 Minute.
0x24 4 Second.

Driver Data

These values do not actually seem to be used. Instead, this data is read from the ghost's filename.

Driver information consists of 12 identical data blocks (one for each driver) of length 0x1c. The first driver slot represents the data of the recorded ghost. If the ghost was recorded while racing against another ghost, the data of this other ghost is stored in the second driver slot with a special CPU flag (see below). Furthermore, the third and fourth slot will in this case be occupied by two Mario's using the standard vehicle combination (Standard Kart, Standard Wheels, Super Glider) with a CPU-flag for unoccupied slots (see below). This could indicate that racing against multiple ghosts at the same time was a planned feature, just like in Mario Kart 7 where up to 7 ghosts can be raced against at the same time. The 2nd through 4th slots are identical to the remainder 8 slots if the ghost was recorded without racing against another ghost. The fifth through 12th slots are always occupied with filler data (mostly FF-values) and the "unoccupied" CPU-flag.

Individual Driver Data
Offset Size Description
0x00 4 Vehicle Body ID.
0x04 4 Tires ID.
0x08 4 Glider ID.
0x0c 4 Character ID.
0x10 1 Character Variant ID.
0x11 1 Mii Weight Class.
0x12 2 Padding. Always 0.
0x14 4 CPU Flag. 0 for a human player, 1 for a CPU, 3 for the vs ghost in the second driver slot, 4 when unoccupied.
0x18 4 Team Flag. 0 for red team, 1 for blue team, 2 for a solo team (always the case in Time Trials or during other solo races).
0x1c End of driver data.

The offsets above are relative to the following offets for each driver.

Driver Data
Offset Description
0x2c Player 1
0x48 Player 2
0x64 Player 3
0x80 Player 4
0x9c Player 5
0xb8 Player 6
0xd4 Player 7
0xf0 Player 8
0x10c Player 9
0x128 Player 10
0x144 Player 11
0x160 Player 12
0x17c End of driver data

Race Settings

Race settings store flags or other identifiers of a race. These are sometimes used in the UI for MKTV Replays.

Race Settings
Offset Size Description
0x17c 4 Track ID.
0x180 4 Online Mode. 0 for local play. 1 for online play.
0x184 4 Game Mode. 0 for Grand Prix, 1 for Time Trial, 2 for VS Race, 3 for Battle Mode.
0x188 4 Unknown. Always 00 00 00 02.
0x18c 4 Unknown. Always 00 00 00 00 or 00 00 00 01.
0x190 4 CC Mode. 0 for 50cc, 1 for 100cc, 2 for 150cc, 3 for 200cc.
0x194 1 Mirror Flag. 0 for standard play, 1 for mirror mode.
0x195 1 Team Flag. 0 for solo play, 1 for team play.
0x196 1 Unknown. Always 03.
0x197 1 Unknown. Always 00.
0x198 16 Unknown.
0x1a8 4 Number of Drivers. Always 1 for Time Trials. Number is equal to the number of drivers with an "unoccupied" CPU slot (see driver data above).

Mii Info

In every single ghost file there is embedded Mii data so that the game can reconstruct the Mii in the case that a Mii was used in the run. Below is the start offset and size of the Mii data. If a Mii was not used during the race, this data will match the Mii of the user profile. Furthermore, this data is used to display the driver name when replaying a ghost.

Mii Info
Offset Size Description
0x244 92 Mii Data. See full description on 3dsbrew. Note that the embedded Mii name is in UTF-16 (little Endian).
0x2a0 2 Probably Padding.
0x2a2 2 CRC-16 XMODEM Checksum.
0x2a4 End of Mii data.

Location

This data is used to display the driver's flag at the end of a race when replaying a ghost.

Location
Offset Size Description
0x2a4 1 Country ID.
0x2a5 1 Sub-region ID.
0x2a6 2 Probably Padding (always 0).
0x2a8 1 Country ID repeat (purpose unknown).
0x2a9 1 Sub-region ID repeat (purpose unknown).

Player Info

Player Info
Offset Size Description
0x304 20 Mii Name in UTF-16 big Endian. Purpose Unknown.

Total Time & Lap Splits

This data is on the overview at the end of a ghost replay. If a track does not have a lap, its lap time is set to 9:59:999 just like in the ghost filename. The exception to this are the lap splits for lap 6 and 7, which are instead 0:00:000 if left unused.

Total Time & Lap Splits
Offset Size Description
Lap 1 Time
0x330 2 Minutes.
0x332 1 Seconds.
0x333 1 Padding (always 0).
0x334 2 Milliseconds.
0x336 6 Probably Padding (always 0).
Lap 2 Time
0x33c 2 Minutes.
0x33e 1 Seconds.
0x33f 1 Padding (always 0).
0x340 2 Milliseconds.
0x342 6 Probably Padding (always 0).
Lap 3 Time
0x348 2 Minutes.
0x34a 1 Seconds.
0x34b 1 Padding (always 0).
0x34c 2 Milliseconds.
0x34e 6 Probably Padding (always 0).
Lap 4 Time
0x354 2 Minutes.
0x356 1 Seconds.
0x357 1 Padding (always 0).
0x358 2 Milliseconds.
0x35a 6 Probably Padding (always 0).
Lap 5 Time
0x360 2 Minutes.
0x362 1 Seconds.
0x363 1 Padding (always 0).
0x364 2 Milliseconds.
0x366 6 Probably Padding (always 0).
Total Time
0x36c 2 Minutes.
0x36e 1 Seconds.
0x36f 1 Padding (always 0).
0x370 2 Milliseconds.
0x372 10 Probably Padding (always 0).
0x37c 4 Unknown.
0x380 4 Probably Padding (always 0).
Lap 6 Time
0x384 2 Minutes.
0x386 1 Seconds.
0x387 1 Padding (always 0).
0x388 2 Milliseconds.
0x38a 6 Probably Padding (always 0).
Lap 7 Time
0x390 2 Minutes.
0x392 1 Seconds.
0x393 1 Padding (always 0).
0x394 2 Milliseconds.
0x396 6 Probably Padding (always 0).

Additionally, (staff) ghosts recorded in v4 of the game store additional 9:59:999 data, which might be placeholders for laps 8, 9, and 10. In earlier versions of the game this data was always 0. It is unknown if this data is properly written to if the lap count of an existing track is increased. Likewise, the behaviour of the UI nor the ghost filename is known if this is done.

Total Time & Lap Splits
Offset Size Description
Lap 8 Time Placeholder?
0x39c 2 Minutes.
0x39e 1 Seconds.
0x39f 1 Padding (always 0).
0x3a0 2 Milliseconds.
0x3a2 6 Probably Padding (always 0).
Lap 9 Time Placeholder?
0x3a8 2 Minutes.
0x3aa 1 Seconds.
0x3ab 1 Padding (always 0).
0x3ac 2 Milliseconds.
0x3ae 6 Probably Padding (always 0).
Lap 10 Time Placeholder?
0x3b4 2 Minutes.
0x3b6 1 Seconds.
0x3b7 1 Padding (always 0).
0x3b8 2 Milliseconds.
0x3ba 6 Probably Padding (always 0).
0x3c0 48 Probably Padding (always 0).
0x400 End of total time & lap splits.

Replay Data

There are several Yaz0 sections at the end of the ghost file that represent the drivers' movement. This section can be exchanged with that of another ghost file, making the game replay the ghost of the injected data (after fixing the CRC and file size elements in the ghost's header). It is currently unknown what they specifically store.

Replay Header

The replay header stores information regarding the yaz0 blocks, and the section data within them when uncompressed, at the end of the file.

Replay Header
Offset Size Description
0x400 4 CRC-32 Checksum over all bytes after this value until the first yaz0 section.
0x404 4 First Yaz0 Section Offset. Offset for the first yaz0 section, relative to the start of the replay data.
0x408 8 Unknown.
0x408 4 Number of subsections in the decompressed yaz0 sections. Always 17.
0x40c 72 Unknown.
0x454 4 Number of sections in the decompressed yaz0 sections.
0x458 8 Unknown.
0x460 4 Number of yaz0 sections at the end of the file.
0x464 32 Probably Padding.


Additional Data

The game must also store the following information, likely in the yaz0 sections:

  • Player position
  • Player rotation
  • Player scale (E.g., when smashed by a thwomp)
  • Drift usage
  • Tricks performed
  • Horn usage
  • Item usage. When hacking in bullet bills or stars, the game properly displays this.
  • Lakitu saves

Tools

The following tools can handle Ghost Data (DAT) files:

  • MK8Leaderboards: Fetches and formats data from the time trial leaderboards. Includes a Discord bot.
  • Poltergust: A (staff) ghost visualisation, extraction, and conversion tool.