Difference between revisions of "BFRES (File Format)"

From MK8
Jump to navigation Jump to search
Line 1: Line 1:
'''BFRES''' is an archive format used on [[Wii U]] and in [[Mario Kart 8]] to store graphics data. Typically, a BFRES file will contain information for one model including textures and animations, but this is not necessarily the case. BFRES seems to be a direct upgrade of the [[:mkw:BRRES (File Format)|BRRES]] format used on [[Wii]]. Due to their vast size, large BFRES files are often compressed using [[YAZ0 (File Format)|YAZ0 compression]].
+
'''BFRES''' is an archive format used on [[Wii U]] and in [[Mario Kart 8]] to store graphics data. Typically, a BFRES file will contain information for one model including textures and animations, but this is not necessarily the case. BFRES seems to be a direct upgrade of the [[:mkw:BRRES (File Format)|BRRES]] format used on [[Wii]]. Due to their vast size, large BFRES files are often compressed using [[YAZ0 (File Format)|YAZ0 compression]] and will then have the file extension ''.szs''.
  
 
__TOC__
 
__TOC__
  
 
= Format =
 
= Format =
 
+
There are different versions of the format and therefor the format varies in different games. This documentation mostly focus on the v3.5.4.X format. The basic architecture of a '''BFRES''' file is any number of subfiles each in one of 12 groups governing format and purpose. Each subfile is something like a model or a texture. Unless otherwise noted, all offsets in BFRES files are relative to themselves, not the beginning of the file which is more common in other formats.
This article describes the '''BFRES''' file format seen in [[Mario Kart 8]]. It is unknown if the file format differs between games like the [[:mkw:BRRES (File Format)|BRRES]] format does. The basic architecture of a '''BFRES''' file is any number of subfiles each in one of 12 groups governing format and purpose. Each subfile is something like a model or a texture. Unless otherwise noted, all offsets in BFRES files are relative to themselves, not the beginning of the file which is more common in other formats.
 
  
 
== Header (FRES) ==
 
== Header (FRES) ==
 
 
Every BFRES file begins with an 0x6C byte '''FRES''' header.
 
Every BFRES file begins with an 0x6C byte '''FRES''' header.
 
{| class="wikitable"
 
{| class="wikitable"
! Offset
+
! Offset !! Size !! Description
! Size
 
! Description
 
 
|-
 
|-
| 0x00
+
| 0x00 || 4 || '''File magic'''. Always ''FRES'' in ASCII.
| 4
 
| "'''FRES'''" File identifier, ASCII string.
 
 
|-
 
|-
| 0x04
+
| 0x04 || 4 || '''Version number''' of the BFRES file format, stored as four bytes. The content of the sub files can vary depending on the version. Version used in [[Mario Kart 8]] are 3.0.0.X, 3.2.0.X, 3.3.0.X and 3.4.0.X where 3.4.0.X is the most common.
| 3
 
| '''Version number''' of the BFRES file format, stored as three bytes. The content of the sub files can vary depending on the version. Version used in [[Mario Kart 8]] are 3.0.0, 3.2.0, 3.3.0 and 3.4.0 where 3.4.0 is the most common.
 
 
|-
 
|-
| 0x07
+
| 0x08 || 2 || '''Byte order mark''' (BOM): 0xFE,0xFF for big endian and 0xFF,0xFE for little endian. Seems to indicate endianness of embedded files.
| 1
 
| {{Unknown|'''Unknown'''. '''0x01, 0x02 or 0x04''' in [[Mario Kart 8]].}}
 
 
|-
 
|-
| 0x08
+
| 0x0A || 2 || {{Unknown|'''Unknown'''. Always has value '''0x0010''' in [[Mario Kart 8]].}}
| 2
 
| '''Byte order mark''' (BOM): 0xFE,0xFF for big endian and 0xFF,0xFE for little endian. Seems to indicate endian-ness of embedded files.
 
 
|-
 
|-
| 0x0A
+
| 0x0C || 4 || '''File length''' including all headers, in bytes.
| 2
 
| {{Unknown|'''Unknown'''. Always has value '''0x0010''' in [[Mario Kart 8]].}}
 
 
|-
 
|-
| 0x0C
+
| 0x10 || 4 || '''File alignment''' a power of 2. Most specific data alignment required in the file. The graphics card often requires data to be aligned to specific power of 2 boundaries. If the file is loaded into memory aligned to this value (a multiply of this value), then all addresses will be correct for the graphics card.
| 4
 
| '''File length''' including all headers, in bytes.
 
 
|-
 
|-
| 0x10
+
| 0x14 || 4 || '''File name''' offset (without file extension).
| 4
 
| '''File alignment''' a power of 2. Most specific data alignment required in the file. The graphics card often requires data to be aligned to specific power of 2 boundaries. If the file is loaded into memory aligned to this value (a multiply of this value), then all addresses will be correct for the graphics card.
 
 
|-
 
|-
| 0x14
+
| 0x18 || 4 || '''[[#String Table|String table]] length'''. Length in bytes of the string table.
| 4
 
| '''File name''' offset (without file extension).
 
 
|-
 
|-
| 0x18
+
| 0x1C || 4 || '''[[#String Table|String table]]''' offset.
| 4
 
| '''String table Length'''. Length in bytes of the String Table.
 
 
|-
 
|-
| 0x1C
+
| 0x20 || 4 × 12 || '''File offsets'''. Offsets to [[#Index Group|index groups]] for each of the 12 subfile types. '''0''' indicates that a particular subfile type is not present.
| 4
 
| '''String table''' offset.
 
 
|-
 
|-
| 0x20
+
| 0x50 || 2 × 12 || '''File counts'''. Number of files in the [[#Index Group|index groups]] for each of the 12 subfile types. '''0''' for subfile types which are not present. This excludes the root entries, and is the same as each of the subfile counts in the index groups.  
| 4 × 12
 
| '''File offsets'''. Offsets to [[#Index Group|index groups]] for each of the 12 subfile types. '''0''' indicates that a particular subfile type is not present.
 
 
|-
 
|-
| 0x50
+
| 0x68 || 4 || {{Unknown|'''Unknown''' always '''0''' in [[Mario Kart 8]] files.}}
| 2 × 12
 
| '''File counts'''. Number of files in the [[#Index Group|index groups]] for each of the 12 subfile types. '''0''' for subfile types which are not present. This excludes the root entries, and is the same as each of the subfile counts in the index groups.  
 
 
|-
 
|-
| 0x68
+
| 0x6C || colspan="2" {{Unknown|End of '''FRES''' header}}
| 4
 
| {{Unknown|'''Unknown''' always '''0''' in [[Mario Kart 8]] files.}}
 
|-
 
| 0x6C
 
| colspan="2" {{Unknown|End of '''FRES''' header}}
 
 
|}
 
|}
<references group="FRES"/>
 
  
 
== Index Group ==
 
== Index Group ==
 
 
The '''BFRES Index Group''' is a data structure that occurs very often within BFRES files and subfiles. It is a binary search tree in which a series of named data pointers are stored which are used to represent subfiles. It has a 0x08 byte header, as follows.
 
The '''BFRES Index Group''' is a data structure that occurs very often within BFRES files and subfiles. It is a binary search tree in which a series of named data pointers are stored which are used to represent subfiles. It has a 0x08 byte header, as follows.
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
! Offset
+
! Offset !! Size !! Description
! Size
 
! Description
 
 
|-
 
|-
| 0x00
+
| 0x00 || 4 || '''Length''' of group in bytes.
| 4
 
| '''Length''' of group in bytes.
 
 
|-
 
|-
| 0x04
+
| 0x04 || 4 || '''Number''' of subfiles in the group (excluding the root entry). Same as file counts in header.
| 4
 
| '''Number''' of subfiles in the group (excluding the root entry). Same as file counts in header.
 
 
|-
 
|-
 
| 0x08
 
| 0x08
 
| colspan="2" {{Unknown|End of index group header}}
 
| colspan="2" {{Unknown|End of index group header}}
 
|}
 
|}
This is then followed by a number of entries equal to the number in the header. The first entry is never an actual data entry, but instead a reference point. Each entry is a 0x10 byte structure as follows.
+
 
 +
This is then followed by a number of entries equal to the number in the header, plus a root entry. The root entry is never an actual data entry, but instead a reference point. It is always the first group in the index group and is not included in the group count. Each entry is a 0x10 byte structure as follows.
 +
 
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
! Offset
+
! Offset !! Size !! Description
! Size
 
! Description
 
 
|-
 
|-
| 0x00
+
| 0x00 || 4 || '''Search value'''. Refers to a single bit in the filename being searched for. It's laid out like so: XXXXXYYY, where X is the index of the last valid byte (From the left) of the name string, and Y is the position of that bit within that byte.
| 4
 
| '''Search value'''. Refers to a single bit in the filename being searched for. It's laid out like so: XXXXXYYY, where X is the index of the last valid byte (From the left) of the name string, and Y is the position of that bit within that byte.
 
 
|-
 
|-
| 0x04
+
| 0x04 || 2 || '''Left index'''. If the bit referred to by the search value is 0, jump to the index group entry with this index. (Index 0 is the dummy node at the beginning)
| 2
 
| '''Left index'''. If the bit referred to by the search value is 0, jump to the index group entry with this index. (Index 0 is the dummy node at the beginning)
 
 
|-
 
|-
| 0x06
+
| 0x06 || 2 || '''Right index'''. If the bit referred to by the search value is 1, jump to the index group entry with this index. (Index 0 is the dummy node at the beginning).
| 2
 
| '''Right index'''. If the bit referred to by the search value is 1, jump to the index group entry with this index. (Index 0 is the dummy node at the beginning).
 
 
|-
 
|-
| 0x08
+
| 0x08 || 4 || '''Name pointer'''. Offset to the name of this entry.
| 4
 
| '''Name pointer'''. Offset to the name of this entry.
 
 
|-
 
|-
| 0x0C
+
| 0x0C || 4 || '''Data pointer'''. Offset to the data of this entry.
| 4
 
| '''Data pointer'''. Offset to the data of this entry.
 
 
|-
 
|-
| 0x10
+
| 0x10 || colspan="2" {{Unknown|End of entry}}
| colspan="2" {{Unknown|End of entry}}
 
 
|}
 
|}
  
Line 179: Line 127:
  
 
== String Table ==
 
== String Table ==
 +
The '''BFRES String Table''' is a table that occurs near the end of the BFRES file. There is no header information for the table. References to the table are stored as offsets to strings directly. The strings are stored in ASCII order (so Z comes before a). The length of each string is stored in the 4 bytes before that string and the strings are also null terminated. Each length is 4 byte aligned. The pointer in the [[#Header (FRES)|FRES header]] points to the length of the first string.
 +
 +
== Embedded File ==
 +
Embedded files are found into the '''11:th''' [[#Index Group|index group]] and can be any data. In [[Mario Kart 8]] only shader files have been seen with one exception. [[Filesystem/content/mapobj/PackunMusic|PackunMusic.bfres]] is a special case because it contains shader source code for a shader. The shader file is called Turbo_UBER.bfsha (Cafe Turbo is the codename of [[Mario Kart 8]]).
 +
<br>The data offset in an [[#Index Group|index group]] points to the following structure:
  
The '''BFRES String Table''' is a table that occurs near the end of the BFRES file. There is no header information for the table. References to the table are stored as offsets to strings directly. The strings are stored in ASCII order (so Z comes before a). The length of each string is stored in the 4 bytes before that string and the strings are also null terminated. Each length is 4 byte aligned. The pointer in the [[#Header (FRES)|FRES header]] points to the length of the first string.
+
{|class="wikitable"
 +
|-
 +
! Offset !! Size !! Description
 +
|-
 +
| 0x00 || 4 || '''Offset''' to the file data.
 +
|-
 +
| 0x04 || 4 || '''Size''' of the file data in bytes.
 +
|}
 +
 
 +
These structures are always placed directly after the [[#Header (FRES)|main header]]. The file data is always placed at the end of the file.
  
 
== Sub Files ==
 
== Sub Files ==
 
 
The format and purpose of subfiles is determined entirely by the index group they're in of the 12 main index groups references by the FRES header. Note that these subfiles are not necessarily continuous. Typically, all the headers for all subfiles occur at the start of the BFRES, followed by the string table, and then finally all the data referenced by the offsets in each of the subfile headers. This is probably a caching optimization, as the CPU generally only deals with the headers of each format, and the graphical data can be sent to the GPU in one go. Grouping together the headers makes them all more likely to fit in the CPU's cache, speeding up file access.
 
The format and purpose of subfiles is determined entirely by the index group they're in of the 12 main index groups references by the FRES header. Note that these subfiles are not necessarily continuous. Typically, all the headers for all subfiles occur at the start of the BFRES, followed by the string table, and then finally all the data referenced by the offsets in each of the subfile headers. This is probably a caching optimization, as the CPU generally only deals with the headers of each format, and the graphical data can be sent to the GPU in one go. Grouping together the headers makes them all more likely to fit in the CPU's cache, speeding up file access.
  
Line 211: Line 172:
 
| 10 || '''FSCN''' || '''Scene''' animation.
 
| 10 || '''FSCN''' || '''Scene''' animation.
 
|-
 
|-
| 11 || '''Embedded''' || '''Embedded file'''. The data offset points to a offset &amp; length pair which describe the embedded file. The embedded files always seem to be placed at the end of the BFRES file. Shader source code has been seen in embedded files, though this is not typically the case.
+
| 11 || '''Embedded''' || '''[[#Embedded File|Embedded file]]'''.
 
|}
 
|}
  
 
= Tools =
 
= Tools =
 
 
The following tool can operate on BFRES files:
 
The following tool can operate on BFRES files:
  

Revision as of 15:09, 9 May 2017

BFRES is an archive format used on Wii U and in Mario Kart 8 to store graphics data. Typically, a BFRES file will contain information for one model including textures and animations, but this is not necessarily the case. BFRES seems to be a direct upgrade of the BRRES format used on Wii. Due to their vast size, large BFRES files are often compressed using YAZ0 compression and will then have the file extension .szs.

Format

There are different versions of the format and therefor the format varies in different games. This documentation mostly focus on the v3.5.4.X format. The basic architecture of a BFRES file is any number of subfiles each in one of 12 groups governing format and purpose. Each subfile is something like a model or a texture. Unless otherwise noted, all offsets in BFRES files are relative to themselves, not the beginning of the file which is more common in other formats.

Header (FRES)

Every BFRES file begins with an 0x6C byte FRES header.

Offset Size Description
0x00 4 File magic. Always FRES in ASCII.
0x04 4 Version number of the BFRES file format, stored as four bytes. The content of the sub files can vary depending on the version. Version used in Mario Kart 8 are 3.0.0.X, 3.2.0.X, 3.3.0.X and 3.4.0.X where 3.4.0.X is the most common.
0x08 2 Byte order mark (BOM): 0xFE,0xFF for big endian and 0xFF,0xFE for little endian. Seems to indicate endianness of embedded files.
0x0A 2 Unknown. Always has value 0x0010 in Mario Kart 8.
0x0C 4 File length including all headers, in bytes.
0x10 4 File alignment a power of 2. Most specific data alignment required in the file. The graphics card often requires data to be aligned to specific power of 2 boundaries. If the file is loaded into memory aligned to this value (a multiply of this value), then all addresses will be correct for the graphics card.
0x14 4 File name offset (without file extension).
0x18 4 String table length. Length in bytes of the string table.
0x1C 4 String table offset.
0x20 4 × 12 File offsets. Offsets to index groups for each of the 12 subfile types. 0 indicates that a particular subfile type is not present.
0x50 2 × 12 File counts. Number of files in the index groups for each of the 12 subfile types. 0 for subfile types which are not present. This excludes the root entries, and is the same as each of the subfile counts in the index groups.
0x68 4 Unknown always 0 in Mario Kart 8 files.
0x6C End of FRES header

Index Group

The BFRES Index Group is a data structure that occurs very often within BFRES files and subfiles. It is a binary search tree in which a series of named data pointers are stored which are used to represent subfiles. It has a 0x08 byte header, as follows.

Offset Size Description
0x00 4 Length of group in bytes.
0x04 4 Number of subfiles in the group (excluding the root entry). Same as file counts in header.
0x08 End of index group header

This is then followed by a number of entries equal to the number in the header, plus a root entry. The root entry is never an actual data entry, but instead a reference point. It is always the first group in the index group and is not included in the group count. Each entry is a 0x10 byte structure as follows.

Offset Size Description
0x00 4 Search value. Refers to a single bit in the filename being searched for. It's laid out like so: XXXXXYYY, where X is the index of the last valid byte (From the left) of the name string, and Y is the position of that bit within that byte.
0x04 2 Left index. If the bit referred to by the search value is 0, jump to the index group entry with this index. (Index 0 is the dummy node at the beginning)
0x06 2 Right index. If the bit referred to by the search value is 1, jump to the index group entry with this index. (Index 0 is the dummy node at the beginning).
0x08 4 Name pointer. Offset to the name of this entry.
0x0C 4 Data pointer. Offset to the data of this entry.
0x10 End of entry

The node indexes seem to be 0 when there is no/NULL node there.

To find a specific filename in the index group using the binary search algorithm, use the following pseudo-Python:

def searchIndexGroup(groupPtr, data, name):

    # groupPtr is an int containing the offset to the index group
    # data is the data for the entire BFRES
    # name is the name we're searching for (bytes, not str)
    # Returns the offset of the start of the file data, or -1 for failure

    firstEntry = groupPtr + 8
    searchval = parse_u32(data, firstEntry + 0)
    leftIdx = parse_u16(data, firstEntry + 4)

    entries = groupPtr + 8
    entry = entries + leftIdx * 16
    nextsearchval = parse_u32(data, entry + 0)
    while searchval > nextsearchval:

        # Get the direction
        charpos = searchval >> 3
        bitpos = searchval & 0b111
        if charpos >= len(name):
            direction = 0
        else:
            direction = (name[charpos] >> bitpos) & 1

        # Go to the left index or right index based on the value of that bit
        searchval = nextsearchval
        if direction == 0:
            leftIdx = parse_u16(data, entry + 4)
            entry = entries + leftIdx * 16
        else:
            rightIdx = parse_u16(data, entry + 6)
            entry = entries + rightIdx * 16
        nextsearchval = parse_u32(data, entry + 0)

    # Ensure that we reached the correct file
    namePtr = entry + 8 + parse_u32(data, entry + 8)

    if not namePtr:
        return -1

    namelen = parse_u32(data, namePtr - 4)
    if namelen != len(name):
        return -1

    if parse_null_terminated_string(data, namePtr) != name:
        return -1

    return entry

String Table

The BFRES String Table is a table that occurs near the end of the BFRES file. There is no header information for the table. References to the table are stored as offsets to strings directly. The strings are stored in ASCII order (so Z comes before a). The length of each string is stored in the 4 bytes before that string and the strings are also null terminated. Each length is 4 byte aligned. The pointer in the FRES header points to the length of the first string.

Embedded File

Embedded files are found into the 11:th index group and can be any data. In Mario Kart 8 only shader files have been seen with one exception. PackunMusic.bfres is a special case because it contains shader source code for a shader. The shader file is called Turbo_UBER.bfsha (Cafe Turbo is the codename of Mario Kart 8).
The data offset in an index group points to the following structure:

Offset Size Description
0x00 4 Offset to the file data.
0x04 4 Size of the file data in bytes.

These structures are always placed directly after the main header. The file data is always placed at the end of the file.

Sub Files

The format and purpose of subfiles is determined entirely by the index group they're in of the 12 main index groups references by the FRES header. Note that these subfiles are not necessarily continuous. Typically, all the headers for all subfiles occur at the start of the BFRES, followed by the string table, and then finally all the data referenced by the offsets in each of the subfile headers. This is probably a caching optimization, as the CPU generally only deals with the headers of each format, and the graphical data can be sent to the GPU in one go. Grouping together the headers makes them all more likely to fit in the CPU's cache, speeding up file access.

Index Type Description
0 FMDL Model data.
1 FTEX Texture data.
2 FSKA Skeleton animation.
3 FSHU Shader parameters.
4 FSHU Color animation.
5 FSHU Texture SRT animation.
6 FTXP Texture pattern animation.
7 FVIS Bone visibility animation.
8 FVIS Material visibility animation.
9 FSHA Shape animation.
10 FSCN Scene animation.
11 Embedded Embedded file.

Tools

The following tool can operate on BFRES files: