Address Corruption Manifesto

Address Corruption in N64 Prototype Dumps

  Several N64 prototype dumps exhibit the same type of data corruption, best described as "address corruption".  In every case only the lower halfword at an address (*2, *6, *A, *E) will contain invalid data, and this data will predictably match the value at a related address.
  "Predictability" is the operative word.  Unlike noise-pattern corruption (electrical dropoff in overly long cables, picking up interference from nearby power cables, etc.) not only is the location of the data predictable but so is the value itself.

  * The point of this document is only to make dumpers aware of the problem, determine what dumping methods are subject to it, and seek to ideintify and redump affected files.  The only solution for a bad dump is a good dump!

  * As of 10/9/2022 it's now known at least two of the prototypes listed below and several anonymous prototypes that exhibit this corruption pattern were dumped via GameSharks.  It's not known if this is exclusive to GameSharks however.  It appears tied to sync issues in the parallel protocol used so may also affect other devices in the same way.  Which protocols, and whether USB-to-parallel adapters are affected is also unknown at this point.

.: Known Affected Publicly-Released Prototypes :.
Turok 3 - Shadow of Oblivion 6.6.2000
	MD5: 69CE88C46A7C829C6F54004DE93EFCEF
Turok 3 - Shadow of Oblivion 7.10.2000
	MD5: C38ACBAE773CC3845EA354421E171998
Derby Stallion 64
	MD5: 1051E1402EE110F3C5E372C9E1C5B338
Glover (PROTOTYPE) [Original Dump]
	MD5: 43C3375BBD6496B6C51D08EC193BC8ED
  * If you know what method any of these prototypes was dumped with, please email.


.: Address and Data Patterns :.
  Data corruption occurs when:
1) the upper 16bit halfword at an address (*0, *4, *8, *C) has four or fewer bits set and...
2) the lower halfword at an address (*2, *6, *A, *E) has twelve to sixteen bits set.
  When both conditions are true, instead of returning the correct value it returns the value at a related address.  That address is, almost exclusively, a mask of the original address:
addr & 0xFFFF01FF = source of bad data

  As an example, from the 7.10.2000 bad dump of Turok 3:
address 	data	source of bad data
0x1E2A42	5249	0x1E0042
0x1EA272	0000	0x1E0072
0x1EA482	0004	0x1E0082
0x1EC48A	0208	0x1E008A
0x1EE32E	0114	0x1E012E
0x1F4A8A	0002	0x1F008A
0x1FADEA	0004	0x1F01EA
0x1FB602	0011	0x1F0002
0x1FD492	0004	0x1F0092
0x1FE75A	FF59	0x1F015A

  The important point is that affected titles *only* have invalid data in the low halfwords.  Upper halfwords are never affected.

  It should be noted there are outliers.  In most cases, the address mask is slightly different, like this sole example from 7.10.2000 Turok 3:
AAF046	0BB0	*AA4046	[addr & 0xFFFF01FF, | 0x4000]

  In the 65 known errors in this Turok 3 build, only one outlier exists.  There's a ratio of 1/570 in Derby Stallion.  Some titles have more, and it's not impossible that some cases that don't adhere to the bit limits also exist.  There is not a full explanation of the behavior as of yet.  However, that also means not all errors are completely detectable, so attempting to fix a bad dump is (mostly) wasted effort.

  Unfortunately, even with these clear rules simply scanning for potential spots where bad data may be present will produce thousands of false positives.  It is best used as a filter when comparing other data sources or using other forms of comparison.


.: Verifying Bad Data Exists :.
  Without a point of reference it would be difficult to know if this corruption pattern really exists.  Confirming there is bad data should be the starting point.

  Firstly: if the game boots on console but the dump's internal checksum is bad, you have a bad dump.
  Anything using a retail CIC will verify the checksum at boot, permalooping on a mismatch.  The exceptions (which don't contain a checksum) are iQue and a devkit bootstrap designed for Indy boards.  If this bootcode (IPL3) is modified or corrupt, console and pickier emulators will permaloop in IPL2.
  Conversely, since this is not a simple sum of bytes, a good checksum means you shouldn't find any problems within the (usually) first 1MB of data.

  If the game has a retail counterpart or a duplicate of its data is found in the overdump region, you have a point of comparison.  Audio tends to be a good starting point, as it rarely changes in significant ways during development.  However, be aware that some titles don't define padding and it may change, providing a false positive.

  If you lack a reference, most titles use some kind of compression for files.  Even the laziest forms--RLE or light LZ--are unlikely to decompress properly when random values are changed.  The pattern affects 16 bits, and most can't tolerate more than 8 in exactly the right place.
  Some compression formats also provide checksums.  Turok 3 uses Rob Northern Compression, providing both an input crc (cyclic redundnacy check value) and an output checksum, in addition to input and output file lengths.  Many will at least provide an output length, and at worst you may need to look at the data itself to make certain it doesn't become garbage.
  If a file is suspected corrupt, check for the pattern and brute-force corrections to confirm.


.: Differences in Dumping Methods :.
  There's a number of different ways data is read from the hardware.  Some methods are software-based, others hardware-based.

  Software methods include devices like GameSharks/Action Replays and the old pirate scene dumpers: v64 & v64jr, z64, cd64, etc.  In these cases, software is loaded to the device and executed, blocks of data are read from the cartridge, and these are transmitted to either the device or a connected PC.
  Reading the data can either be done word-by-word (32bits at a time), or a PI request can be sent that reads a variable-sized block at a time.  In both cases, potential programming mistakes are not checking the PI is ready, not waiting until all the data is available to transmit it, mishandling your buffers between the software and your recording device, and mishandling the data cache.
  Those connecting to PCs are susceptible to picking up noise on the cables running between the devices.  Keep them away from other noisy lines, power cables, and keep them short so there isn't a problem.

  Examples of hardware-based methods are the Ultrasave + 64drive, Sanni dumper, and the console itself.  They directly drive the bus through the address lines, reading from the data lines.
  Prototypes are effectively flashcarts.  One concern is that the devices provide enough power for its on-board FPGA (usually an Altera chip) as well as its array of flash chips.  This requires significantly more power than 1-2 dedicated maskroms.
  It's unknown if prototypes are more or less sensitive to changes in bus protocol, but it's fairly unlikely.

  No two methods are quite the same, and all have advantages and disadvantages.  A corruption pattern is an indication that one of these methods has a flaw--at least when dumping prototypes.  It's important to identify which it is so it can either be mitigated or avoided in the future.

-Zoinkity