catlify

toc:

noncas:
  tag 0f #optional
  bundles 01
    id 07
    offset 09
    size 09
  chunks 01
  name 07 #optional
  alwaysEmitSuperbundle 06 #optional, True or False both work

cas:
  bundles 01
    id 07
    offset 09
    size 08
  chunks 01
  cas 06
  name 07
  alwaysEmitSuperbundle 06

sb:

noncas (my own object names):
  ebxEntries
    name
    sha1
    size
    originalSize
  resEntries
    name
    sha1
    size
    originalSize
    resType
    resMeta
  chunkEntries
    id
    size
    sha1
    v1
    v2
  chunkMeta # this can be copied directly between cas and noncas
    h32 08
    meta 02


cas:
  path 07, the name of the bundle itself (found in the toc already)
  magicSalt 08 #no idea, this is always the same in all files
  ebx 01
    name 07
    sha1 10
    size 09 #this is the exact payload size found in cascat too
    originalSize 09
    (idata) 13
  dbx 01 # are these necessary?
    name 07
    sha1 10
    size 09
    originalSize 09
    (idata) 13
  res 01
    name 07
    sha1 10
    size 09
    originalSize 09
    resType 08
    resMeta 13
    (idata) 13
  chunks 01
    id 0f
    sha1 10
    size 09
    (rangeStart) 08
    (rangeEnd) 08
    (logicalOffset) 08
    (idata) 13
  chunkMeta # this can be copied directly between cas and noncas
    h32 08 #hash32. This is found in ebx files to access chunks?
    meta 02
  alignMembers 06, True
  totalSize 09, sum of payload sizes (i.e. "size") from ebx+dbx+res+chunks

note: essential are only
  path 07
  magicSalt 08
  alignMembers 06
  totalSize 09

for sbcas:
When idata exists, it contains the entire payload (always zlib?)! The file still exists in the
cascat archive, and the data there is EXACTLY the same as given in idata.
This hopefully means that idata is completely optional and can be omitted.
size is exactly the size of the payload found in cascat. Additionally the given sha1 matches the sha1 of the payload.


rangeStart, rangeEnd, logicalOffset always appear together.
chunk.meta is not nullbyte <=> chunk has range entries. This is false for noncas.
ALWAYS:
rangeEnd = size
rangeStart < size
rangeStart=0 <=> logicalOffset=0

rangeStart is the start of one zlib component (i.e. right before the two size ints). At rangeStart+8 there is the zlib header 78da.
logicalOffset is the position in the unzipped file for that.
The two separate parts divided by rangeStart/logicalOffset appear to be the same kind of file.
possible logicalOffset values:
0, 1024, 2048, 2560, 5120, 8192, 10240, 16384, 20480, 32768, 40960, 65536, 81920, 131072, 163840, 262144, 327680,
524288, 655360, 1048576, 1310720, 2097152, 2621440,          4194304, 5242880, 8388608, 10485760, 20971520


for sbnoncas:
chunks have three fields
v1, v2, logicalOffset
the payload size is v2-v1. If v1==0: sha1 matches the given sha1. else: SHA1 DOES NOT MATCH.
v1=0 <=> logicalOffset=0
v1 < v2
v1 <= logicalOffset
v2 can be larger or smaller than logicalOffset.
possible logicalOffset values:
0, 1024,             5120, 8192, 10240, 16384, 20480, 32768, 40960, 65536, 81920, 131072, 163840, 262144, 327680,
524288, 655360, 1048576, 1310720, 2097152, 2621440, 2752512, 4194304, 5242880, 8388608, 10485760, 20971520


unzipped => v1=0, v1!=0 => zipped
zipped:
   34159 v1=0
   29788 v1!=0

zipped:
   38340 meta=0
   25607 meta!=0
unzipped:
   11300 meta=0
   10593 meta!=0

v1!=0 => zlib format (recall v1!=0 => zipped) consists of several parts (and is never just one part)

logicalOffset!=0 => logicalOffset = (unzipped file size)*3 - 40 #many exceptions for this


noncas. Total number of steps in the file depending on logicalOffset
{392: 8, 43692: 9, 43792: 11, 520: 9, 10952: 10, 21896: 11, 10924: 8, 11024: 10, 11016: 11, 87440: 11, 22032: 11, 5488: 8, 174800: 12, 87408: 10, 87432: 12, 1544: 10,
 174792: 13, 5468: 8, 21872: 9, 2796204: 51, 349584: 15, 5512: 10, 1416: 9, 349544: 15, 10960: 9, 2796216: 52, 2832: 9, 87400: 11, 2768: 8, 174864: 13, 174776: 12,
 43720: 11, 3088: 10, 712: 8, 699080: 21, 1424: 8, 2824: 10, 699088: 20, 10936: 9, 21844: 8, 43728: 10, 349552: 14, 21852: 9, 5520: 9, 44048: 12, 699064: 20, 1384: 8,
 1398120: 31, 2744: 8, 1398128: 30, 5480: 9, 5648: 10, 21904: 10, 43784: 12, 87380: 9, 21864: 10, 43704: 10, 349524: 13, 2760: 9}




NONCAS
##########################
sha1 does not match for v1!=0
meta has nothing to do with zip status
meta has nothing to do with v1
v1!=0 => zipped, unzipped => v1=0
##########################


CAS
##########################
meta=None <=> unranged, meta!=None <=> ranged.
ranged => zipped, unzipped => unranged
##########################



cas:
2801 chunks
971 chunks with range entries
463 nonzlib chunks (many audio 4800000c, some others too). For these chunks chunkMeta.meta="\x00"
0 nonzlib chunks with range
70 chunks with idata
1830 chunks with chunkMeta.meta="\x00"

1830+971=2801 =>  range entries <=> chunkMeta.meta!="\x00"

chunkMeta.meta values
00 => None
0866697273744d6970000000000000 => 0
0866697273744d6970000100000000 => 1
0866697273744d6970000200000000 => 2
  f i r s t m i p




chunkMeta.h32: 32bit hash.



If rangeStart is defined, then meta != \x00. If rangeStart is undefined, then meta = \x00
The size given in sb is the same size found in cascat. The sha1 is always the sha1 of the payload.
The ranged chunks are always in zlib format (sometimes uncompressed). Some unranged chunks are audio in non-zlib format.


logicalOffset vs steps in ranged chunks: Recall that ranged chunks are always zlib.

To obtain the number of steps I do the following:
Read the uncompressed and compressed size. Add uncompressed size to the total uncompressed size.
If the total uncompressed size == logicalOffset, return the number of steps so far. Otherwise jump over compressed size in the payload and repeat.



There is exactly one specific number of steps for each logicalOffset. It's the same even between bf3 and mohw.
[(0, 0), (1024, 1), (2048, 1), (2560, 2), (5120, 2), (8192, 1), (10240, 2), (10752, 3), (16384, 1), (20480, 2), (21504, 3), (32768, 1), (40960, 2), (43008, 3),
(43520, 4), (65536, 1), (81920, 2), (86016, 3), (87040, 4), (131072, 2), (163840, 3), (172032, 4), (174080, 5), (262144, 4), (327680, 5), (344064, 6),
(348160, 7), (349184, 8), (524288, 8), (655360, 10), (688128, 11), (696320, 12), (1048576, 16), (1310720, 20), (1376256, 21), (1392640, 22),
(2097152, 32), (2621440, 40), (2752512, 42), (2785280, 43), (4194304, 64), (5242880, 80), (5505024, 84), (5570560, 85), (8388608, 128), (10485760, 160),
(11010048, 168), (20971520, 320), (22020096, 336)]

There are several stepnumbers to read the complete chunk.
{655360: set([20, 21]), 163840: set([11, 12, 13, 14]), 262144: set([13, 14, 15]), 4194304: set([94]), 1310720: set([30, 31]), 131072: set([12]), 1024: set([10]), 8192: set([10]), 524288: set([20]), 16384: set([10]), 5242880: set([94, 95]), 20480: set([10, 11]), 1048576: set([30, 31]), 32768: set([10, 11]), 5120: set([11]), 10240: set([10]), 40960: set([10, 11, 12]), 2097152: set([52, 53]), 10485760: set([180, 181]), 65536: set([11]), 327680: set([16, 13, 14, 15]), 2621440: set([52, 53]), 81920: set([10, 11, 12])}


meta=0 <=> unranged, meta!=0 <=> ranged.

ranged => zipped only (56017)
unranged:
   33615 zipped
   53041 unzipped


##########################
meta=0 <=> unranged, meta!=0 <=> ranged.
ranged => zipped
##########################








##################
65536 65335 78da
65536 65423 78da
65536 65440 78da
65536 65437 78da
65536 65387 78da
65536 65380 78da
65536 65481 78da
65536 65401 78da
65536 65415 78da
65536 65377 78da
65536 65381 78da
65536 65390 78da
65536 65379 78da
65536 65445 78da
65536 65391 78da
65536 65395 78da
65536 65536 b44f

Traceback (most recent call last):
  File "D:\TOCSB FILES\cas vs noncas\bundles cas.py", line 186, in <module>
    ha()
  File "D:\TOCSB FILES\cas vs noncas\bundles cas.py", line 151, in ha
    data,data2=cat.grabPayload(sha1)
  File "D:\TOCSB FILES\cas vs noncas\bundles cas.py", line 45, in grabPayload
    payload2=self.zlibb(cas,entry)
  File "D:\TOCSB FILES\cas vs noncas\bundles cas.py", line 73, in zlibb
    tr=zlib.decompress(tr0)
error: Error -3 while decompressing data: unknown compression method
############





files without 78da magic:

sha1 60f9416124ac9384abf18b3cad5b4b52dcec9ddf
size 349577
rangeStart 0
rangeEnd 349577
dif 349577
meta 0
logicalOffset 0

sha1 7415170f48f0e79ccb2b881e20116079d49169cd
size 1395961
rangeStart 1309917
rangeEnd 1395961
dif 86044
meta 2
logicalOffset 1310720


sha1 8dbc4e03fc44d9c01f2694cc5f56db89ce8d636a
size 1397590
rangeStart 1310100
rangeEnd 1397590
dif 87490
meta 2
logicalOffset 1310720







cascat archives:
cat:
The size must be >= the actual size of the payload.
1<=casnum<=255

cas:
The faceoff header might as well be omitted entirely, the game does not mind it being replaced with ff bytes.
zlib texture with rangeStart!=0 and firstMip 1: the uncompressed size of the first piece may be larger or smaller than the actual uncompressed size. The uncompressed size late in the file may be >= the actual uncompressed size. The game gets stuck in the loading screen with smaller sizes. => The game only decompresses the pieces starting from logicalOffset and skips the highres texture unless told otherwise.





D:\TOCSB FILES\cas vs noncas\casbundles\COOP_007\bundles\Win32\levels\coop_007\coop_007.bundle
dc16ae4638d8bc91588e611ea0ecd833e820b8b5
9
08594588225c143bcaadfc57f6918636f36166d4
17

D:\TOCSB FILES\cas vs noncas\casbundles\MP_013\bundles\Win32\levels\mp_013\mp_013.bundle
75d0530106c5cb473e274cd111cc5842b08951df
35

D:\TOCSB FILES\cas vs noncas\casbundles\SP_Bank\bundles\Win32\levels\sp_bank\highway_03_props.bundle
104bb008799c07ddb0816d49a25a195992e475fa
5

D:\TOCSB FILES\cas vs noncas\casbundles\SP_Earthquake\bundles\Win32\levels\sp_earthquake\meatmarket.bundle
7474364d2b7a74032fbac73139dac01033f20a98
2

D:\TOCSB FILES\cas vs noncas\casbundles\SP_Earthquake\bundles\Win32\levels\sp_earthquake\sp_earthquake.bundle
d0932a5c7643627e4df0e4712af90a73f7f1674d
17

D:\TOCSB FILES\cas vs noncas\casbundles\SP_Paris\bundles\Win32\levels\sp_paris\street.bundle
3e18daf5392a18d895d38182ce8d18c6b0a97589
325

D:\TOCSB FILES\cas vs noncas\casbundles\SP_Tank_b\bundles\Win32\levels\sp_tank_b\ambush.bundle
7682297a175a1717702369c5bd75497d7fd9d93a
17