MKWii Custom Character with perfect shading tutorial

This is a tutorial that shows how to make Custom Characters with perfect shading for Mario Kart Wii. This tutorial assumes you know basic modeling, and it won't explain how to create a model.

Requirements:
— 3ds Max (I use 3ds Max 2015, any version after 2010 should work fine).
— BrawlBox.
— Any Hex Editor.
— A 3D model of the character you want to import. Can be from any other game, made by yourself, or whatever.

• Step 1 - Exporting an original character model from BrawlBox
It is necessary to export the original character model you want your model to rig over. Please note that this does not mean that your character is replacing this original character, since you can actually use a character's rigging over any other character slot from the same weight class (e.g. you can rig a character over Mario and put it over Peach, but not over Bowser).

Simply open the original character file and export the model as DAE (by right clicking the MDL0 file and then choosing Export). Please note that you need to reset the bone scale values before exporting. This is the case of Bowser, who has a bigger scale value for his hands. Just expand the MDL0, go to bones, and find the ones that do not have a scale value of (1, 1, 1), and change them to that same value. Now you are ready to import your model in 3ds Max!


• Step 2: Importing the model in 3ds Max
Since your model was exported from BrawlBox as a DAE file, it must be imported in 3ds Max with a different scale rule than default. After selecting your model to import in 3ds Max, the FBX Importer window will open, you'll need to set Advanced Options → Units → Centimeters. After that, press OK. http://prntscr.com/9xjzyk


• Step 3: Removing original skin attributes and bones
If your model has any bones, you must select all them, right click and delete.

If you exported the model with bones from BrawlBox as DAE, you'll probably get an error when importing (http://prntscr.com/9xk0po), but don't worry, everything is correct. The model and its bones should appear correctly in the program. Now, you must delete all the bones. http://prntscr.com/9xk2u6

After this, you'll need to remove each "Skin" modifier from each object. Select them, go to the modifiers table and right click "Skin", then delete. http://prntscr.com/9xk4oj


• Step 4: Model reducing and optimizing
This is probably the hardest part to make a good model. First of all, toggle the display polygon/vertex counter by pressing 7. I also use the "Shaded" lighting option from the viewport (might not be in newer versions of 3ds Max): http://prntscr.com/9xk613

Now, you'll want to reduce polygon count. The simplest way to do this is by selecting all the objects and using the "ProOptimizer" modifier. After that, select these options: http://prntscr.com/9xk811


RED arrow → You need to check this (or else you will get a solid color model).
GREEN arrow → Optional: if textures are messed up with reduction (http://prntscr.com/9xk9js), select this, and it will be slightly fixed. This has some disadvantages, because it can take the double of polygon count (http://prntscr.com/9xkaj8), and it could even need a more precise and accurate UV map fix; abusing of this (reducing a lot with this feature) will make your polygons look like meshes reduced to a single vertex (http://prntscr.com/9xkbj6).
BLUE arrow → This is optional, but helps leaving unified vertices and faces. It takes only a little more of polygon count.


Now, select your reduce value percentage. Your model should not exceed 1500 polygons (maximum recommended for MKWii characters), although the size is actually determined by the BRRES size (driver_model.brres is recommended to be 250 KB or smaller), so you can also help by reducing the textures, the LOD model, animations, removing vertex colors, etc.

After reducing, your model could look like it has some errors, which will require a manual fix. When you finish the optimization, right click the modifier and click on "Collapse All", if a window pops up, click "Yes".


Nowadays, I don't use the ProOptimizer method anymore because it's not exactly precise. If you want a perfectly reduced model, consider using more advanced methods, like vertex welding and mirroring. This means more time to spend but for a much better result.

To perform a manual fix, you need patience. Moving polygons looks easy, but the adjustement is tricky sometimes. First of all, locate the incorrect objects which are out of the model's boundaries (http://prntscr.com/9xkeec). Fixing them will require adding vertices.

In my case, I'll move the eyes manually (http://prntscr.com/9xkfgu), add missing vertices to the feet (http://prntscr.com/9xkifm), and fix UV maps for the legs (http://prntscr.com/9xko9m). It could take some time, but with experience it becomes something easier and faster each time to perform.


When you are done with model fixing, it's convenient to attach all the objects with the same material into a single object. This prevents large file size and multiple model nodes (http://prntscr.com/9xkrra press CTRL+click to add an object to selection). You might also want to use the Vertex Weld modifier with the minimum threshold value to unify all the split vertices that are in the same position.


• Step 5: Setting up your model for the game
Now that you have finished fixing your model, it's time to import the original MKWii model you want its bones over. In my case, I used ma_mii_f's (Medium Female Mii) bone structure.


Import the MKWii character model (exported from BrawlBox as DAE), with the same attributes (Advanced Options → Units → Centimeters) in 3ds Max. Now, fit your custom character model with the MKWii model (http://prntscr.com/9xku65). If your model does not fit correctly (longer arms/legs, bigger feet/hands, etc.), it's recommended to adjust them. Select from the Editable Mesh modifier the vertices you want to adjust, and scale, rotate or move as much as you need (http://prntscr.com/9xkwr9). After that, remove the original MKWii model (except the bones) and apply a Skin modifier to your custom character. Then, click on "Add" and select all the bones. Now select skl_root, click on "Edit Envelopes" and select all the models' vertices. Go to "Abs. Effect" and set it to 1. Then, your model should turn all red. http://prntscr.com/9xl04z


• Step 6: BrawlBox "model patching"
This is a trick I discovered and I use to perform better model lighting and normal fix. Before rigging the rest of the bones, first export the entire model as DAE, with Advanced Options → Units → Inches. When you get your DAE, import it in BrawlBox (v0.77 preferably). Use the following settings when importing your DAE: http://prntscr.com/9xl2lh Also, you can import the textures now for a better preview.


After importing, check this setting:
http://prntscr.com/9xl36g


And then, apply the original material and shader settings to the model. To do this, you can either copy the values from existing models, or export and replace them. When replacing a material, make sure you delete all texture references it has, else it might become corrupted. Export the shaders and materials when finished. If you save the BRRES and open it again with BrawlBox, you'll notice it has incorrect lighting (http://prntscr.com/9xl4td). To fix this, export the model as DAE and import it again in 3ds Max, with the Centimeters scaling rule. Remove again the Skin modifiers from your model (Step 2).


• Step 7: High-performance editing
If you want perfect shading/lighting for your model in MKWii, you'll need to apply vertex colors and fix the normals, which is optional but prefered. Select all objects (not the bones) and use the "Edit Normals" modifier. Now select all the normals (CTRL+A), and they'll turn red. Check the "Use Threshold" box and press the "Selected" button. If your model looks weird when doing this, check "Unify/Break to Average" and press on "Unify". http://prntscr.com/9xl7t7 Then, right click the modifier and press "Collapse All". This will unify split normals, which will usually fix a model's polygon roughness, making it smooth. Although this method is useful, sometimes you will prefer to split some normals (for example, on shoe soles), so some polygons won't be smooth with the rest of polygons. To do this, uncheck "Unify/Break to Average", select the normals you want to split, and click on "Break". If you want to unify them back, make sure you check "Unify/Break to Average" again and then press "Unify".


Next you do is applying vertex colors. Use the "VertexPaint" modifier for this. A window will pop up, then, you will need to select all vertices. Next, click on the following buttons: http://prntscr.com/9xl8tb and then, right click on the modifier and Collapse All. Make sure you select the full white color (0xFFFFFF, R:255, G:255, B:255). You can also add some shadows by painting some vertices gray, or add color to determined parts if you want to have better lighting effects.


If your model looks fine for now, you can go to the next step. If there are one or some polygons with wrong lighting (like this: http://prntscr.com/9xla7v), the Edit Normals modifier should be enough to fix it, else, try choosing the "Edit Poly" modifier, select all the models' faces and press the "Auto Smooth" button.


• Step 8: Rigging bones
Since rigging is a complex modeling feature, I can't explain it very well. You must use the "Skin" modifier and add all bones to the list again. Then press on "Edit Envelopes", select all vertices and apply 1.0 in "Abs. Effect" for the skl_root bone.

For the rest of bones, I recommend applying the rigging in the bone tree order. I do like this:
— Select the spin bone and apply to the top half of the body.
— Select the face bone and apply to the head and the vertices that connect it to the neck, if any.
— If there's a mouth bone and your model has a 3D mouth (separated polygons for the jaws), apply to the bottom jaw. If it doesn't have it, skip this bone.
— If there's a tonge bone and your model has a 3D tonge, apply to the whole tonge except the vertices that connect it to the mouth. It might have more than one bone for the tonge, if so, rig it from parent to child, being the parent tonge_1, and the last child being applied to the tip of the tonge.
— If there are hair bones, apply them to the mane/ponytail if any, else, skip.
— Select the arm_l1 bone and apply to the whole left arm, from the shoulder to the fingers/hand. Don't include the armpit bones.
— Select the arm_l2 bone and apply from the left elbow to the fingers/hand.
— Select the wrist_l1 bone and apply to the whole left hand.
— Repeat the last three steps for arm_r1, arm_r2 and wrist_r1, but for the right arm instead.
— If there are tail bones and your model has a 3D tail, apply them to the whole tail from parent to child, excluding the vertices that connect to the butt or wherever the tail is connected to.
— Select the leg_l1 bone and apply to the whole left leg until the left foot. Pick some vertices from the pelvis if needed. If your model has no legs (like Toad), skip.
— Select the leg_l2 bone and apply from the left knee to the left foot.
— Select the ankle_l1 bone and apply to the left foot/shoe. Even if the name suggest the ankle, sometimes it's better to leave the ankle rigged to the leg_l2 bone instead. If your model has no legs (like Toad), rig the whole foot/shoe to this bone, and if it has no feet, skip.
— Repeat the last three steps for leg_r1, leg_r2 and ankle_r1, but for the right leg instead.

Make sure "Abs. Effect" is set to 1.0 when the vertices only affect one bone. If it affects two bones, you can divide the value in the proportions you want for each bone (0.5 for each, or 0.25 for one and 0.75 for the other, etc. http://prntscr.com/9xlckp). This step could take some time, depending on how complex your model is. You need a lot of patience to do this, and I can tell that it won't always work fine at the first try. Testing your model's rigging by moving the bones is necessary, but be careful and remember to reset the bones to their original position after doing this (you can use CTRL+Z to undo).


• Step 9: Importing your rigged and fixed model
Finally, you need to export the 3ds Max scene as DAE, and import in BrawlBox with the same properties as in Step 5. Now, replace Shader0 and the materials with the ones you exported before. Remember you must delete any texture reference found into the imported material before replacing, else the game might crash when using it.

Last step is fixing color nodes. Open the MDL0 and look for a folder named "Colors". They should look like this: http://prntscr.com/9xlep8

Now, you must change any of the colors' alpha value to a value that is not 255 (I use 254, on the first color). Do this for all color nodes (each color file found into the folder, not each color value), and export them. In a Hex Editor, you must find the alpha value (if you put 254, it will be 0xFE), and replace it with 0xFF (255). Save the file and reimport it in BrawlBox.


Before finishing, you can now check if the animations work fine in BrawlBox's previewer. You might also want to edit the texture pattern and SRT animations that control the eyes, if you also included multiple eye textures.

Now that you're done, save your BRRES and open it again in BrawlBox. If your model looks like this http://prntscr.com/9xlj9d check in the materials' texture references that lm_0 and lm_1 have the "HasTextureMatrix" property set to "True", when possible. http://prntscr.com/9xljwc

Your model should now look perfect! http://prntscr.com/9xlkba


Tutorial written by AtlasOmegaAlpha.
YouTube Channel → https://www.youtube.com/user/atlasomegaalpha
Twitter → https://twitter.com/AtlasOmegaAlpha
Discord → Atlas#5057