Unite 2018 Mirishita Presentation Notes

Please refer to the slides when reading these notes, otherwise they won't make any sense.

This is a (rather rough) TL of the Unite 2018 Mirishita presentation. It contains a lot (and I mean A LOT) of
technical terms, so I apologize in advance - I'm no master of Unity.

I hope this helps you understand what went into the development process at least!
_______________________________________________________________________________________________________________________________________

Nanto! Unite!
What is this "AKANE Great Strategy" that supports Mirishita?

Presented by:
Hayato Ikeda: Bandai Namco Studios/Lead Programmer
Masaki Katou: Bandai Namco Studios/Programmer

- Hayato Ikeda:
 - Entered Bamco in 1996 as a hardware engineer
 - In 2003, became a programmer for mobile apps
 - Is the client engineer for Mirishita, manages version control via Perforce
 - Is a Noriko, JuliaP

- What is Mirishita?
 - iDOLM@STER Million Live! Theater Days: A Bandai Namco Entertainment Android/iOS Idol Live and Produce Game, aka Mirishita

- Live Information

- At release
 - 5 idols on stage dancing together
 - Can change positions and outfits at will
 - 5 people have specific motions when dancing to a song (formation)
 - 51 characters with different vocal parts for certain songs
 - Special appeal (with a new motion) during Lives
- During game's lifetime
 - Lives with 4, 3 and 2 people singing and dancing
 - 52 characters with different vocal parts for certain songs
 - Songs with transformations in them
- Latest addtion
 - Songs with 13 idols on stage

- Model data
 - Each model is made of 10,000 polygons (1,500 for face, 3,000 for hair, 5,500 for body)
 - Costume textures are one 1024x1024 page
 - Facial expressions are made via blend shapes.
- Stage data
 - Each stage is made of about 15,000 polygons (10,000 for the stage, 5,000 for the performance effects)
 - Stage textures are two 1024x1024 pages
 - The audience's concert lights are made of 1 mesh, making graphical load light.

- Post-effects and resolution
 - Depth of Field (DoF)
 - Bloom
 - Blur
 - Flare
  - Unity Image Effects, while high-quality, are too heavy for mobile, so they created their own lightweight, independent shaders
  - Uses an intermediary buffer to save memory
  - Realizes DoF by creating depth textures while rendering without passing
 - Resolution: 1280x720
 - Low resolution + MSAA (multisample antialiasing)
  - Created low resolution mode for devices with low GPU output, but heavily scrutinised due to jagged edges
  - While it does increase GPU load, in scenes with large draw distances the effects of low-res are higher, so there is a merit

- Timing sheet (for Stage events)
 - Timeline tool is a new tool for the Home version of Unity
 - Used to stage the Live scenes
 - Continuously improved with input from the Game Designer and Visual Artist
 - Complete control over camera, character positions, character expressions, stage movements, concert lights
 - 2 kinds, for Camera and Stage Direction

- Songs with seperate vocals
 - For home iM@S titles, there is a function where the vocals of a song change with the idol in a specific position (for this game, eg. Thank You, Welcome, etc.)
 - With 52 idols and 5-man units, number of permutations is 52P5 (311,875,200 total). With each song being about 2.5 minutes, it would require 46.5 billion seconds to listen to them all (that's about 1474 years, or 538194 days!)
 - Sound tools: CRIWARE ADX2
  - To accommodate this, each vocal file is given its own channel, and different vocals are achieved by muting the other channels.

- Development schedule
 - Apr 2016: Engineer and Technical Artist implement a 3D Live prototype build
 - Jun 2016: Version α1, implemented Title Screen -> Theater Menu -> Story -> Live functions
 - Nov 2016: Version α2, implemented all necessary functions
 - Jan 2017: Began Optimization Project
 - Apr 2017: Version β1, began use of AssetBundle/Server communication
 - May 2017: Version β2, began review of the entire game
 - 26th Jun, 2017: Last minute Unity version update, submitted app to the store
 - 28th Jun, 2017: Started distribution
 - 29th Jun, 2017: Began service

- Mobile development process
 - Developed on the latest model of iPhone, worked fine
 - Problem: does not work correctly on iPhone 5s, Android
 - Since it cannot be released without dealing with these issues, independent from the development project, the Optimization Project was set up.

- "AKANE Great Strategy"
 - Optimization project to make Mirishita work on Android
 - Started around the end of development cycle, with a seperate line from the development team (a special team)
 - Codenamed "AKANE Great Strategy"
  - Android Kousaku-ka And NativE-ka Great Strategy (Android Work And Native Strategy)
- AKANE consists of:
 - 2D Drawing Bench
 - 2D Drawing Lightweight Memo
 - Getting POV of the in-house team
 - AKANE Great Strategy/Is String.StartsWith too slow?
 - AKANE Great Strategy/Latest (6/19) Live Processing Load
 - AKANE Great Strategy/Character Motion Capacity Reduction
 - AKANE Great Strategy/Theater, Mid-Commu Debug Button
 - AKANE Great Strategy/Testing Benchmark Data
 - AKANE Great Strategy/Model, Bone Investigation
 - AKANE Great Strategy/Live UI and Improvements
 - AKANE Great Strategy/Increasing Live UI speed
 - AKANE Great Strategy/Mid-Live Debug Button
 - AKANE Great Strategy/Reducing Load Times
 - AKANE Great Strategy/Verifying Data Items in the List
 - AKANE Great Strategy/Independent Application of Animation System
 - AKANE Great Strategy/Background Effect List
 - AKANE Great Strategy Minutes
 - AKANE Great Strategy Weekly Report
 - Summary of other Android models
 - Android model trends
 - Review version summary
 - Criteria for small loads on characters
 - Screenshots
 - Questions for those in other professions
 - Theater・ADV Analysis page
 - Summary of procession load of each game part
 - Regarding the settings of each Rendering Mode
 - Measurement results (XperiaZ4)
 - Regarding how to read time measurement
 - Load Countermeasure Information

- "AKANE Great Strategy" Approach
 - Review project settings
 - Speeding up 3D rendering
  - Have to look at CPU and GPU to measure efficiency
 - Speeding up 2D rendering
  - From Canvas to Sprite, cannot have useless rendering of transparent parts, image atlases, etc.
 - Scheme to prevent spikes
  - Must reduce GC (Garbage Collection)

- "AKANE Great Strategy" Goals
 - To reduce SetPassCall(DrawCall)
 - Reduce load of GPU
  - Introduce texture atlases (aka spritesheets)
  - Integration of Index buffers
  - Reduce adjustment times for Material Rendering
 - Reduce CPU load
  - Increase processing speed to deal with shaking objects
  - Processing threads (will not work if CPU does not work)
 - Optimizing memory
  - Any scheme to ensure no GC?
- In the end, managed to get within 16.6666....ms of 60FPS.

- "AKANE Great Strategy" Policy
 - Do not reduce visual quality
 - Make sure it can work on many models (draw settings)
 - Must do your best with the engineers until the final moments
  - "No post effects...shaking objects...review LoD...Frame skips...reducing render resolution...reduce number of Bone influences..."

- Profile
 - Rough CPU processing can be seen with UnityProfiler
 - GPU is analysed with SnapdragonProfiler, distributed by Qualcomm
 - Preparing analysing environment, digitize and compare
  - Cannot measure anything worth if the optimization is blindly implemented, must profile and make comparisons with every change
  - Major changes will not affect much, but minor changes have a big influence. It is important to monitor and analyze the environment every time.
 - Must know the habits of each terminal
  - Performance and operational habits depends on the model, must have the mindset that everything that happens is an everyday occurence
  - Must understand system clock to understand heat generation

- Improvement Idea
 - Various ideas for "AKANE Great Strategy" were put in a list.
  - If we can reduce even by a few micro-seconds, it is worth a try
  - Important not to aim for victory, but to dare to try
   - May work out for different things!
  - Must try with your hands, not just with your head

- Implementation and testing
 - P: Prioritize ideas for optimization
 - D: Implement
 - C: Build and Measure (Profile)
 - A: Come up with further ideas
  - ※Plan → Do → Check → Act
 - For easier testing, have the ability to turn ON/OFF optimization data
  - Easier for the visual artist to check
 - Test with heavy (load-wise) characters (5 of them)

- Regarding Unity
 - Despite knowing how to program a game engine, many things I still need to learn
   - Just learnt how to perform multi-thread
 - Won't do things that cannot be implemented in a short time
 -  Also, many Unity functions to deal with
 - Unite's forum videos, Unity tutorials were very useful
  - Also used Unity Forum and Unity Issue Tracker
 - Cannot update Unity after release, so had to be careful with that
  - Tried to use the latest versions as much as possible during development

- Daily improvements
 - Optimization still going on even after release!
  - Post-effect repairs
  - Material integration
  - Stage direction optimizations
  - Reducing data size by implementing an independent animation system
  - Reducing GC
 - Profile is done periodically due to the implementation of these new functions

- 13 Man Live
 - Around the end of 2017, there were discussions: "Can we have more that 5 people in a Live?"
 - Since they had prepared testing tools for 5 or more characters, they performed a project that could have 15 people on stage at once.
 - Due to this, 13-Man implementation was created within 3 months
 - No special memory reduction or load reduction functions for 13 Man
 - The accumulation of the above optimizations

END OF FIRST HALF
_______________________________________________________________________________________________________________________________________
SECOND HALF (Refer to the slides - this part gets quite confusing even with them)

- Profile: Masaki Katou
 - Visual Programmer
  - Works with motions, graphics. It is not as cool as you think. (w)

- Mission:
 - Reduce the number of draw calls for characters
 - Do not "rebuild assets" without changing the appearance

- REPORT
 - Integration of submesh
  - Reduced number of draw call for a character from 25 to 17
 - Command Buffer
  - Can ensure fine control of model rendering

- What a character model consists of
 - Costume, Sequins, Skin, Character Color (refer to slides for illustration)
 - Shader:
  - Pass 1: Actual form
  - Pass 2: Border outlines
 - Border outlines
  - Render with inverted polygons
  - Expand using vertex shaders
  - Share objects with meshes

- Submesh obstacles
 - Draw call - certain objects don't make it past the submesh wall!

- How rendering works
 - Vertex buffer
  - Vertex position, UV
 - Index buffer
  - Order to join vertices
  - Materials necessary for rendering
 - Material
  - (Omitted for now)

- Mesh Class - System Method
 - Submesh designation
  - Need: SetIndices(), Index, Topology
  - Don't need: SetVertices(), SetNormals(), SetTangents(), SetUVs(), SetColors(), boneWeights(), Position, UV, Tangent, Normal vector, Color, Influence
 - Vertice-related methods do not require submesh designation (Note: Unity 5.6 used)

- "Mesh" and "Submesh"?
 - Go look up Unity mesh and submesh lol

- Native Implementation
 - Ensure only 1 index buffer
 - Render part or whole based on index range specifications

- Is it OK to add SubMesh?
 - Combine all submeshes into 1

- Testing
 - Adding submeshes - no changes
 - Expand range of materials, add materials with outlines - Comes out properly!

- Result
 - 25 ->17
 - Not really decreased by much?
 - 5 people (now 13!)

- Before Optimization, 44 cycles
- After, 36 cycles (reduced by 8)

- Next goal
 - Material LoD, combine Costume and sequins, reduce 1 draw cycle
 - Use Command Buffer, render submesh unit
  - CommandBuffer.DrawRenderer

- How to turn off SkinnedMeshRenderer rendering?
 - Wrong: Set enabled to false - won't stop until skinning
 - Correct: Leave Renderer.materials [] empty
  - Skinning and frustum culling occur

- Applications:
 - Ensure submesh units are visible
 - Wire on shade (for debug purposes)

- Optimization concept - "AKANE Great Strategy"
 - Realizing common rendering methods in Unity
 - Not enough to know Unity's specs, must know native implementation to speed up performance
 - Is Unity a "Black Box"? It's not "Mystery Technology"

- Frequently Asked Question - Why the "AKANE Great Strategy"?
 - Began accelerating development around New Year's 2017
 - 2017 Zodiac was the rooster (酉)
 - 西 ← 酉 → 茜(Akane)
 - Thus "AKANE Great Strategy" was named! (It's a lie~)