THE SCIENCE OF DANMAKU BY ALBERT WANG

Original text from http://web.archive.org/web/20090227001726/http://www.ocf.berkeley.edu/~albie/pseudoscience.htm

Abstract:

Sakuya is cute.
THE SCIENCE OF DANMAKU
BY ALBERT WANG

One of the pastimes I most frequently engage in, after drawing and crying over my empty shell of an existence, is practicing the Touhou games (which you may remember from the Art section.) Recently I was really bored and got to thinking what the most effective method of training for Touhou gameplay would be. I decided to find the answer... with science. (At least, as scientific as you can get with one tester, who also happens to be the researcher.)

I attempted to measure the effects of four distinct training methods, covering a continuum from an emphasis on memorization to a focus on a general skill factor (hereafter referred to as a g-factor, because that term sounds totally ace.) The first method consisted of playing in practice mode with a restart on every error. The second method was similar to the first method, but more lenient, calling for play to the end of the stage. The third method involved playing above my competency level, in an attempt to rapidly increase general skill. The fourth, and least memorization-intensive, was to play a different game entirely, but with similar enough gameplay to use the same skill set.

I. Practice With Restarts

[Sakuya Izayoi]

The first method I tested, one with which I had had some prior experience, was playing a stage in Practice Mode, and restarting the stage every time I was hit. The problem lay in empirically measuring the results of this form of training. Simply gauging my performance based on score would be inadequate, as score in Touhou games depends on many factors other than progress within a given stage. I settled on the "Cherry" system, the in-house performance-tracking system employed in Perfect Cherry Blossom (the seventh game in the Touhou series,) as it returned data as one number, and gave what I believed at the time to be a reasonably accurate representation of the player's progress. I settled on PCB's first stage on Lunatic difficulty as my practice arena; I chose as my character Sakuya Izayoi (of Art Section fame,) because she's cute I was comfortable using her and thus would be able to bypass the initial learning curve.
[Perfect Cherry Blossom Stage 1 - Lunatic difficulty]

I made 30-40 attempts at the stage per day for a week, recording the amount of Cherry points I had when an attempt was ended either by being hit or by successful completion of the stage. My expectation was that with continuing trials, my final Cherry would increase (allowing, of course, for some fluctuation,) and for several days, my expectations were more than met. On June 1, the first day of my study, I averaged 20,122.81 Cherry; on June 4, I averaged 52,832.94. However, on June 5, I relocated from my family's home in Foster City to a rental room in Berkeley, a move that had serious and unforeseen implications.
[A scatter plot of my final Cherry against the number of trials completed]

At my new location, I had the ability to play at a greater distance from my computer monitor, an opportunity which I immediately took advantage of. At the same time, however, my performance quickly declined and hit a plateau at approximately 20,000 Cherry points. Indeed, I was stuck at one point in the stage--Cirno's Spell Card. Even more perplexingly, I failed to recover from the crash throughout the remainder of the week. I had difficulty believing that such an apparently trivial factor as physical distance from the monitor could have such a dramatic effect on performance, but the possibility was confirmed when I returned to Stage 1 on June 11 (after a few days of no Touhou gameplay) in order to test the effects of the passage of time on skills. I performed the first dozen trials sitting close to my monitor, then I decided to retreat to the further position to see what would happen, and instantly my final Cherry values went from a range between 10,000 and 60,000 to a range between 19,000 and 22,000.

I regret that, due to my relocation and my underestimation of the effects of player position, the results of the practice-with-restart method could not be accurately and empirically measured. However, going on my results prior to the move, this practice method comes highly recommended for the short term. By the way, Shrinemaiden.com forums member Pilpsie devised a way to deal with the inherently high variation in Cherry; see the appendix for details.

II. Practice Without Restarts

[Perfect Cherry Blossom Stage 1 - Lunatic difficulty]

My second method was broadly similar to the first, with the main distinction being that rather than restarting on every error, I would play the stage to completion. Again I played on Stage 1 at Lunatic difficulty, but although I used Sakuya again, I opted this time for her alternate attack type (another suggestion of Pilpsie's.) The trials lasted a week again, but, as I would obviously be spending more time on each attempt, I played only ten times a day.

My expectation was that as I continued to practice and accrued experience, I would finish with more and more Cherry points each time. Indeed, over the week of practice with this method, I saw a fairly steady increase in Cherry, averaging a gain of 346.54 per day. On the final day of trials, I averaged 76,762 Cherry, as opposed to 45,652 on the first day. Even more encouragingly, I retained the benefits of the practice even after a period of no play; three days after the test concluded, I did ten additional trials, and averaged 73,041 Cherry. While my immediate gains were not as impressive as with the previous method, the steadiness and permanence of this method make it a solid choice.

III. Higher-Level Practice

[Perfect Cherry Blossom Stage 4 - Lunatic difficulty]

The third method I considered brought me away from the memorization school. For a week, I played Perfect Cherry Blossom's fourth stage on Lunatic difficulty (the highest level I was able to obtain for Practice Mode) for an hour a day, using Sakuya. If there was a general factor of skill involved in Touhou performance, I reasoned, practicing above my level of competence would raise it greatly. The expected end result of the week of practice on Stage 4 was thus an improvement in my performance on Stage 1, due to heightened skill. It is worth noting that at this point I decided to abandon using Cherry as a measure of proficiency, as it suffered from not only high variation, but it also rewarded behaviors not necessarily associated with skillful gameplay, such as being hit during a Supernatural Border while on a Spell Card. Instead, I opted to measure performance directly, keeping a record of how many lives and bombs I had remaining at the end of each playthrough of a stage, as well as how many Spell Cards I had captured (passed with zero errors.) It should be noted that the bomb count is less significant, as losing a life resets the bomb counter to four.
[Perfect Cherry Blossom Stage 4 - Lunatic difficulty]

At first, I had trouble completing Stage 4, but in a couple of days I became able to do so regularly. Somewhat disturbingly, I stayed at that same level of ability for the remainder of the week; on the final day, I was still barely completing the stage. Another surprise came when I did my trials on Stage 1, and compared the results to those I had done prior to the week of Stage 4 practice. Although I went from 6.2 average lives at the finish to 6.7, 3.35 average bombs to 3.75 and 2.05 to 2.25 Spell Cards captured on average, in none of the three areas did my gain exceed the standard deviation. It would seem that any general skill I had gained from my practice had had only a minor, statistically insignificant effect. My improvement under the first two methods was probably due more to memorization of the particulars of Stage 1 than to an increase in general skill.

IV. Tangential Practice

[Phantasmagoria of Flower View - Lunatic difficulty]

The fourth and final method I tested removed me even further from my Perfect Cherry Blossom control. My training avenue of choice this time was an entirely different game--Phantasmagoria of Flower View, the ninth game in the Touhou series. It was chosen because while its gameplay is distinct from that of Perfect Cherry Blossom, it nevertheless requires the same dexterity and perception that its predecessor does. If a g-factor did indeed exist, I theorized, a period of practice in Phantasmagoria of Flower View should be beneficial for my performance in Perfect Cherry Blossom. For a week, I played ten matches a day against a Lunatic-difficulty AI opponent. For each game, I used Sakuya, and selected the computer's character at random. (In the screenshots, I'm the one on the left.)
[Phantasmagoria of Flower View - Lunatic difficulty]

Over the course of the week, I saw modest improvements in my performance, although I was never able to defeat the AI. Unexpectedly, however, not only did my practice not generalize to Perfect Cherry Blossom, but it seemed to have actually had a detrimental effect on my performance. Out of 20 plays of Perfect Cherry Blossom Stage 1 on Lunatic difficulty (done the day after I finished my week of Phantasmagoria of Flower View,) I averaged 6.05 lives (with a standard deviation of 1.05,) 3.9 bombs (SD 0.3078) and 2.2 Spell Cards captured (SD 0.8944.) Although neither of the decreases I suffered was statistically significant, this outcome was surprising nonetheless. Based on my data (which can be found in full in the appendix,) I cannot recommend this method to Touhou players.

V. Conclusion

The implications of the comparative success of the first two, memorization-oriented methods are clear. My data strongly suggest that memorization played a greater role in Touhou proficiency for me than did general skill. Whether this phenomenon is due to my own personal idiosyncracies or a general truth about Touhou games is unfortunately beyond the scope of this study. Further research in this area is a thoroughly ridiculous notion. However, if it were not, I would advise that researchers be wary of such hidden confounding variables as physical location of players, and a more methodologically valid study of my first method would be particularly valuable. I hope newer Touhou players find this article helpful.

Appendix

At Pilpsie's suggestion, I tried a different method of analysis for my data on the first method, considering for each trial only whether or not I was able to complete the stage, and then taking the final Cherry amount for those successful attempts only. My successful trials were concentrated around the last day before my move from Foster City, with one outlier on the day I came back to Perfect Cherry Blossom after my period of inactivity. Interestingly, with the passage of time I saw a general decrease in the amount of Cherry with which I finished my successful trials. More information is available in the full data set, which you can download in Microsoft Excel format here.