Pixel Aspect Ratio in retro systems.

PIXEL ASPECT RATIO
Perhaps you've witnessed the abomination of a stream or youtube video stretching a "4:3" retro game over a 16:9 apsect ratio canvas and laughed along as someone made fun of it. Well, guess what -- unless you're streaming N64, 4:3 isn't correct, either. While it is MUCH closer to correct it still isn't faithful to the way games appeared on our Cathode Ray Tube TVs, which I will argue is how the best game artists intended the game to be seen.   Some retro game streamers have been doing it right long before now, but many still are newcomers or think themselves incapable of getting it right. Though most of this pastebin is aimed at explaining the industry standards that led to unsquare pixels and other technical stuff with links to resources, I created these templates to use in basic layouts with  correct pixel aspect ratios.
https://imgur.com/a/PTRa7

BACKGROUND
I'm EunosXX.  I've been streaming speedruns of The Legend of Zelda and racing Zelda Randomizer (by fcoughlin) on SRL for barely over a year. Others know me for racing the even younger Zelda 2 Randomizer (by digshake) or even for doing wood carving on stream.  As it happens, I also make furniture -- bed frames, tables, cabinets, bookshelves, and smaller novelty items like boxes and rocking horses. Attention to detail is what I do. Another thing I learned in the shop came from my vastly more experienced friend; and that is that over-attention to detail can be a time-waster.  The underside of a table doesn't need to be as smooth as the top, the hidden framing doesn't need to look good.  Furthermore, there's more than one way to copy a piece of overpriced furniture from the Pottery Barn catalog (SOOOOO OVERPRICED). Likewise, the purpose of displaying games in the correct PAR is for the subconscious appreciation of aesthetics, not so that everyone understands the inner workings of analog motion picture. This is why I didn't bother going through great lengths to explain it previously. Finally and ironically when it comes to accepting technical, mathematical concepts (as I expect some have come to know about me), I compulsively must fully understand and be able to do it on my own or completely surrender all hope of being capable. There is no middle ground.

MY STREAM
For the first five months I streamed, I was playing exclusively on emulator. It was messing around with the video config that I learned the NES resoltuion is 256x240. Other than being able to tell that Link looked thinner than I remember, it never occurred to me that the pixels should be anything but square. People who came to my stream frequently remarked about the good game image quality, so that reinforced that everything was fine.  Then I garbage picked a nice CRT and got myself a GV-USB2 to capture from. I was becoming aware of differences in the capture resolution from the resolution on emu. The GV-USB2 could capture NES in 720x480 mode or 704x480 mode, but there was junk outside the area I'm used to seeing on emulator that needed covered up. (Black space, a white line, NES BG color... aka overscan)  I just wanted to make my analog capture look as good as possible for the viewers. Last fall, based on the advice of acmlm I went from streaming in downscaled 1080p to normal 480p and also used the unscaled capture size from my GV-USB2. I tested multiple capture programs and deinterlacing filters in each, and finally went with Yadif2x in OBS at 29.97, encoding to 60fps.  As far as I was concerned, the capture was as good as possible on the given hardware. Since the capture area is pretty close to 640x480 (a 4:3 ratio) I just assumed everything was correct.

WATERSHED MOMENT
Then one day I was talking in my twitch channel with Myria who, to our gaming community, goes back to the 90s with the FFV English translation. I don't recall the context (not that she needs any =^.^= ), but she pointed out that on our analog displays the game pixels aren't square. This reminded me that during a recent fit of curiosity & perfectionism I had actually taken a tape measure to my TV and measured the tiles on the dungeon floor in Zelda.  They were about 10% wider than tall. Having bigger claims on my attention, I never did anything with that piece of information nor did it seem all that relevant. To make the cognitive connection from that to the point of this pastebin on one's own involves a bit of prerequisite knowledge about the digital consoles we are playing, the analog outputs they produce, and the devices we are using to view them. For me to provide you with all that knowledge is beyond the scope of this pastebin about Pixel Aspect Ratio.

VIDEO GAMES ARE FUN FOR THE FAMI[LY]COM
Though it seems both obvious and tangential, I find it necessary to point out that the Famicom is a bit different from a "home computer" like an AppleII which came with a monitor. Though home computers were a breakthrough in the late70s and early 80s, computing was arcane to the average person and at $600-2500 the price wasn't inviting to the uninitiated. However, a television was an established household item. Nintendo's goal with the Family Computer was to bring video games to the television set in the family's living room. But in 1983, a niche industry like video games did not influence the design standards for consumer-grade TVs, nor did consumers-at-large purchase TVs primarily for use in gaming. (In fact, some people still owned black&white TVs due to their operational longevity.) Thus, designers for video game consoles needed to work with what TVs the consumers already have.  Now, let's get a little more technical.

NTSC - I KNOW THIS ACRONYM
As a result of this marketing goal, Nintendo used NTSC standards since Japan's TVs were NTSC spec -- same as North America.  What are the basic guidelines of NTSC?  You can find out that and who decided it here and in the following section: https://en.wikipedia.org/wiki/NTSC#History . There's a lot of technical stuff there, and I don't understand it well enough to teach it.   In short, those standards combined with 8-bit limitations dictated that early computers should render a 240 line resolution, the Famicom/NES being just one of many systems to do so.  For an added wrinkle, consumer NTSC TVs commonly but not uniformly cut off some lines from the top and bottom. So, the people designing software to be displayed on consumer TVs worked under that assumption as well. For example, in Super Mario Bros. there are 16 pixels of empty sky above the text in the HUD (score, etc), and TWO full tiles or 32px below Mario's feet in 1-1.  Thus, the game's full aesthetic and function is preserved in 208 visible lines out of the 240 rendered. Furthermore, many games (such as MM3, SMB3) have artifacts that exist outside the typical boundary of an NTSC TV, which the designers left there since we weren't supposed to see it.

GOOD LORD SNES IS SO MUCH BETTER LOOKING THAN NES
The SNES used the same output resolution as NES, but added a 224-line display mode developers could and did most often use to just replace the commonly unseen area with black space. Note, that the full 240 lines (actually 241.5) are all still there, so when you capture SNES games using the 224-line mode, there should be 32px or so of black bar assuming an typical SD 480i capture. However, this is not enough to conclude Nintendo had a definitive uniform specification or even guidelines for first-party games and its licensees about how to artistically represent their games over the stretched display.

HOW FAMICOM SPECS ARE INFLUENCED by 1920s FILM
So far we've only talked about the Y-axis of the display resolution, but for a specific reason. It's the one that's limitations were dictated by consumer forces.  For a long time, 4:3 was the recommended aspect ratio for TV sets because it was the aspect ratio for 35mm film. (Wait, 35mm film is even older than NTSC!) Come to think of it, how the heck does a 256x240 or a 256x224 image fit on a 4:3 ratio TV -- those resolutions are 16:15 and 8:7 aspect ratio.  Wouldn't it have to be something like 320x240 to fit on a 4:3 aspect TV set? Actually, it doesn't necessarily matter how many pixels there are width-wise because a CRT TV doesn't create the image from individual pixels, it creates the image in horizontal lines at a specific refresh rate. (That refresh rate is ~15,700Hz, which is the source of your CRT's high pitched hum)  A computer system can control the output rate so that when it interacts with the TV set the image is displayed at the intended width. Given how the systems' outputs were influenced by film and TV standards, keep in mind the question: Did game artists design with these factors in mind?

CHRONO TRIGGER
Here's the memorable scene from Chrono Trigger of Magus' Castle silhouetted by the moon.  http://imgur.com/C3T8FwV   The bottom half is from an emulator displaying the rendered and cropped 256x224 image in perfectly square pixels in an exact 2x scale. It's pretty obvious that the moon is oblong. The top pic is captured from the analog output of a SNES rendering the exact same resolution in ~4:3 aspect ratio. The moon seems a lot closer to a circle. This artistic evidence persuaded me that the artists do create with the knowledge that pixels will be stretched. Actually, if you start measuring the emulator's is 208x236 pixels, while the moon in the NTSC image will come out to about 260x252. The NTSC capture still has common mistake where it's trying to correct the ratio by vertically stretching away the black bars so that the 640x480 (4:3 ratio) space is filled completely and missing nothing, but this is also not completely correct. What is correct, then and how is it determined?

HOW THE F*** DO YOU MATH?
Let's just say you were a CT fanatic and only cared about capturing this game in it's artistic intention. The simplest approach would be to stretch the area of your game capture or emulator until that moon was perfectly round.  You could also measure screenshots pasted to MS Paint.  This approach doesn't necessarily apply to every game or answer to the uniformity of this approach by game designers. If you did believe that CT's moon was the Rosetta stone for pixel stretching, all you'd have to do is solve for 236/208 to find the pixel aspect ratio (about 1.14, in this case). However, without this Rosetta stone, let's look at the system-wide 224-line mode as a standard -- that the 256x224 resolution is what is intended to stretch and fill the 4:3 area.  Is it as simple as multiplying the 224 lines that are standard to the SNES mode and approximately standard to a TV by 4/3?  Let's say you do that (ignoring doubling for now), and get 298.66666 pixels of width to 224 height. If you then increase that by 4:3 you will arrive at 320x240 and 640x480. Problem solved, right?

THIS IS WHY WORD PROBLEMS WERE IMPORTANT.
Hold on, isn't that ultimately what the captured image of CT's moon did and arrived at a just slightly over-stretched moon?  How did we make that mistake?  Well, what we needed to find was the width of the pixels, not the number of digital square pixels to make a 4:3 ratio.  Let's try to put it into words.  We need to find out the ratio by which to multiply the width of our 256 original rendered pixels in order to simulate the 4:3 total aspect.  We're going to do it by using the ratio of simulated pixels in the 4:3 ratio to original pixels.  This means that { (298.66666/256) * X = 4/3 }  298.66666:256 turns out to be 7:6 so we can just solve for X easily.  { X = (4/3)*(6/7) }.  X = 24/21 which simplifies to 8/7 or 1.142857 repeating. This means the pixels are 14.2857% wider than they are tall and that 8 pixels on the y-axis should have the same length as 7 pixels on the X-axis.

WHY DIDN'T I JUST LEAD WITH THIS?
Don't believe it? Well, here's the simple technical data to go with the more artistic proof.
To quote http://wiki.nesdev.com/w/index.php/NTSC_video#Basics
"NTSC Master clock is 21.47727273 MHz and each PPU pixel lasts four clocks"
reference that with http://pineight.com/mw/index.php?title=Dot_clock_rates
"Frequency: 5.37 MHz 	Exact Frequency: 945/176 MHz	PAR [pixel aspect ratio]: 1.14	Exact PAR: 8:7	Devices: The 256px display of the TMS9918 (TI-99/4A, ColecoVision, MSX, and SG-1000), its descendants (MSX2, Master System, Genesis), and other VDPs inspired by it (NES, TurboGrafx-16,[4] Super NES), and 256px mode of the original PlayStation[5]
To translate this jargon into jargon we care about: the NES PPU was specifically designed with that exact frequency so that when used on an NTSC TV the image would stretch 256 pixels wide enough to fit into a 4:3 space.

WHAT TO DO ABOUT IT.
A typical analog capture device (Dazzle DVC100, GV-USB2, etc) captures the above NTSC systems using a 480 line mode which works fine for our purposes. For any 240p system the correct width for the game area is { horizontal resolution * scaling factor * pixel aspect ratio }.  For the case of a NES being captured in a 480 line mode (480/240 = scaling factor of 2) it looks like this { 256 * 2 * 1.142857 = 585 } or 585x480. It doesn't matter if you're capturing a SNES in 224-line mode with black bars at the top and bottom. Leave them there, put your overlay on top of them, or crop them out.

My preference is to crop down to 224 and fill the layout vertically to create the largest artistically intended image. In this case, I need another multiplier to get a new width. This multiplier is the original vertical resolution divided by the resolution you crop to {240/224}. In full: { 256*1.142857*240/224*X } where X is the scaling factor of your stream or recording. For 480p {x=2}; 720p, {x=3}, 1080p {x=4.5}. Alternatively, now that you know the pixel aspect ratio, you can apply this knowledge using trial and error rather than algebra. Just take a screenshot of your OBS window, paste it to Paint and select a 7 tile wide by 8 tile tall space in the game.  If the dimensions are square your PAR is dead on.  You can also use this template in OBS which provides you with the most common layout options. https://imgur.com/a/PTRa7

For the record, my GV-USB2 capture device at 720x480 or 704x480 closely doubles the 240p raster to a height of 480i lines the way a TV would, but it stretches the 256 pixel game area to 644 pixels. 644/512 or 322/256 gives a pixel aspect ratio of over 1.25:1 -- that's almost as far off from correct as your 1:1 pixel emulator.

PAR CORRECT SHAPES IN OTHER POPULAR GAMES
Some other titles which make correct shapes with unsquare pixels: In The Legend of Zelda (NES), the triforce on the title screen is 71x71px.  If you stretch the width by 8/7 and it will be ~81.14px along the top. Use half that for A in do Pythagorean theorem (a^2 + b^2 = c^2), the height of tringle for B and C will be 81.77. Pretty close for a NES game. LTTP also has equilateral tringles.
I think that between the technical specs of the NES & SNES PPU, first- and third-party releases accounting for unsquare pixels in game art to make squared objects we can conclude that the 8:7 pixel aspect ratio was known and is worthy of the small amount of attention to detail that is required to recreate it.