tyler infrastructure 1

# Pastebin ntXQr6kp
adacic — Today at 10:26 AM
***`zenon de kition @DeKition 145 Following 90 Followers
***These days, supercomputers are constellations of processor blades with super-high-speed interconnect fabrics.

The cost of building supercomputers can be low - I’ve built what could be described as a supercomputer using a bunch of processor boards with 1.6GHz four-core ARM processors and gig-E interconnect. My total cost in is a couple of grand. Going much bigger, I soon discover that my interconnect costs dwarf my CPU costs…

Same is true at larger scale - even if you’re using blades with multiple Xeon processors, your interconnect now needs to be 10 or 100 gig Infiniband, and you soon discover that the couple-grand-per-blade cost on the host adapters gets swallowed in building deep enough Clos fabrics out of switches which don’t have nearly enough ports…

Then you get to the real killers - power, and uptime.

Power on supercomputers is a thing, to the point that the most useful measure is now in “teraflops per megawatt”. Power - including HVAC for cooling, power distribution, power backup, etc - is the big soul sucking cost involved in running a supercomputer center.

Uptime, that’s a thing too - give a very large number of cores (say, oh, I dunno, 100k cores or bigger) and very complex problems, the odds of having a CPU core failure or a memory failure or some other failure that blows out one of the nodes in your computing constellation are nearly guaranteed to mean that you have to deal with that, in real time. So, you can’t write software that just naively assumes that everything is fine all the time, and you can’t count on the OS to know that things are fine or not because there’s no “OS” for the constellation as a whole, just for the individual blades…***
`If you’re trying to make a supercomputer for your own use, you will find that it’s not too hard, for some definition of “supercomputer”. If you’re trying to make one to sell to others? Or to attack nation-state level problems? That’s a whole different kettle of fish…*``**