"one of the things i often, usually to no avail, try to explain to people, is that there is no such thing as 'efficiency.' there is only efficiency -at- something."

-- Stanislav Datskovskiy, http://log.bitcoin-assets.com/?date=16-11-2014#925565


Consummate with the preceding quote, I shall explain exactly what I would like to do, why the current offerings are unacceptable and what is required to meet my needs. The first idea I cannot currently express is a program for doing chemistry I described in #lisp earlier:

As it stands there are a bunch of databases (some of them on paper) detailing properties of different compounds etc. there are a few programs which understand molecular formulas and some of the diagrams we use to depict them. In organic chemistry courses worldwide students are taught they should memorize reaction charts. This is amazingly primitive. An intelligently designed system would have all of these charts computerized - instantaneous access to any database you might desire, taking into account intellectual property in some fashion (not according to the currently laws, this much is certain) and also allow one to change between any given representation of the structure in question on a whim, modify it on a whim - you can play with Doug Hoyte's molview program if you'd like to get a very crude sense for what I'm thinking of. Also, one should be able to specify "I want to go from here to there" in the reaction charts and either get a 'recipe' for their lab (ie, program 4 r0botz) or run up against the current human understanding of the subject; or the limits of your own fork of the "chemistry" program that takes into account your lab data and bent. Mathematica wants to be this, but isn't and never will be for a number of reasons I'll address another day. The generalization of this is the ability for all programs in some context to seamlessly interoperate with each other without any loss of expressivity. The studious will note that this is a completely solved problem that no machine learning algorithm will be able to approximate anytime soon.

The second idea is to redesign cities. Today they're optimized for cars and the results are without distinction terrible. I've visions of a gleaming metropolis from my childhood dreams I'd like to visit sometime and such a thing isn't available anywhere on earth. Before we get into the implementation details let's read a passage from the Torah:


"It's time to reform transportation.

Ist of all, cars are not cool. Cars aren't something the rich, the rebel, the young, the cool drive around anymore than Camels are something the sophisticated light up. I get it, car companies spend a lot on advertising. Who cares ?

Cars are a tool, much like that long snake plumbers use, rooting through your shit to clear pipes. Driving is for truck drivers, driving is for weird pakis coming by the clan to NY in search of \"a better life\" and taking turns sleeping in the trunk. Driving your car to wherever you're going is not so much unlike showing up in your overalls, with a plunger or whatever in your hand.

For this reason cars really have no business whatsoever above ground, not anymore than rats do, at any rate. The modern town would limit car traffic to tunnels strictly, running parallel with the sewage tunnels. Under the streets and buildings where actual people go. Nothing but a succession of garages and pipes taking cars from garage to garage. You want to see the light of day, you want to interact with human beings ? Drop the car off at 22 purple or whatever, hop in the elevator and here you are.

II. Bikes are not cool. They're fucking modified wheelchairs, nothing more, nothing else. Using a wheelchair for no reason whatsoever, just to bother people and get in their way is a misdemeanor, using a bike AT ALL should be a felony by default. It's clear you're not handicapped, or else you'd be in a proper wheelchair. You're just bothering people and getting in their way, fuck you for being a shitty sociopath, off to jail for one year to meditate on what a little shit your parents brought into the world and to cut to your heart's content. Remember : along, not across.

All this infestation of the perfect form of the human body with ill thought, aesthetically offensive mechanical contraptions is frankly nauseating. Walking is so great, so noble, so close to true nature. Why not enjoy it for what it is, away and protected from the insanity of they in perpetual search of crutches ?

I tell you, cars underground, where they belong, like the slaves in any Roman bath, like the slaves in Disneyland. Doing their thing quietly away from actual human society. Bikes verboten on account of being evil in the eyes of the lord. Hapiness. Greatness. Perfection.

PS. The discussion about mechanical contraptions above extends to anything and everything, like all the weird shit 1900s people came up with as \"inventions\", but it specifically excludes high heels. High heels are great, and even if we ever move to public nudism high heels on nubile women remain mandatorily required."

-- Mircea Popescu, http://trilema.com/2013/time-to-reform-transportation/


The "vision" being sold to the general public on the cities of the future by Google etc. is lame. I've seen the self-driving car many times and it is just that - a car that drives itself. By attempting to mutate the car into something that understands urban sprawl instead of taking the opportunity to redesign cities they've turned what could have been a straightforwards engineering problem into an open research problem with icky political implications.

We can most certainly build a "succession of garages and pipes taking cars from garage to garage" with standard interfaces that only accept cars of certain dimensions so as to prevent any undesirable organisms from making their way in. Ensuring that the cars don't run into each other isn't that difficult, model the car/pod network as ~fluid flow, start from 5mph and 2 cars on the road and scale up the simulation in both real life and on computers. Incidentally, this version of the car, a decadent self-driving living room that allows me to travel from here to there in a disorder of girls and psychoactive substances is far superior than anything I've seen coming out of Google. Why, pray tell, do I want to have the same old boring seats, trunks, compartments etc. when I could have leg room, loveseats, computers & anything else I care to bring with me? I and every other person on the planet knows exactly how to work the thing, just make it happen already. I parse Google's lack of vision naively. If they had a plan they'd be yammering about it nonstop because GOD DAMN THIS IS EXCITING.

Underground opulence-pods is a good starting point for what I'm getting at, so let's work our way up. If we're building standard interfaces, we should probably settle on an abstraction like the city block. Let's generalize it to three dimensions - a voxel (to overload a gaming term). All we need is a standard edge length on our voxels, standard interfaces for electricity, internet, algorithms for decoding / encoding common abstractions (opulence-pod ducts, hospitals, housing, transportation, sewers etc.) and you've got a system that can be constructed by robots. More specifically, teaching a robot to understand a regular 3D cube structure isn't terribly difficult: have a programmer write some path-finding algorithms to traverse the voxels by ~electromagnetically locking onto a face, rotating in some manner onto the adjacent face, `defmethods' specializing on the type of voxel, orientation in 3D space, and type of face you're adding / removing to the structure, etc..

Computers are much better than humans at doing well specified tasks, and cities constructed thusly will be far superior to anything currently available. New and better versions of some voxel are released? decode the old voxels, re-encode with replacements and move the people back in, drones to move human things from point A to point B, with point B being a standard transportation chute that can ship things (or people, as fdesired) at 300mph (ie, https://www.youtube.com/watch?v=9oVx0sq2JXM). City planners today (whatever they're called - civil engineers or something) cannot programmatically specify that given the price of voxel faces according so-and-so voxel-face future we've taken out, construct N more housing units, M hospitals per day within a radius R of each population center while decoding the slums and moving them under the ducts, all along replacing them with brand-spanking-new gentrification voxels at a rate of 100 slum voxels to 1 gentrification voxel/hr, though this rate is tied to the # of traversal-bots on hand which can be purchased at a price of .02 BTC each and will arrive within the hour.

The political implications of such a habitat are non-trivial. Today intelligent persons worldwide are beholden (to various degrees) to the host state and moving elsewhere is a tedious, time consuming process. Being able to pack your town onto a few ships overnight is a useful political tool. Morons managed to make their way into power while you were busy with research? Leave. When the "people" of earth finally get the message that brainpower is the limiting regent in life they'll soon start to change their tune - or find themselves living in a ~leper colony/africa.

This description is sufficiently general that every reader will have a different vision of construction-bots, opulence-pods, drones and the gleaming metropolis. This is why I would like a sane 3D library. Obviously, the way to start constructing this is to build a simulation for housing first, see how that works, generalize to hospitals, labs, self-driving living rooms, compare the simulation to the prototype houses being built in parallel etc. A simulation provides the opportunity to ask interesting questions: are voxels really the correct standard 3d abstraction? should we use equilateral triangles as the standard 2d plane? what are the structural integrity & cost trade-offs associated with different materials? etc. It seems to me that the current distinction between "an artist's rendering" and "engineering documents" is entirely due to the deficiencies of current programs. Designing should quite literally be writing Lisp code - sketch a general idea and begin to fill in the details, allowing the solution to take its shape from the curvatures of the problem space.

All existing 3D systems one could use to simulate such a idea are unacceptable. I don't feel the need to defend the statement that almost everything computer-related today is either undocumented or drowning in undocuments. I could learn, say, OpenGL or blender but doing so would prevent me from working on any of the other things that interest me purely because of the required time investment. I want expressive power, not just expression of this particular, also, I don't enjoy damaging my brain on their poor abstractions. While we're on the subject, what specifically is wrong with OpenGL? [Note: there are some people who have 'plans' to fix OpenGL, and they're all entirely unrealistic.[1]]

1. The documentation for OpenGL is either poorly written, non-existent or so outdated that it actually manages to destroy understanding. The red book claims to contain the information necessary to write "modern OpenGL" but is little more than a poorly written scam to drain desperate programmers of their precious time and money. Very few programs require more than docstrings and perhaps a single document containing the vocabulary necessary to discuss the conceptual territory. In any case, the common lisp hyperspec clearly demonstrates that large programs can be adequately documented online - no book required.

2. OpenGL is entirely dependent on the the underlying hardware, and there are numerous issues at this level of abstraction. In the 22 years since its inception no one has created an open source program to inform you as to which features are available on your computer, vendors regularly ignore bug reports, disregard the specification[3] and the hardware doesn't necessarily work as advertised[4].

3. GLSL. It has (some of) the syntax of C and none of the semantics, the language specification is a joke and as designed, there are serious performance issues.[5] There is no reason for it to exist and it needs to die.

4. The surrounding ecosystem is intellectually bankrupt. OpenGLUT, SDL and X may not be part of OpenGL, but they're necessary for using it. All are broken in various ways.[6] EG, when running some cl-opengl examples that make use of glut, if you press a key, with the glut window focused, it'll throw an error, and cause repaint errors (at the X level? screenshot: http://i.imgur.com/A2lY4zn.png) I realize that by abstracting over the debugging system of X, OpenGL and glut with the CL condition system, it's possible to have a sane development experience, but as far as i know no one has done this. The idiots who write SDL/GLUT/X etc. are perfectly happy to live with this defectiveness, but it wastes a massive amount of time of anyone who wishes to build even mildly sophisticated programs.

There are three strategies that immediately come to mind for attacking this problem:

1. Ignore 3D.

2. Abstract over OpenGL. Anyway this is done it is going to be hellish to due to the lack of documentation. The easiest way to approach this problem would be to exclusively target a set of hardware that get high enough OpenGL performance and is well supported by open source / reverse engineered drivers, SBCL & a few other CL compilers. If you want sane 3D development, purchase said hardware. A specific version of OpenGLX (a subset of opengl that ostensibly runs on many different chips) should be chosen as the reference implementation and completely abstracted over in a programmatic fashion using the XML versions of the specification as a basis.

https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/wgl.xml
https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/gl.xml
https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/glx.xml
https://cvs.khronos.org/svn/repos/ogl/trunk/doc/registry/public/api/readme.pdf

In the process the linux graphics stack should be cleaned up by tossing out any code that one can get rid of. See http://blog.mecheye.net/2012/06/the-linux-graphics-stack/ and http://libv.livejournal.com/22502.html for more information. Before anyone mentions it, wayland is not a solution to linux graphics woes, It addresses none of the above problems, doesn't address the problems that caused X to become such a disaster in the first place and is written by uninspired sheep. When the OpenGLX wrapper is completed, build a test farm that uses image recognition to verify that a particular codebase runs as intended. When this is completed one can begin supporting new hardware, OpenGL extensions, the remainder of OpenGL etc. A list of non-compliant chips / drivers should be published (publicly) and also mailed to the parties responsible on a rolling basis. A template like the following should suffice:

"Scientists, engineers and thinking men of all walks of life depend on your hardware / software to run mission critical systems. Version X of Y is broken in Z fashion. By producing such shoddy work you are directly preventing yourself from having sophisticated military, aerospace, healthcare systems etc. at your disposal. We urge you <Manager of K team> to fire everyone involved, open source your drivers / hardware designs and leave it in the hands of more competent people.

If you fail to do this, at the very least take the time to read the Common lisp Hyperspec and compare it against your published documentation. Then fix your docs... etc."

3. Write a sane 3D api that ignores GPU acceleration entirely. Such a codebase could be actually understood (as you're not hacking around completely undocumented hardware such as GPUs) and re-write the relevant bits  if hardware acceleration ever makes itself available. This will result in the most sane codebase as the abstractions will carry down to the metal. I don't know what sort of speed you could get by removing all of X, OpenGL and writing inline ASM, but suspect that it would be much faster than the pile of crud we've got today. Sure, you're not going to have shaders, lighting etc. and instead of textures you'll have stipple patterns/grayscale.

So, more specifically, what I need is the ability to have a single language at all levels of abstraction of my program (with an exception being made for inline assembler/microcode - extension languages are a terrible idea in any and all cases). I'd like for this to be something I'll use 150 years from now, therefore it must be of high quality. Communicating my ideas to other thinking people should be trivial - eg, I must be able to send arbitrary GPG-signed data structures across the network. The preceding requirements force the code base to be written in CL. Everything that isn't written in Lisp (and possibly forth & apl, language families I've not had a chance to spend enough time with - though I must say so far I prefer the former to the latter) predictably turns into incomprehensible monstrosities.[6]

Some more specifics, X is shit and must be replaced. I know that I'm not going to get shiny renderings anytime soon and don't care. Colored wireframes, stipple patterns and numbered faces are fine: expressive power, not crysis-style graphics.

Much of the implementation details of how to do XYZ necessary for getting rid of X and OpenGL is buried in mountains of C code. This needs to be addressed in a manner different than it has in the past (reading C). IMO, the only sane way is to perform queries on the ast of various C codebases until you can find the tables you're looking for. Yes, you're still reading Cee on some level, but I'd much prefer to have that programmatic power than not.

Fortunately someone has already written a complete C99 parser that can parse the Linux Kernel - the haskell package Language.C.AST[7] - Writing a program that round trips C source -> C AST -> pretty printed C source is absolutely trivial. It just needs a sexp printer hack and then I'll be able to work it into my CL toolchain.

[note: this post was originally a message to someone named |3b| on a irc channel]

As for |3b|'s comment: ah, "i saw a picture of it once and assumed it was solved", even better... "your field isn't even worth thinking about, so i'll dismiss it out of hand"

When I say that 3D is "solved" I mean that the things I desire could be done on Lisp machines in thee 80s/90s. Today I can't even get a 3D CLOS wire-frame that I can play around with in a presentation style format. I know you could do this back then because:

1. pictures / movies / one of Bill Gosper's cellular automata papers

2. I have talked to people who previously used LispMs and they've told me that I could do the basic things I desire. EG, 3D-wireframe plot in black and white that I can rotate on any axis, scale, place next to other plots, view source of etc.

Graphics in general will never be "solved", we will continue to build better algorithms and I find this exciting. I also find it exciting that Lisp is flexible enough that when I disagree with the design decisions someone else made, I can single-handledly redesign the codebase and take advantage of the work that they've done - without it being a particularly big deal. Knowing that the Symbolics code was written in CL means that even if it were not perfect (*gasp*) I know I could fix it up to do what I want. I can by no stretch of imagination do this with OpenGL.

Footnotes
================================================================================

[1]

(dolist (k '("http://www.joshbarczak.com/blog/?p=99" "http://richg42.blogspot.com/2014/05/things-that-drive-me-nuts-about-opengl.html" "http://richg42.blogspot.com/2014/06/how-i-learned-to-stop-worrying-and-love.html" "http://timothylottes.blogspot.se/2014/05/re-joshua-barczaks-opengl-is-broken.html" "http://www.joshbarczak.com/blog/?p=196")) (browse-url k))

[2] http://richg42.blogspot.com/2014/05/the-truth-on-opengl-driver-quality.html
[3] https://dolphin-emu.org/blog/2013/09/26/dolphin-emulator-and-opengl-drivers-hall-fameshame/
[4] http://www.joshbarczak.com/blog/?p=154 see reason #3
[5] how we got here. i read this book and found it informative and entertaining: http://richard.esplins.org/static/downloads/unix-haters-handbook.pdf
[6]

Case Study 1: Clang
===================

Clang is a compiler front-end for LLVM co-developed by Apple, Google, Sony, ARM and Intel that serves as an umbrella project for static analysis and tooling such as the C++ to AST parser. Suppose a programmer wishes to return the name and file location of all functions in a given program. the correct way to go about this is to walk the program's AST and return all nodes on the tree that are functions. The Clang codebase for manipulating C++'s abstract syntax tree weighs in at >100k LoC.

A roughly equivalent (common) Lisp program (the required amount of code will be similar in any Lisp dialect):

(with-open-file (stream "/root/quicklisp/local-projects/masamune/dashboard.lisp" :direction :input)
	   (let* ((out))
	     (remove-if-not (lambda (l) (eq 'defun (car l)))
			    (handler-case (loop while t do (push (read stream) out))
			      (error nil (nreverse out))))))

Please note that this is just one of the many tools in the Clang project. The remainder are equally bloated and unnecessary.

Case Study 2: Autotools
================================================================================

"The GNU build system, also known as the Autotools, is a suite of programming tools designed to assist in making source code packages portable to many Unix-like systems. ... widely used in many free software and open source packages."

-- http://en.wikipedia.org/wiki/GNU_build_system

compare autotools to ASDF (another system definition facility), the CL equivalent ([[http://www.dwheeler.com/sloccount/][data generated using David Wheeler's Sloccount]]).

| Codebase         | Estimated Development Cost in $ | Man Years | Lines of Code | # Programming Languages |
|------------------+---------------------------------+-----------+---------------+-------------------------|
| Autotools        | 5,248,975                       | 38.86     | 151,168       | 8                       |
| ASDF             | 242,245                         | 1.79      | 8,077         | 1                       |
| ASDF + Quicklisp | 378,768                         | 2.8       | 12,755        | 1                       |

someone will inevitably complain that Autotools has some feature or another ASDF doesn't, so I've added in the metrics for the CL package manager Quicklisp (hands down the best package manager available for any language). in addition to the eight languages making up Autotools, the project introduces at least one language of its own (makefiles). this ensures that no single human being understands it and that no one ever will. there are simply better things to do with one's time than think about how to fix a braindamaged build system written in 8 similarly braindamaged languages. Autotools is of course, required for building critically important software projects eg, freebsd, Emacs.

Case Study 3: the Web
================================================================================

The web was obsolete at its inception. There is no hypertext document format or stylesheets there are only programs. there is no cloud, there is only another's disk etc. The failure to use the terminology of computer science gives rise to plugins, extension languages, frameworks and other such definitionless monstrosities.

Various parties claim that they're going to 'fix the web'. To those innocent of the implementation details it's not clear this is impossible. in practice, unless the powers that be are willing to scrap everything that the web currently is and replace it with something sane, these projects are doomed to failure through obscurity, or by succeeding only in impressing idiots. Consider google's pnacl. Tfhe goal is to have a system that allows a programmer to intermix html, css and javascript with system level resources by compiling any language down to a subset of llvm bytecode. To produce anything on top of this tower of nonsense requires one to know the (moving) specifications for the 3 web languages + the language or languages being compiled down to llvm bytecode. Such a system is the antithesis of comprehensibility, and thus, progress. Last I checked, You couldn't so much as get error messages through pnacl and it's written in c++. Yes, I know they're "working on it" and oracle is still trying to work around the idiocy that is java and africa is still trying to work around the genetic/epigenetic/cultural relationship that leads the inhabitants to destroy anything resembling order. The philosophy "if we continue to add more code order and sophistication will somehow emerge from the chaos!" produces precisely the opposite of the intended result. Order and sophistication blossom from comprehensibility, not layers of obsolescent crud.

This position is diametrically opposed to that of ~every startup & corporation in silicon valley. I shall take a moment to illustrate an alternative to some of their madness. consider backend architectures, eg, code callable through an http/tcp interface. The way this is done today is by wiring (generic)functions on the backend to a url scheme invented on the fly. Successful 'apis' are wrapped by hand (string hacking!) and made available as client libraries because it's more convenient to call a function (which has semantics) in one's native language than make http requests by concatenating strings (devoid of semantics).

The correct way to go about this is to use rpc (http://en.wikipedia.org/wiki/Remote_procedure_call). A programmer wishes to make a codebase available over the internet. He gives a list of functions he'd like to make available to another program. This program walks the ast, retrieves the documentation and argument lists of said functions, wires them to a standard calling interface over http or tcp, and exports a tree (as s-expressions) notating this information. Any programmer can now walk the tree in any language they wish and transform it into a client library (the end game for any sufficiently useful or popular service). Layering rate limiting, encryption, public key authentication, a standard gui, etc. on this is straightforwards.

The slime codebase for Emacs embodies this basic functionality + many entirely unrelated goodies in 46,158 loc. how many millions of loc, man hours and dollars have been spent on building 'apis' in the past 3 years? How many millions on web browsers? Some will say that "the Lisp people never had anything like the web!" this is because "the web" is an idiotic idea and networking computers together so crass that no one thought it worth differentiating from everyday interaction with a computer (https://www.youtube.com/watch?v=7DxYj32cvoE).

I shall note that I've already written the whole basis for the rpc described previously i.e., walk this ast and spit out a client library + the working sketch of the rpc and le fecking gasp - I ran into a problem at the level of linux networking. Anyways, it's currently on pause until I've time to muck around with C networking nonsense.

[7] This is the best of the lot - I reviewed the first 5 pages of google results on a few searches that approximate to "I'd like the C AST please" this turned up several schemes, none of which were anywhere near as complete or sane. Clang is there, sure, but it is a mass of C++. I don't like haskell (so far anyways), but I prefer it to C++ any day of the week.