* Noweb3 source code

Norman's first commit is the c/ directory, containing web files.
Began my reading at

  https://raw.githubusercontent.com/nrnrnr/noweb3/719d924b59ba3fa685af416c327dd897d5cb9d6e/c/readme.nw

The notangle program begins in c/main.nw. (Option -L writes #line
information to C source code.)  There is no noweave program on the
initial revision.

--8<---------------cut here---------------start------------->8---
Lol. ``This nuisance shows up because functions must be declared
before use.''  I also find that so annoying.
--8<---------------cut here---------------end--------------->8---

Files which receive source code are called ``modules.''  For instance,
if we have ``<<pairs.c>>='', then pairs.c is a module.

--8<---------------cut here---------------start------------->8---
What is the abstraction of a module?  A module is a linked list (see
modules.nw) of either lines to be printed or a reference to another
``module name''.  How is a file generated?  After all chunks are read
into a module linked list, the linked list is written to disk in the
following way: if the current linked list element is a string, it is
printed; if it is a module, the module is expanded (right away).
There are only these possibilities.
--8<---------------cut here---------------end--------------->8---

Fast-foward to

  https://github.com/nrnrnr/noweb3/commit/e6bd16cdc1d1c1f8efcb6431ac01d0ac4404aaa3

Back in 1993, stdio did not have getline, so let's rename Norman's getline.

--8<---------------cut here---------------start------------->8---
@@ -29,6 +29,7 @@ static buf_size = START_SIZE;         /* size of both buffers if non-NULL */
 char *getline_expand (FILE *fp);
   /* grab a line in buffer, return new buffer or NULL for eof
      tabs in line are expanded according to tabsize */
+#define getline noweb3_getline
 char *getline (FILE *fp);
   /* grab a line in the buffer, return a new buffer or NULL for eof
      no expansion of tabs */
--8<---------------cut here---------------end--------------->8---

The next problem is that now stdio has vfprint.  Let's remove it.

--8<---------------cut here---------------start------------->8---
%git diff errors.nw
diff --git a/c/errors.nw b/c/errors.nw
index f9dfc74..a2c9f3b 100644
--- a/c/errors.nw
+++ b/c/errors.nw
@@ -38,11 +38,13 @@ void errormsg(enum errorlevel level, char *s, ...);
 #define impossible(s) errormsg(Impossible, "This can't happen: %s", s)
 #define checkptr(P) do { if (!(P)) overflow("memory"); } while (0)
 <<C functions>>=
+/*
 extern int _doprnt(const char *format, va_list args, FILE *stream);

 int vfprintf(FILE *stream, const char *format, va_list arg) {
     return _doprnt(format,arg,stream);
 }
+*/

 void errormsg(enum errorlevel level, char *s,...) {
     va_list args;                       /* see K&R, page 174 */
%
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
make: *** No rule to make target 'markmain.nw', needed by 'markmain.o'.  Stop.
--8<---------------cut here---------------end--------------->8---

Norman's lack of attention. Perhaps markmain.nw will show up in the
next commits, but even by 1995, markup still can't be built because
there is no main function for it.  Let's move to 1997 where
lua-2.5+-nw was added.

We're at

  https://github.com/nrnrnr/noweb3/commit/3ecd8061d8824964d3bb9bd6895376b8cb188a9f

No sensible Makefile still.  No Makefile for lua-2.5.

How to build luac?

--8<---------------cut here---------------start------------->8---
%LDLIBS='../inout.o ../parser.o dump.o print.o ../table.o ../func.o
../fallback.o ../hash.o ../lex.o ../mem.o ../opcode.o ../tree.o
../undump.o' make luac
%file luac
luac: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for
GNU/Linux 3.2.0,
BuildID[sha1]=5ed65240bbdb7aeafc1d0a20fb912791f4b6893a, not stripped
%./luac
usage: luac [-dlpv] [-o output] file ...
--8<---------------cut here---------------end--------------->8---

How to compile the libs?

--8<---------------cut here---------------start------------->8---
%pwd
/home/dbastos/anyweb/noweb3/lua-2.5+nw/clients/lib

%gcc -c *.c -I../../include

%ls
iolib.c  iolib.o  mathlib.c  mathlib.o  old  strlib.c  strlib.o
%
--8<---------------cut here---------------end--------------->8---

How to create lua.a?

--8<---------------cut here---------------start------------->8---
%pwd
/home/dbastos/anyweb/noweb3/lua-2.5+nw/src
%ar r lua.a *.o
--8<---------------cut here---------------end--------------->8---

How to compile the Lua interpreter?

--8<---------------cut here---------------start------------->8---
%pwd
/home/dbastos/anyweb/noweb3/lua-2.5+nw/clients/lua

%gcc -c lua.c
%gcc -o lua lua.o ../lib/iolib.o ../lib/mathlib.o ../lib/strlib.o ../../src/lua.a -lm

%printf 'print("Hello world")' | ./lua
Hello world
%
--8<---------------cut here---------------end--------------->8---

How to execute compiled lua code?

--8<---------------cut here---------------start------------->8---
%pwd
/home/dbastos/anyweb/noweb3/lua-2.5+nw/src/luac
%

%cat hello.lua
print("Hello compiler.")

%./luac hello.lua
%

%pwd
/home/dbastos/anyweb/noweb3/lua-2.5+nw/clients/lua
%

%./lua ../../src/luac/hello.lua
Hello compiler.
%
--8<---------------cut here---------------end--------------->8---

Fully functional Lua.

Does 1997-noweb3 uses this lua compiler at all?  No.  So Norman added
lua-2.5 to 1997-noweb3, but doesn't use it.  Lua comes into use
in 1998.  We're now at

  https://github.com/nrnrnr/noweb3/commit/319cc82c549f6cc8c281cad158def70a820509f4

Norman's idea for noweb3 is, therefore, to write both notangle and
noweave in Lua.

How to produce these Lua programs?

--8<---------------cut here---------------start------------->8---
%pwd
/home/dbastos/anyweb/noweb3/lua
%notangle -Rnotangle.lua tangle.nw > notangle.lua
%notangle -Rnoweave.lua weave.nw > noweave.lua
--8<---------------cut here---------------end--------------->8---

We got a problem:

--8<---------------cut here---------------start------------->8---
%../lua-2.5+nw/clients/lua/lua notangle.lua notangle.lua
lua: syntax error;
> last token read: "a" at line 10 in file notangle.lua
Active Stack:
        `error' fallback
%

%nl notangle.lua | head -11
     1  markup = LIB .. "/markup"   -- not portable to dos
     2    -- should be changed to internal markup
     3  opt = {}
     4  arg = {}
     5  markopt = {}
     6  filters = {}

     7  while length(argv) > 0 do
     8    a = get(argv)
     9    globmatch a of
    10    | "-ml", "-m3", "-awk", "-icn", "-icon", "-pascal",
%
--8<---------------cut here---------------end--------------->8---

Let's move on.  This old code anyway.

We're now at

  https://github.com/nrnrnr/noweb3/commit/df9c016f871b8f82dd8feb2d36cd9f2067d977db

Lua code in place as can be seen by env-lua.nw.  That's the only one
in c/.  Fast-forward a few days, I see the following construction

<<*>>=
<<nullsink.c>>
<<header>>=
<<nullsink.h>>

That's a way to give chunks multiple names.  He defined nullsink.c.
If you don't specify any root chunk to notangle, then nullsink.c will
be output.  (This is useful.)

%notangle nullsink.nw
#include <stdio.h>
#include "nwbuffer.h"

int null_sink(void* out, NowebBuffer in) {
        FILE* fp = out;
        fprintf(fp, "Executing nullsink...\n");
        return 0;
}

If you specify nullsink.h, you get what you ask.  You get the same if
you say -Rheader.  So he's giving them multiple names.  Alright.

CII has appeared as early as

  736709ecb84cecf62ce7957a9ae38d74eb3f2e80.

Stages appeared as early as

  736709ecb84cecf62ce7957a9ae38d74eb3f2e80

We need to understand how that works.  Take a look at toascii.c.  It
looks like an implementation of a backend.  So a backend is internally
called a Sink.  The function toascii_sink() is producing the markup of
some NW file.  It gets a NowebReader and scans it pair by pair.  In
each pair, a token is found.

Each backend implementation is registered in c/stages.nw.

Lol.  The file xpipe.nw is added empty.  At

  https://raw.githubusercontent.com/nrnrnr/noweb3/544fe0b18517633c2376851907afbecd569c3cc9/c/xpipe.nw

we have a decent explanation of the XPipe idea.  XPipe_program()
creates a pipeline.  XPipe_connect() connects the stdout of LEFT to
the stdin of RIGHT, where LEFT and RIGHT are external UNIX programs.
Finally, XPipe_execute() puts it all in motion.

struct XPipe_Program {
  char* path;
  char** argv;
  char** envp;
};

struct XPipe_Stage {
  enum { Prog, Fun } tag;
  union {
    struct XPipe_Program prog;
    struct XPipe_Function fun;
  } u;
};

struct XPipe {
  List_T stages;
};

So you can see that XPipe_Stage is either an external program or an
internal function and XPipe is a list.

These interfaces --- so far --- are all platform-independent, but
XPipe_execute() is of concern for Windows support.

Notice XPipe_print() is a cool function.  It should display what the
pipeline looks like.

Most of the rest of the code is platform-dependent and so far there's
only UNIX support.  So, no, no chance for Windows in this commit.

What is nocond all about?  Nocond is a filter!

Looking at

  https://raw.githubusercontent.com/nrnrnr/noweb3/90b456d638423af80c87983df1f26f89ec1efd28/c/util.nw

we can see that the environment variable NWPATH sets the directory
where Lua scripts can be found.  If NWPATH is not set, current
directory seems to be a fallback.

Looking at

  https://raw.githubusercontent.com/nrnrnr/noweb3/1c49db3384fb006de46806b079189a144d2ff2fe/c/ipipe-lua.nw

I get the idea that a C program will initiate the work on the Lua
side.  So notangle and noweave (most likely noweave usually), will be
invoked from C but the work will be done by Lua scripts.  So C will
prepare things on the Lua side.  Prepare things looks like a UNIX
pipeline, composed of Lua scripts mostly.  That's what I guess right
now.  Take a look at apply_filter.  A function argument is pushed onto
the stack, buffers in and out are set, a function f is called and a
result of the function is extracted, returning to the C caller.

Looking at

  https://raw.githubusercontent.com/nrnrnr/noweb3/6eeca2640976ec01ec29c5f842d2c92f642c0fe3/lua/config.nw

we see the important noweb.cfg file.  That looks totally new relative
to noweb2.  It's a Lua script that loads pipes.cfg, sys.cfg, util.cfg,
list.nws, totex.nws.

Every pipeline script must be a UNIX executable program.

function Unix.isExecutable(file)
  return XPipe.run(XPipe.stage("test", {"-x", file})) == 0
end

That's a problem for Windows.  Norman is aware of that and prepares
some solution right away.

case PLATFORM of
| "unix" => MD = Unix
else => error("Unknown target platform ", PLATFORM)
end

Except for UNIX, all others are unknown platforms.  That's alright.

How does noweave work?  Here's an early version of a Lua implementation.

  https://raw.githubusercontent.com/nrnrnr/noweb3/78b8aa250d2c611fe9980f49e1ec3a8cd5ddc5b2/lua/weave.nw

What do we see?  A [M]ixed pipeline is built and eventually run.

P = MPipe
[...]
pipeline = P.connect(P.connect(src, filters), backend)
[...]
exit(P.run(pipeline))

So, noweave is completely implemented in Lua.  Why the C sources?
Again, I think what Norman is doing is writing in C his ``own
language''.  Given the portability of C, he can always take his
friendly Lua programming language with him everywhere he goes and
doesn't have to keep on writing C to extend and fix problems in noweb.
That's the way to go about it.  Noweave's UNIX manual page is there.

Looking at

  https://raw.githubusercontent.com/nrnrnr/noweb3/6cf04907ae4e3512dd87886a77e8b3686d0bd396/lua/test.lua

What is this?  This is a test file for Lua infrastructure.  He's
seeing whether his Lua code works.  A pipeline P is created.  As its
source, the markup program is defined to process the source totex.nw,
which is a noweb file.  A processor program totex will process
totex.nw as a source.  The pipeline is programmed by connecting the
source (f) to the processor (b = totex).  It is then run.

This is effectively what noweave will do, but taking orders from the
user on the command line.  But the effect is the same: that kind of
Lua infrastructure will be invoked and executed.

Norman is UNIX minded.  He has always been doing noweb{2,3} in the
UNIX pipeline way.  The difference in noweb3 relative to noweb2 is
that he's focusing on Lua instead of so many sh, awk, icon programs.

We've gone up to the year 2000 and still no markmain.nw file!  We
don't have the main function of markup.  Why do you do this to us,
Norman?  Lol.

Guess what?  At

  https://github.com/nrnrnr/noweb3/blob/811ec71e2d9b78163c97feb8e3b7f92d148b81b4/c/Makefile

he seems to have solved this problem, but surely introduced others.
For instance, nwprocess.nw is not included in the commit.  Why do you
do this to us, Norman?  Lol.

He gives us a new build system for liblua.a.  Let's see.

  https://github.com/nrnrnr/noweb3/commit/e9ff3f1038ec6c6028401e2602461799cd8cfa14

Not very useful at all.

Guess what?  At

  https://github.com/nrnrnr/noweb3/commit/07fe8e76a43b4193e27011f4e7652bbbfbd76758

he seems to have a real build system now.

It's still broken, but this looks like the first would-be-release.
Notice nwprocess.nw was added, finally.  Guess what?  Markup's main
function was added.  Just a few years later.  It's in markup-run.nw.

At

  https://github.com/nrnrnr/noweb3/commit/5f7948927817fad02704742f2035e5ad779418f5

we can build the guide.dvi and convert it to guide.pdf to read.  This
is the hacker's guide for noweb3.  Everything we said above will help
in the reading of the guide.  (It's interesting how, without the code,
the guide seems close to useless.  There's no royal road to code
reading.  Hardest path seems the best one.  Only then reading
documentation will help.)

I think we were right all along in these notes, even though we could
only run the first versions --- as there is no working build until
now.

What's next?  Let's spend some time hacking noweb3.  Notice ``no'' is
a Lua interpreter.

%./no
  print("Hello world")
Hello world
  %

It turns out, ``no'' can take at least two arguments, weave and
tangle.  Here's weave.

--8<---------------cut here---------------start------------->8---
%./no weave
hello
hello
<<*>>=
def pythonf():
  return 1
@ We are done.
\documentclass{article}\usepackage{noweb}\pagestyle{noweb}\noweboptions{}[...]
hello
\nwenddocs{}\nwbegincode{1}\moddef{*}\endmoddef\nwstartdeflinemarkup[...]
def pythonf():
  return 1
\nwendcode{}\nwbegindocs{2}We are done.
\nwenddocs{}\end{document}

%
--8<---------------cut here---------------end--------------->8---

We can learn a few things here.  The numbering of chunks and
documentation and other obvious stuff.  I'd like to see the markup.

We can probably write our own filter and invoke it with ``no'' because
``weave'' seems to be a filter too.  Here's how to look at the effect
of markup.  With the following filter

--8<---------------cut here---------------start------------->8---
%cat ../lib3/noweb3/test4
P = MPipe

p = P.source(Stages.markup, {})
p = p .. P.sink(Stages.toascii)
exit(P.run(p))
%
--8<---------------cut here---------------end--------------->8---

we get the following behavior.

--8<---------------cut here---------------start------------->8---
%./no test4
intro...
<<*>>=
root chunk, as usual..
@ back to doc
@file
@begin docs 0
@text intro...
@nl
@end docs 0
@begin code 1
@defn *
@nl
@text root chunk, as usual..
@nl
@end code 1
@begin docs 2
@text back to doc
@nl
@end docs 2
%
--8<---------------cut here---------------end--------------->8---

From this, we can infer: what markup does is to say @file, @begin docs
<integer>, @text ..., @nl, @end docs <integer>, @defn chunk, @nl,
... So markup is really the filter that does the hard job.  All others
will read this nicely formatted output and do something about it.  For
instance, weave might open verbatim mode when it finds @begin code
<integer> and close it when it finds @end code <integer>.  So the
plain filter is really Stages.toascii because it seems to do no
transformation to the input.  Let's use it to see the behavior of
other filters.

If you create a new filter, remember to load it in noweb.cfg.  You
need a ``do_nwfile("newfilter.nws")'' there.

So what is noweb3?  It is a lua interpreter with scripts included that
can implement literate programming Ramsey-style.  What do we want from
it?  We want to run it on Windows.  That is all, really.  All features
are already there for us.  We just can't compile it on Windows.  Even
with Cygwin?  Even with cygwin, which is very strange.

We couldn't compile on Cygwin because the compiler we were using was
made for the MinGW.  MinGW's GCC is really made for Windows, so it
doesn't come with header files such as sys/wait.h.  For true
portability, we'll need to write a portable wait() function, along
with WEXISTATUS.  See

  https://stackoverflow.com/questions/5487249/how-write-posix-waitpid-analog-for-windows

Cygwin's GCC, however, does come with such header files.  We installed
it and compiled noweb3.  Notice noweb3 does not come with noweb.sty.
(Thanks!)  I copied it from my noweb2 installation and was able to
produce documentation.  The program ``no tangle'' also works.

I don't think the option ``-autodefs c'' works, though.  So we cannot
produce documentation for C programs.  The problem seems to be the
lack of files such as autodefs.c in the installation.  We have a very
broken source code package, after all.  Maybe we can find such files
in the noweb2 installation.

I found them.  The problem, though, is that autodefs.c is a binary
ICON program.  How will we run it on Windows?

--8<---------------cut here---------------start------------->8---
%strings autodefs.c | head -10
#!/bin/sh
IXBIN=/usr/bin/iconx
IXLCL=`echo $0 | sed 's=[^/]*$=iconx='`
[ -n "$ICONX" ] && exec "$ICONX" $0 ${1+"$@"}
[ -x "$IXLCL" ] && exec "$IXLCL" $0 ${1+"$@"}
[ -x "$IXBIN" ] && exec "$IXBIN" $0 ${1+"$@"}
exec iconx $0 ${1+"$@"}
[executable Icon binary follows]

I9.0.00/32

%./autodefs.c
error in startup code
insufficient memory, corrupted icode file, or wrong platform
%
--8<---------------cut here---------------end--------------->8---

So we will have to compile these ICON programs on Windows too.
Compiling was not impossible, but running them successfully is the
question.  Check this out.

--8<---------------cut here---------------start------------->8---
%/usr/local/noweb/bin/no weave -autodefs c < crush.nw

%
--8<---------------cut here---------------end--------------->8---

No errors at all.  Thanks!  This behavior is due to the fact that
autodefs.c is being search for in lib2/ and I placed it in lib3/.
Putting autodefs.c in lib2/ produces now an error message.

%/usr/local/noweb/bin/no weave -autodefs c < crush.nw
Could not execute /usr/local/noweb/lib2/autodefs.c
%

%/usr/local/noweb/lib2/autodefs.c
-bash: /usr/local/noweb/lib2/autodefs.c: impossível executar o arquivo binário: Exec format error
%

Why does this happen?  Because autodefs.c is not an executable.  It is
just ICON compiled code.  (Not even ``file'' knows ICON.  This
language really didn't last very long.)

%file /usr/local/noweb/lib2/autodefs.c
/usr/local/noweb/lib2/autodefs.c: data
%

We will have to make it into an executable.  The trick is to make it
into a shell script that /execs/ ``iconx''.  See the noweb2
installation on UNIX to see how it is done.  This wasn't built
properly on Windows because, clearly, Windows' ICONT does not produce
the shell script that's built on UNIX.  It's a portability problem.

I turned it into an executable which calls iconx, but iconx doesn't
like the compiled code.

--8<---------------cut here---------------start------------->8---
%./autodefs.c
error in startup code
insufficient memory, corrupted icode file, or wrong platform
%
--8<---------------cut here---------------end--------------->8---

Yes, thanks for the or-like error message.  Now I need only check 3
causes and likely found neither one is correct.  Hypothesis: my
/iconx/ was built for native Windows and I'm running it on Cygwin.
Each version does a different thing, so we're living a nightmare.

Conclusion: although we can run noweb3 on Cygwin, we cannot run
autodefs.c, so we don't have a typesetting system for C programming,
unless we do mark the definitions ourselves, which isn't the end of
the world.  At least we have ``dpp -tex'' working properly.  We can
actually work on Windows by using cygwin and noweb3.

Now installing noweb3 on Windows still is a work to be done.  We can
package it up now, excluding all icon work.  This is in fact a
motivation for someone to write autodefs in Lua.  It should actually
be easy to do that because Lua is such a nice language and autodefs is
such an easy to write program.  Perhaps even I will write it
eventually.  Chapter closed.