=== Table of contents ===

 - 1. [[#about : About LuaExec]]
  - 1.1. [[#overview : Overview]]
  - 1.2. [[#demarcation : What it does and doesn't]]
  - 1.3. [[#implementation : Implementation details]]
  - 1.4. [[#status : Status, outlook]]
  - 1.5. [[#license : License]]
  - 1.6. [[#links : Authors, links and downloads]]

 - 2. [[#buildinstall : Installation]]
  - 2.1. [[#requirements : Requirements]]
  - 2.2. [[#buildenv : Adjusting the build environment]]
  - 2.3. [[#builing : Building]]
  - 2.4. [[#installation : Installation]]
  - 2.5. [[#gendoc : Documentation system]]

 - 3. [[#examples : Examples]]
  - 3.1 [[#tests : Tests]]
  - 3.2 [[#httpserver : HTTP server]]

 - 4. [[#tutorial : Quickstart tutorial]]

=== Reference ===

 * [[manual.html#tek.lib.exec : LuaExec module reference]]
 * [[manual.html : Complete class and library reference manual]]
 * [[changes.html : Changelog]]
 * [[copyright.html : COPYRIGHT]]


---------------------------------------------------------------------

==( about : 1. About LuaExec )==

===( overview : 1.1 Overview )===

LuaExec is another take on multitasking for Lua. 

LuaExec is mainly focused on general applicability, simplicity and
concise semantics. It provides multiple threads of execution in the
most abstract sense, behaving understandably, platform-independently
and unencumbered by OS specific details.

===( demarcation : 1.2 What it does and doesn't )===

 * runs a Lua interpreter per task
 * is recursive, a child task can itself require the module, run
 child tasks, and see all other tasks
 * data exchange using messages, signals, arguments and return values
 * allows to synchronize on: task completion, signals, messages
 * abstract signals: a, t, c, m (abort, terminate, child, message)
 * tasks can find each other by name or by parent
 * no shared data, no marshalling, this is at the user's discretion
 * no operating system details (e.g. priorities)

===( implementation : 1.3 Implementation details )===

 * LuaExec is a regular module using only the legal Lua API.
 * It should work with Lua 5.1, 5.2, 5.3 on POSIX and Windows.
 Not all combinations are fully tested yet.
 * Child tasks are subject to garbage collection in their parent
 that created them. Their underlying OS threads are always joined
 and ''never killed forcibly'', and all resources requested in any
 task are meant to be returned, at the latest through garbage
 collection.
 * A Lua error in a task raises a signal, a (abort), that is by
 default propagated to the parent task.
 * Child tasks and tasks that spawned children are, by default,
 supplied with a debug hook that checks for the abortion signal,
 where it raises an error, signals its parent, and so forth.
 * The hook can be disabled on a per-task basis. In such tasks
 abortion strikes only when they are suspended in wait, sleep etc.

===( status : 1.4. Status, outlook )===

This project is a prototype and request for comments. Additions,
suggestions, improvements, bug fixes are likely needed and welcome.
Performance was no consideration yet. In its current stage LuaExec
is about getting semantics right. Some considerations:

 * Because LuaExec is recursive in that child tasks can create new
 tasks that can see other tasks (and can be seen by other tasks),
 LuaExec could easily provide a shared data or registry mechanism.
 This is currently being investigated. 

 * LuaExec's protocol and API should be transparent as to whether
 it is implemented as threads or processes, or as threads or
 processes on different hosts.
 
 * Support for serialization or marshalling

 * Given sufficient interest, this library might be extended in a way
 that other Lua modules and libraries can be written against
 LuaExec's C and Lua APIs, so that future Lua sockets and I/O,
 libraries and applications can converge and be synchronized on
 atomically in a platform-independent, Lua-ish way.
 
This project is a spin-off from the
[[tekUI project][http://tekui.neoscientists.org/]]. The LuaExec 0.80
module is fully contained in (but usable independently of) tekUI
version 1.11. The reason for outsourcing was to have a separate
testbed and package that is unencumbered by the requirements and
complexity of graphical input and output. There are a few examples
in this package that can be run against tekUI, and there are a few
more tasks examples in the tekUI package.

===( license : 1.5. License )===

LuaExec is free software under the same license as Lua itself: It can
be used for both academic and commercial purposes at no cost. It
qualifies as Open Source software, and its license is compatible with
the GPL – see [[copyright][copyright.html]].


===( links : 1.6. Authors, links and downloads )===

Authors, contributors:
 - Timm S. Müller <tmueller at schulze-mueller.de>
 - Tobias Schwinger <tschwinger at isonews2.com>

Open source project website:
 - http://luaexec.neoscientists.org/

Releases:
 - http://luaexec.neoscientists.org/releases/

Online source code repository:
 - http://hg.neoscientists.org/luaexec


---------------------------------------------------------------------

==( buildinstall : 2. Building and installing )==

===( requirements : 2.1. Requirements )===

 * Lua 5.1, 5.2, 5.3
 * GCC or LLVM/clang
 * MinGW for building for the Windows platform

LuaExec is supposed to work with LuaJIT 2.0, provided that LuaJIT
implements or can be made to implement {{lua_newstate()}} without
reporting an error. On x86_64 however this has been found to not be
the case. 

==== External Lua modules ====

LuaExec does not strictly depend on any external Lua modules. The
following modules are supported:

 * [[LuaFileSystem][http://www.keplerproject.org/luafilesystem/]]
 for the HTTP server example and documentation generator
 * [[LuaSocket][http://w3.impa.br/~diego/software/luasocket/]]
 for the HTTP server example
 
===( buildenv : 2.2. Adjusting the build environment )===

If building fails for you, you may have to adjust the build
environment, which is located in the {{config}} file on the topmost
directory level. Supported build tools are {{gmake}} (common under
Linux) and {{pmake}} (common under FreeBSD). More build options and
peculiarities are documented in the tekUI package, many of which
apply to the LuaExec package as well.

==== Windows notes ====

We recommend using the MSYS shell and MinGW under Windows, or to
cross-build for Windows using GCC or clang. An example configuration
is provided in {{etc/}}.

==== BSD notes ====

On FreeBSD and NetBSD you need a Lua binary which is linked with the
{{-pthread}} option, as LuaExec is using multithreaded code in shared
libraries, which are dynamically loaded by the interpreter.

===( building : 2.3. Building )===

To see all build targets, type

  # make help

The regular build procedure is invoked with

  # make all

===( installation : 2.4. Installation )===

A system-wide installation of LuaExec is not strictly required. Once
it is built, it can be worked with and developed against, as long as
you stay in the top-level directory of the distribution; all required
modules will be found if programs are started from there.

If staying in the top-level directory is not desirable, then LuaExec
must be installed globally. By default, the installation paths are

 - {{/usr/local/lib/lua/5.1}}
 - {{/usr/local/share/lua/5.1}}

It is not unlikely that this is different from what is common for
your operating system, distribution or development needs, so be sure
to adjust these paths in the {{config}} file. The installation is
conveniently invoked with

  # sudo make install
  
===( gendoc : 2.5 Documentation system )===

LuaExec comes with a documentation generator. It is capable of
generating a function reference and hierarchical class index from
specially crafted comments in the source code. To regenerate the
full documentation, invoke

  # make docs

Note that you need
[[LuaFileSystem][http://www.keplerproject.org/luafilesystem/]] for
the document generator to process the file system hierarchy.


---------------------------------------------------------------------

==( examples : 3. Examples )==

===( tests : 3.1 Tests )===

Examples from the tutorial, HTTP server, and tests are in {{bin/}}.

===( httpserver : 3.2 HTTP server example )===

One of the use cases LuaExec was developed against was to supply
a regular Lua application with an inbuilt HTTP server. It can be
run in the background like this:

  local exec = require "tek.lib.exec"
  local httpd = exec.run(function()
    require "tek.class.httpd":new {
      Listen = arg[1], DocumentRoot = "doc"
    }:run()
  end, arg[1] or "localhost:8080")
  -- ... do something else ...
  httpd:terminate() -- or httpd:join()

To start it serving LuaExec's documentation (default localhost:8080):

  # bin/webserver.lua [host:port]

It requires luasocket and lfs. The server itself is not
multithreaded, due to limitations of the current Lua library
ecosystem, but it supports copas/coxpcall if available. 

These links should work when you have loaded this document from the
provided HTTP server:

 * CGI test page: [[/webserver/testcgi.lua]]
 * Formtest Lua page: [[/webserver/formtest.lhtml]]
 * Webserver control Lua page: [[/webserver/control.lhtml]]

Please note that the HTTP/1.1 part of the server was not thoroughly
written against specifications. It is included as an example and may
get improved and become a separate package some day.


---------------------------------------------------------------------

==( tutorial : 4. Quickstart tutorial )==

For the first part of this tutorial, please run Lua in interactive
mode, because this is going to be fun, and hopefully instructive!
Start off by requiring the LuaExec module:

  # lua
  > exec = require "tek.lib.exec"

What's our task's name?

  > print(exec.getname())
  => main

Sleep a second (1000 milliseconds):

  > exec.sleep(1000)

Now to starting a new task:

  > exec.run(function() end)
  =>
  
Nothing happened? Well, it probably just ran and exited.
Let's have it do something:

  > exec.run(function() print "hello" end)
  => hello

Seems reasonable. Now let's have it do some work:

  > exec.run(function() while true do end end)
  
So far, so good. Do you notice some change in the behaviour of your
computer? Probably not. Ok, then try it again...

  > exec.run(function() while true do end end)

Still nothing? Then try this:
  
  > for i = 1, 4 do exec.run(function() while true do end end) end

Wait a moment. Some fans spinning up possibly?

I think it's time to quit interactive mode now rather quickly.
I hope your Lua interpreter has proper interactive mode, so that
it is orderly closed down and you see the intended effect.
Press CTRL-D now (not CTRL-C):

  > 
  luatask: received abort signal
  stack traceback:
          stdin:1: in function <stdin:1>
          [C]: ?
  luatask: received abort signal
  stack traceback:
          stdin:1: in function <stdin:1>
          [C]: ?
  luatask: received abort signal
  stack traceback:
          stdin:1: in function <stdin:1>
          [C]: ?
  luatask: received abort signal
  stack traceback:
          stdin:1: in function <stdin:1>
          [C]: ?
  luatask: received abort signal
  stack traceback:
          stdin:1: in function <stdin:1>
          [C]: ?
  luatask: received abort signal
  stack traceback:
          stdin:1: in function <stdin:1>
          [C]: ?
  #

Each of the tasks started above running an infinite loop should now
receive an abortion signal from Lua closing down and performing
garbage collection. This causes each task to raise an error, and
consequently to report that error. This is a very important
aspect of LuaExec. (And by dissipating heat we just may have helped
the universe to approach its destiny a little sooner, but this
remains to be seen.)

The next examples are performed in scripts. To reiterate what was
said above, run the following script. You'll get a stack trace from
the child, because it has not finished voluntarily when the main
program is ending:

  local exec = require "tek.lib.exec"
  exec.run(function() 
    while true do end
  end)
  => 
  luatask: received abort signal
  stack traceback:
          bin/tutorial.lua:3: in function <bin/tutorial.lua:2>
          [C]: in ?

On the other hand, if it has finished voluntarily, everything
remains silent:

  local exec = require "tek.lib.exec"
  exec.run(function() 
  end)
  exec.sleep(100)
  =>
  
I have added the last line to drive home that point, because your
funny operating system could theoretically decide to postpone
execution of the child task just long enough, so that we ended up
with an error and a stracktrace nonetheless. (Told in confidence,
this is not true, ''this'' example wouldn't raise a signal even without
sleeping in the parent, but this is an implementation detail.)

Note that although this is the plushy world of Lua, and LuaExec tries
its best to hide the nastiness of multithreaded programming from you,
trailing behind a bunch of dangling threads is by no means an
encouraged programming technique. What ''is'' encouraged is that you
synchronize your program on the completion of each task. This also
allows you to take delivery of return values:

  local exec = require "tek.lib.exec"
  local c = exec.run(function()
    return "successful", "indeed"
  end)
  print(c:join())
  =>
  true    successful      indeed

Next we should ask ourselves what threads are actually good for.
Especially in the Lua world this seems to be hotly debated -

In computations, actually, it is difficult to put them to good use,
but matters improved a bit in that regard, with projects like
SETI@home, mining bitcoins, and so on. There are plenty of CPU cycles
to be wasted in fun computations, especially if you run them in a
scripting language. Wouldn't it be a pity if we couldn't saturate
eight cores from a one-liner in Lua?

Irony aside, threads have always been useful for long-running tasks,
regardless of their computational demand, and they are even more
useful if they are bound to I/O operations that are precisely ''not''
demanding computationally, but stuttering, jerky, and unpredictable,
and spend most of their time waiting, but it is impossible to tell
for how long, and if the wait will even turn out a result. Also note
that from a computer's viewpoint a human could be seen as a very
demanding I/O device, making it a good candidate for a thread or two
to deal with, although that may not be the most popular take on that
matter.

So in I/O, that's where computing leaves academia and hits the hard
cobbles of reality:
Imagine you were writing a virus scanner with a graphical user
interface (you wouldn't want to write such a software, for all the
money in the world, I know! This is just an example) - would you
start a task each time the scan button is clicked? No, you would
start that task when the application starts, put it to sleep, and in
it wait for the order to scan, that is what you'd do.
Or rather, that's what I'd do, and I may be completely off the track,
because I would allow the user to enter the configuration panels and
other sections of the program even while such a scan is in progress.

First let's see how we can start and stop such an application worker:

  local exec = require "tek.lib.exec"
  local c = exec.run(function()
    local exec = require "tek.lib.exec"
    print "worker running"
    print(exec.wait())
    return "worker done"
  end)
  exec.sleep(1000)
  print(c:terminate())
  =>
  worker running
  t
  true    worker done

There is a new function to end the child task, terminate, and
another new function, wait, that by sheer chance waits until the main
program calls terminate on the child. What wait really does, however,
is that it waits until it receives any one or more signals from a set
of possible signals, and returns the ones received to the caller. The
signal '''t''' is included in the default set, and it is important
enough to be supported by a function on the other end, terminate, that
really does nothing but to send the '''t''' signal, and then to behave
exactly like join. So this would be equivalent to the last line:

  c:signal("t")
  print(c:join())

There are more signals, but the t signal is the most gentle one, with
the predefined meaning "pretty please, could you quit?". There is
nothing wrong if you received this signal and did nothing in reaction,
and nothing keeps you from assigning it a different meaning.

By now it has probably dawned upon Unix connoisseurs that
LuaExec signals have got '''nil''' to do with Unix signals, although
there are some artifical similarities in their names. The meanings
reserved to them are completely different.

Rowdies and bullies among you (who have an exceptional attention
span to be still following this tutorial) surely want to see the
brutal termination of a task. We can't help but to admit that this
needs to be mentioned, so pay attention, we will do this only once,
because we wouldn't want to further support your urge to destroy
things:

  local exec = require "tek.lib.exec"
  local c = exec.run(function()
    local exec = require "tek.lib.exec"
    for i = 1, 1000000000 do end
    exec.wait()
  end)
  exec.sleep(1000)
  c:abort()
  =>
  luatask: received abort signal
  stack traceback:
          bin/tutorial.lua:4: in function <bin/tutorial.lua:2>
          [C]: in ?

It doesn't matter what the child task is doing, as long as it is in
Lua code or a LuaExec function, it will be interrupted. Here again
the {{abort}} method is just the short form for sending a signal, in
this case '''a''' for abort, and then to {{join}}. But why would you
want to call abort at all? In exchange you are getting an ugly stack
trace that you somehow need to hide from your audience whom you
wanted to convince of using your program, and this is even done
automatically for you when your program ends and if there are still
tasks running. We leave it at that it is a rather exceptional thing
to do.

Signals are important, even essential, but their number is limited,
only four in fact, of which we know two already, and of which there
is only one free for the user, '''t'''. There could be more signals,
user signals 1, 2, 3, {{SIGFOO}}, you name them. But we asked
ourselves, why even stop there? No, we have kept the number of
signals at the bare minimum so that you won't have to ponder
needlessly on freaked out meanings that could be assigned to them.

Because, there is a tool that is so much more powerful, based on,
but transcending signals.
This tool is called messages. Messages can be any data, you can
reserve all meaning in the world to them, and with them, you can
even do without signals:

  local exec = require "tek.lib.exec"
  local c = exec.run(function()
    local exec = require "tek.lib.exec"
    print "your orders please!"
    while true do
      local order, sender = exec.waitmsg()
      print(order, sender)
      if order == "quit" then
        break
      end
    end
  end)
  exec.sleep(1000)
  c:sendmsg("scan")
  exec.sleep(1000)
  c:sendmsg("quit")
  c:join()
  =>
  your orders please!
  scan    main
  quit    main

You can send any string (even ones containing unprintable data)
around in messages, and not only that, you will also know who has
sent you a message, in this example the main program, {{"main"}}.

By receiving the sender together with the message you can write a
worker that is completely ignorant of the rest of the world, it just
takes the next order, executes it, and sends the result back to the
originator. Our first real worker will compute 5 random numbers
between 1 and 10:

  local exec = require "tek.lib.exec"
  local c = exec.run(function()
    local exec = require "tek.lib.exec"
    while true do
      local order, sender = exec.waitmsg()
      if order == "getrandom" then
        exec.sendmsg(sender, math.random(1, 10))
      elseif order == "quit" then
        break
      end
    end
  end)
  for i = 1, 5 do
    c:sendmsg("getrandom")
    print(exec.waitmsg())
  end
  c:sendmsg("quit")
  c:join()
  =>
  8       task: 0x7f4e60000af0    m
  3       task: 0x7f4e60000af0    m
  7       task: 0x7f4e60000af0    m
  7       task: 0x7f4e60000af0    m
  9       task: 0x7f4e60000af0    m

Some commentary is probably advised. The first column in the output
is the returned message body, the second the name of the sender, as
we already know from the previous example. Because here the sender
is the child task and the child task has no name, LuaExec has given
it a name for identification.

The third return value from waitmsg is the signal that caused waitmsg
to return, and by this we also get to know the third signal, '''m'''
for message, which indicates that, you guessed it, the task has
received a message. Please note, however, that it is entirely
possible that there is no signal that causes waitmsg to return (in
which case the third return value will be '''nil'''), because
returning the signal to the caller also clears it from the task, and
there may be more messages arriving in the meantime, in which case
waitmsg will return the next message immediately.

To conclude this introductory tutorial, didn't we say eight cores?

Well, of course none of these exercises in futility have got anything
to do with cores, but here's the same worker again (this time we have
given it a name), and ten other tasks requesting random numbers from
it:

  local exec = require "tek.lib.exec"
  local c = exec.run({ taskname = "worker", func = function()
    local exec = require "tek.lib.exec"
    while true do
      local order, sender = exec.waitmsg()
      if order == "getrandom" then
        exec.sendmsg(sender, math.random(1, 10))
      elseif order == "quit" then
        break
      end
    end
  end })
  for i = 1, 10 do
    exec.run(function()
      local exec = require "tek.lib.exec"
      for i = 1, 10 do
        exec.sendmsg("worker", "getrandom")
        print(exec.waitmsg())
      end
    end)
  end
  exec.sleep(1000)
  c:sendmsg("quit")
  c:join()
  =>
  1       worker  m
  10      worker  nil
  510     worker          workerm
  m
  8       worker  m
  ...

You may find the output a bit scrambed sometimes, like I do on my
computer. This is harmless and due to the fact that multiple tasks
are writing to standard output at the same time. The exact behaviour
may vary between different operating systems, but even if you don't
get a scrambled output on your computer, you may still get the point.

Of course we can still do worse than that. To expose this tutorial
to ridicule once and for all, in the final example the results will
be passed on to the main task, which prints them, so the main task
becomes the stdout worker, in a sense:

  local exec = require "tek.lib.exec"
  local c = exec.run({ taskname = "worker", func = function()
    local exec = require "tek.lib.exec"
    while true do
      local order, sender = exec.waitmsg()
      if order == "getrandom" then
        exec.sendmsg(sender, math.random(1, 10))
      elseif order == "quit" then
        break
      end
    end
  end })
  for i = 1, 10 do
    exec.run(function()
      local exec = require "tek.lib.exec"
      for i = 1, 10 do
        exec.sendmsg("worker", "getrandom")
        local number = exec.waitmsg()
        exec.sendmsg("*p", number)
      end
    end)
  end
  for i = 1, 100 do
    print(exec.waitmsg())
  end
  c:sendmsg("quit")
  c:join()

Besides prettier output, here we have another feature of LuaExec,
addressing a task by the name {{"*p"}}, which sends a message to a
task's parent task.


--------------------------------------------------------------------