demon.slide

Demon
A tool for running demos
18:00 16 May 2018
Tags: golang, demo, websocket, http

Michael Lore
Principal Architect, SAP Concur
@ancientlore



* About Me

- Central Architecture Team at SAP Concur
- Designed the booking engine powering Concur Travel
- Playing with Go since 2010 (see [[https://github.com/ancientlore/go-avltree][go-avltree]])
- Interested in concurrent applications in the travel space
- ...which explains my interest in Go

.image media/gopher.png

[[https://twitter.com/ancientlore][@ancientlore]]
.link https://github.com/ancientlore/demon

.background media/plane_bg.png

: Hello.



* The Idea
.background media/plane_bg.png



* The problem with demos
.image media/demo_pain.png 530 _
.background media/plane_bg.png

: Switching windows.
: Copy/paste commands, to the right window.
: Typos, missing steps...
: More stress while giving demo.
: Also more confusing for the audience trying to follow the steps.



* A solution
.image media/demo_solved_notes.png 550 _
.background media/plane_bg.png

: Show web pages in iframes.
: Websocket access to external processes like bash or kubectl exec.
: Demo script with commmands included.
: Demo script knows where to send commands!
: Less stress! Easier to follow!



* Can it be mostly done in two days?

- JSON config file - very easy in Go
- Go's `os/exec` package can be used to launch processes
- Using [[https://github.com/gorilla/websocket][github.com/gorilla/websocket]] for websockets, we can send commands to processes and receive the output
- Use Go's templates to render the HTML site
- Find/steal some decent CSS
- Write some simple JavaScript - less fun than Go, but unavoidable

.caption Let's go for it!

.background media/plane_bg.png

: Go provides a lot of what we need to do this quickly.
: External packages provide the rest.



* Try out the os/exec package
.background media/plane_bg.png



* Simple command execution
.play examples/output/output.go
.background media/plane_bg.png
: os package can be used to execute processes and connect pipes.
: os/exec was designed to make process execution easier.
: One-liner gets the result of executing bash.



* Combining stdout and stderr
.play examples/combinedOutput/combinedOutput.go
.background media/plane_bg.png
: Using CombinedOutput we can capture stdout and stderr in the same stream.
: os/exec makes sure writes are synchronized.



* Using a Command with stdin/stdout
.play examples/command/command.go
.background media/plane_bg.png
: For our project, we need to deal with streams, so let's try out streams.
: You can assign cmd.Stdin, cmd.Stdout, and cmd.Stderr.
: If they are actual os files, they are passed over as such.
: Otherwise, a pipe is automatically created, including a goroutine to read or write.
: cmd.Run() executes the command using the streams assigned.



* Running a Command in the background
.play examples/background/background.go
.background media/plane_bg.png
: We need to be able to run the command in the background, not all at once.
: For this, we use cmd.Start.
: Once the I/O is done (in this example there isn't any), we call cmd.Wait.



* Using pipes
.play examples/pipe/pipe.go /func main/,/^}/
.background media/plane_bg.png
: You can create a pipe directly and write to it like any stream.
: The pipe is closed when the command exits.
: Note that some commands require the input to be closed before they will exit.



* Orchestrate bash - setup Command
.code -numbers examples/bbash/bbash.go /func main/,/launch/
.background media/plane_bg.png
: Create command and create pipes to use.
: Make Stderr and Stdout the same writer!
: Note: If Stdout and Stderr are the same writer, and have a type that can be compared with ==, at most one goroutine at a time will call Write.



* Orchestrate bash - I/O and Wait
.play -numbers examples/bbash/bbash.go /launch/,/^}/
.background media/plane_bg.png
: Write commands, closing the input when done.
: Read the output.
: Then (and only then) call cmd.Wait.
: You must do all your reads before calling cmd.Wait.



* Coding the utility
.background media/plane_bg.png



* Config File - sites and processes
.code demo.json /^\{/,/^\s*"steps"/
.background media/plane_bg.png

: Sites are URLs to load into an iframe.
: Processes are external programs to run (like a bash shell).
: Each has a key name in the map that can be used later.
: The position specifies the order the windows are loaded.



* Config File - steps
.code demo.json /^\s*"steps"/,/"Get some XML"/
.background media/plane_bg.png

: Steps are basically your demo script.
: Each step can optionally include an input command to send to one of the processes.
: The id specifies which process to send the command to.



* Loading the Config File
.code config.go /^\s*type config/,/c\.Validate/
.background media/plane_bg.png

: JSON tags provide hints to the JSON marshaler.
: The configuration is read from a file and the config type is unmarshaled from JSON.
: Validation checks that the configuration is consistent.



* The process type
.code config.go /^\s*type process/,/^\}/

- Start() error
- Wait() error
- Write(b []byte) (int, error) // io.Writer
- Read(b []byte) (n int, err error) // io.Reader

See [[https://golang.org/pkg/os/exec/][golang.org/pkg/os/exec]]
.background media/plane_bg.png

: The process type doesn't know about websockets.
: Was written during testing before websockets were added.
: It allows you to start and stop the process.
: Use Write (implementing io.Writer) to write to stdin.
: Use Read (implementing io.Reader) to read the combined stdout/stderr.


* Starting a process
.code process.go /exec.Command/,/cmd.Start/
.background media/plane_bg.png

: Create the command.
: Assign the directory to run the command in.
: Create input and output pipes. (Could perhaps be done just assigning Stdin and Stdout.)
: Make stderr use the same output pipe.
: Start the command.

* Ending a process
.code process.go /p.stdin != nil/,/return nil/
.background media/plane_bg.png

: Close the input, optionally sending an exit command. This is typically used to get the shell to stop.
: Wait on the wait group - this waits for the reader to exit on EOF.
: Want on the command to finish.



* Websockets - Upgrading the connection
.code socket.go /ServeHTTP/,/p\.Wait/
.background media/plane_bg.png

: Websocket code borrowed largely from https://github.com/gorilla/websocket/tree/master/examples/command.
: Connection must be upgraded to a websocket.
: Start the process.



* Processing the websocket
.code socket.go /stdoutDone/,/^\}/
.background media/plane_bg.png

: Use goroutines to process output and input from websockets.
: Websocket is closed during graceful server shutdown.



* Client-side websocket
.code home.go /window\["WebSocket"\]/,/\} else \{/
.background media/plane_bg.png

: Messages received are added to the DOM.



* Client-side websocket - writes
.code home.go /submitters\[k\] = function/,/\};/
.background media/plane_bg.png

: Messages received are added to the DOM.



* Serving the templates

.code home.go /<body>/,/\{\{end\}\}/

The handler:
.code home.go /var tpl/,/^\}/
.background media/plane_bg.png

: Using html/template.
: THere is an outer loop that allows the elements to show up where sorted.
: The loop is a little confusing (too clever) but works nicely.



* Finishing up
.background media/plane_bg.png



* Demo
.iframe http://localhost:8080 600 1000
.background media/plane_bg.png

: Don't forget to start the program...



* Thoughts

- Pretty easy to "scratch your own itch" using Go.
- gorilla/websocket handles SIGINT, I'd rather be able to control it.
- os/exec: having both `.Stdin` and `.StdinPipe()` is a bit confusing.
- There is a bit of mismatch between `os/exec` and `gorilla/websocket`, but their examples are good.
- CSS is messy; beg, borrow, or steal if you need to move quickly.
- You can do a lot with JavaScript even without a fancy framework.
- Might not work in every browser.

.background media/plane_bg.png

: Finish.