-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathintro.html
362 lines (343 loc) · 21.7 KB
/
intro.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
<!DOCTYPE html>
<html lang="en-us">
<head>
<title>intro.nim</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2280%22>🐳</text></svg>">
<meta content="text/html; charset=utf-8" http-equiv="content-type">
<meta content="width=device-width, initial-scale=1" name="viewport">
<link rel='stylesheet' href='https://unpkg.com/normalize.css/'>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/kognise/water.css@latest/dist/light.min.css">
<link rel='stylesheet' href='https://cdn.jsdelivr.net/gh/pietroppeter/nimib/assets/atom-one-light.css'>
</head>
<body>
<header>
<div id="header-box">
<span id="home"><a href=".">🏡</a></span>
<span id="header-title"><code>intro.nim</code></span>
<span id="github"><a href="https://github.com/ajusa/binarylang-fun"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" width="1.2em" height="1.2em" style="vertical-align: middle;" preserveAspectRatio="xMidYMid meet" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59c.4.07.55-.17.55-.38c0-.19-.01-.82-.01-1.49c-2.01.37-2.53-.49-2.69-.94c-.09-.23-.48-.94-.82-1.13c-.28-.15-.68-.52-.01-.53c.63-.01 1.08.58 1.23.82c.72 1.21 1.87.87 2.33.66c.07-.52.28-.87.51-1.07c-1.78-.2-3.64-.89-3.64-3.95c0-.87.31-1.59.82-2.15c-.08-.2-.36-1.02.08-2.12c0 0 .67-.21 2.2.82c.64-.18 1.32-.27 2-.27c.68 0 1.36.09 2 .27c1.53-1.04 2.2-.82 2.2-.82c.44 1.1.16 1.92.08 2.12c.51.56.82 1.27.82 2.15c0 3.07-1.87 3.75-3.65 3.95c.29.25.54.73.54 1.48c0 1.07-.01 1.93-.01 2.2c0 .21.15.46.55.38A8.013 8.013 0 0 0 16 8c0-4.42-3.58-8-8-8z" fill="#000"></path></svg></a></span>
</div>
<style>
div#header-box {
display: flex;
align-items: center;
justify-content: space-between;
}
</style>
<hr>
</header>
<main>
<h2>Introduction to Binarylang</h2>
<p>So one day, I'm working on a project and I realize that I need to do some socket level HTTP
request sending and receiving. Since this is at the socket level, I can't really use the code that is
a part of the standard library as easily, since parsing HTTP stuff is baked into other functions.</p>
<p>So, I'm given a response that looks a little something like this:</p>
<pre><code class="nim hljs"><span class="hljs-keyword">echo</span> msg</code></pre>
<pre><samp>HTTP/1.1 404 Not Found
Date: Sun, 18 Oct 2012 10:36:20 GMT
Server: Apache/2.2.14 (Win32)
Content-Length: 10
Connection: Closed
Content-Type: text/html; charset=iso-8859-1
here is me</samp></pre>
<p>Now there are quite a few ways to parse this. Regex is probably a pretty good cross language
way of handling it, although then you'll have <a href="https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/">two problems instead of one.</a>
Nim's <code>scanf</code> module would also probably handle this pretty well, as would some sort
of generic parser using <code>parseutils</code> or <code>npeg</code>.</p>
<p>However, none of these tools really address an important issue: how do I serialize the object?</p>
<p>If I want to be able to both parse the HTTP header from a server, and create one as a client,
shouldn't that code be pretty much identical? Nim has strong meta-programming capabilities,
why can't I just define what my format looks like and have a parser/serializer generated from that?</p>
<p>Enter <a href="https://github.com/sealmove/binarylang">binarylang</a>.</p>
<p>Now, I can <em>declare</em> what my type looks like, and let it generate everything else for me. Let's try parsing just
the first line in that HTTP header, shall we?</p>
<pre><code class="nim hljs">struct(http):
s: _ = <span class="hljs-string">"HTTP/"</span>
s: version
s: _ = <span class="hljs-string">" "</span>
s: code
s: _ = <span class="hljs-string">" "</span>
s: msg
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
print toHTTP(msg)</code></pre>
<pre><samp>toHttp(msg)=Http(version:"1.1", code:"404", msg:"Not Found")</samp></pre>
<p>So what's going on here? First, we tell binarylang to go ahead and create a type called http, for parsing.
Next, we use it to define the format of a header. A header has a string, that starts with HTTP/, and then a version.
The version is also a string, so we prefix it with a <code>s</code>. Then, we look for a space to separate the two, and continue on
in a similar fashion. Here, everything we are parsing happens to behave like a string, so we can have all the types be string.
An underscore (_) simply signifies that we don't care enough about that value to name it. It is good for what are called
"magic" values, or to skip to the field you actually care about.</p>
<p>Since binarylang operates on bitstreams, we turn the string into one, and then tell it to parse into an object.</p>
<p>So, we have a functioning parser for the example header. What do we do if we wanted to generate a header though?</p>
<pre><code class="nim hljs"><span class="hljs-keyword">var</span> httpHeader = <span class="hljs-type">HTTP</span>(version: <span class="hljs-string">"1.1"</span>, code: <span class="hljs-string">"200"</span>, msg: <span class="hljs-string">"OK"</span>)
<span class="hljs-keyword">echo</span> httpHeader.fromHTTP</code></pre>
<pre><samp>HTTP/1.1 200 OK</samp></pre>
<p>Wow. It pretty much just works.</p>
<p>Next are the headers. While we could proceed as we did earlier, for each of the headers, it won't quite work.
HTTP headers can be in a different order, and more importantly, they can be <em>anything</em>. So, we need some way to parse
a sequence of headers. First, let's define a type for the header itself.</p>
<pre><code class="nim hljs">struct(header):
s: name
s: _ = <span class="hljs-string">": "</span>
s: value
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
print <span class="hljs-string">"Server: Apache/2.2.14 (Win32)</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>.toHeader</code></pre>
<pre><samp>toHeader("Server: Apache/2.2.14 (Win32)\n")=Header(name:"Server", value:"Apache/2.2.14 (Win32)")</samp></pre>
<p>Fantastic! We now have a way to parse a single header line. Of course, we need to handle a list of these
somehow. Thankfully, binarylang has us covered.</p>
<pre><code class="nim hljs">struct(http2):
s: _ = <span class="hljs-string">"HTTP/"</span>
s: version
s: _ = <span class="hljs-string">" "</span>
s: code
s: _ = <span class="hljs-string">" "</span>
s: msg
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
*header: {headers}
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
print msg.toHTTP2</code></pre>
<pre><samp>toHttp2(msg)=Http2(
version:"1.1",
code:"404",
msg:"Not Found",
headers:@[
Header(name:"Date", value:"Sun, 18 Oct 2012 10:36:20 GMT"),
Header(name:"Server", value:"Apache/2.2.14 (Win32)"),
Header(name:"Content-Length", value:"10"),
Header(name:"Connection", value:"Closed"),
Header(name:"Content-Type", value:"text/html; charset=iso-8859-1")
]
)</samp></pre>
<p>Hold on, what's going on here? What's up with all of the weird * and {}? Doesn't * mean a
public property in Nim?</p>
<p>The * can be used for two different things in binarylang. It can be used to either make a field public,
or to refer to an existing parser being used as a type. In this case, we can use it to refer to the header
type that we defined earlier. As for the <code>{headers}</code>, the curly braces denote "read into a seq until the next value can be parsed".
So, what happens is that we try to parse each header, and after parsing each one we see if the next thing on the stream is a
newline. If it is, we stop parsing headers and finish, otherwise we keep adding on to that seq. Since HTTP headers use newlines
to delimit the different sections, this works out fine.</p>
</main>
<footer>
<hr>
<span id="made">made with <a href="https://github.com/pietroppeter/nimib">nimib 🐳</a></span>
<button id="show" onclick="toggleSourceDisplay()">Show Source</button>
<section id="source">
<pre><code class="nim hljs"><span class="hljs-keyword">import</span> binarylang, sequtils, strutils, print, json, tables
printColors = <span class="hljs-literal">false</span>
<span class="hljs-keyword">import</span> nimib
nbInit
nbText: <span class="hljs-string">"""
## Introduction to Binarylang
So one day, I'm working on a project and I realize that I need to do some socket level HTTP
request sending and receiving. Since this is at the socket level, I can't really use the code that is
a part of the standard library as easily, since parsing HTTP stuff is baked into other functions.
So, I'm given a response that looks a little something like this:
"""</span>
<span class="hljs-keyword">var</span> msg = <span class="hljs-string">"""
HTTP/1.1 404 Not Found
Date: Sun, 18 Oct 2012 10:36:20 GMT
Server: Apache/2.2.14 (Win32)
Content-Length: 10
Connection: Closed
Content-Type: text/html; charset=iso-8859-1
here is me"""</span>
nbCode:
<span class="hljs-keyword">echo</span> msg
nbText: <span class="hljs-string">"""
Now there are quite a few ways to parse this. Regex is probably a pretty good cross language
way of handling it, although then you'll have [two problems instead of one.](https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/)
Nim's `scanf` module would also probably handle this pretty well, as would some sort
of generic parser using `parseutils` or `npeg`.
However, none of these tools really address an important issue: how do I serialize the object?
If I want to be able to both parse the HTTP header from a server, and create one as a client,
shouldn't that code be pretty much identical? Nim has strong meta-programming capabilities,
why can't I just define what my format looks like and have a parser/serializer generated from that?
Enter [binarylang](https://github.com/sealmove/binarylang).
Now, I can *declare* what my type looks like, and let it generate everything else for me. Let's try parsing just
the first line in that HTTP header, shall we?
"""</span>
nbCode:
struct(http):
s: _ = <span class="hljs-string">"HTTP/"</span>
s: version
s: _ = <span class="hljs-string">" "</span>
s: code
s: _ = <span class="hljs-string">" "</span>
s: msg
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
print toHTTP(msg)
nbText: <span class="hljs-string">"""
So what's going on here? First, we tell binarylang to go ahead and create a type called http, for parsing.
Next, we use it to define the format of a header. A header has a string, that starts with HTTP/, and then a version.
The version is also a string, so we prefix it with a `s`. Then, we look for a space to separate the two, and continue on
in a similar fashion. Here, everything we are parsing happens to behave like a string, so we can have all the types be string.
An underscore (_) simply signifies that we don't care enough about that value to name it. It is good for what are called
"magic" values, or to skip to the field you actually care about.
Since binarylang operates on bitstreams, we turn the string into one, and then tell it to parse into an object.
So, we have a functioning parser for the example header. What do we do if we wanted to generate a header though?
"""</span>
nbCode:
<span class="hljs-keyword">var</span> httpHeader = <span class="hljs-type">HTTP</span>(version: <span class="hljs-string">"1.1"</span>, code: <span class="hljs-string">"200"</span>, msg: <span class="hljs-string">"OK"</span>)
<span class="hljs-keyword">echo</span> httpHeader.fromHTTP
nbText: <span class="hljs-string">"""Wow. It pretty much just works."""</span>
nbText: <span class="hljs-string">"""
Next are the headers. While we could proceed as we did earlier, for each of the headers, it won't quite work.
HTTP headers can be in a different order, and more importantly, they can be *anything*. So, we need some way to parse
a sequence of headers. First, let's define a type for the header itself.
"""</span>
nbCode:
struct(header):
s: name
s: _ = <span class="hljs-string">": "</span>
s: value
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
print <span class="hljs-string">"Server: Apache/2.2.14 (Win32)</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>.toHeader
nbText: <span class="hljs-string">"""
Fantastic! We now have a way to parse a single header line. Of course, we need to handle a list of these
somehow. Thankfully, binarylang has us covered.
"""</span>
nbCode:
struct(http2):
s: _ = <span class="hljs-string">"HTTP/"</span>
s: version
s: _ = <span class="hljs-string">" "</span>
s: code
s: _ = <span class="hljs-string">" "</span>
s: msg
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
*header: {headers}
s: _ = <span class="hljs-string">"</span><span class="hljs-meta">\n</span><span class="hljs-string">"</span>
print msg.toHTTP2
nbText: <span class="hljs-string">"""
Hold on, what's going on here? What's up with all of the weird * and {}? Doesn't * mean a
public property in Nim?
The * can be used for two different things in binarylang. It can be used to either make a field public,
or to refer to an existing parser being used as a type. In this case, we can use it to refer to the header
type that we defined earlier. As for the `{headers}`, the curly braces denote "read into a seq until the next value can be parsed".
So, what happens is that we try to parse each header, and after parsing each one we see if the next thing on the stream is a
newline. If it is, we stop parsing headers and finish, otherwise we keep adding on to that seq. Since HTTP headers use newlines
to delimit the different sections, this works out fine.
"""</span>
<span class="hljs-comment"># However, working with a sequence of headers is a little strange. </span>
<span class="hljs-comment"># It would be nice if it could be accessed as if it were a table in Nim,</span>
<span class="hljs-comment"># with a string to string mapping. Thankfully, binarylang provides `@get` and `@set`, which define procs that can</span>
<span class="hljs-comment"># be ran on field access or assignment. First, let's define some helper procs to convert between the two.</span>
<span class="hljs-comment"># nbCode:</span>
<span class="hljs-comment"># type myTable = TableRef[string, string]</span>
<span class="hljs-comment"># template tableHeaderGet(parse, parsed, output: untyped) =</span>
<span class="hljs-comment"># parse</span>
<span class="hljs-comment"># var tmp = newTable[string, string]()</span>
<span class="hljs-comment"># for header in parsed:</span>
<span class="hljs-comment"># tmp[header.name] = header.value</span>
<span class="hljs-comment"># output = tmp</span>
<span class="hljs-comment"># template tableHeaderPut(encode, encoded, output: untyped) =</span>
<span class="hljs-comment"># var tmp: seq[Header]</span>
<span class="hljs-comment"># for name, value in encoded:</span>
<span class="hljs-comment"># tmp.add(Header(name: name, value: value))</span>
<span class="hljs-comment"># output = tmp</span>
<span class="hljs-comment"># encode</span>
<span class="hljs-comment"># # proc toTable(headers: seq[Header]): auto =</span>
<span class="hljs-comment"># # proc toHeaders(headers: TableRef[string, string]): seq[Header] =</span>
<span class="hljs-comment"># # for name, value in headers:</span>
<span class="hljs-comment"># # result.add(Header(name: name, value: value))</span>
<span class="hljs-comment"># # print msg.toHTTP2.headers.toTable</span>
<span class="hljs-comment"># # print {"Content-Length": "10"}.newTable.toHeaders</span>
<span class="hljs-comment"># nbText: """</span>
<span class="hljs-comment"># I'm using a `TableRef` here due to the `print` macro from [treeform](https://github.com/treeform/print/blob/master/tests/test.nim#L88)</span>
<span class="hljs-comment"># currently being broken on normal tables.</span>
<span class="hljs-comment">#</span>
<span class="hljs-comment"># Anyway, how do we hook this up to have a nicer interface? Now, our type looks like this:</span>
<span class="hljs-comment"># """</span>
<span class="hljs-comment"># nbCode:</span>
<span class="hljs-comment"># struct(http3):</span>
<span class="hljs-comment"># s: _ = "HTTP/"</span>
<span class="hljs-comment"># s: version</span>
<span class="hljs-comment"># s: _ = " "</span>
<span class="hljs-comment"># s: code</span>
<span class="hljs-comment"># s: _ = " "</span>
<span class="hljs-comment"># s: msg</span>
<span class="hljs-comment"># s: _ = "\n"</span>
<span class="hljs-comment"># *header {tableHeader[myTable]}: {headers}</span>
<span class="hljs-comment"># # *header {@get: _.toTable, @set: _.toHeaders}: {headers}</span>
<span class="hljs-comment"># s: _ = "\n"</span>
<span class="hljs-comment"># echo msg.toHTTP3.headers["Content-Length"]</span>
<span class="hljs-comment"># nbText: """</span>
<span class="hljs-comment"># That's pretty neat! All that's missing now is the actual, you know, content bit. We want to read a string</span>
<span class="hljs-comment"># that is as long as the content length. Here's what that looks like:</span>
<span class="hljs-comment"># """</span>
<span class="hljs-comment"># nbCode:</span>
<span class="hljs-comment"># struct(http4):</span>
<span class="hljs-comment"># s: _ = "HTTP/"</span>
<span class="hljs-comment"># s: version</span>
<span class="hljs-comment"># s: _ = " "</span>
<span class="hljs-comment"># s: code</span>
<span class="hljs-comment"># s: _ = " "</span>
<span class="hljs-comment"># s: msg</span>
<span class="hljs-comment"># s: _ = "\n"</span>
<span class="hljs-comment"># *header {@get: _.toTable, @set: _.toHeaders}: {headers}</span>
<span class="hljs-comment"># s: _ = "\n"</span>
<span class="hljs-comment"># s: content(headers["Content-Length"].parseInt)</span>
<span class="hljs-comment"># print msg.toHTTP4</span>
<span class="hljs-comment"># nbText: """We can now parse an entire HTTP response. Remember how at the start I said that we can also use</span>
<span class="hljs-comment"># this to serialize a full HTTP response? Let's try that.</span>
<span class="hljs-comment"># ```nim</span>
<span class="hljs-comment"># print HTTP4(version: "1.1", code: "200", msg: "OK", content: "Here I am!").fromHTTP4</span>
<span class="hljs-comment"># # Error: unhandled exception: key not found: Content-Length [KeyError]</span>
<span class="hljs-comment"># ```</span>
<span class="hljs-comment"># This doesn't quite work. We could provide the actual Content-Length manually, but that's</span>
<span class="hljs-comment"># a hassle, and somewhat error prone. Instead, we can try using `@hook` from binarylang, which</span>
<span class="hljs-comment"># runs code whenever a field is mutated.</span>
<span class="hljs-comment"># """</span>
<span class="hljs-comment"># nbCode:</span>
<span class="hljs-comment"># proc updateLength(headers: TableRef[string, string], length: string): auto =</span>
<span class="hljs-comment"># headers["Content-Length"] = $length.len</span>
<span class="hljs-comment"># return headers</span>
<span class="hljs-comment"># struct(http5):</span>
<span class="hljs-comment"># s: _ = "HTTP/"</span>
<span class="hljs-comment"># s: version</span>
<span class="hljs-comment"># s: _ = " "</span>
<span class="hljs-comment"># s: code</span>
<span class="hljs-comment"># s: _ = " "</span>
<span class="hljs-comment"># s: msg</span>
<span class="hljs-comment"># s: _ = "\n"</span>
<span class="hljs-comment"># *header {@get: _.toTable, @set: _.toHeaders}: {headers}</span>
<span class="hljs-comment"># s: _ = "\n"</span>
<span class="hljs-comment"># s {@hook: (headers = headers.updateLength(_))}: content(headers["Content-Length"].parseInt)</span>
<span class="hljs-comment"># var response = HTTP5(version: "1.1", code: "200", msg: "OK")</span>
<span class="hljs-comment"># response.content = "Here I am!"</span>
<span class="hljs-comment"># echo response.fromHTTP5</span>
nbSave
</code></pre>
</section>
<script>
function toggleSourceDisplay() {
var btn = document.getElementById("show")
var source = document.getElementById("source");
if (btn.innerHTML=="Show Source") {
btn.innerHTML = "Hide Source";
source.style.display = "block";
} else {
btn.innerHTML = "Show Source";
source.style.display = "none";
}
}
</script>
<style>
span#made {
font-size: 0.8rem;
}
button#show {
font-size: 0.8rem;
}
button#show {
float: right;
padding: 2px;
padding-right: 5px;
padding-left: 5px;
}
section#source {
display:none
}
</style>
</footer>
</body>
</html>