Patgen platform

[ginty]

This PR adds the following:

• An AST processor platform suitable for handling both pattern and program generation
• Some simple example AST processors
• Plumbing for a JTAG driver
• Hook up of reg write/verify actions to controller write/verify_register handler methods
• A new block file type, services.py, for hooking up things like drivers and other services
• Changed the object returned by my_block.sub_blocks to be a regular Python dict

The long description:

AST Generation

All AST-related stuff lives in origen/src/generator and all processors live in origen/src/generator/processors. Expect in future this will expand to ../generator/processors/v93k, etc.

There is a singleton object defined by ../generator/test_manager.rs which is instantiated and available globally from origen::TEST and this is a representation of the current test being generated.
This wraps up a single AST instance to make it easy to push nodes from anywhere without having to worry about getting a mutable lock on it.

AST nodes are comprised of three main pieces of information:

• The node type itself carries semantic information, e.g. a reg write vs. verify (vs. any other operation) is identified by the node type being either RegWrite or RegVerify
• The node's attributes. These are specific to the node type, and in the case of a register operation carries things like a reference to the target register ID, the data being written, etc.
• The node's children which is a vector (array) of other nodes.

There is also provision made for meta data, e.g. to store source file and line number info but that is not currently hooked up to anything.

All nodes are defined in ../generator/ast.rb.
Program generator nodes and all ATE-specific pattern nodes should also go in here, so this will become a big file, but I think it will be useful when writing processors to have a single place to refer to the full library of possible nodes.

Nodes are created via the Node::new() method where the node type and attributes must be supplied.
For example, the cycle node is defined as:

Cycle(u32, bool), // repeat (0 not allowed), compressable

To create a node representing a single cycle and which can be compressed:

use crate::generator::ast::*;

let cyc = Node::new(Attrs::Cycle(1, true));

a macro is available to make this more succinct:

use crate::generator::ast::*;
use crate::node;

let cyc = node!(Cycle, 1, true);

Creating a node does not automatically add it to the current AST, to do that simply push it to the TEST manager:

origen::TEST.push(cyc);

Nodes can be cloned (which will also clone all of their children) and to push the same node multiple times it must be cloned:

for _i in 0..10 {
  origen::TEST.push(cyc.clone());
}

To add a child to a node:

my_parent_node.add_child(my_other_node);

Note that there is no differentiation between node types that can accept children and those that can't - they can all accept children. In practice though many nodes will be considered as terminal nodes and will be used as such, the cycle here being one such example.

In most cases children would not be added directly like this and would instead be added via the TEST manager API.

An additional method named push_and_open(<node>) is available which will add a node to the AST and then leave it open such that all future nodes will be pushed as children of this node until a corresponding TEST.close() is called.
To help catch coding errors where the user has forgotten to close a node, a reference is returned when leaving a node open and this same reference must be supplied when closing it.
This is best shown by example:

// Create a reg write node
let reg_write = node!(RegWrite, 10, 0x12345678_u32.into(), None, None);
// Now push it, leave it open and save the reference
let tid = TEST.push_and_open(reg_trans);
// These cycles will be added as children of the register transaction
let cyc = node!(Cycle, 1, true);
for _i in 0..5 {
    TEST.push(cyc.clone());
}
// Now close the transaction
TEST.close(tid).expect("Closed reg trans properly");

In the console the current test's AST can be viewed by calling:

_origen.test()

This is considered temporary and in future I expect this will be hooked up to something like tester.pattern.ast or similar.

AST Processing

The AST processing system is greatly inspired by the Ruby lib that was used for the O1 program generator. This remains a good and relevant reference on how the O2 processors work - http://whitequark.github.io/ast/AST/Processor/Mixin.html

Here is the boilerplate to create a new processor:

use crate::generator::ast::*;
use crate::generator::processor::*;

pub struct MyProcessor {
}

impl MyProcessor {
    pub fn run(node: &Node) -> Node {
        let mut p = MyProcessor { };
        node.process(&mut p).unwrap()
    }
}

impl Processor for MyProcessor {
}

The run function is the entry point to the processor, it takes a node as an input (an AST is just the top-most node in the tree) and returns a new node as the output - which is basically a representation of the input node which has been transformed in some way by the processor.
The run function should instantiate the processor and call the process method on the node, giving the (mutable) processor instance as an argument.

The run function is a convention rather than being a hard requirement and deviations are allowed where appropriate. For example, the function could return some result value rather than a new node.

Fields should be added to the processor's struct to handle any temporary storage that it needs.

All processors must implement the Processor trait. This trait provides default methods to handle all node types and so no methods are actually required.
By default the processor will iterate through all nodes and return an unmodified clone of them, therefore the processor above will return a clone of the input node.

To implement a handler for a particular node type define a function named after the underscored node type preceeded by on_, for example on_comment, on_cycle, on_reg_write, etc.

The arguments for the handler correspond to the node attributes and all handlers take the node itself as the last argument.
For example, the comment node type has the attribute signature:

Comment(u8, String), // level, msg

and here is a simple handler to upcase all comments:

impl Processor for MyProcessor {
    fn on_comment(&mut self, level: u8, msg: &str, node: &Node) -> Return {
        let new_node = node.replace_attrs(Attrs::Comment(level, msg.to_uppercase()));
        Return::Replace(new_node)
    }
}

Here is an example of a processor which will remove all comments below the level given to the processor:

use crate::generator::ast::*;
use crate::generator::processor::*;

pub struct CommentLevelProcessor {
    min_level: u8,
}

impl CommentLevelProcessor {
    pub fn run(node: &Node, min_level: u8) -> Node {
        let mut p = CommentLevelProcessor { min_level: min_level };
        node.process(&mut p).unwrap()
    }
}

impl Processor for CommentLevelProcessor {
    fn on_comment(&mut self, level: u8, msg: &str, node: &Node) -> Return {
        if level < self.min_level {
            Return::None
        } else {
            Return::Unmodified
        }
    }
}

All handler functions return a special processor::Return type which is defined in ../generator/processor.rs, which provides option like whether to remove the node, keep it unmodified, process its children (but otherwise keep it unmodified), or replace it with a new node or nodes created by the handler function.

There are also some special handler functions, on_all will be called for every node type which does not have a dedicated handler defined, and on_end_of_block will be called at the end of processing every node which has children.
It is expected that more special functions will be added over time as processors are developed.

See ../generator/processors for some more examples.

TEST AST Processing and Combining Processors

The main TEST AST is well wrapped up such that it is not possible to get a direct handle on it.
A special method is provided to process the AST which takes a closure function and the ast will be passed into this:

let new_ast = TEST.process(&|ast| CommentLevelProcessor::run(ast, 3))

Once you have the initially processed AST it is a simple matter to then make further transforms:

let new_ast = TEST.process(&|ast| CommentLevelProcessor::run(ast, 3))
let new_ast = UpcaseComments::run(new_ast);
let new_ast = CompressCycles::run(new_ast);
// new_ast now is the original AST with all comments below level 3 removed,
// all remaining comments uppercased, and all cycles compressed

Adding New Nodes

Defining a new node type requires it to be added to 3 places:

• To the Attrs enum in ../generator/ast.rs
• To the Node::process function immediately below the Attrs enum in ../generator/ast.rs
• A generic handler for the new node must be added to ../generator/processor.rs

It is expected that we will eventually end up with hundreds of node types so try and group related nodes together and use comment blocks for organization.

Note that the match block in the Node::process function will be compiled like a C switch statement, meaning that it will execute in constant time and will not slow down as more nodes are added. i.e. it will be implemented like a jump table in assembly and not like if/else logic which has to be evaluated in turn.

JTAG and Services

To provide something to generate a meaningful AST I hooked up the register actions to the controllers and added a simple register action handler in the controller of the example DUT:

# example/blocks/dut/controller.py
from origen.controller import TopLevel

class Controller(TopLevel):
    def write_register(self, reg_or_val, size=None, address=None, **kwargs):
        self.jtag.write_ir(0xF, size=8)
        self.jtag.write_dr(reg_or_val, size)

    def verify_register(self, reg_or_val, size=None, address=None, **kwargs):
        self.jtag.write_ir(0x1F, size=8)
        self.jtag.verify_dr(reg_or_val, size)

I realized that the JTAG driver couldn't be instantiated like a sub-block in O1 without significant internal changes since its not so easy to store different types in the one collection in Rust compared to Python or Ruby.
So I introduced a new block file type and collection called services:

#  example/blocks/dut/services.py
from origen.services import *

Service("jtag", JTAG())

Basically to define a service on a block (model) you define what you want it to be called and supply an instantiated object.
The above is just a placeholder and the JTAG contructor will probably accept pin arguments and so on in future.

A service implemented in Rust must implement a set_model() method as shown here for JTAG - https://github.com/Origen-SDK/o2/blob/patgen_platform/rust/pyapi/src/services.rs#L30
This is responsible for creating the service in Rust and attaching it to the parent model.

Python implemented service can implement the same method if they want to, but it is not required and any object can be given as a service (though currently there are no examples of this).

There is a stub of the JTAG driver implemented here which generates a few dummy cycles currently to make some example AST content - https://github.com/Origen-SDK/o2/blob/patgen_platform/rust/origen/src/services/jtag/service.rs

Note that the functions accept a Value type which is a newly added type which can be used to respresent sized data values that could be either a BitCollection or a dumb value.
It has the following signature:

pub enum Value<'a> {
    Bits(BitCollection<'a>, Option<u32>), // bits holding data, optional size
    Data(BigUint, u32),                   // value, size
}

Example AST

Here are a few console operations to show this in action:

stephen@falcon:~/Code/github/o2/example$ origen i
Origen 2.0.0-pre0
>>> _origen.test()
Test("ad-hoc")

>>> write(dut.areg0.set_data(0x1234))
>>> _origen.test()
Test("ad-hoc")
    RegWrite(2, BigUint { data: [52] }, Some(BigUint { data: [] }), None)
        JTAGWriteIR(8, BigUint { data: [15] }, None, None)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
        JTAGWriteDR(32, BigUint { data: [52] }, Some(BigUint { data: [] }), None)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)

>>> verify(dut.areg0.diff)
>>> _origen.test()
Test("ad-hoc")
    RegWrite(2, BigUint { data: [52] }, Some(BigUint { data: [] }), None)
        JTAGWriteIR(8, BigUint { data: [15] }, None, None)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
        JTAGWriteDR(32, BigUint { data: [52] }, Some(BigUint { data: [] }), None)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
    RegVerify(2, BigUint { data: [52] }, Some(BigUint { data: [32] }), Some(BigUint { data: [] }), Some(BigUint { data: [] }), None)
        JTAGWriteIR(8, BigUint { data: [31] }, None, None)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
        JTAGVerifyDR(32, BigUint { data: [52] }, Some(BigUint { data: [32] }), Some(BigUint { data: [] }), Some(BigUint { data: [] }), None)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)
            Cycle(1, true)

[ginty]

Hi @coreyeng, when writing this I realized that I forgot to add an API method to replace and insert nodes into the AST to support something like the runtime cycle compressor.
I'll add that over the next few days and either push here or open a new PR.

[ginty]

Probably also worth saying that this deviated quite a bit from the wiki proposal, mainly with regards to how the derivative ASTs are stored.
With this implementation only the master AST is stored in origen::TEST (and we will have a similar origen::FLOW in future I think) and no provision is made for static storage of processed ASTs.

I think that might be OK since once generation switches to the backend it shouldn't need to swtich back and forth between Python and Rust any more.
If it does need to be stored while making a trip back to Python-land then either it can be stored in some dedicated Rust-implemented Python class, or else we can add a static origen::ASTS vector where it could be temporarily pushed before going to Python and then pop'd back off on return.

[ginty]

Another thing missing is test support, we will need some helpers for creating test cases to verify that generators produce the correct output.
I'll add that soon too.

[coreyeng]

@ginty Thanks for putting this together! It does differ quite a bit from wiki but I think I'm still following what's going on.

Regarding the services, can a service implemented only in Python interact directly with the AST? For example, if we had an internal protocol similar to JTAG but implemented only in Python would there be a way to inject a new node into the AST? Or would it have to boil down to pin operations, which would disrupt any protocol-aware ATE-side generation? Or could we extend meta to handle something like this (I can try extending meta if we want to go this route)?

[ginty]

Hi @coreyeng, there's nothing currently that would allow a service or any other object in the Python domain to interact with the AST directly.

I think node creation should almost always be a side effect of calling an API like pin.drive, etc. but I can also see the case for a custom transaction node type per your scenario.
I propose that we have a generic transaction node type, similar to a jtag write/verify but with maybe an additional string field to allow its protocol to be identified.
Maybe just a String and then a few ints for the app to use as it sees fit:

Attrs::GenericTransaction(String, BigUint, Option<BigUint>, Option<BigUint>, Option<BigUint>)

Then we would have some (tester?) API method to create them:

ref = tester.start_transaction("my_trans_type", 0x1234, 0x1234, None, None)
tester.cycle
tester.cycle
tester.close_transaction(ref)

Could add some optional string fields too I suppose for extra flexibility.

I would say the bigger challenge is how to process such a node into a PA-aware output - I guess ultimately for something like that we would also be talking about a custom generator implemented in Python.

I think that means that we also need to be able to create AST processors in Python.
We could add a ToPython processor in pyapi/ and anytime a new node is added this would be a 4th place where it would have to be defined - to handle how it should be converted to Python.

That would allow the AST to go over the wall and after that it shouldn't be hard to add the ability to create processors in Python - the Ruby equivalent base processor is only about 20 lines or so.
Your tester should expose some low level methods to Python, like "tester.open_pattern_for_write()", "tester.print(line)", etc.

What do you think?

[coreyeng]

Hi @ginty, yes, I think a generic node like that would work fine. There definitely would be a custom generator involved, though now that I think more about it, the generator could key off of the regwrite and regread nodes instead and that might actually be the cleaner/safer thing to do, at least for use case I'm envisioning. Though, it may still worthwhile to have a generic node as a fallback for anything else that may come up. Then I think the processors would need to come along as you've described as well.

[ginty]

OK, I'll work on adding some of that, plus having the AST in python might also be useful for unit testing.

[ginty]

Hi @coreyeng,

I added some additional methods:

// Replace the current node at position n - 0 with the given node
TEST.replace(node, 0);

// Insert the given node at position n - 0
TEST.insert(node, 0);

// Get (a copy of) the node at position n - 0
TEST.get(0);

That should be enough to implement something like the cycle optimizer we talked about:

// If the last node was a cycle
if let Attrs::Cycle(repeat, compressable) = TEST.get(0)?.attrs {
    if compressable {
        TEST.replace(node!(Cycle, repeat + 1, true), 0)?;
    }
}

[coreyeng]

Looks good!

[ginty]

Hi @coreyeng,

OK, I added the ability to transfer the AST over to Python and to create processors in Python, you can get a handle on it from origen.test_ast().

I used the Serde crate to serialize the AST into a byte format called Pickle which seems to be Python's equivalent to Ruby's Marshal, and that was remarkably easy, that Serde crate is really good.
On the Python side I didn't change the AST format at all from what come out of Rust, so I think that means we can throw it back over the wall and de-serialize back into Rust if we want to, although I haven't tried or set that up yet.

The way you create a processor in Python is much the same as in Rust, the only real difference is that you just get the node as the argument passed into the handler methods rather than the de-composed attributes.

There are a couple of examples in these tests:
https://github.com/Origen-SDK/o2/blob/patgen_platform/example/tests/ast_processor_test.py#L18

So I think I'm done with this PR unless you can think of anything else that should be added at this stage?

If you are happy to merge approve when you are ready.

[coreyeng]

Looks good to me!