Split parser functionality into its own package?

[GuySartorelli]

I'm writing a tool that helps me find all of the API breaking changes between two versions of a large, multi-repo project, and I've found that a lot of the parser tools this repository has (e.g. referencing a "project" at the top level, from which I can find reflection about various items) is immensely helpful.

I was wondering if you'd consider splitting the parser functionality out as its own package, which would then be a dependency of doctum?

[williamdes]

I was wondering if you'd consider splitting the parser functionality out as its own package, which would then be a dependency of doctum?

Maybe reach out to the phpdocumentor devs to have a repo in their org ?

I am not sure what is the end goal and if this is the right project for the task. As here it is mostly parse and display stuff

Let me know more

[GuySartorelli]

Maybe reach out to the phpdocumentor devs to have a repo in their org ?

The code I'm using is specifically in this package right now - I'm proposing it gets separated out into a separate package, whether that ends up being in a different GitHub organisation or not isn't really for me to decide or negotiate.

I am not sure what is the end goal and if this is the right project for the task. As here it is mostly parse and display stuff

My tool is using code that's currently in doctum for getting full class hierarchy API surface data across two different versions, which will then be compared in a separate step.

Basically, doctum has two steps:

1. collect data (aka parse())
2. render docs (aka render())

My tool will work similarly:

1. collect data (exact same data doctum needs, so makes sense to use your code rather than reinvent the wheel)
2. compare data (I'll have custom code for this relying heavily on your reflection classes)

Expand to see the code I'm playing with

#!/usr/bin/env php
<?php

use Doctum\Parser\ClassVisitor;
use Doctum\Parser\CodeParser;
use Doctum\Parser\DocBlockParser;
use Doctum\Parser\Filter\DefaultFilter;
use Doctum\Parser\NodeVisitor;
use Doctum\Parser\Parser;
use Doctum\Parser\ParserContext;
use Doctum\Parser\ProjectTraverser;
use Doctum\Project;
use Doctum\Store\JsonStore;
use Gsartorelli\DoctumPlayground\RecipeFinder;
use Gsartorelli\DoctumPlayground\RecipeVersionCollection;
use PhpParser\NodeTraverser;
use PhpParser\NodeVisitor\NameResolver;
use PhpParser\ParserFactory;
use PhpParser\PrettyPrinter\Standard as PrettyPrinter;

require_once __DIR__ . '/vendor/autoload.php';

$collection = new RecipeVersionCollection();
$store = new JsonStore();

$project = new Project(
    $store,
    $collection,
    [
        'build_dir' => '/home/gsartorelli/dump/temp/doctum-dump/%version%/',
        'cache_dir' => '/home/gsartorelli/dump/temp/doctum-cache/%version%/',
        'include_parent_data' => true,
    ]
);

$iterator = new RecipeFinder($collection);
$iterator
    ->files()
    ->name('*.php')
    ->exclude('thirdparty')
    ->exclude('examples')
    ->exclude('tests');

$parserContext = new ParserContext(new DefaultFilter(), new DocBlockParser(), new PrettyPrinter());

$traverser = new NodeTraverser();
$traverser->addVisitor(new NameResolver());
$traverser->addVisitor(new NodeVisitor($parserContext));

$codeParser = new CodeParser($parserContext, (new ParserFactory())->create(ParserFactory::PREFER_PHP7), $traverser); // @TODO use PHP version detection per recipe version

$visitors = [
    new ClassVisitor\InheritdocClassVisitor(),
    new ClassVisitor\MethodClassVisitor(),
    new ClassVisitor\PropertyClassVisitor($parserContext),
];

$projectTraverser = new ProjectTraverser($visitors);

$parser = new Parser($iterator, $store, $codeParser, $projectTraverser);

$project->setParser($parser);

$project->parse(); // @todo can use callback to output current step
// @TODO comparison step now we've parsed all relevant versions of all relevant packages

Edit: See https://github.com/GuySartorelli/silverstripe-deprecation-checker for the tool I'm making. Particularly the GenerateCommand::parseModules() method and the BreakingChangesComparer class.

[GuySartorelli]

One of the major benefits to splitting out the parsing functionality is it will be easier for downstream users like me to help resolve problems like #76 since the parsing functionality won't be directly tied to the documentation rendering logic anymore.

There are quite a few inconsistencies I've found while working on my tool and I'd be very keen to help iron those out.

[williamdes]

I think the parser can be moved into another package at code-lts. But it should be re-written from zero. The current code is an ocean of arrays, quite a mess. Are you willing to re-build a parser ?
See: #51

PS: sorry for not replying to all, it requires some thinking. But I will get to all of your contributions and post replies

[GuySartorelli]

I would prefer to refactor rather than rebuild from scratch. The whole reason I'm using your code in the first place is to avoid building this from scratch :p