Skip to content

Commit

Permalink
PNPM workspaces (#663)
Browse files Browse the repository at this point in the history
* remove baseUrl/typeRoots

linting starts and then fails because tests can't resolve their types
packages

* WIP dtslint checks

I'm at the point of checking `version` and I think I need to copy code
from definition-parser maybe. Visions of semver parsing...

* check version in package.json

* update to ts@next to match DT

* Switch from no-outside-dependencies to import/no-extraneous-dependencies

It's already written!

Also make deepEqual traverse ojbects as well as arrays.

* Fix used-files calculation and improve error output

Show all unused files at once, not just the first unused one.

* allow all @types deps

* Mostly done switching to package.json dep/devDeps

Deletes a whole bunch of stuff and simplifies or reworks others.

I noticed that we still want to walk the file looking for globals SO I
have to put back in huge chunks of code I took out. Creating a commit so
I can diff on github.

* Restore code for finding global values in source

Everything builds now!

* definitions-parser tests pass

* *now* definitions-parser tests all pass

* Make generate-packages tests pass

* npm i -> pnpm i (compiles but hasn't run yet)

* Delete installDependencies entirely

It's only one line of code now

* Remove pathMapping check in definition-parser

Also all (?) the infrastructure needed to support it.

* slice multiple prefixes from resolved filenames

self-references sometimes result in doubled
'node_modules/@types/self-package-name' prefixes.

* add debug entrypoint for dtslint-runner

* Switch from index.d.ts header to package.json

Lots of code churn and probable bugs, but no real functionality change
right now. In the future this makes it much easier to mimic real ES
module layouts.

* Require either contributor url XOR githubUsername

Plus update typescript again to match DT

* Switch to reporting errors at end, not throwing one at a time

* 1. Remove npm consistency checks in dtslint-runner. They are unused.
2. Simplify and reduce JSON parsing in dtslint-runner. Pass around
AllPackages object instead of writing it to disk and reading it over and
over.
3. Sort AllPackages object before returning it in addition to before
writing it to disk.

* Switch to more semver

Still not 100% semver, but the common paths now use it.

* Package publication works now.

Only a few fixes needed this time!

* Fix merge conflict mistakes

* Test fixes

Still a couple of failures, but:

1. Fix bad merge in typescript-versions. (Re-update tests for 5.3)
2. Update patch version .0 -> .99999 everywhere.
3. In definitions-parser, stop formatting versions manually and use the
utility function. Which is correct, unlike the several different
mistakes I made elsewhere when formatting them.

* 1. Add two new external dependencies
2. Allow two more keys in package.json -- these are not yet checked!
3. Fix npm-naming test in dtslint -- another .0 -> .99999
4. Uncomment a throw that I had turned in a console.log for debugging.
This makes a test pass but might break actual runs; I haven't checked
yet.

* dprint

* fix lint

* Fix version of packages after parsing

Should be N.M.99999 not N.M.0
Probably.

* Minor fixes

1. allow 'path' in tsconfig, not in package.json
2. In dts-critic, throw when header-parser finds errors instead of
squelching.

* realpath when enumerating files

* Rename name-related fields (#752)

* Rename name-related fields

* Update related tests

* Type check all tests on pnpm build

* Fix remaining tests

* Fix typo

Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com>

* Delete unused function

* Don’t use pkg.desc for output directory

* My most common precedence mistake

* Fix contributor printing in README

---------

Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com>

* Include nonNpm in published package.json

* Pnpm workspaces: fix dtslint-runner incremental (#753)

* WIP: fix dtslint-runner incremental

Switch to `pnpm ls` to detect dependencies, not our own code

* Everything working

Lots of cleanup left though

* add another note

* Address PR comments:

1. change jakebailey/pnpm-workspaces-working back to master and make
settings constants for both.
2. Use semver.satisfies for finding packages that depend on deleted
packages.

* getAffectedPackages:array of errors, don't throw

Enables reporting multiple errors in one run.

* fix lint

* Cleanup and TODOs

1. Check pnpm overrides
2. Remove done TODOs, or long-term ones that are now recorded in #751
3. Clean up some unused code.
4. Rename some properties to be shorter.

* Move more things to header-parser

1. checkPackageJson*
2. getLicenseFromPackageJson

* Parse with semver

Instead of initially parsing ourselves and then parsing LATER with
semver.

* require devDeps to have self-ref w/workspace:.

* Move semver deps

From dtslint-runner and utils to header-parser

* Forbid references to old /v* packages

* make definitions-parser error handling more like header-parser

* new validation for tsconfig "paths"

* format files

* Remove lint + a couple of TODOs

* lint rule to forbid old header

* new lint: forbid devDep imports in .d.ts

* Allow self-reference in no-import-of-dev-dependencies

Also improve `pnpm install --filter` instructions in dtslint.

* forbid pinning on-DT @types/ packages

* update READMEs

* make dtslint example files more generic

* Edits from self-review of PR

* npmNaming: report all errors always

* dtslint-runner --clone SHA runs pnpm install once again

* fix direction of clone check

* Remove globals, make praser not crash on empty directories, remove dead code

* Remove obsolete test

* Finish removing globals stuff

* Package.json: change properties and patch version

1. Patch version is now .9999 not .99999
2. contributors -> owners
3. typeScriptVersion -> minimumTypeScriptVersion

Note that the publisher should still produce a package.json with
a contributor property.

* address PR comments

* Update dts-critic message to reference package.json instead of header comment

* dts-critic correctly passes typesVersions to header-parser

To correctly allow/disallow types/typesVersion in tsconfig

* allow 0.0 versions to mismatch npm again

* get-affected-packages: check for dependents by package name

* dtslint-runner: Simplify getAffectedPackagesFromDiff

It has a bunch of dead code!

* Add changeset for shipping

---------

Co-authored-by: Andrew Branch <andrew@wheream.io>
Co-authored-by: Andrew Branch <andrewbranch@users.noreply.github.com>
Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com>
  • Loading branch information
4 people authored Oct 17, 2023
1 parent e81ee12 commit 024c3e7
Show file tree
Hide file tree
Showing 111 changed files with 3,073 additions and 3,749 deletions.
14 changes: 14 additions & 0 deletions .changeset/new-items-explode.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
"@definitelytyped/typescript-versions": patch
"@definitelytyped/definitions-parser": patch
"@definitelytyped/dtslint-runner": patch
"@definitelytyped/eslint-plugin": patch
"@definitelytyped/header-parser": patch
"@definitelytyped/dts-critic": patch
"@definitelytyped/publisher": patch
"@definitelytyped/dtslint": patch
"@definitelytyped/retag": patch
"@definitelytyped/utils": patch
---

Update @definitelytyped for Definitely Typed's monorepo upgrade
46 changes: 41 additions & 5 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,47 @@
"version": "0.2.0",
"configurations": [
{
"type": "pwa-chrome",
"name": "dtslint-runner local all",
"program": "${workspaceFolder}/packages/dtslint-runner/dist/index.js",
"request": "launch",
"name": "Launch Chrome against localhost",
"url": "http://localhost:8080",
"webRoot": "${workspaceFolder}"
}
"skipFiles": [
"<node_internals>/**"
],
"args": [
"--path", "../DefinitelyTyped", "--selection", "all", "--localTypeScriptPath", "../../ts/built/local",
],
"type": "node"
},
{
"name": "jest",
"program": "${workspaceFolder}/node_modules/.bin/jest",
"request": "launch",
"skipFiles": [
"<node_internals>/**"
],
"args": [ "definitions-parser"],
"type": "node"
},
{
"name": "dts-critic test",
"program": "${workspaceFolder}/packages/dts-critic/dist/index.js",
"request": "launch",
"skipFiles": [
"<node_internals>/**"
],
"args": [ "--dts", "packages/dts-critic/testsource/tslib/index.d.ts", ],
"type": "node"
},
{
"name": "dtslint aframe",
"program": "${workspaceFolder}/packages/dtslint/dist/index.js",
"request": "launch",
"skipFiles": [
"<node_internals>/**"
],
"args": [ "../DefinitelyTyped/types/aframe", ],

"type": "node"
},
]
}
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
A monorepo for formerly disparate DefinitelyTyped-related tools:

- [definitions-parser](packages/definitions-parser): the part of [microsoft/types-publisher](https://github.com/microsoft/types-publisher) that reads DefinitelyTyped repository data
- [dtslint](packages/dtslint): [microsoft/dtslint](https://github.com/microsoft/dtslint)
- [dtslint-runner](packages/dtslint-runner): [DefinitelyTyped/dtslint-runner](https://github.com/DefinitelyTyped/dtslint-runner)
- [dts-critic](packages/dts-critic): [DefinitelyTyped/dts-critic](https://github.com/DefinitelyTyped/dts-critic)
- [header-parser](packages/header-parser): [microsoft/definitelytyped-header-parser](https://github.com/microsoft/definitelytyped-header-parser)
- [dtslint](packages/dtslint): [microsoft/dtslint](https://github.com/microsoft/dtslint), make sure a package is correct for DT
- [dtslint-runner](packages/dtslint-runner): [DefinitelyTyped/dtslint-runner](https://github.com/DefinitelyTyped/dtslint-runner), test all packages, or all changed packages, on DT with dtslint
- [eslint-plugin](packages/eslint-plugin): provides DT-specific lint rules (except for the few remaining tslint rules, still in dtslint)
- [dts-critic](packages/dts-critic): [DefinitelyTyped/dts-critic](https://github.com/DefinitelyTyped/dts-critic), issue errors when a types packages mismatches its original Javascript package.
- [header-parser](packages/header-parser): [microsoft/definitelytyped-header-parser](https://github.com/microsoft/definitelytyped-header-parser), check and extract DT-related info from package.json
- [publisher](packages/publisher): the rest of [microsoft/types-publisher](https://github.com/microsoft/types-publisher)
- [retag](packages/retag): [DefinitelyTyped/dt-retag](https://github.com/DefinitelyTyped/dt-retag)
- [retag](packages/retag): [DefinitelyTyped/dt-retag](https://github.com/DefinitelyTyped/dt-retag), script to add ATA tags to `@types` packages. Run weekly.
- [typescript-versions](packages/typescript-versions): the part of [definitelytyped-header-parser](https://github.com/microsoft/definitelytyped-header-parser) that tracked which TypeScript versions are published to npm and supported on DefinitelyTyped
- [utils](packages/utils): shared utilities, mostly extracted from [microsoft/types-publisher](https://github.com/microsoft/types-publisher)

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"@typescript-eslint/eslint-plugin": "^5.55.0",
"@typescript-eslint/parser": "^5.55.0",
"eslint": "^7.31.0",
"eslint-plugin-import": "^2.22.1",
"eslint-plugin-import": "^2.27.5",
"eslint-plugin-jsdoc": "^30.7.8",
"jest": "^29.5.0",
"prettier": "^2.6.2",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -605,6 +605,7 @@ bl
bookshelf
boxen
broadcast-channel
broccoli-node-api
broccoli-plugin
bson
buffer
Expand Down Expand Up @@ -724,6 +725,7 @@ log4js
logform
loglevel
logrocket
long
lottie-web
magic-string
markdownlint
Expand Down
169 changes: 71 additions & 98 deletions packages/definitions-parser/src/get-affected-packages.ts
Original file line number Diff line number Diff line change
@@ -1,109 +1,82 @@
import { mapDefined, mapIterable, sort } from "@definitelytyped/utils";
import {
TypingsData,
AllPackages,
PackageId,
PackageBase,
getMangledNameForScopedPackage,
formatDependencyVersion,
} from "./packages";

export interface Affected {
readonly changedPackages: readonly TypingsData[];
readonly dependentPackages: readonly TypingsData[];
allPackages: AllPackages;
import { assertDefined, execAndThrowErrors, mapDefined, withoutStart } from "@definitelytyped/utils";
import { sourceBranch, sourceRemote } from "./lib/settings";
import { AllPackages, PackageId, formatTypingVersion, getDependencyFromFile } from "./packages";
import { resolve } from "path";
import { satisfies } from "semver";
export interface PreparePackagesResult {
readonly packageNames: Set<string>;
readonly dependents: Set<string>;
}

/** Gets all packages that have changed on this branch, plus all packages affected by the change. */
export function getAffectedPackages(allPackages: AllPackages, changedPackageIds: PackageId[]): Affected {
const resolved = changedPackageIds.map((id) => allPackages.tryResolve(id));
// If a package doesn't exist, that's because it was deleted.
const changed = mapDefined(resolved, (id) => allPackages.tryGetTypingsData(id));
const dependent = mapIterable(collectDependers(resolved, getReverseDependencies(allPackages, resolved)), (p) =>
allPackages.getTypingsData(p)
);
return { changedPackages: changed, dependentPackages: sortPackages(dependent), allPackages };
}

/** Every package name in the original list, plus their dependencies (incl. dependencies' dependencies). */
export function allDependencies(allPackages: AllPackages, packages: Iterable<TypingsData>): TypingsData[] {
return sortPackages(transitiveClosure(packages, (pkg) => allPackages.allDependencyTypings(pkg)));
}

/** Collect all packages that depend on changed packages, and all that depend on those, etc. */
function collectDependers(
changedPackages: PackageId[],
reverseDependencies: Map<PackageId, Set<PackageId>>
): Set<PackageId> {
const dependers = transitiveClosure(changedPackages, (pkg) => reverseDependencies.get(pkg) || []);
// Don't include the original changed packages, just their dependers
for (const original of changedPackages) {
dependers.delete(original);
}
return dependers;
}

function sortPackages(packages: Iterable<TypingsData>): TypingsData[] {
return sort<TypingsData>(packages, PackageBase.compare); // tslint:disable-line no-unbound-method
}

function transitiveClosure<T>(initialItems: Iterable<T>, getRelatedItems: (item: T) => Iterable<T>): Set<T> {
const all = new Set<T>();
const workList: T[] = [];

function add(item: T): void {
if (!all.has(item)) {
all.add(item);
workList.push(item);
export async function getAffectedPackages(
allPackages: AllPackages,
deletions: PackageId[],
definitelyTypedPath: string
): Promise<PreparePackagesResult> {
const allDependents = [];
const filters = [`--filter '...[${sourceRemote}/${sourceBranch}]'`];
for (const d of deletions) {
for (const dep of allPackages.allTypings()) {
for (const [name, version] of dep.allPackageJsonDependencies()) {
if (
"@types/" + d.typesDirectoryName === name &&
(d.version === "*" || satisfies(formatTypingVersion(d.version), version))
) {
filters.push(`--filter '...${dep.name}'`);
break;
}
}
}
}

for (const item of initialItems) {
add(item);
}

while (workList.length) {
const item = workList.pop()!;
for (const newItem of getRelatedItems(item)) {
add(newItem);
}
const changedPackageNames = await execAndThrowErrors(
`pnpm ls -r --depth -1 --parseable --filter '[${sourceRemote}/${sourceBranch}]'`,
definitelyTypedPath
);
// Chunk into 100-package chunks because of CMD.COM's command-line length limit
for (let i = 0; i < filters.length; i += 100) {
allDependents.push(
await execAndThrowErrors(
`pnpm ls -r --depth -1 --parseable ${filters.slice(i, i + 100).join(" ")}`,
definitelyTypedPath
)
);
}

return all;
return getAffectedPackagesWorker(allPackages, changedPackageNames, allDependents, definitelyTypedPath);
}

/** Generate a map from a package to packages that depend on it. */
function getReverseDependencies(
/** This function is exported for testing, since it's determined entirely by its inputs. */
export function getAffectedPackagesWorker(
allPackages: AllPackages,
changedPackages: PackageId[]
): Map<PackageId, Set<PackageId>> {
const map = new Map<string, [PackageId, Set<PackageId>]>();
for (const changed of changedPackages) {
map.set(packageIdToKey(changed), [changed, new Set()]);
}
for (const typing of allPackages.allTypings()) {
if (!map.has(packageIdToKey(typing.id))) {
map.set(packageIdToKey(typing.id), [typing.id, new Set()]);
}
}
for (const typing of allPackages.allTypings()) {
for (const [name, version] of Object.entries(typing.dependencies)) {
const dependencies = map.get(packageIdToKey(allPackages.tryResolve({ name, version })));
if (dependencies) {
dependencies[1].add(typing.id);
}
}
for (const dependencyName of typing.testDependencies) {
const version = typing.pathMappings[dependencyName] || "*";
const dependencies = map.get(packageIdToKey(allPackages.tryResolve({ name: dependencyName, version })));
if (dependencies) {
dependencies[1].add(typing.id);
}
}
}
return new Map(map.values());
changedOutput: string,
dependentOutputs: string[],
definitelyTypedPath: string
): PreparePackagesResult {
const dt = resolve(definitelyTypedPath);
const changedDirs = mapDefined(changedOutput.split("\n"), getDirectoryName(dt));
const dependentDirs = mapDefined(dependentOutputs.join("\n").split("\n"), getDirectoryName(dt));
const packageNames = new Set(
changedDirs.map(
(c) =>
assertDefined(
allPackages.tryGetTypingsData(assertDefined(getDependencyFromFile(c + "/index.d.ts"), "bad path " + c))
).subDirectoryPath
)
);
const dependents = new Set(
dependentDirs
.map(
(d) =>
assertDefined(
allPackages.tryGetTypingsData(assertDefined(getDependencyFromFile(d + "/index.d.ts"), "bad path " + d)),
d + " package not found"
).subDirectoryPath
)
.filter((d) => !packageNames.has(d))
);
return { packageNames, dependents };
}

function packageIdToKey(pkg: PackageId): string {
return getMangledNameForScopedPackage(pkg.name) + "/v" + formatDependencyVersion(pkg.version);
function getDirectoryName(dt: string): (line: string) => string | undefined {
return (line) =>
line && line !== dt ? assertDefined(withoutStart(line, dt + "/"), line + " is missing prefix " + dt) : undefined;
}
Loading

0 comments on commit 024c3e7

Please sign in to comment.