Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MAINTAINERS file + associated policy / tools #13602

Closed
edolstra opened this issue Mar 1, 2016 · 64 comments
Closed

Add MAINTAINERS file + associated policy / tools #13602

edolstra opened this issue Mar 1, 2016 · 64 comments
Assignees
Milestone

Comments

@edolstra
Copy link
Member

edolstra commented Mar 1, 2016

Following up on http://lists.science.uu.nl/pipermail/nix-dev/2016-February/019811.html.

To prevent the Nixpkgs from descending into chaos, we need some policy about who is allowed to commit/merge what. We already have meta.maintainers for packages, but not for other parts of Nixpkgs/NixOS. In particular it's completely unclear who "owns" central parts like lib/ so it risks becoming a free-for-all.

So I would like to propose the following:

  • Add a MAINTAINERS file (in the style of https://www.kernel.org/doc/linux/MAINTAINERS) listing who is responsible for particular topics / subsystems. This would generally only list things that cannot be expressed via the meta.maintainers field in packages or modules, such as cross-cutting concerns or general responsibility for a set of related packages like KDE. Example entry would be

    HASKELL INFRASTRUCTURE
    H: peti
    F: pkgs/top-level/haskell-packages.nix
    F: pkgs/development/compilers/ghc/*
    F: pkgs/development/haskell-modules/lib.nix
    

    where "H:" is a GitHub user name and "F:" are filename-matching regexps to help identify whether a particular change affects this subsystem (not required).

  • Add meta.maintainers fields to NixOS modules.

  • Document the policy (e.g. changes need approval from the responsible maintainers).

  • Make a little tool that given a diff tells you who the responsible maintainers are. It would do this by 1) matching the modified files in the diff against the "F:" regexps in the MAINTAINERS file; and 2) looking for a meta.maintainers field in the modified files. Also make a GitHub bot that applies this tool to incoming PRs and pings/assigns the maintainers.

Does this sound reasonable?

@edolstra edolstra added the 0.kind: enhancement Add something new label Mar 1, 2016
@edolstra edolstra self-assigned this Mar 1, 2016
@7c6f434c
Copy link
Member

7c6f434c commented Mar 1, 2016

   HASKELL INFRASTRUCTURE
   H: peti
   F: pkgs/top-level/haskell-packages.nix
   F: pkgs/development/compilers/ghc/*
   F: pkgs/development/haskell-modules/lib.nix

where "H:" is a GitHub user name and "F:" are filename-matching regexps to help identify whether a particular change affects this subsystem (not required).

Do I understand correctly, that H: line can also be repeated?

  • Add meta.maintainers fields to NixOS modules.
  • Document the policy (e.g. changes need approval from the responsible maintainers).
  • Make a little tool that given a diff tells you who the responsible maintainers are. It would do this by 1) matching the modified files in the diff against the "F:" regexps in the MAINTAINERS file; and 2) looking for a meta.maintainers field in the modified files. Also make a GitHub bot that applies this tool to incoming PRs and pings/assigns the maintainers.

Does this sound reasonable?

What do we do with PRs effectively creating new subareas?

What do we do with tools/misc and similar?

How can I mark a package «feel free to ping me for review/ask for
help/etc., but I also trust any willing NixPkgs committer to apply
a change that is not obvoiusly bad and keeps the thing building»
(this is meaningful for many small leaf packages, I think).

For small things the key (in my obviously wrong opinion) thing we lack
is even more manpower for keeping them up-to-date and not-bit-rotted;
for core and complicated things, stability matters, of course.

@heydojo
Copy link
Contributor

heydojo commented Mar 1, 2016

👍
100% agree @edolstra

@edolstra
Copy link
Member Author

edolstra commented Mar 1, 2016

@7c6f434c

  • Yes there could be multiple maintainers.
  • This would probably fall under a general "Nixpkgs scope" topic (presumably maintained by me).
  • Well, pkgs/tools/misc are just regular packages with meta.maintainers attributes, so no need to list them in MAINTAINERS.
  • The kernel's MAINTAINERS file has an "R: ..." field to list designated reviewers, which could be useful for this (and maybe an analogous meta.reviewers attributes). And we could have some way to mark a package that anybody can update (maybe simply something like meta.maintainers = [ community ]).

@jagajaga
Copy link
Member

jagajaga commented Mar 1, 2016

👍
But I thing that we should also have people who can coordinate others in PRs. Like nowadays I usually merge things myself but if I'm not sure about the patch or it's too complicated for me I call the responsible guys. (Also I have great experience with our repository and know well it's infrastructure that's why I know who can help us to solve problems or just help in a PR). Also nowadays @pSub do @joachifm do a great job with labeling pkgs.
And I thing we must leave several core people with current rights as they have them, like @peti @domenkozar @vcunat @abbradar @aszlig @pSub @7c6f434c @nckx @wizeman @edwtjo @ttuegel @bjornfor @jagajaga (myself) @shlevy @lethalman if they want to.

@7c6f434c
Copy link
Member

7c6f434c commented Mar 1, 2016

  • This would probably fall under a general "Nixpkgs scope" topic (presumably maintained by me).

Please assign some more trusted reviewers for such things, we need all
of you alive.

Adding things should have a lower risk of breaking everything,
hopefully…

  • Well, pkgs/tools/misc are just regular packages with meta.maintainers attributes, so no need to list them in MAINTAINERS.

Ah, so not all packages fall into such groups. Noted.

  • The kernel's MAINTAINERS file has an "R: ..." field to list designated reviewers, which could be useful for this (and maybe an analogous meta.reviewers attributes). And we could have some way to mark a package that anybody can update (maybe simply something like meta.maintainers = [ community ]).

meta.reviewers, probably — there are maintainers who can help with
debugging an update, but a working update can be reviewed by whoever
agrees (in contrast to some other areas where there are some people who
are good at debugging such things, but wouldn't accept the
responsibility of the final review).

@ttuegel
Copy link
Member

ttuegel commented Mar 1, 2016

👍

And I thing we must leave several core people with current rights as they have them, like @peti @domenkozar @vcunat @abbradar @aszlig @pSub @7c6f434c @nckx @wizeman @edwtjo @ttuegel @bjornfor and @jagajaga (myself) @shlevy @lethalman if they want to.

If I understand correctly, this is not about restricting the actual permissions of committers, but establishing policy and developing tools to notify the appropriate maintainers. At this point, I am comfortable trusting all the committers to follow an established policy; the only problem is that we don't as yet have one 😃

@jagajaga
Copy link
Member

jagajaga commented Mar 1, 2016

Oh! I misunderstood. I agree wholeheartedly with establishing policy!

@abbradar
Copy link
Member

abbradar commented Mar 1, 2016

Yay! I returned again and again to a thought "how would I specify that I (or someone else) is responsible for this directory tree / feature / function?". This would scratch my itch.

@7c6f434c
Copy link
Member

7c6f434c commented Mar 1, 2016

And I thing we must leave several core people with current rights as they have them, like @peti @domenkozar @vcunat @abbradar @aszlig @pSub @7c6f434c @nckx @wizeman @edwtjo @ttuegel @bjornfor and @jagajaga (myself) @shlevy @lethalman if they want to.

If I understand correctly, this is not about restricting the actual permissions of committers, but establishing policy and developing tools to notify the appropriate maintainers. At this point, I am comfortable trusting all the committers to follow an established policy; the only problem is that we don't as yet have one 😃

It could be nice to have brief recommendations for people with bigger
NixPkgs experience about gotchas for updating various libraries.

For example, a few years ago GNU TLS updates loved breaking libsoup and
WebKit-using browsers using libsoup… I guess there are some such things
now and when updating (say) LibreOffice and needing a newer version of
some dependency knowing what to check would make things go smoother.

@lucabrunox
Copy link
Contributor

Different proposal

I think linux long flat file is nice, but maybe there's something better we can think nowadays, and different because of our nixpkgs layout. Also, consider that regexp are not much of any use in our tree because most of stuff is a single file in a directory.

  • Put a MAINT.md in a subtree. The first line of the file looks like: "Maintainer: peti". The second line of the file looks like: "Subtree: Haskell Infrastructure", or something like that.
  • In the MAINT.md you can add other stuff that briefly document how that subtree works in terms of maintenance. It might help other people jump on maintaining that subtree as well, or describe some maintenance design decisions that do not belong to any of our current manuals.
  • The little tool would achieve point 1) of the @edolstra proposal by going upwards in the tree until a MAINT.md is found, and returning the first line of the file.

I think this will also make us think about subtrees differently: more maintenance-wise than category-wise. Like, why do we have pkgs/development/compilers/ghc and pkgs/development/haskell-modules ? Maybe we should just forget about "compilers" category being useful to anybody, and put everything under pkgs/lang-support/haskell for example.

EDIT: slightly changed the header of the MAINT.md, but that's only a detail of this proposal.

@maurer
Copy link
Member

maurer commented Mar 1, 2016

@lethalman
Won't the hierarchical MAINT.md method have issues with e.g. pkgs/top-level/haskell-packages.nix being delegated to the appropriate maintainers? Additionally, you end up with two MAINT.md files for a logical subtree in the case of the gnome3 subtree or similar ones which simultaneously have files in nixos/modules and in pkgs.

Re: the compilers category, it might be a useful designation for packages which are known to gate many other packages and whose build failure should be taken more seriously when cutting releases.

The overall oddity of the MAINT.md approach seems to be that it expects the directory structure of nixpkgs to match the maintenance structure. I suppose this is possible, but it's not immediately clear to me that this is ideal.

I agree that having the description of the subtree procedures sounds useful, as does not having everything in one monster file, just trying to note some potential complications to address beforehand.

P.S.: I'm mostly a user, not a bigshot maintainer, so these are just my two cents, sorry if I'm bothering you folks.

@lucabrunox
Copy link
Contributor

@maurer I don't see any issue in creating more MAINT.md with the same maintainer. It's just your github username. About nixos and pkgs, it may or may not be a problem. You can have two different descriptions. At least for gnome, the two descriptions would be very different. Also it does not expect the tree to match the maintenance structure, it only induces to have a more maintenance-wise structure.

@benley
Copy link
Member

benley commented Mar 2, 2016

There is a pretty decent semi-standard file format for exactly what we're trying to accomplish here:
https://github.com/bkeepers/OWNERS

It's used in chromium's git repos:
https://www.chromium.org/developers/owners-files
https://chromium.googlesource.com/chromium/tools/depot_tools/+/master/owners.py

Another related implementation, as a Gerrit plugin:
https://gerrit.googlesource.com/plugins/owners

More related tooling:
https://github.com/Nextdoor/git-change#owners-scope

Some commentary about origins and reasoning:
bkeepers/OWNERS#1 (comment)

The net effect is comparable to the linux MAINTAINERS file, but this can be distributed throughout a codebase like @maurer's MAINT.md proposal.

@dezgeg
Copy link
Contributor

dezgeg commented Mar 2, 2016

I don't think per-subdirectory files would work well in this specific case which was about cross-cutting concerns, which kind of tend to touch files all across the directory tree. For instance I'd be thinking of something like this for myself:

ARM AND OTHER ARCHITECTURE SUPPORT
F: pkgs/misc/uboot
F: pkgs/top-level/platforms.nix
F: pkgs/stdenv/linux/make-bootstrap-tools-cross.nix
F: nixos/modules/installer/cd-dvd/sd-image*.nix
F: nixos/modules/system/boot/loader/generic-extlinux-compatible
F: nixos/modules/system/boot/loader/raspberrypi

...which is already all over the place and directories like pkgs/top-level contain files that other people maintain.

Also if we use the MAINTAINERS format from kernel, we can also steal their script (https://github.com/torvalds/linux/blob/master/scripts/get_maintainer.pl) which does other kinds of useful stuff already (like ignore files and using git blame to find additional reviewers like Mention Bot)

@bendlas
Copy link
Contributor

bendlas commented Mar 2, 2016

Can somebody give perspective on OWNERS vs MAINTAINERS differences, with respect to encodable information vs tooling? I have seen both things but don't know the first thing about them. bkeepers/OWNERS#1 (comment) seems to suggest that OWNERS is more geared towards machine-readability.
Their usage seems to be about equal:


MAINTAINERS.md: 1,859
https://github.com/search?utf8=%E2%9C%93&q=filename%3AMAINTAINERS.md&type=Code&ref=searchresults

MAINTAINERS: 219,266
https://github.com/search?utf8=%E2%9C%93&q=filename%3AMAINTAINERS&type=Code&ref=searchresults


OWNERS.md: 323
https://github.com/search?utf8=%E2%9C%93&q=filename%3AOWNERS.md&type=Code&ref=searchresults

OWNERS: 224,132
https://github.com/search?utf8=%E2%9C%93&q=filename%3AOWNERS&type=Code&ref=searchresults

@edolstra
Copy link
Member Author

edolstra commented Mar 2, 2016

@lethalman What @maurer said. Some topics don't correspond to subtrees, in fact some may not correspond to any specific files at all. (E.g., "Nixpkgs coding style", "New packages / modules", ...)

@lucabrunox
Copy link
Contributor

@edolstra I wonder what means being maintainer of nixpkgs coding style.

@edolstra
Copy link
Member Author

edolstra commented Mar 2, 2016

Regarding existing tooling, we'll probably have to write/adapt our own anyway because it will need to be able to handle meta.maintainers in addition to MAINTAINERS/OWNERS.

@matthiasbeyer
Copy link
Contributor

HASKELL INFRASTRUCTURE
H: peti
F: pkgs/top-level/haskell-packages.nix
F: pkgs/development/compilers/ghc/*
F: pkgs/development/haskell-modules/lib.nix

I like the general idea, though I wouldn't bind that file to the github infrastructure so closely, but allow email entries like so:

HASKELL INFRASTRUCTURE
H: Github-User-Name
M: github_user_name@some_mail.ext
F: pkgs/top-level/haskell-packages.nix
F: pkgs/development/compilers/ghc/*
F: pkgs/development/haskell-modules/lib.nix

So we can also be reached by non-github users via email.

@7c6f434c
Copy link
Member

7c6f434c commented Mar 2, 2016

I like the general idea, though I wouldn't bind that file to the github infrastructure so closely, but allow email entries like so:

HASKELL INFRASTRUCTURE
H: Github-User-Name
M: github_user_name@some_mail.ext
F: pkgs/top-level/haskell-packages.nix
F: pkgs/development/compilers/ghc/*
F: pkgs/development/haskell-modules/lib.nix

So we can also be reached by non-github users via email.

Add the name of maintainers.nix entry then. And display name.

@domenkozar
Copy link
Member

Wouldn't it be better to add a list of paths maintained to lib/maintainers.nix or some other file so it can reference the maintainer name and email? Otherwise we have to maintain two list of maintainers.

@jagajaga
Copy link
Member

jagajaga commented Mar 2, 2016

@matthiasbeyer

M: github_user_name@some_mail.ext

Why github_user_name?

@matthiasbeyer
Copy link
Contributor

@jagajaga This was just an example. Of course it would be @matthiasbeyer and mail at beyermatthias [dot] de for me.

@joachifm
Copy link
Contributor

joachifm commented Mar 2, 2016

To my mind, the assignment of ownership is what makes this proposal valuable. That there is ownership is more important than who owns what or how it is recorded. I hope this doesn't get bogged down by trying to hash out all the details up front ... What @edolstra proposes seems perfectly workable as-is to me, though @domenkozar's idea of re-using maintainers.nix makes a lot of sense, esp. for the long term. Just my 2 NOK.

@Profpatsch
Copy link
Member

Also make a GitHub bot that applies this tool to incoming PRs and pings/assigns the maintainers.

Will we keep the bot that looks up blame information and cc’s the responsible users then? I like having everyone interested in cc, but I’m afraid of creating too much noise in general.
Maybe keep both active until too many people complain?

Another thing I loathe about the current system is that there is sometimes no connection between mail address, IRC user name and Github user name.
That bugs me when I look up git information in my local repository and want to find out the handle of that person on Github. Maybe add that to the maintainer file?

@7c6f434c
Copy link
Member

7c6f434c commented Mar 4, 2016

Another thing I loathe about the current system is that there is sometimes no connection between mail address, IRC user name and Github user name.
That bugs me when I look up git information in my local repository and want to find out the handle of that person on Github. Maybe add that to the maintainer file?

You may want to run my vanity counter from maintainers/scripts and use
its output (it does deduplication, so it tries to match names, emails
and GitHub usernames even when a person varies the name spelling or
changes the email while keeping the name spelling the same). The commit
counts would be a useless side effect for you.

@abbradar abbradar closed this as completed Mar 4, 2016
@abbradar abbradar reopened this Mar 4, 2016
FRidh added a commit to FRidh/nixpkgs that referenced this issue Jul 28, 2017
Some time ago GitHub introduced the CODEOWNERS file. The file is similar
to the MAINTAINERS file that was proposed in
NixOS#13602. Code owners will
automatically receive a review request.
@aszlig
Copy link
Member

aszlig commented Jul 30, 2017

Hm, what about running our own mention bot instead (we could even patch it to incorporate maintainers meta info), because CODEOWNERS looks a bit redundant to me.

@edolstra
Copy link
Member Author

Yeah, a patched mentionbot that uses meta.maintainers would be ideal.

@vcunat
Copy link
Member

vcunat commented Aug 6, 2017

Nitpick: then it would be a rather important requirement to enforce matching maintainer IDs with github account names or add some extra mapping for that. (For instance, Eelco's names don't match :)

@7c6f434c
Copy link
Member

7c6f434c commented Aug 6, 2017

Maybe add optional fields like Nix-related GitHub username, Nix-related GitLab user name, etc.?

(Yes, my maintainers.nix name is my Nix SVN username back from when it made sense, and my GitHub username is the same user-part as the email I use in connection to Nix stuff. These two are different)

@domenkozar
Copy link
Member

Very relevant thoughts: http://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html

@dezgeg
Copy link
Contributor

dezgeg commented Aug 21, 2017

So what's the next actionable thing to do given that the CODEOWNERS file was merged in #27499?

@vcunat
Copy link
Member

vcunat commented Aug 26, 2017

Some support for meta.maintainers, I guess? (Assuming that we don't want to duplicate that information in CODEOWNERS.)

@edolstra
Copy link
Member Author

edolstra commented Sep 7, 2017

As to the next actionable things:

  • @vcunat pointed out that CODEOWNERS allows us to require review from owners. Enabling this might be a good idea, since it fixes the current anarchic situation where any of a few dozen committers can change anything, including core components.

  • If we use CODEOWNERS for mandatory reviews (rather than as a mentionbot replacement) then we should document this policy somewhere.

  • @aszlig expressed some interest in writing a bot to use meta.maintainers for finding reviewers, though I don't think he's working on it at the moment.

@vcunat
Copy link
Member

vcunat commented Sep 7, 2017

I think there would be the downside that the feature can only be used if we disallow direct pushing, i.e. even noncritical changes would AFAIK have to go through PRs, which tends to be annoying in my experience for trivial/safe changes. (I use it in gitlab for my paid job, but the speed of changes is much slower in there.)

@matthiasbeyer
Copy link
Contributor

I think there would be the downside that the feature can only be used if we disallow direct pushing

My experience from "allow direct pushing" is "I can do this, I know my code works" ... except when it doesn't. So I'd opt for disallowing direct pushing everywhere, even for security patches (for example, I cannot push to master in my own projects, where I even work alone on the codebase - just because "My code always works" does simply not exist).

But that's just my 2ct.

@vcunat
Copy link
Member

vcunat commented Sep 7, 2017

Well, if we had some (more) reliable CI for PRs, that would be a different question... (But even there – it feels strange to wait for functional tests when you only want to fix a typo in docs.)

@7c6f434c
Copy link
Member

7c6f434c commented Sep 7, 2017 via email

@vcunat
Copy link
Member

vcunat commented Sep 7, 2017

CODEOWNERS might help with this problem, via faster notification of interested people with push access, at least in some parts of the nixpkgs tree.

@TheNeikos
Copy link
Contributor

@vcunat Regarding the typo fixes, one could look at how rust-lang/rust does it. They have two kinds of patches, 'real' patches and fixups. Fixups get batched and periodically run, removing pressure for such changes.

@matthiasbeyer
Copy link
Contributor

@vcunat Regarding the typo fixes, one could look at how rust-lang/rust does it. They have two kinds of patches, 'real' patches and fixups. Fixups get batched and periodically run, removing pressure for such changes.

That's indeed a good idea! I would even include smaller package updates into such a branch (like if abook gets updated... nobody cares that much to have the latest abook as fast as possible). If these kind up package updates appear once a week or twice a month in unstable, everything should be fine, IMHO.

@7c6f434c
Copy link
Member

7c6f434c commented Sep 7, 2017

Erm. I do actually care about some of the leaf-package updates specifically at the moment I perform them.

@vcunat
Copy link
Member

vcunat commented Sep 7, 2017

We actually batch the "larger updates", as they are often mass rebuilds :-) EDIT: I suppose Rust's motivation is partially similar and they have some extensive test suite which isn't economical to run for each small fix.

@nbp
Copy link
Member

nbp commented Oct 31, 2017

Add meta.maintainers fields to NixOS modules.

This now exists, and is used.

@nbp nbp modified the milestones: 17.09, 18.03 Oct 31, 2017
@matthewbauer matthewbauer modified the milestones: 18.03, 18.09 Apr 17, 2018
@edolstra
Copy link
Member Author

Closing in preference of NixOS/rfcs#19.

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/missing-documentation-of-makeautostartitem/20496/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests