Illumos 4958, 5164, 5165

[FransUrbo]

4958 zdb trips assert on pools with ashift >= 0xe
Reviewed by: Matthew Ahrens mahrens@delphix.com
Reviewed by: Max Grossman max.grossman@delphix.com
Reviewed by: George Wilson george.wilson@delphix.com
Reviewed by: Christopher Siden christopher.siden@delphix.com
Approved by: Garrett D'Amore garrett@damore.org

References:
https://www.illumos.org/issues/4958
illumos/illumos-gate@2a104a5

Porting notes:
Keep the ZIO_FLAG_FASTWRITE define, because that is used in Linux but not in *BSD.

5164 space_map_max_blksz causes panic, does not work
5165 zdb fails assertion when run on pool with recently-enabled spacemap_histogram feature
Reviewed by: George Wilson george.wilson@delphix.com
Reviewed by: Christopher Siden christopher.siden@delphix.com

References:
https://www.illumos.org/issues/5164
https://www.illumos.org/issues/5165
illumos/illumos-gate@b1be289

Ported by: Turbo Fredriksson turbo@bayour.com

[FransUrbo]

@behlendorf This have now been accepted into illumos-gate...

[behlendorf]

This depends on https://www.illumos.org/issues/4958, we'll need to port illumos/illumos-gate@2a104a5 before this can be merged.

[behlendorf]

As an aside, we really need to come up with some system for tracking which Illumos patches we have merged and which ones we haven't.

[FransUrbo]

@behlendorf Do you want that into a separate PR or is it ok to put it in this one?

It might also be dependent on something more. It (Illumos 4958) removes ZIO_FLAG_FASTWRITE, but ZoL still have a lot of references to it...

[FransUrbo]

As an aside, we really need to come up with some system for tracking which Illumos patches we have merged and which ones we haven't.

Yeah, I got nothing... You?

[behlendorf]

@FransUrbo @kpande what would be idea (at least from my point of view) is if we could automate the process of automatically creating a new issue in the ZoL tracker for each change as it's merged in to Illumos. Arguably we should be more involved in the entire review process going on for Illumos but there are only so many hours in the day.

[behlendorf]

@behlendorf Do you want that into a separate PR or is it ok to put it in this one?

It's OK to put it in this pull request, just make sure it's a separate patch.

It might also be dependent on something more. It (Illumos 4958) removes ZIO_FLAG_FASTWRITE, but ZoL still have a lot of references to it...

Support for ZIO_FLAG_FASTWRITE was never pushed upstream to Illumos and they never spent the time to port it. It's a Linux-only feature at the moment.

[FransUrbo]

Support for ZIO_FLAG_FASTWRITE was never pushed upstream to Illumos and they never spont the time to port it. It's a Linux-only feature at the moment.

Ok, so should we keep that then?

[behlendorf]

Yes. It's something we need to keep in mind when porting the patch.

[FransUrbo]

Ok, I put back ZIO_FLAG_FASTWRITE in the Illumos 4958 port.

[FransUrbo]

if we could automate the process of automatically creating a new issue in the ZoL tracker for each change as it's merged in to Illumos.

@behlendorf I'm not sure that's a good idea. A lot/some (?) of the commits isn't ZFS related…

But I hacked this http://bayour.com/misc/tail-illumos-gate-commits.txt real quick.

It will get a list of commits since last check (configured in /var/cache/illumos-gate) and will send them as separate mails (starting with the OLDEST it find).=

[behlendorf]

Right. We'd want to somehow restrict this to zfs related changes.

[FransUrbo]

@behlendorf I've updated it to do simple check and see if it's relevant.

[behlendorf]

Neat, I'll check it out. It would be great if we could automate the majority of this process. Even if it isn't foolproof.

[FransUrbo]

@behlendorf I've added my "check illumos commits" script to my scripts repo - https://github.com/FransUrbo/scripts/blob/master/tail-illumos-gate-commits.pl.

I've updated it somewhat to also include the issue link(s) in the mail. I've started running this on my own machine once a day to see it "in action"...

[behlendorf]

@FransUrbo Regarding a "check illumos commits" script, I took at look at what you've written and it's a nice start. What I was thinking would be nice first step would be to add a utility like this to the scripts directory. But all it would do is the following:

• Checkout the Illumos source to a temporary directory, /var/tmp/xxx
  • Maybe provide an option to point it at an existing tree.
• Checkout the ZoL source to a temporary directory, /var/tmp/xxx
  • Maybe provide an option to point it at an existing tree.
• Run through the last N commits in the Illumos source.
  • Automatically identify ZFS commits by some reasonable heuristic (perhaps filenames)
  • Compare the Illumos bug numbers in the commit against ZoLs commit history or a known white list of Illumos bug numbers which have been merged.
  • We could leave a state file committed in the ZoL tree which includes
    • The commit hash to start scanning from
    • A white list of ZFS hashes which have already been merged
    • A black list of non-ZFS hashes to be ignored as false positives
  • The script would output all ZFS commits in Illumous but not in ZoL.
• If it were written in bash it would be able to source the common.sh script like many of the other helper scripts. This would make easy to run certain commands in tree.
• Add changes were merged from Illumos we'd need to make a point to update the state file.

This might provide as a sane way to track what has and has not been pulled in from Illumos. If we wanted to go one step further we could extend it to attempt to auto-generate patches from those commits which apply. What do you think?

Also, I should be able to get these changes merged today. I've tweaked thing slightly and assuming testing doesn't uncover anything unexpected they'll go in.

[behlendorf]

Merged as:

9635861 Illumos 5164-5165 - space map fixes
b02fe35 Illumos 4958 zdb trips assert on pools with ashift >= 0xe