json implemented in zfs CLI #3938

goulvenriou · 2015-10-19T14:07:01Z

Hi, in response to the pull request #3483 Alyseo's team has refactored the code to deliver single commit including all the jsonify code, the json output has been implements in all CLI commands.
Regards.
Goulven

behlendorf · 2015-10-19T16:05:33Z

Awesome, thanks for squashing the patch stack it will make it much easier to review.

ryao · 2015-10-21T00:29:19Z

@goulvenriou @gwilson told me that you have libzfs_core extensions as part of the JSON changes. Where can I find those?

goulvenriou · 2015-10-21T14:17:51Z

@ryao , Hi , the libzfs_core isn't affected by the JSON implementation.

hadees · 2015-11-12T20:02:55Z

Awesome! I've really been wanting to write a web interface for my box and this will make it way easier. 👍

yada · 2015-11-12T20:16:00Z

@hadees Was one of the main goal : we think it will facilitate many new community projects based on openzfs ;-) Happy if it helps and please test, review, comment or create bug report if you find any. Thanks for your support.

hadees · 2015-12-08T05:02:52Z

@yada is this live yet? How do I use it?

yada · 2015-12-08T15:29:31Z

@hadees The branch has been updated by our team (it is up to date now) and you check this readme file if you want to build : https://github.com/Alyseo/zfs/blob/json-0.6.5/json/install.md

fractalram · 2016-01-11T04:37:43Z

Is this part of the newly release zfs-0.6.5.4?

yada · 2016-01-11T16:48:55Z

@fractalram Still not, we will push again a new and fresh rebase to help core dev review and hope it will happen soon ;-) @behlendorf may have more info on timeframe ?

yada · 2016-01-18T16:54:18Z

New rebase done, hope it helps ;-)

yada · 2016-01-28T23:10:18Z

Hi,

We understand it is a pretty disruptive append to the code base but at least it is not IMHO introduce disruptive changes.

We are closed to the final step now, all commands have been implemented using two formats (json and ldjson), schemas descriptions are online (including orderly), we regularly push new rebase to help code review... So now we just need some love :) or in other words, we need to know if you guys (OpenZFS community) agree and get buy on the fundamental approach and give their approval for merging our code ?

For infos (output and format...), see :
https://github.com/Alyseo/zfs/tree/json-0.6.5/json

BTW, we follow and like #3907
We have different approaches but I strongly think that both can be very complementary and can co-exist since they serve slightly different roles. To help code review on these two importants pull requests, maybe we can help each others (code review) ? @ryao let me know what you think about it and if it can helps ?

PS: Also posted to [OpenZFS Developer]

Regards,
Yacine

goulvenriou · 2016-02-04T17:07:25Z

new rebase and squash Done , your comments and reviews are welcome . the diff is available here : https://github.com/zfsonlinux/zfs/pull/3938/files
Regards,
Goulven
edited on 02/09/2016

yada · 2016-02-11T15:07:24Z

The usual tests failed I guess, can you guys confirm we have nothing to fix on our side to improve this : @behlendorf ?

behlendorf · 2016-02-11T22:29:14Z

@yada all the tests normally pass. It looks like your changes have accidentally introduced at least one change in behavior. It seems filebench is failing because the zfs set compression command is failing, see the CentOS 7 test log as an example.

sudo -E zfs set compression=lz4 tank/filebench
missing property=value argument(s)
usage:
    set [-jJ] <property=value> ... <filesystem|volume|snapshot> ...

The following properties are supported:
...

yada · 2016-02-12T17:22:34Z

@behlendorf : thanks for the info and issue fixed, all passed now and branch is up to date so happy review ;-)

update zfs create for test suite

behlendorf · 2016-05-05T17:19:55Z

@yada how goes the process of adding test cases to verify the json output?

yada · 2016-05-09T15:02:04Z

@behlendorf Sorry, team been busy on big project... will find some spare time to work on it asap.

yada · 2016-05-28T12:43:58Z

Can you describe and explain your major issue !?
All schema are complete for ages... See :
https://github.com/Alyseo/zfs/blob/json-0.6.5/json/STATUS.md

What is missing from your point of view ?
Will love to understand your latest sentence and why current work done is not useful...

denji · 2016-05-28T17:21:07Z

libxo - A Library for Generating Text, XML, JSON, and HTML Output

goulvenriou · 2016-06-01T14:35:24Z

@denji
the choice of the library was made during exchange on the mailing list zfs before implementation, you can see the archive here : https://www.mail-archive.com/developer@open-zfs.org/msg00880.html ,
we have indeed chosen to use the libnvpair , modified by Joyent

FYI : 
- Long term, we'd like the kernel to do more of the property processing, so this could eventually turn into simply:
     - do an ioctl to get the requested properties of the requested datasets, returning an nvlist
     - print nvlist as json (which has already been implemented, e.g. by Joyent)

behlendorf

Test cases needed.

wspeirs

I really only looked at the schema files as the plan is to add something to the tests to verify that that JSON output validates against the schema files. However, the errors in the files need to be fixed first... most of them appear to be copy-paste issues.

@yada any thoughts on a good tool to use for validating the JSON output against the schema files? There are a few tools (http://json-schema.org/implementations.html) but mileage varies greatly.

wspeirs · 2016-09-27T20:40:24Z

json/README.md

+            "items":
+                {
+                    "type":"object",
+                    "type": "type of volume",


type seems to be a duplicate key here, and overrides the schema restriction. Should "type of volume" be a property instead?

wspeirs · 2016-09-27T20:50:12Z

json/schemav1.0/schema/schema-zfs-list.json

+            "items":
+                {
+                    "type":"object",
+                    "type": "type of volume",


type seems to be a duplicate key here, and overrides the schema restriction. Should "type of volume" be a property instead?

wspeirs · 2016-09-27T20:51:30Z

json/schemav1.0/schema/schema-zpool-status.json

+        }
+    }
+}
+```


Erroneous markdown

wspeirs · 2016-09-27T20:53:03Z

json/schemav1.0/schema/schema-zpool-split.json

+    "type":"object",
+    "name": "zpool split -J",
+    "version": "1.0",
+    "description": "create a new pool by splitting a mirrored zfs storage poo"l,


Invalid JSON, need to move the l inside the quotes

wspeirs · 2016-09-27T20:56:22Z

json/schemav1.0/schema/schema-zpool-list.json

+                        "health": {
+                            "type":"string",
+                            "name": "health",
+                            "description": "   The current health of the pool. Health can be "ONLINE", "DEGRADED",  "FAULTED",  "  OFF‐LINE", "REMOVED", or "UNAVAIL"",


Cannot have double-quotes in values without escaping them with a backslash

wspeirs · 2016-09-27T21:07:13Z

json/schemav1.0/schema/schema-zfs-list.json

+}
+
+
+```


Erroneous markdown

wspeirs · 2016-09-27T21:09:04Z

json/schemav1.0/schema/schema-zfs-groupspace.json

+            "items":
+                {
+                    "type":"object",
+                    "type": "type of volume",


type seems to be a duplicate key here, and overrides the schema restriction. Should "type of volume" be a property instead?

wspeirs · 2016-09-27T21:10:18Z

json/schemav1.0/schema/schema-zfs-diff.json

+            "items":
+                {
+                    "type":"object",
+                    "type": "type of volume",


type seems to be a duplicate key here, and overrides the schema restriction. Should "type of volume" be a property instead?

wspeirs · 2016-09-27T21:10:58Z

json/schemav1.0/schema/schema-zfs-destroy.json

@@ -0,0 +1,27 @@
+zfs destroy  -J :


Erroneous sring

wspeirs · 2016-09-27T21:11:54Z

json/schemav1.0/schema/schema-zfs-create.json

+}
+
+
+```


Erroneous markdown

wspeirs · 2016-09-27T23:44:11Z

@yada I also believe there are errors in your schema files compared to what is generated. Using zfs list -J on a system with no ZFS volumes yields the following output:

{"cmd":"zfs list","schema_version":"1.0","stdout":[],"stderr":{"error":"no datasets available"}}

However, according to the schema, stderr should be a string:

        "stderr": {
            "type":"string",
            "title": "stderr",
            "description": "error output of command",
            "required":true
        },

I imagine this is broken in a few places...

Also, it appears you're using Draft3 of the JSON Schema, should this be updated to Draft4? One thing I noticed that has changed is the required restriction.

Finally, why did you use Orderly (the orderly-json.org site seems to be down) instead of straight JSON schema? I think the next step to creating tests would be to run commands and verify the output against the schema files, but currently too much is broken for this to work. Happy to help, but I need to better understand the design decisions here.

alek-p · 2017-06-22T21:59:18Z

Looks like this PR has been inactive for a while, @goulvenriou, @yada what's the status on this?
We'd like to see this integrated so I can offer assistance to move this along.

yada · 2017-06-29T21:15:36Z

@alek-p : Yep sorry guys but new job and no ressources anymore to move forward on this project.
I think this feature will be great for the openzfs community and hope some folks can take over...

sevagh · 2018-02-13T07:05:08Z

Hello. I've had to write some ugly Python to parse the output of zfs status, and searched for JSON output capabilities.

I'm interested in continuing the work on this PR. Should I continue from the last commit on this PR, or is it better to start over?

behlendorf · 2018-02-13T18:28:43Z

There was work done by ClusterHQ https://github.com/ClusterHQ/pyzfs which is worth looking at.

alek-p · 2018-02-13T19:51:01Z

We've made some progress on JSON output support for zfs cmd at datto. What we've done is piggyback JSON output support on top of channel programs (#6558). This way the JSON output support is targeted to scripting use cases and is easily maintainable since it really only touches one function (zfs_do_channel_program()).
Of course, this only adds the ability to get JSON output for zfs commands, we're still thinking about how to cleanly do this for zpool commands and welcome collaboration on that.

Since channel programmes got merged recently, I've dug out the JSON patch and am rebasing it, plus adding tests. Once that's done it will be put out for review in ZoL.

yzgyyang · 2021-01-11T20:57:02Z

We've made some progress on JSON output support for zfs cmd at datto. What we've done is piggyback JSON output support on top of channel programs (#6558). This way the JSON output support is targeted to scripting use cases and is easily maintainable since it really only touches one function (zfs_do_channel_program()).
Of course, this only adds the ability to get JSON output for zfs commands, we're still thinking about how to cleanly do this for zpool commands and welcome collaboration on that.

Since channel programmes got merged recently, I've dug out the JSON patch and am rebasing it, plus adding tests. Once that's done it will be put out for review in ZoL.

Any news regarding zpool channel programs?

alek-p · 2021-01-27T05:29:44Z

We've made some progress on JSON output support for zfs cmd at datto. What we've done is piggyback JSON output support on top of channel programs (#6558). This way the JSON output support is targeted to scripting use cases and is easily maintainable since it really only touches one function (zfs_do_channel_program()).
Of course, this only adds the ability to get JSON output for zfs commands, we're still thinking about how to cleanly do this for zpool commands and welcome collaboration on that.
Since channel programmes got merged recently, I've dug out the JSON patch and am rebasing it, plus adding tests. Once that's done it will be put out for review in ZoL.

Any news regarding zpool channel programs?

I think you're asking about json output support for channel programs - that was implemented here #7281

goulvenriou mentioned this pull request Oct 19, 2015

implementing json in zfs list cli command #3483

Closed

behlendorf added the OpenZFS Review label Oct 20, 2015

behlendorf mentioned this pull request Dec 10, 2015

Accessing zfs information programatically is not working #2908

Closed

behlendorf added the Type: Feature Feature request or new feature label Jan 28, 2016

goulvenriou force-pushed the json-0.6.5 branch 2 times, most recently from 70b96eb to b3c8794 Compare February 9, 2016 16:34

goulvenriou force-pushed the json-0.6.5 branch 4 times, most recently from fcb2dd7 to 3fa363c Compare March 11, 2016 14:55

goulvenriou force-pushed the json-0.6.5 branch 5 times, most recently from 104942e to 70eb170 Compare April 7, 2016 11:03

goulvenriou force-pushed the json-0.6.5 branch 2 times, most recently from d176b47 to 080f5ee Compare April 15, 2016 13:42

goulvenriou force-pushed the json-0.6.5 branch from 080f5ee to ac73eab Compare April 18, 2016 15:20

goulven added 3 commits April 19, 2016 10:52

json implemented in zfs CLI

8be165e

fix args error in zfs_do_set

9fb160e

update json output for zpool status

1db7ef1

update zfs create for test suite

goulvenriou force-pushed the json-0.6.5 branch from ac73eab to 1db7ef1 Compare April 19, 2016 08:52

behlendorf mentioned this pull request May 5, 2016

Add -lhHpw options to "zpool iostat" for avg latency, histogram, & queue stats #4433

Closed

behlendorf mentioned this pull request May 20, 2016

zpool status always returns true #4670

Closed

behlendorf requested changes Sep 17, 2016

View reviewed changes

wspeirs suggested changes Sep 27, 2016

View reviewed changes

behlendorf added the Status: Work in Progress Not yet ready for general review label Oct 7, 2016

This was referenced Jun 30, 2017

ZFS Support storaged-project/udisks#42

Open

Request: json output format #740

Closed

behlendorf closed this Oct 12, 2017

FireDrunk mentioned this pull request Dec 26, 2017

[discussion] How stable is the JSON Output in regards to ABI/API Compatibility #6967

Closed

alek-p mentioned this pull request Mar 7, 2018

add JSON output support to channel programs #7281

Merged

13 tasks

		}


		```

		}


		```

json implemented in zfs CLI #3938

json implemented in zfs CLI #3938

Conversation

goulvenriou commented Oct 19, 2015

behlendorf commented Oct 19, 2015

ryao commented Oct 21, 2015

goulvenriou commented Oct 21, 2015

hadees commented Nov 12, 2015

yada commented Nov 12, 2015

hadees commented Dec 8, 2015

yada commented Dec 8, 2015

fractalram commented Jan 11, 2016

yada commented Jan 11, 2016

yada commented Jan 18, 2016

yada commented Jan 28, 2016

goulvenriou commented Feb 4, 2016

yada commented Feb 11, 2016

behlendorf commented Feb 11, 2016

yada commented Feb 12, 2016

behlendorf commented May 5, 2016

yada commented May 9, 2016

yada commented May 28, 2016

denji commented May 28, 2016 • edited Loading

goulvenriou commented Jun 1, 2016

behlendorf left a comment

Choose a reason for hiding this comment

wspeirs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wspeirs commented Sep 27, 2016 • edited Loading

alek-p commented Jun 22, 2017

yada commented Jun 29, 2017

sevagh commented Feb 13, 2018 • edited Loading

behlendorf commented Feb 13, 2018

alek-p commented Feb 13, 2018

yzgyyang commented Jan 11, 2021

alek-p commented Jan 27, 2021 • edited Loading

denji commented May 28, 2016 •

edited

Loading

wspeirs commented Sep 27, 2016 •

edited

Loading

sevagh commented Feb 13, 2018 •

edited

Loading

alek-p commented Jan 27, 2021 •

edited

Loading