diff --git a/content/docs/guides/using-data-package.md b/content/docs/guides/using-data-package.md index f197006b..e286a58c 100644 --- a/content/docs/guides/using-data-package.md +++ b/content/docs/guides/using-data-package.md @@ -4,6 +4,63 @@ sidebar: order: 1 --- -:::caution -This section is under development +There are many alternatives when it comes to Data Package Standard implementations. We will cover a few the most popular options which will be a good starting point. + +:::tip +Please take a look at the full list of Data Package [Software](../../standard/software/) to find other implementations. ::: + +## Open Data Editor + +The simplest way to start using the Data Package Standard is by installing [Open Data Editor](https://opendataeditor.okfn.org/) (currently, in beta): + +[![Open Data Editor](../../../assets/software/ode.png)](https://opendataeditor.okfn.org) + +You can use the visual interface as you usually do in any modern IDE, adding and moving files, validating data, etc. Under the hood, Open Data Editor will be creating Data Package descriptors for your datasets (can be explicitly done by creating a dataset), inferring metadata, and data types. When the data curation work is done a data package can be validated and published, for example, to CKAN. + +Please refer to the [Open Data Editor's documentation](https://opendataeditor.okfn.org) to read about all the features. + +## frictionless-py + +If you prefer a command-line interface, or Python, there is [frictionless-py](https://framework.frictionlessdata.io/), a complete framework for managing data packages. Here are main commands available in CLI: + +```bash +frictionless describe # to describe your data +frictionless explore # to explore your data +frictionless extract # to extract your data +frictionless index # to index your data +frictionless list # to list your data +frictionless publish # to publish your data +frictionless query # to query your data +frictionless script # to script your data +frictionless validate # to validate your data +frictionless --help # to get list of the command +frictionless --version # to get the version +``` + +Please refer to the [frictionless-py's documentation](https://framework.frictionlessdata.io/) to read about all the features. + +## frictionless-r + +For the R community, there is [frictionless-r](https://docs.ropensci.org/frictionless/) package that allows managing data packages in R language. For example: + +```r +library(frictionless) + +# Read the datapackage.json file +# This gives you access to all Data Resources of the Data Package without +# reading them, which is convenient and fast. +package <- read_package("https://zenodo.org/records/10053702/files/datapackage.json") + +package + +# List resources +resources(package) + +# Read data from the resource "gps" +# This will return a single data frame, even though the data are split over +# multiple zipped CSV files. +read_resource(package, "gps") +``` + +Please refer to the [frictionless-r's documentation](https://docs.ropensci.org/frictionless/) to read about all the features. diff --git a/content/docs/specifications/data-package.md b/content/docs/specifications/data-package.md index 417f4b11..8320f397 100644 --- a/content/docs/specifications/data-package.md +++ b/content/docs/specifications/data-package.md @@ -173,7 +173,7 @@ Here is an example: ``` - `name`: The `name` `MUST` be an [Open Definition license ID](http://licenses.opendefinition.org/) -- `path`: A [url-or-path](../data-resource/#url-or-path) string, that is a fully qualified HTTP address, or a relative POSIX path. +- `path`: A [URL or Path](../glossary/#url-or-path) string, that is a fully qualified HTTP address, or a relative POSIX path. - `title`: A human-readable title. ### `profile` @@ -210,7 +210,7 @@ A URL for the home on the web that is related to this data package. An image to use for this data package. For example, when showing the package in a listing. -The value of the image property `MUST` be a string pointing to the location of the image. The string `MUST` be a [url-or-path](../data-resource/#url-or-path), that is a fully qualified HTTP address, or a relative POSIX path. +The value of the image property `MUST` be a string pointing to the location of the image. The string `MUST` be a [URL or Path](../glossary/#url-or-path), that is a fully qualified HTTP address, or a relative POSIX path. ### `version` @@ -277,6 +277,6 @@ The raw sources for this data package. It `MUST` be an array of Source objects. ``` - `title`: title of the source (e.g. document or organization name) -- `path`: A [url-or-path][] string, that is a fully qualified HTTP address, or a relative POSIX path (see [the url-or-path definition in Data Resource for details][url-or-path]). +- `path`: A [URL or Path](../glossary/#url-or-path) string, that is a fully qualified HTTP address, or a relative POSIX path. - `email`: An email address - `version`: A version of the source diff --git a/content/docs/specifications/data-resource.md b/content/docs/specifications/data-resource.md index dc1457eb..688616d8 100644 --- a/content/docs/specifications/data-resource.md +++ b/content/docs/specifications/data-resource.md @@ -98,7 +98,7 @@ A resource `MUST` contain a property describing the location of the data associa #### Single File -If a resource have only a single file then `path` `MUST` be a string that a "url-or-path" as defined in [URL of Path](#url-or-path) section. +If a resource have only a single file then `path` `MUST` be a string that a "url-or-path" as defined in the [URL of Path](../glossary/#url-or-path) definition. #### Multiple Files @@ -227,34 +227,6 @@ A Data Resource `MAY` have a `schema` property to describe the schema of the res The value for the `schema` property on a `resource` MUST be an `object` representing the schema OR a `string` that identifies the location of the schema. -If a `string` it must be a [url-or-path as defined above](#url-or-path), that is a fully qualified http URL or a relative POSIX path. The file at the location specified by this url-or-path string `MUST` be a JSON document containing the schema. +If a `string` it must be a [URL or Path](../glossary/#url-or-path), that is a fully qualified http URL or a relative POSIX path. The file at the location specified by this [URL or Path](../glossary/#url-or-path) string `MUST` be a JSON document containing the schema. NOTE: the Data Package specification places no restrictions on the form of the schema Object. This flexibility enables specific communities to define schemas appropriate for the data they manage. As an example, the [Tabular Data Package](https://specs.frictionlessdata.io/tabular-data-package/) specification requires the schema to conform to [Table Schema](../table-schema/). - -## URL or Path - -A `url-or-path` is a `string` with the following additional constraints: - -- `MUST` either be a URL or a POSIX path -- [URLs](https://en.wikipedia.org/wiki/Uniform_Resource_Locator) `MUST` be fully qualified. `MUST` be using either http or https scheme. (Absence of a scheme indicates `MUST` be a POSIX path) -- [POSIX paths](https://en.wikipedia.org/wiki/Path_%28computing%29#POSIX_pathname_definition) (unix-style with `/` as separator) are supported for referencing local files, with the security restraint that they `MUST` be relative siblings or children of the descriptor. Absolute paths `/`, relative parent paths `../`, hidden folders starting from a dot `.hidden` `MUST` NOT be used. - -Example of a fully qualified url: - -```json -{ - "path": "http://ex.datapackages.org/big-csv/my-big.csv" -} -``` - -Example of a relative path that this will work both as a relative path on disk and online: - -```json -{ - "path": "my-data-directory/my-csv.csv" -} -``` - -:::caution[Security] -`/` (absolute path) and `../` (relative parent path) are forbidden to avoid security vulnerabilities when implementing data package software. These limitations on resource `path` ensure that resource paths only point to files within the data package directory and its subdirectories. This prevents data package software being exploited by a malicious user to gain unintended access to sensitive information. For example, suppose a data package hosting service stores packages on disk and allows access via an API. A malicious user uploads a data package with a resource path like `/etc/passwd`. The user then requests the data for that resource and the server naively opens `/etc/passwd` and returns that data to the caller. -::: diff --git a/content/docs/specifications/glossary.md b/content/docs/specifications/glossary.md index 7fb20075..1512008a 100644 --- a/content/docs/specifications/glossary.md +++ b/content/docs/specifications/glossary.md @@ -1,7 +1,6 @@ --- title: Glossary sidebar: - hidden: true order: 6 --- @@ -20,22 +19,30 @@ The key words `MUST`, `MUST NOT`, `REQUIRED`, `SHALL`, `SHALL NOT`, `SHOULD`, `S ## Definitions -:::caution -This section is under development -::: - -### Metadata Descriptor +### URL or Path -### Metadata Profile +A `URL or Path` is a `string` with the following additional constraints: -### Tabular Data +- `MUST` either be a URL or a POSIX path +- [URLs](https://en.wikipedia.org/wiki/Uniform_Resource_Locator) `MUST` be fully qualified. `MUST` be using either http or https scheme. (Absence of a scheme indicates `MUST` be a POSIX path) +- [POSIX paths](https://en.wikipedia.org/wiki/Path_%28computing%29#POSIX_pathname_definition) (unix-style with `/` as separator) are supported for referencing local files, with the security restraint that they `MUST` be relative siblings or children of the descriptor. Absolute paths `/`, relative parent paths `../`, hidden folders starting from a dot `.hidden` `MUST` NOT be used. -### Physical Level +Example of a fully qualified url: -### Logical Level +```json +{ + "path": "http://ex.datapackages.org/big-csv/my-big.csv" +} +``` -### Data Consumer +Example of a relative path that this will work both as a relative path on disk and online: -### Data Producer +```json +{ + "path": "my-data-directory/my-csv.csv" +} +``` -### Implementation +:::caution[Security] +`/` (absolute path) and `../` (relative parent path) are forbidden to avoid security vulnerabilities when implementing data package software. These limitations on resource `path` ensure that resource paths only point to files within the data package directory and its subdirectories. This prevents data package software being exploited by a malicious user to gain unintended access to sensitive information. For example, suppose a data package hosting service stores packages on disk and allows access via an API. A malicious user uploads a data package with a resource path like `/etc/passwd`. The user then requests the data for that resource and the server naively opens `/etc/passwd` and returns that data to the caller. +::: diff --git a/content/docs/specifications/security.md b/content/docs/specifications/security.md index 35f6d04d..96b3975a 100644 --- a/content/docs/specifications/security.md +++ b/content/docs/specifications/security.md @@ -17,7 +17,7 @@ Security considerations around Data Packages and Data Resources. The key words `MUST`, `MUST NOT`, `REQUIRED`, `SHALL`, `SHALL NOT`, `SHOULD`, `SHOULD NOT`, `RECOMMENDED`, `MAY`, and `OPTIONAL` in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) -## Library users +## Usage Perspective Data packages is a container format that allows the creator to specify payload data (Resources) either as JSON objects/arrays or via pointers. There are two pointer formats: @@ -36,7 +36,7 @@ ONLY in a trusted environment (eg. your own computer during development of Data all kinds of Resource pointers. In every other environment, you MUST keep the various attack scenarios in mind and filter out potentially dangerous Resource pointer types -## Dangerous Descriptor/Resource pointer combinations +### Dangerous Descriptor/Resource pointer combinations How to read the table: if your "datapackage.json"-file comes from one of the sources on the left, you should treat Resources in the format on the top as: @@ -47,7 +47,7 @@ Resources in the format on the top as: ![Security Matrix](./assets/security-matrix.png) -### Descriptor source is a URL +#### Descriptor source is a URL If your descriptor is loaded via URL, and the server to which the URL points is not fully trusted, you SHOULD NOT allow Data Packages with Resource pointers in @@ -64,7 +64,7 @@ each could point to very large CSV files hosted somewhere. The Data Package proc those CSV files which might overwhelm the user's computer. If an attacker were able to spread such a malicious Data Package, this could exhaust the resources of a hosting service. -### Descriptor source is a local relative path +#### Descriptor source is a local relative path If your descriptor is loaded via a local relative path, and the source of the Data Package is not fully trusted, you SHOULD NOT allow Data Packages with Resource pointers in @@ -82,7 +82,7 @@ as well as crafting malicious Data Packages. In the above table, this case is th If Data Package parsing is part of a service offered to computers across subnets on the same LAN or even open to the internet, it NEVER safe to accept Data Packages containing URL-based Resource pointers. -### Descriptor source is a local relative path +#### Descriptor source is a local relative path While it is never safe to accept absolute file paths for Resources, it is perfectly safe to accept them for Descriptor files. If your descriptor is loaded via a local absolute path, and the source of the Data Package is not fully @@ -101,7 +101,7 @@ as well as crafting malicious Data Packages. In the above table, this case is th If Data Package parsing is part of a service offered to computers across subnets on the same LAN or even open to the internet, it NEVER safe to accept Data Packages containing URL-based Resource pointers. -### Descriptor source is a JSON object +#### Descriptor source is a JSON object If the Descriptor is not loaded from file but created in-memory and the source of the Data Package is not fully trusted, you SHOULD NOT allow Data Packages with Resource pointers in @@ -120,13 +120,13 @@ as well as crafting malicious Data Packages. In the above table, this case is th If Data Package parsing is part of a service offered to computers across subnets on the same LAN or even open to the internet, it NEVER safe to accept Data Packages containing URL-based Resource pointers. -### Descriptor source is a self-created JSON object +#### Descriptor source is a self-created JSON object If the Descriptor is not loaded from file or created via a third-party application but by your software, it is generally assumed you know what you do and therefore, loading Resources from URLs or file is considered safe. You still SHOULD NOT use absolute paths as a matter of precaution - and implementing libraries should filter them out. -## Library creators +## Implemention Perspective Two kinds of Resource pointers can never be guaranteed to be totally safe: diff --git a/content/docs/specifications/table-schema.md b/content/docs/specifications/table-schema.md index 5a146adb..8bcbc5b2 100644 --- a/content/docs/specifications/table-schema.md +++ b/content/docs/specifications/table-schema.md @@ -349,7 +349,7 @@ See [Field Constraints](#field-constraints) #### `missingValues` -A list of missing values for this field as per [Missing Values](#missing-values) definition. If this property is defined, it takes precedence over the schema-level property and completely replaces it for the field without combining the values. +A list of missing values for this field as per [Missing Values](#missingvalues) definition. If this property is defined, it takes precedence over the schema-level property and completely replaces it for the field without combining the values. For example, for the Table Schema below: diff --git a/content/docs/standard/changelog.md b/content/docs/standard/changelog.md index e50f248f..bac6d856 100644 --- a/content/docs/standard/changelog.md +++ b/content/docs/standard/changelog.md @@ -34,13 +34,13 @@ This change allows omitting `title` property for the `contributor` and `source` ##### Added `contributor.given/familyName` -This change adds two new properties to the `contributor` object: `givenName` and `familyName`. Please read more about [`package.contributors`](../../specifications/data-resource/#contributors) property. +This change adds two new properties to the `contributor` object: `givenName` and `familyName`. Please read more about [`package.contributors`](../../specifications/data-package/#contributors) property. > [Pull Request -- #20](https://github.com/frictionlessdata/datapackage/pull/20) ##### Added `contributor.roles` property -This change adds a new `contributors.roles` property that replaces `contributor.role`. Please read more about [`package.contributors`](../../specifications/data-resource/#contributors) property. +This change adds a new `contributors.roles` property that replaces `contributor.role`. Please read more about [`package.contributors`](../../specifications/data-package/#contributors) property. > [Pull Request -- #18](https://github.com/frictionlessdata/datapackage/pull/18) @@ -54,7 +54,7 @@ This change adds omitted `version` property to the Data Package profiles. ##### Relaxed `resource.name` rules but keep it required and unique -This change relaxes requirements to `resource.name` allowing it to be any string. This property still needs to present and be unique among resources. Please read more about [`resource.name`](../../specifications/data-resource/#name) property. +This change relaxes requirements to `resource.name` allowing it to be any string. This property still needs to present and be unique among resources. Please read more about [`resource.name`](../../specifications/data-resource/#name-required) property. > [Pull Request -- #27](https://github.com/frictionlessdata/datapackage/pull/27) @@ -110,7 +110,7 @@ This change adds a new constraint for the `object` and `array` fields. Please re ##### Support `groupChar` for integer field type -This change adds support for providing integers with group chars. Please read more about [`field.groupChar`](../../specifications/table-schema/#groupchar) property. +This change adds support for providing integers with group chars. Please read more about [`field.groupChar`](../../specifications/table-schema/#integer) property. > [Pull Request -- #6](https://github.com/frictionlessdata/datapackage/pull/6) diff --git a/content/docs/standard/contributing.md b/content/docs/standard/contributing.md index de291f9a..32014340 100644 --- a/content/docs/standard/contributing.md +++ b/content/docs/standard/contributing.md @@ -2,6 +2,7 @@ title: Contributing sidebar: order: 9 + hidden: true --- :::caution diff --git a/content/docs/standard/extensions.mdx b/content/docs/standard/extensions.mdx index 23e57cf6..167ad863 100644 --- a/content/docs/standard/extensions.mdx +++ b/content/docs/standard/extensions.mdx @@ -18,13 +18,13 @@ One of the key strengths of the Data Package Standard lies in its extensibility. @@ -34,24 +34,12 @@ One of the key strengths of the Data Package Standard lies in its extensibility. - - - - diff --git a/content/docs/standard/guides.mdx b/content/docs/standard/guides.mdx index 17e1db66..afc59ead 100644 --- a/content/docs/standard/guides.mdx +++ b/content/docs/standard/guides.mdx @@ -21,13 +21,3 @@ The least formal part of the standard containing various guides on how to get st href="/guides/using-data-package" /> - -## Extensions - - - - diff --git a/content/docs/standard/recipes.mdx b/content/docs/standard/recipes.mdx index dc72b75d..5a3915ca 100644 --- a/content/docs/standard/recipes.mdx +++ b/content/docs/standard/recipes.mdx @@ -62,12 +62,6 @@ Recipes are various approaches for solving common problems, in ways that are not href="/recipes/caching-of-resources" /> - - + + @@ -97,14 +97,14 @@ Recipes are various approaches for solving common problems, in ways that are not />