From fe4498e6ff2ad5023925e629f9060fe3fdc6d100 Mon Sep 17 00:00:00 2001 From: Yamil Date: Wed, 11 Aug 2021 08:47:31 -0400 Subject: [PATCH] Reorganization of migration docs and addition of video links (#1884) --- .../migration-islandora-workbench.md | 82 +++++++++++ .../migration-migrate-api.md | 71 ++++++++++ .../migration-overview.md | 130 +++--------------- .../migration-rest-api.md | 24 ++++ mkdocs.yml | 39 +++--- 5 files changed, 214 insertions(+), 132 deletions(-) create mode 100644 docs/technical-documentation/migration-islandora-workbench.md create mode 100644 docs/technical-documentation/migration-migrate-api.md create mode 100644 docs/technical-documentation/migration-rest-api.md diff --git a/docs/technical-documentation/migration-islandora-workbench.md b/docs/technical-documentation/migration-islandora-workbench.md new file mode 100644 index 000000000..88bed98cc --- /dev/null +++ b/docs/technical-documentation/migration-islandora-workbench.md @@ -0,0 +1,82 @@ +## Islandora Workbench + + +### Repository + + + +### Overview + +More tailored for end users with less technical knowledge or limited server access. + +- Uses Islandora’s REST API +- Runs on your computer +- “CSVs and a pile of scans” +- Cross Platform - Python + +### Islandora Workbench highlights + +- Opinionated + - MUCH less configuration. Decisions made for you. +- No Processing + - CSV has to be in the right format +- Write Operations + - Create, Update, and Delete content +- Bumpers On + - Configuration and CSV are validated + +### Islandora Workbench basics + +- Column names are field names +- If your value contains a comma, wrap it in double quotes +- Multiple values are pipe delimited +- Entity references are done via numeric id (nid, mid, tid) + +Islandora Workbench - Taxonomy Terms: + +- Can use term id, term name, or both + - 26 + - Cats + - 26|Cats +- If using multiple vocabularies, prefix with vocabulary id: + - cats:Calico|dogs:Dachshund +- Terms that don’t exist can be created + +Islandora Workbench - More Field Types: + +- Typed Relations - Prefix term ids with namespace:rel: + - relators:pht:30 + - Relators:pht:30|relators:pub:45 +- Geolocation fields - “Lat,Long” + - "49.16667,-123.93333" + +Paged Content - Two Ways: + +- Metadata on Parent + - Simple directory structure and filename convention +- Page Level Metadata + - Parent and page metadata in same CSV + + +### Videos + + +#### Summer of Islandora Workbench: Introduction to Islandora Workbench + +This [video](https://www.youtube.com/watch?v=bHMPsbYG-4c) (July 29, 2021) is an introduction of Islandora Workbench. + +[![Summer of Islandora Workbench: Introduction to Islandora Workbench](https://img.youtube.com/vi/bHMPsbYG-4c/0.jpg)](https://www.youtube.com/watch?v=bHMPsbYG-4c) + + +#### Islandora Workbench Demo + +This [video](https://www.youtube.com/watch?v=hNS5ouqdcfk) (Dec 17, 2020) is a demo of Islandora Workbench. + +[![Islandora Workbench Demo](https://img.youtube.com/vi/hNS5ouqdcfk/0.jpg)](https://www.youtube.com/watch?v=hNS5ouqdcfk) + + +#### Islandora Online: Islandora Migration Tools + +This [video](https://www.youtube.com/watch?v=95Bnix-z1zY) (Aug 10, 2020) provides an overview of the **Islandora Workbench** and the two other options available to migrate data into an Islandora 8 installation. + +[![Islandora Online: Islandora Migration Tools](https://img.youtube.com/vi/95Bnix-z1zY/0.jpg)](https://www.youtube.com/watch?v=95Bnix-z1zY) diff --git a/docs/technical-documentation/migration-migrate-api.md b/docs/technical-documentation/migration-migrate-api.md new file mode 100644 index 000000000..9f57ad91e --- /dev/null +++ b/docs/technical-documentation/migration-migrate-api.md @@ -0,0 +1,71 @@ +## Migrate API + +Uses the Drupal 8 [Migrate API](https://www.drupal.org/docs/8/api/migrate-api/migrate-api-overview), which "provides services for migrating data from a source system to Drupal 8. + +The "source system" can be almost anything: + +- an Islandora 7 system +- a group of scanned images and their metadata inside a CSV file +- a web API + +Why use the Migrate API? + +- You can (potentially) do everything with configs! +- Leverage contrib module plugins. +- Making plugins for more complex sources and processes is (relatively) simple. +- Updating metadata is as simple as: +`drush mim node --update` + +### A Migration Configuration defines an Extract, Transform, Load (ETL) process + +- Source plugins extract data from a source +- Process plugins transform the data +- Destination plugins load the data (create new entities) + +### We’ve built two tools for you using the Migrate API + +- **migrate_islandora_csv** + - + - Tutorial with a sample migration using some files and a CSV + - Documentation section on [migrate_islandora_csv](migrate-csv.md) +- **migrate_7x_claw** + - + - A tool to get all your Islandora 7 content migrated over + - Documentation section on [migrate_7x_claw](migrate-7x) + +#### Recap of migrate_islandora_csv + +- CSVs + - Everyone understands and knows how to work with CSVs +- Documented + - It’s a step-by-step walkthrough +- Process Metadata + - Clean up / transform the metadata using processors +- Build Relationships + - Migrations can reference other migrated content or generate new content on the fly + +#### Recap of migrate_7x_claw + +- Designed to migrate Islandora 7 data to Islandora 8. +- DATASTREAMS + - All of your datasteams, including the audit trail, are migrated +- METADATA + - Migrate metadata from Solr or any XML datastream +- CUSTOMIZABLE + - Migrate_7x_claw is a starting point, meant to be tailored to your metadata + +##### To make migrate_7x_claw work you need + +- Access + - You need credentials to both your Islandora 7 and 8 installs. +- Migrate API Knowledge + - The tutorial for migrate_islandora_csv +Is still relevant +- Config Sync + - You need to understand Drupal config synchronization. Features knowledge helps too. +- Command Line Skills + - This is best done with shell access and drush + +### Migrate API demo video + +Check out this video that demonstrates the Drupal Migrate API migration process: [Islandora Webinar: Migrating from Islandora 7 to Islandora 8](migrate-7x.md) (Nov 21, 2019) diff --git a/docs/technical-documentation/migration-overview.md b/docs/technical-documentation/migration-overview.md index 32a12c29e..07c9968e7 100644 --- a/docs/technical-documentation/migration-overview.md +++ b/docs/technical-documentation/migration-overview.md @@ -1,13 +1,17 @@ ## Islandora 8 Migration Overview -This video provides an overview of the various options available to migrate data into an Islandora 8 installation. +This [video](https://www.youtube.com/watch?v=95Bnix-z1zY) (Aug 10, 2020) provides an overview of the various options available to migrate data into an Islandora 8 installation. [![Islandora Online: Islandora Migration Tools](https://img.youtube.com/vi/95Bnix-z1zY/0.jpg)](https://www.youtube.com/watch?v=95Bnix-z1zY) -The three options are: -- REST API -- Migrate API -- Islandora Workbench +The three main migration options are: + +- [REST API](migration-rest-api.md) +- [Migrate API](migration-migrate-api.md) + - migrate_islandora_csv + - migrate_7x_claw +- [Islandora Workbench](migration-islandora-workbench.md) + ### REST API @@ -16,28 +20,13 @@ Why use the rest API? - **Works anywhere**: You don’t have to work on the Drupal server. Migrate from your laptop! - **No PHP required**: Use any language that can make an http request. Even cURL will do just fine. - **JSON**: Why use XML if you don’t have to? +- Relies on Drupal’s own REST API -BONUS: It’s just Drupal’s REST API - -#### Islandora only provides two additional API endpoints - -- /media/{mid}/source - - PUT a file to this endpoint to create/update a Media’s file -- /node/{nid}/media/{media_type}/{taxonomy_term} - - PUT a file to this endpoint to create/update a Media for a Node +Visit the [REST API](migration-rest-api.md) migration documentation section for more details. -Just be aware, you are writing everything yourself! (In other words you are making all of the migration decisions yourself.) ### Migrate API -Uses the Drupal 8 [Migrate API](https://www.drupal.org/docs/8/api/migrate-api/migrate-api-overview), which "provides services for migrating data from a source system to Drupal 8. - -The "source system" can be almost anything: - -- an Islandora 7 system -- a group of scanned images and their metadata inside a CSV file -- a web API - Why use the Migrate API? - You can (potentially) do everything with configs! @@ -46,106 +35,19 @@ Why use the Migrate API? - Updating metadata is as simple as: `drush mim node --update` -#### A Migration Configuration defines an Extract, Transform, Load (ETL) process - -- Source plugins extract data from a source -- Process plugins transform the data -- Destination plugins load the data (create new entities) - -#### We’ve built two tools for you using the Migrate API - -- [migrate_islandora_csv](https://github.com/Islandora/migrate_islandora_csv) - - Tutorial with a sample migration using some files and a CSV -- [migrate_7x_claw](https://github.com/Islandora-Devops/migrate_7x_claw) - - A tool to get all your Islandora 7 content migrated over - -##### Recap of migrate_islandora_csv - -- CSVs - - Everyone understands and knows how to work with CSVs -- Documented - - It’s a step-by-step walkthrough -- Process Metadata - - Clean up / transform the metadata using processors -- Build Relationships - - Migrations can reference other migrated content or generate new content on the fly - -##### Recap of migrate_7x_claw - -- Designed to migrate Islandora 7 data to Islandora 8. -- DATASTREAMS - - All of your datasteams, including the audit trail, are migrated -- METADATA - - Migrate metadata from Solr or any XML datastream -- CUSTOMIZABLE - - Migrate_7x_claw is a starting point, meant to be tailored to your metadata +Two tools that use the Migrate API are [migrate_islandora_csv](migrate-csv.md) and [migrate_7x_claw](migrate-7x). -###### To make migrate_7x_claw work you need +Visit the [Migrate API](migration-migrate-api.md) migration documentation section for more details. -- Access - - You need credentials to both your Islandora 7 and 8 installs. -- Migrate API Knowledge - - The tutorial for migrate_islandora_csv -Is still relevant -- Config Sync - - You need to understand Drupal config synchronization. Features knowledge helps too. -- Command Line Skills - - This is best done with shell access and drush - -#### Migrate API demo video - -Check out this video that demonstrates the Drupal Migrate API migration process: [Islandora Webinar: Migrating from Islandora 7 to Islandora 8](migrate-7x.md) ### Islandora Workbench -https://github.com/mjordan/islandora_workbench - -More tailored for end users with less technical knowledge or limited server access. +Why use the Migrate API? +- More tailored for end users with less technical knowledge or limited server access. - Uses Islandora’s REST API - Runs on your computer - “CSVs and a pile of scans” - Cross Platform - Python -#### Islandora Workbench highlights - -- Opinionated - - MUCH less configuration. Decisions made for you. -- No Processing - - CSV has to be in the right format -- Write Operations - - Create, Update, and Delete content -- Bumpers On - - Configuration and CSV are validated - -#### Islandora Workbench basics - -- Column names are field names -- If your value contains a comma, wrap it in double quotes -- Multiple values are pipe delimited -- Entity references are done via numeric id (nid, mid, tid) - -Islandora Workbench - Taxonomy Terms: - -- Can use term id, term name, or both - - 26 - - Cats - - 26|Cats -- If using multiple vocabularies, prefix with vocabulary id: - - cats:Calico|dogs:Dachshund -- Terms that don’t exist can be created - -Islandora Workbench - More Field Types: - -- Typed Relations - Prefix term ids with namespace:rel: - - relators:pht:30 - - Relators:pht:30|relators:pub:45 -- Geolocation fields - “Lat,Long” - - "49.16667,-123.93333" - -Paged Content - Two Ways: - -- Metadata on Parent - - Simple directory structure and filename convention -- Page Level Metadata - - Parent and page metadata in same CSV +Visit the [Islandora Workbench](migration-islandora-workbench.md) migration documentation section for more details. diff --git a/docs/technical-documentation/migration-rest-api.md b/docs/technical-documentation/migration-rest-api.md new file mode 100644 index 000000000..315eabc8c --- /dev/null +++ b/docs/technical-documentation/migration-rest-api.md @@ -0,0 +1,24 @@ +## REST API + +Why use the REST API? + +- **Works anywhere**: You don’t have to work on the Drupal server. Migrate from your laptop! +- **No PHP required**: Use any language that can make an http request. Even cURL will do just fine. +- **JSON**: Why use XML if you don’t have to? + +BONUS: It’s just Drupal’s REST API + +### Islandora only provides two additional API endpoints + +- /media/{mid}/source + - PUT a file to this endpoint to create/update a Media’s file +- /node/{nid}/media/{media_type}/{taxonomy_term} + - PUT a file to this endpoint to create/update a Media for a Node + +Just be aware, you are writing everything yourself! (In other words you are making all of the migration decisions yourself.) + +### Videos + +This [video](https://www.youtube.com/watch?v=95Bnix-z1zY) (Aug 10, 2020) provides an overview of the **REST API** and the two other options available to migrate data into an Islandora 8 installation. + +[![Islandora Online: Islandora Migration Tools](https://img.youtube.com/vi/95Bnix-z1zY/0.jpg)](https://www.youtube.com/watch?v=95Bnix-z1zY) diff --git a/mkdocs.yml b/mkdocs.yml index 137b57caa..f2f2fccc7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -57,16 +57,16 @@ nav: # moved from "Installation" section; alternatively duplicate a simplified version in the overview section # also see Architecture Diagram in Developer documentation # also see user-intro.md#architecture for the cheeseburger/bento box analogy - # Conceptual/ reference, all user roles/ sys admins/ developers: + # Conceptual/ reference, all user roles/ sys admins/ developers: # procedural information should be moved out installation guides - 'Modelling content in Islandora 8 vs. 7': 'user-documentation/objects_to_resource_nodes.md' # conceptual, all user roles # moved from User documentation > Content in Islandora 8 - Islandora Defaults: 'reference/islandora_defaults_reference.md' - Installation: - # Procedural, different user roles: clarify who the audience is for a - # particular recipe. The ISLE documentation page was written with a - # 'non-technical' user in mind who wants to test. The Manual installation + # Procedural, different user roles: clarify who the audience is for a + # particular recipe. The ISLE documentation page was written with a + # 'non-technical' user in mind who wants to test. The Manual installation # guides mention that users will need additional knowledge about server administration. - 'Docker (ISLE)': - 'Introduction to ISLE': 'installation/docker-introduction.md' @@ -75,7 +75,7 @@ nav: - 'Installing a Demo Server': 'installation/docker-demo.md' - 'Installing a Development Server': 'installation/docker-local.md' - 'Installing a Staging/Production Server': 'installation/docker-custom.md' - - Maintaining Islandora: + - Maintaining Islandora: - 'Basic Usage': 'installation/docker-basic-usage.md' - 'Maintaining Drupal': 'installation/docker-maintain-drupal.md' - 'Maintaining ISLE': 'installation/docker-maintain-isle.md' @@ -83,7 +83,7 @@ nav: - 'Available Configuration': 'installation/docker-available-configuration.md' - 'Troubleshooting' : 'installation/docker-troubleshooting.md' - 'Ansible Playbook': 'installation/playbook.md' - - Manual Installation: + - Manual Installation: - 'Introduction': 'installation/manual/introduction.md' - 'Preparing a LAPP Webserver': 'installation/manual/preparing_a_webserver.md' - 'Installing Composer, Drush, and Drupal': 'installation/manual/installing_composer_drush_and_drupal.md' @@ -97,16 +97,16 @@ nav: # procedural, sysadmin # moved from System Administrator Documentation - Tutorials: - # Audience for this section is someone who is extremely new to doing + # Audience for this section is someone who is extremely new to doing # the kinds of things that we are introducing. This is someone's first - # foray into this domain so each tutorial should accomplish something + # foray into this domain so each tutorial should accomplish something # satisfying, and work 100% of the time. - # Tutorials are not conceptual. They tell you exactly what to do without - # explaining why, and let the user decide if they want to look deeper at - # what these concepts are. + # Tutorials are not conceptual. They tell you exactly what to do without + # explaining why, and let the user decide if they want to look deeper at + # what these concepts are. - 'Create a Resource Node': 'tutorials/create-a-resource-node.md' - # procedural, for people learning what islandora is. + # procedural, for people learning what islandora is. - 'Create a Collection': 'tutorials/how-to-create-collection.md' # procedural, repository managers/ curators - 'Configure Blocks': 'tutorials/blocks.md' @@ -117,7 +117,7 @@ nav: # Add/link to conceptual docs about Views? - 'Video Documentation': 'user-documentation/video-docs.md' # mostly procedural, repository managers - + - Documentation: - 'Introduction': 'user-documentation/user-intro.md' # Conceptual, 'site admins/repository managers' @@ -128,12 +128,12 @@ nav: # conceptual/ reference, repository managers/ developers - Content in Islandora 8: # largely conceptual, repository managers and to some extent curators - # currently duplicates a section also present under "User documentation", - # develop into procedural documentation for repository + # currently duplicates a section also present under "User documentation", + # develop into procedural documentation for repository # admins about how to manage content in I8 - 'Resource Nodes': 'user-documentation/resource-nodes.md' - # largely conceptual, repository managers; move out procedural - # information (e.g. move procedure for setting display hints to + # largely conceptual, repository managers; move out procedural + # information (e.g. move procedure for setting display hints to # documentation about contexts) - 'Media': 'user-documentation/media.md' # conceptual and procedural, repository managers @@ -204,6 +204,9 @@ nav: - 'Alpaca Tips': 'technical-documentation/alpaca_tips.md' - Migration: - 'Migration Overview': 'technical-documentation/migration-overview.md' + - 'REST API': 'technical-documentation/migration-rest-api.md' + - 'Migrate API': 'technical-documentation/migration-migrate-api.md' + - 'Islandora Workbench': 'technical-documentation/migration-islandora-workbench.md' - 'CSV': 'technical-documentation/migrate-csv.md' - 'Islandora 7': 'technical-documentation/migrate-7x.md' - Contributing: @@ -212,7 +215,7 @@ nav: - 'Checking Coding Standards': 'technical-documentation/checking-coding-standards.md' - 'Contributing Workflow': 'contributing/contributing-workflow.md' - 'Creating GitHub Issues': 'contributing/create_issues.md' - - 'Editing Documentation': 'contributing/editing-docs.md' + - 'Editing Documentation': 'contributing/editing-docs.md' - 'How to Build Documentation': 'technical-documentation/docs-build.md' # moved from Developer documentation - 'Documentation Style Guide': 'contributing/docs_style_guide.md'