Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorganization of migration docs and addition of video links #1884

Merged
merged 1 commit into from
Aug 11, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/technical-documentation/migration-islandora-workbench.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
## Islandora Workbench


### Repository

<https://github.com/mjordan/islandora_workbench>

### Overview

More tailored for end users with less technical knowledge or limited server access.

- Uses Islandora’s REST API
- Runs on your computer
- “CSVs and a pile of scans”
- Cross Platform - Python

### Islandora Workbench highlights

- Opinionated
- MUCH less configuration. Decisions made for you.
- No Processing
- CSV has to be in the right format
- Write Operations
- Create, Update, and Delete content
- Bumpers On
- Configuration and CSV are validated

### Islandora Workbench basics

- Column names are field names
- If your value contains a comma, wrap it in double quotes
- Multiple values are pipe delimited
- Entity references are done via numeric id (nid, mid, tid)

Islandora Workbench - Taxonomy Terms:

- Can use term id, term name, or both
- 26
- Cats
- 26|Cats
- If using multiple vocabularies, prefix with vocabulary id:
- cats:Calico|dogs:Dachshund
- Terms that don’t exist can be created

Islandora Workbench - More Field Types:

- Typed Relations - Prefix term ids with namespace:rel:
- relators:pht:30
- Relators:pht:30|relators:pub:45
- Geolocation fields - “Lat,Long”
- "49.16667,-123.93333"

Paged Content - Two Ways:

- Metadata on Parent
- Simple directory structure and filename convention
- Page Level Metadata
- Parent and page metadata in same CSV


### Videos


#### Summer of Islandora Workbench: Introduction to Islandora Workbench

This [video](https://www.youtube.com/watch?v=bHMPsbYG-4c) (July 29, 2021) is an introduction of Islandora Workbench.

[![Summer of Islandora Workbench: Introduction to Islandora Workbench](https://img.youtube.com/vi/bHMPsbYG-4c/0.jpg)](https://www.youtube.com/watch?v=bHMPsbYG-4c)


#### Islandora Workbench Demo

This [video](https://www.youtube.com/watch?v=hNS5ouqdcfk) (Dec 17, 2020) is a demo of Islandora Workbench.

[![Islandora Workbench Demo](https://img.youtube.com/vi/hNS5ouqdcfk/0.jpg)](https://www.youtube.com/watch?v=hNS5ouqdcfk)


#### Islandora Online: Islandora Migration Tools

This [video](https://www.youtube.com/watch?v=95Bnix-z1zY) (Aug 10, 2020) provides an overview of the **Islandora Workbench** and the two other options available to migrate data into an Islandora 8 installation.

[![Islandora Online: Islandora Migration Tools](https://img.youtube.com/vi/95Bnix-z1zY/0.jpg)](https://www.youtube.com/watch?v=95Bnix-z1zY)
71 changes: 71 additions & 0 deletions docs/technical-documentation/migration-migrate-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
## Migrate API

Uses the Drupal 8 [Migrate API](https://www.drupal.org/docs/8/api/migrate-api/migrate-api-overview), which "provides services for migrating data from a source system to Drupal 8.

The "source system" can be almost anything:

- an Islandora 7 system
- a group of scanned images and their metadata inside a CSV file
- a web API

Why use the Migrate API?

- You can (potentially) do everything with configs!
- Leverage contrib module plugins.
- Making plugins for more complex sources and processes is (relatively) simple.
- Updating metadata is as simple as:
`drush mim node --update`

### A Migration Configuration defines an Extract, Transform, Load (ETL) process

- Source plugins extract data from a source
- Process plugins transform the data
- Destination plugins load the data (create new entities)

### We’ve built two tools for you using the Migrate API

- **migrate_islandora_csv**
- <https://github.com/Islandora/migrate_islandora_csv>
- Tutorial with a sample migration using some files and a CSV
- Documentation section on [migrate_islandora_csv](migrate-csv.md)
- **migrate_7x_claw**
- <https://github.com/Islandora-Devops/migrate_7x_claw>
- A tool to get all your Islandora 7 content migrated over
- Documentation section on [migrate_7x_claw](migrate-7x)

#### Recap of migrate_islandora_csv

- CSVs
- Everyone understands and knows how to work with CSVs
- Documented
- It’s a step-by-step walkthrough
- Process Metadata
- Clean up / transform the metadata using processors
- Build Relationships
- Migrations can reference other migrated content or generate new content on the fly

#### Recap of migrate_7x_claw

- Designed to migrate Islandora 7 data to Islandora 8.
- DATASTREAMS
- All of your datasteams, including the audit trail, are migrated
- METADATA
- Migrate metadata from Solr or any XML datastream
- CUSTOMIZABLE
- Migrate_7x_claw is a starting point, meant to be tailored to your metadata

##### To make migrate_7x_claw work you need

- Access
- You need credentials to both your Islandora 7 and 8 installs.
- Migrate API Knowledge
- The tutorial for migrate_islandora_csv
Is still relevant
- Config Sync
- You need to understand Drupal config synchronization. Features knowledge helps too.
- Command Line Skills
- This is best done with shell access and drush

### Migrate API demo video

Check out this video that demonstrates the Drupal Migrate API migration process: [Islandora Webinar: Migrating from Islandora 7 to Islandora 8](migrate-7x.md) (Nov 21, 2019)
130 changes: 16 additions & 114 deletions docs/technical-documentation/migration-overview.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
## Islandora 8 Migration Overview

This video provides an overview of the various options available to migrate data into an Islandora 8 installation.
This [video](https://www.youtube.com/watch?v=95Bnix-z1zY) (Aug 10, 2020) provides an overview of the various options available to migrate data into an Islandora 8 installation.

[![Islandora Online: Islandora Migration Tools](https://img.youtube.com/vi/95Bnix-z1zY/0.jpg)](https://www.youtube.com/watch?v=95Bnix-z1zY)

The three options are:
- REST API
- Migrate API
- Islandora Workbench
The three main migration options are:

- [REST API](migration-rest-api.md)
- [Migrate API](migration-migrate-api.md)
- migrate_islandora_csv
- migrate_7x_claw
- [Islandora Workbench](migration-islandora-workbench.md)


### REST API

Expand All @@ -16,28 +20,13 @@ Why use the rest API?
- **Works anywhere**: You don’t have to work on the Drupal server. Migrate from your laptop!
- **No PHP required**: Use any language that can make an http request. Even cURL will do just fine.
- **JSON**: Why use XML if you don’t have to?
- Relies on Drupal’s own REST API

BONUS: It’s just Drupal’s REST API

#### Islandora only provides two additional API endpoints

- /media/{mid}/source
- PUT a file to this endpoint to create/update a Media’s file
- /node/{nid}/media/{media_type}/{taxonomy_term}
- PUT a file to this endpoint to create/update a Media for a Node
Visit the [REST API](migration-rest-api.md) migration documentation section for more details.

Just be aware, you are writing everything yourself! (In other words you are making all of the migration decisions yourself.)

### Migrate API

Uses the Drupal 8 [Migrate API](https://www.drupal.org/docs/8/api/migrate-api/migrate-api-overview), which "provides services for migrating data from a source system to Drupal 8.

The "source system" can be almost anything:

- an Islandora 7 system
- a group of scanned images and their metadata inside a CSV file
- a web API

Why use the Migrate API?

- You can (potentially) do everything with configs!
Expand All @@ -46,106 +35,19 @@ Why use the Migrate API?
- Updating metadata is as simple as:
`drush mim node --update`

#### A Migration Configuration defines an Extract, Transform, Load (ETL) process

- Source plugins extract data from a source
- Process plugins transform the data
- Destination plugins load the data (create new entities)

#### We’ve built two tools for you using the Migrate API

- [migrate_islandora_csv](https://github.com/Islandora/migrate_islandora_csv)
- Tutorial with a sample migration using some files and a CSV
- [migrate_7x_claw](https://github.com/Islandora-Devops/migrate_7x_claw)
- A tool to get all your Islandora 7 content migrated over

##### Recap of migrate_islandora_csv

- CSVs
- Everyone understands and knows how to work with CSVs
- Documented
- It’s a step-by-step walkthrough
- Process Metadata
- Clean up / transform the metadata using processors
- Build Relationships
- Migrations can reference other migrated content or generate new content on the fly

##### Recap of migrate_7x_claw

- Designed to migrate Islandora 7 data to Islandora 8.
- DATASTREAMS
- All of your datasteams, including the audit trail, are migrated
- METADATA
- Migrate metadata from Solr or any XML datastream
- CUSTOMIZABLE
- Migrate_7x_claw is a starting point, meant to be tailored to your metadata
Two tools that use the Migrate API are [migrate_islandora_csv](migrate-csv.md) and [migrate_7x_claw](migrate-7x).

###### To make migrate_7x_claw work you need
Visit the [Migrate API](migration-migrate-api.md) migration documentation section for more details.

- Access
- You need credentials to both your Islandora 7 and 8 installs.
- Migrate API Knowledge
- The tutorial for migrate_islandora_csv
Is still relevant
- Config Sync
- You need to understand Drupal config synchronization. Features knowledge helps too.
- Command Line Skills
- This is best done with shell access and drush

#### Migrate API demo video

Check out this video that demonstrates the Drupal Migrate API migration process: [Islandora Webinar: Migrating from Islandora 7 to Islandora 8](migrate-7x.md)

### Islandora Workbench

https://github.com/mjordan/islandora_workbench

More tailored for end users with less technical knowledge or limited server access.
Why use the Migrate API?

- More tailored for end users with less technical knowledge or limited server access.
- Uses Islandora’s REST API
- Runs on your computer
- “CSVs and a pile of scans”
- Cross Platform - Python

#### Islandora Workbench highlights

- Opinionated
- MUCH less configuration. Decisions made for you.
- No Processing
- CSV has to be in the right format
- Write Operations
- Create, Update, and Delete content
- Bumpers On
- Configuration and CSV are validated

#### Islandora Workbench basics

- Column names are field names
- If your value contains a comma, wrap it in double quotes
- Multiple values are pipe delimited
- Entity references are done via numeric id (nid, mid, tid)

Islandora Workbench - Taxonomy Terms:

- Can use term id, term name, or both
- 26
- Cats
- 26|Cats
- If using multiple vocabularies, prefix with vocabulary id:
- cats:Calico|dogs:Dachshund
- Terms that don’t exist can be created

Islandora Workbench - More Field Types:

- Typed Relations - Prefix term ids with namespace:rel:
- relators:pht:30
- Relators:pht:30|relators:pub:45
- Geolocation fields - “Lat,Long”
- "49.16667,-123.93333"

Paged Content - Two Ways:

- Metadata on Parent
- Simple directory structure and filename convention
- Page Level Metadata
- Parent and page metadata in same CSV
Visit the [Islandora Workbench](migration-islandora-workbench.md) migration documentation section for more details.
24 changes: 24 additions & 0 deletions docs/technical-documentation/migration-rest-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## REST API

Why use the REST API?

- **Works anywhere**: You don’t have to work on the Drupal server. Migrate from your laptop!
- **No PHP required**: Use any language that can make an http request. Even cURL will do just fine.
- **JSON**: Why use XML if you don’t have to?

BONUS: It’s just Drupal’s REST API

### Islandora only provides two additional API endpoints

- /media/{mid}/source
- PUT a file to this endpoint to create/update a Media’s file
- /node/{nid}/media/{media_type}/{taxonomy_term}
- PUT a file to this endpoint to create/update a Media for a Node

Just be aware, you are writing everything yourself! (In other words you are making all of the migration decisions yourself.)

### Videos

This [video](https://www.youtube.com/watch?v=95Bnix-z1zY) (Aug 10, 2020) provides an overview of the **REST API** and the two other options available to migrate data into an Islandora 8 installation.

[![Islandora Online: Islandora Migration Tools](https://img.youtube.com/vi/95Bnix-z1zY/0.jpg)](https://www.youtube.com/watch?v=95Bnix-z1zY)
Loading