Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BAD-308: schema and BQ description updates #116

Merged
merged 2 commits into from
Nov 25, 2022
Merged

BAD-308: schema and BQ description updates #116

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
777bdd5
BAD-308: updated schemas file names. Removed unnecessary scheams in t…
Nov 2, 2022
bf8f153
BAD-308: Updated schema file names. Removed dates. Removed unused sch…
Nov 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/oaebu_project/overview/background.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Link to public dashboards embedded in the University of Michigan Press website:
* <https://ebc.press.umich.edu/impact/#oa-book-usage>

Direct link to the University of Michigan Press public dashboards:
* <https://tinyurl.com/umpress-public>
* <https://uomp-public.book-analytics.org>

## Pilot Project Dashboard Partners
The pilot project dashboard partners that took part in the pilot project were: the University of Michigan Press, Wits University Press, UCL Press, ANU Press, SpringerNature and OAPEN.
Expand Down
4 changes: 2 additions & 2 deletions oaebu_workflows/dags/onix_telescope.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
project_id = organisation.project_id
download_bucket = organisation.download_bucket
transform_bucket = organisation.transform_bucket
dataset_location = "us"
data_location = "us"
date_regex = workflow.extra.get("date_regex")
date_format = workflow.extra.get("date_format")

Expand All @@ -42,7 +42,7 @@
project_id=project_id,
download_bucket=download_bucket,
transform_bucket=transform_bucket,
dataset_location=dataset_location,
data_location=data_location,
date_regex=date_regex,
date_format=date_format,
workflow_id=workflow.id,
Expand Down
0 ...nu_press_google_analytics_1900-01-01.json → ...se/schema/anu_press_google_analytics.json
View file
Open in desktop
File renamed without changes.
274 changes: 179 additions & 95 deletions oaebu_workflows/database/schema/book_product_2022-03-31.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,118 +439,202 @@
"fields": [
{
"mode": "NULLABLE",
"name": "average_time",
"type": "FLOAT",
"description": "Average time (in seconds) spent on each page"
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "Country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "Number of sessions"
}
],
"mode": "REPEATED",
"name": "country",
"type": "RECORD",
"description": "Unique views per users’ country, derived from their IP addresses or Geographical IDs."
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of views"
}
],
"mode": "REPEATED",
"name": "views_total_country",
"type": "RECORD",
"description": "The total number of views per country"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "Referrer name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "Number of unique views"
}
],
"mode": "REPEATED",
"name": "referrer",
"type": "RECORD",
"description": "Unique views per referrer, the full referring URL including the hostname and path"
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of downloads"
}
],
"mode": "REPEATED",
"name": "downloads_total_country",
"type": "RECORD",
"description": "The total number of downloads per country"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "Social network name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "Number of unique views"
}
],
"mode": "REPEATED",
"name": "social_network",
"type": "RECORD",
"description": "Unique views per social network. This is related to the referring social network for traffic sources; e.g., Google+, Blogger."
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of PDF book downloads"
}
],
"mode": "NULLABLE",
"name": "unique_views",
"mode": "REPEATED",
"name": "downloads_pdf_book_country",
"type": "RECORD",
"description": "Unique views for several different dimensions. Unique views is the number of sessions during which the specified page was viewed at least once. A unique pageview is counted for each page URL + page title combination."
"description": "PDF book downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads",
"type": "INTEGER",
"description": "Number of total downloads summed from downloads_pdf_book, downloads_pdf_chapter, downloads_html_chapter, downloads_epub_book, downloads_epub_chapter, downloads_mobi_chapter"
},
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of PDF chapter downloads"
}
],
"mode": "REPEATED",
"name": "downloads_pdf_chapter_country",
"type": "RECORD",
"description": "PDF chapter downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads_pdf_book",
"type": "INTEGER",
"description": "Number of PDF book downloads"
},
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of HTML book downloads"
}
],
"mode": "REPEATED",
"name": "downloads_html_book_country",
"type": "RECORD",
"description": "HTML book downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads_pdf_chapter",
"type": "INTEGER",
"description": "Number of PDF chapter downloads"
},
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of HTML chapter downloads"
}
],
"mode": "REPEATED",
"name": "downloads_html_chapter_country",
"type": "RECORD",
"description": "HTML chapter downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads_html_chapter",
"type": "INTEGER",
"description": "Number of HTML chapter downloads"
},
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of EPUB book downloads"
}
],
"mode": "REPEATED",
"name": "downloads_epub_book_country",
"type": "RECORD",
"description": "EPUB book downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads_epub_book",
"type": "INTEGER",
"description": "Number of ePUB book downloads"
},
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of EPUB chapter downloads"
}
],
"mode": "REPEATED",
"name": "downloads_epub_chapter_country",
"type": "RECORD",
"description": "EPUB chapter downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads_epub_chapter",
"type": "INTEGER",
"description": "Number of ePUB chapter downloads"
},
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of MOBI book downloads"
}
],
"mode": "REPEATED",
"name": "downloads_mobi_book_country",
"type": "RECORD",
"description": "MOBI book downloads per country"
},
{
"mode": "NULLABLE",
"name": "downloads_mobi_chapter",
"type": "INTEGER",
"description": "Number of MOBI chapter downloads"
}
"fields": [
{
"mode": "NULLABLE",
"name": "name",
"type": "STRING",
"description": "The country name"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "INTEGER",
"description": "The total number of MOBI chapter downloads"
}
],
"mode": "REPEATED",
"name": "downloads_mobi_chapter_country",
"type": "RECORD",
"description": "MOBI chapter downloads per country"
}
],
"mode": "NULLABLE",
"name": "google_analytics",
Expand Down
0 ...e/schema/google_analytics_1900-01-01.json → ...ows/database/schema/google_analytics.json
View file
Open in desktop
File renamed without changes.
0 ...schema/google_books_sales_1900-01-01.json → ...s/database/schema/google_books_sales.json
View file
Open in desktop
File renamed without changes.
0 ...hema/google_books_traffic_1900-01-01.json → ...database/schema/google_books_traffic.json
View file
Open in desktop
File renamed without changes.
0 ...base/schema/jstor_country_1900-01-01.json → ...kflows/database/schema/jstor_country.json
View file
Open in desktop
File renamed without changes.
0 .../schema/jstor_institution_1900-01-01.json → ...ws/database/schema/jstor_institution.json
View file
Open in desktop
File renamed without changes.
12 changes: 9 additions & 3 deletions oaebu_workflows/database/schema/oaebu_publisher_book_product_author_metrics_2022-03-31.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@
"fields": [
{
"mode": "NULLABLE",
"name": "unique_views",
"name": "views_total_country",
"type": "INTEGER",
"description": "Number of unique views"
"description": "Number of page views aggregated over all countries"
},
{
"mode": "NULLABLE",
Expand Down Expand Up @@ -67,12 +67,18 @@
"type": "INTEGER",
"description": "Number of ePUB chapter downloads"
},
{
"mode": "NULLABLE",
"name": "downloads_mobi_book",
"type": "INTEGER",
"description": "Number of MOBI book downloads"
},
{
"mode": "NULLABLE",
"name": "downloads_mobi_chapter",
"type": "INTEGER",
"description": "Number of MOBI chapter downloads"
}
}
],
"mode": "NULLABLE",
"name": "google_analytics",
Expand Down
12 changes: 9 additions & 3 deletions oaebu_workflows/database/schema/oaebu_publisher_book_product_metrics_2022-03-31.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@
"fields": [
{
"mode": "NULLABLE",
"name": "unique_views",
"name": "page_views",
"type": "INTEGER",
"description": "The number of unique views"
"description": "Number of page views"
},
{
"mode": "NULLABLE",
Expand Down Expand Up @@ -99,12 +99,18 @@
"type": "INTEGER",
"description": "Number of ePUB chapter downloads"
},
{
"mode": "NULLABLE",
"name": "downloads_mobi_book",
"type": "INTEGER",
"description": "Number of MOBI book downloads"
},
{
"mode": "NULLABLE",
"name": "downloads_mobi_chapter",
"type": "INTEGER",
"description": "Number of MOBI chapter downloads"
}
}
],
"mode": "NULLABLE",
"name": "google_analytics",
Expand Down
Loading