This repository has been archived by the owner on Oct 28, 2019. It is now read-only.
CKAN setup is too easily skipped #7
Comments
ThrawnCA
referenced
this issue
in qld-gov-au/opswx-ckan-cookbook
Feb 12, 2019
ThrawnCA
referenced
this issue
in qld-gov-au/opswx-ckan-cookbook
Mar 15, 2019
* [QOL-4495] revert most Solr conf changes, just add newer schema * [QOL-4495] provide Redis URL to handle CKAN 2.7+ * [QOL-4495] continue with CKAN installation if INI file doesn't exist, mitigates github #7 * [QOL-4495] oops close the 'unless' block', github #7 * [QOL-4495] oops missed a quote * [QOL-4495] update Redis configuration to use ElastiCache - update URLs to point to provided Redis - provide a unique site_id so multiple CKANs can share the cache * [QOL-4495] move EFS mount point to application-specific location * [QOL-4495] ensure that the app-specific EFS directory is created before mounting it * [QOL-4495] use the OpsWorks app ID as the CKAN site ID * [QOL-4495] use consistent app ID for mounting EFS - app_name isn't good enough since it's different for different OpsWorks apps * [QOL-4495] use app-specific names when looking up stack data * [QOL-4495] oops create directory before using it as a mount point * [QOL-4495] fix external site ID and add properties for QGOV content * [QOL-4495] remove default permissions to match existing CKAN * [QOL-4495] include environment in searches to ensure we identify the right application * [QOL-4495] oops change single quotes to double so we can use single quotes in our lookup * [QOL-4495] oops use correct variable to assemble Redis URL * [QOL-4495] improve guard condition on Solr install - We want to run the installation even if the data files already exist on EFS - TODO: Split the general installation work from the creation of the service file, so we can install the service on multiple machines while only setting up the data once. * [QOL-4495] increase bucket size to handle larger Nginx server names * [QOL-4495] split Solr data creation from service registration * [QOL-4495] drop redundant Solr config if needed * [QOL-4495] use the shared EFS directory to store web content - this is persistent across machines and already gives us write permissions * [QOL-4495] add QGOV extension to the list of known extensions and tighten the check that determines whether it's already installed * [QOL-4495] make /etc/ckan real and put a link under it - if /etc/ckan is a link, then we can't put other config under it, like the DataPusher settings * [QOL-4495] enable datastore if installing DataPusher * [QOL-4495] don't try to overwrite /etc/ckan/default symlink * [QOL-4495] use app ID, instead of version, to assemble the Solr domain name - otherwise Data and Publications step on each other * [QOL-4495] redirect HTTP traffic to HTTPS in Nginx - for unknown reasons, CKAN is sometimes redirecting to plain HTTP. Also, people may type the domain in their browsers without HTTPS. * [QOL-4531] replace SendMail with the AWS SMTP Relay - this allows us to send email using IAM roles instead of SMTP credentials * [QOL-4531] fix location of STMP relay files - put them all in an archive and retrieve it from wherever it's available to a standard location * [QOL-4495] fix database load - disable Python scripts since we want to manage databases ourselves eg migrating existing data - fix datastore usernames to match the standard CKAN pattern * [QOL-4495] update front-end CKAN resources after installing extensions - this wasn't needed in older CKAN versions * [QOL-4495] build the initial Solr index unless it's already done * [QOL-4495] fix path to web content * [QOL-4495] oops 'exists' check needs a question mark * [QOL-4495] don't try to install PostGIS since it will fail and we don't want it - could fix up the credentials so this can actually work, but there's no need * [QOL-4533] add recipes to set up the CKAN DataPusher - two parts: setting up the DataPusher server, and enabling the extension in CKAN. The first is meant to be tied to an OpsWorks layer. The second is tied to the CKAN layer but can be disabled. * [QOL-4533] cleanup - don't install unused DB setup scripts - replace deprecated File.exists with File.exist - trim trailing whitespace * [QOL-4531] set the email From address to match current - except that current always uses the production domain, whereas we will use whichever domain we're coming from * [QOL-4495] update metadata for forked cookbook * [QOL-4570][QOL-4571] run CKAN setup commands as non-root user when possible - this keeps directory ownership in the right hands - also extract more repeated paths to variables, and simplify the logic for when resources should run * [QOL-4495] extract more configuration file options to parameters with default values - we probably won't often want to override these, except maybe the feedback email address, but the option is there * [QOL-4495] oops fix syntax error when invoking virtualenv paster - we had a leftover dot from previous code that activated the virtualenv first - also replace more paths with variables * [QOL-4495] update CKAN config to better match existing settings - enable page tracking - enable activity streams and notifications - increase max resource size to 200MB - set desired locale order - configure Google Analytics if present - enable CORS * [QOL-4587] set Beaker secret in CKAN config file - leaving this as the hardcoded, publicly known version would be VERY bad * [QOL-4572] update tracking data before rebuilding search index * [QOL-4531] split SMTP relay archive and update init script to handle status return code properly - also throw away the unnecessary separate start and stop scripts * [QOL-4572] add scheduled jobs to crontab * [QOL-4610] configure Apache to allow access to DataPusher * [QOL-4495] properly handle extensions with 'ckanext' partway through their names * [QOL-4531] use verified sender email address * [QOL-4570][QOL-4571] set site title properly so test automation can work * [QOL-4495] provide fallbacks when searching for stacks and apps - this will allow us to use either app-specific stacks, or a shared stack * [QOL-4495] oops use regex instead of plain string for pattern matching * [QOL-4531] send error emails to Online Products - previous address bounced, which is not OK * [QOL-4495] clean up file permissions to ensure we can successfully rebuild frontend files * [QOL-4495] adjust Solr index rebuild to run non-destructively and unconditionally * [QOL-4531] disable error emails to match existing environment, and send from our domain * [QOL-4495] drop obsolete Redis recipes * [QOL-4495] use private addressing to connect to Solr and DataPusher - we don't want to have public IPs for our EC2 instances, only for the load balancer and CDN * [QOL-4495] only reindex missing packages on CKAN creation - this allows us to start new instances much faster, while still picking up changes if Solr has been recreated * [QOL-4570][QOL-4571] write logs to separate directories per instance - this is necessary to properly support high-availability configurations, otherwise they write to the same log files and clash * [QOL-4495] query CloudFront and Route 53 to determine the best public domain name on server start - use the public DNS name if available, else CloudFront distribution domain name if available, else leave it untouched (load balancer domain name) * [QOL-4495] reload Apache after configuring site_url * [QOL-4495] restart Apache, not just reload * [QOL-4610] don't upgrade bleach beyond what CKAN needs - upgrading triggers ckan/ckan#3968 * [QOL-4639] extend logrotate configuration to cover our subdirectories * QOL-4495 update smtp relay for override and ssm paramater store lookup * [QOL-4495] install Java 8 rather than 6 or 7 - pre-8 versions are obsolete * [QOL-4495] restart SMTP relay upon setup if it already exists * [QOL-4495] enable SSM in AWS SMTP relay * [QOL-4495] use Brisbane timezone * [QOL-4531] remove Java 7 to ensure we use 8
ThrawnCA
referenced
this issue
in qld-gov-au/opswx-ckan-cookbook
Mar 19, 2019
* [QOL-4495] revert most Solr conf changes, just add newer schema * [QOL-4495] provide Redis URL to handle CKAN 2.7+ * [QOL-4495] continue with CKAN installation if INI file doesn't exist, mitigates github #7 * [QOL-4495] oops close the 'unless' block', github #7 * [QOL-4495] oops missed a quote * [QOL-4495] update Redis configuration to use ElastiCache - update URLs to point to provided Redis - provide a unique site_id so multiple CKANs can share the cache * [QOL-4495] move EFS mount point to application-specific location * [QOL-4495] ensure that the app-specific EFS directory is created before mounting it * [QOL-4495] use the OpsWorks app ID as the CKAN site ID * [QOL-4495] use consistent app ID for mounting EFS - app_name isn't good enough since it's different for different OpsWorks apps * [QOL-4495] use app-specific names when looking up stack data * [QOL-4495] oops create directory before using it as a mount point * [QOL-4495] fix external site ID and add properties for QGOV content * [QOL-4495] remove default permissions to match existing CKAN * [QOL-4495] include environment in searches to ensure we identify the right application * [QOL-4495] oops change single quotes to double so we can use single quotes in our lookup * [QOL-4495] oops use correct variable to assemble Redis URL * [QOL-4495] improve guard condition on Solr install - We want to run the installation even if the data files already exist on EFS - TODO: Split the general installation work from the creation of the service file, so we can install the service on multiple machines while only setting up the data once. * [QOL-4495] increase bucket size to handle larger Nginx server names * [QOL-4495] split Solr data creation from service registration * [QOL-4495] drop redundant Solr config if needed * [QOL-4495] use the shared EFS directory to store web content - this is persistent across machines and already gives us write permissions * [QOL-4495] add QGOV extension to the list of known extensions and tighten the check that determines whether it's already installed * [QOL-4495] make /etc/ckan real and put a link under it - if /etc/ckan is a link, then we can't put other config under it, like the DataPusher settings * [QOL-4495] enable datastore if installing DataPusher * [QOL-4495] don't try to overwrite /etc/ckan/default symlink * [QOL-4495] use app ID, instead of version, to assemble the Solr domain name - otherwise Data and Publications step on each other * [QOL-4495] redirect HTTP traffic to HTTPS in Nginx - for unknown reasons, CKAN is sometimes redirecting to plain HTTP. Also, people may type the domain in their browsers without HTTPS. * [QOL-4531] replace SendMail with the AWS SMTP Relay - this allows us to send email using IAM roles instead of SMTP credentials * [QOL-4531] fix location of STMP relay files - put them all in an archive and retrieve it from wherever it's available to a standard location * [QOL-4495] fix database load - disable Python scripts since we want to manage databases ourselves eg migrating existing data - fix datastore usernames to match the standard CKAN pattern * [QOL-4495] update front-end CKAN resources after installing extensions - this wasn't needed in older CKAN versions * [QOL-4495] build the initial Solr index unless it's already done * [QOL-4495] fix path to web content * [QOL-4495] oops 'exists' check needs a question mark * [QOL-4495] don't try to install PostGIS since it will fail and we don't want it - could fix up the credentials so this can actually work, but there's no need * [QOL-4533] add recipes to set up the CKAN DataPusher - two parts: setting up the DataPusher server, and enabling the extension in CKAN. The first is meant to be tied to an OpsWorks layer. The second is tied to the CKAN layer but can be disabled. * [QOL-4533] cleanup - don't install unused DB setup scripts - replace deprecated File.exists with File.exist - trim trailing whitespace * [QOL-4531] set the email From address to match current - except that current always uses the production domain, whereas we will use whichever domain we're coming from * [QOL-4495] update metadata for forked cookbook * [QOL-4570][QOL-4571] run CKAN setup commands as non-root user when possible - this keeps directory ownership in the right hands - also extract more repeated paths to variables, and simplify the logic for when resources should run * [QOL-4495] extract more configuration file options to parameters with default values - we probably won't often want to override these, except maybe the feedback email address, but the option is there * [QOL-4495] oops fix syntax error when invoking virtualenv paster - we had a leftover dot from previous code that activated the virtualenv first - also replace more paths with variables * [QOL-4495] update CKAN config to better match existing settings - enable page tracking - enable activity streams and notifications - increase max resource size to 200MB - set desired locale order - configure Google Analytics if present - enable CORS * [QOL-4587] set Beaker secret in CKAN config file - leaving this as the hardcoded, publicly known version would be VERY bad * [QOL-4572] update tracking data before rebuilding search index * [QOL-4531] split SMTP relay archive and update init script to handle status return code properly - also throw away the unnecessary separate start and stop scripts * [QOL-4572] add scheduled jobs to crontab * [QOL-4610] configure Apache to allow access to DataPusher * [QOL-4495] properly handle extensions with 'ckanext' partway through their names * [QOL-4531] use verified sender email address * [QOL-4570][QOL-4571] set site title properly so test automation can work * [QOL-4495] provide fallbacks when searching for stacks and apps - this will allow us to use either app-specific stacks, or a shared stack * [QOL-4495] oops use regex instead of plain string for pattern matching * [QOL-4531] send error emails to Online Products - previous address bounced, which is not OK * [QOL-4495] clean up file permissions to ensure we can successfully rebuild frontend files * [QOL-4495] adjust Solr index rebuild to run non-destructively and unconditionally * [QOL-4531] disable error emails to match existing environment, and send from our domain * [QOL-4495] drop obsolete Redis recipes * [QOL-4495] use private addressing to connect to Solr and DataPusher - we don't want to have public IPs for our EC2 instances, only for the load balancer and CDN * [QOL-4495] only reindex missing packages on CKAN creation - this allows us to start new instances much faster, while still picking up changes if Solr has been recreated * [QOL-4570][QOL-4571] write logs to separate directories per instance - this is necessary to properly support high-availability configurations, otherwise they write to the same log files and clash * [QOL-4495] query CloudFront and Route 53 to determine the best public domain name on server start - use the public DNS name if available, else CloudFront distribution domain name if available, else leave it untouched (load balancer domain name) * [QOL-4495] reload Apache after configuring site_url * [QOL-4495] restart Apache, not just reload * [QOL-4610] don't upgrade bleach beyond what CKAN needs - upgrading triggers ckan/ckan#3968 * [QOL-4639] extend logrotate configuration to cover our subdirectories * QOL-4495 update smtp relay for override and ssm paramater store lookup * [QOL-4495] install Java 8 rather than 6 or 7 - pre-8 versions are obsolete * [QOL-4495] restart SMTP relay upon setup if it already exists * [QOL-4495] enable SSM in AWS SMTP relay * [QOL-4495] use Brisbane timezone * [QOL-4531] remove Java 7 to ensure we use 8 * [QOL-4570] fix AWS query expression when an alias actually exists * [QOL-4570] silence output from cron jobs so they don't email us * [QOL-4570] drop unnecessary and broken cron job to manage admins * [QOL-4570][QOL-4571] remove obsolete cron job * [QOL-4659] only run CKAN cron jobs on one instance - instances will coordinate by logging their name and instance ID on the EFS while they are running. Each instance can then examine the list of active instances and use a consistent algorithm to pick a job server, eg the instance with the highest number.
ThrawnCA
referenced
this issue
in qld-gov-au/opswx-ckan-cookbook
Apr 30, 2019
…-ckan-extensions to develop * commit '535683515337f041b6c8522f421c60c7bfadd5cd': [QOL-4867] add prefixes for Salsa CKAN extensions
ThrawnCA
referenced
this issue
in qld-gov-au/opswx-ckan-cookbook
Apr 30, 2019
* [QOL-4495] revert most Solr conf changes, just add newer schema * [QOL-4495] provide Redis URL to handle CKAN 2.7+ * [QOL-4495] continue with CKAN installation if INI file doesn't exist, mitigates github #7 * [QOL-4495] oops close the 'unless' block', github #7 * [QOL-4495] oops missed a quote * [QOL-4495] update Redis configuration to use ElastiCache - update URLs to point to provided Redis - provide a unique site_id so multiple CKANs can share the cache * [QOL-4495] move EFS mount point to application-specific location * [QOL-4495] ensure that the app-specific EFS directory is created before mounting it * [QOL-4495] use the OpsWorks app ID as the CKAN site ID * [QOL-4495] use consistent app ID for mounting EFS - app_name isn't good enough since it's different for different OpsWorks apps * [QOL-4495] use app-specific names when looking up stack data * [QOL-4495] oops create directory before using it as a mount point * [QOL-4495] fix external site ID and add properties for QGOV content * [QOL-4495] remove default permissions to match existing CKAN * [QOL-4495] include environment in searches to ensure we identify the right application * [QOL-4495] oops change single quotes to double so we can use single quotes in our lookup * [QOL-4495] oops use correct variable to assemble Redis URL * [QOL-4495] improve guard condition on Solr install - We want to run the installation even if the data files already exist on EFS - TODO: Split the general installation work from the creation of the service file, so we can install the service on multiple machines while only setting up the data once. * [QOL-4495] increase bucket size to handle larger Nginx server names * [QOL-4495] split Solr data creation from service registration * [QOL-4495] drop redundant Solr config if needed * [QOL-4495] use the shared EFS directory to store web content - this is persistent across machines and already gives us write permissions * [QOL-4495] add QGOV extension to the list of known extensions and tighten the check that determines whether it's already installed * [QOL-4495] make /etc/ckan real and put a link under it - if /etc/ckan is a link, then we can't put other config under it, like the DataPusher settings * [QOL-4495] enable datastore if installing DataPusher * [QOL-4495] don't try to overwrite /etc/ckan/default symlink * [QOL-4495] use app ID, instead of version, to assemble the Solr domain name - otherwise Data and Publications step on each other * [QOL-4495] redirect HTTP traffic to HTTPS in Nginx - for unknown reasons, CKAN is sometimes redirecting to plain HTTP. Also, people may type the domain in their browsers without HTTPS. * [QOL-4531] replace SendMail with the AWS SMTP Relay - this allows us to send email using IAM roles instead of SMTP credentials * [QOL-4531] fix location of STMP relay files - put them all in an archive and retrieve it from wherever it's available to a standard location * [QOL-4495] fix database load - disable Python scripts since we want to manage databases ourselves eg migrating existing data - fix datastore usernames to match the standard CKAN pattern * [QOL-4495] update front-end CKAN resources after installing extensions - this wasn't needed in older CKAN versions * [QOL-4495] build the initial Solr index unless it's already done * [QOL-4495] fix path to web content * [QOL-4495] oops 'exists' check needs a question mark * [QOL-4495] don't try to install PostGIS since it will fail and we don't want it - could fix up the credentials so this can actually work, but there's no need * [QOL-4533] add recipes to set up the CKAN DataPusher - two parts: setting up the DataPusher server, and enabling the extension in CKAN. The first is meant to be tied to an OpsWorks layer. The second is tied to the CKAN layer but can be disabled. * [QOL-4533] cleanup - don't install unused DB setup scripts - replace deprecated File.exists with File.exist - trim trailing whitespace * [QOL-4531] set the email From address to match current - except that current always uses the production domain, whereas we will use whichever domain we're coming from * [QOL-4495] update metadata for forked cookbook * [QOL-4570][QOL-4571] run CKAN setup commands as non-root user when possible - this keeps directory ownership in the right hands - also extract more repeated paths to variables, and simplify the logic for when resources should run * [QOL-4495] extract more configuration file options to parameters with default values - we probably won't often want to override these, except maybe the feedback email address, but the option is there * [QOL-4495] oops fix syntax error when invoking virtualenv paster - we had a leftover dot from previous code that activated the virtualenv first - also replace more paths with variables * [QOL-4495] update CKAN config to better match existing settings - enable page tracking - enable activity streams and notifications - increase max resource size to 200MB - set desired locale order - configure Google Analytics if present - enable CORS * [QOL-4587] set Beaker secret in CKAN config file - leaving this as the hardcoded, publicly known version would be VERY bad * [QOL-4572] update tracking data before rebuilding search index * [QOL-4531] split SMTP relay archive and update init script to handle status return code properly - also throw away the unnecessary separate start and stop scripts * [QOL-4572] add scheduled jobs to crontab * [QOL-4610] configure Apache to allow access to DataPusher * [QOL-4495] properly handle extensions with 'ckanext' partway through their names * [QOL-4531] use verified sender email address * [QOL-4570][QOL-4571] set site title properly so test automation can work * [QOL-4495] provide fallbacks when searching for stacks and apps - this will allow us to use either app-specific stacks, or a shared stack * [QOL-4495] oops use regex instead of plain string for pattern matching * [QOL-4531] send error emails to Online Products - previous address bounced, which is not OK * [QOL-4495] clean up file permissions to ensure we can successfully rebuild frontend files * [QOL-4495] adjust Solr index rebuild to run non-destructively and unconditionally * [QOL-4531] disable error emails to match existing environment, and send from our domain * [QOL-4495] drop obsolete Redis recipes * [QOL-4495] use private addressing to connect to Solr and DataPusher - we don't want to have public IPs for our EC2 instances, only for the load balancer and CDN * [QOL-4495] only reindex missing packages on CKAN creation - this allows us to start new instances much faster, while still picking up changes if Solr has been recreated * [QOL-4570][QOL-4571] write logs to separate directories per instance - this is necessary to properly support high-availability configurations, otherwise they write to the same log files and clash * [QOL-4495] query CloudFront and Route 53 to determine the best public domain name on server start - use the public DNS name if available, else CloudFront distribution domain name if available, else leave it untouched (load balancer domain name) * [QOL-4495] reload Apache after configuring site_url * [QOL-4495] restart Apache, not just reload * [QOL-4610] don't upgrade bleach beyond what CKAN needs - upgrading triggers ckan/ckan#3968 * [QOL-4639] extend logrotate configuration to cover our subdirectories * QOL-4495 update smtp relay for override and ssm paramater store lookup * [QOL-4495] install Java 8 rather than 6 or 7 - pre-8 versions are obsolete * [QOL-4495] restart SMTP relay upon setup if it already exists * [QOL-4495] enable SSM in AWS SMTP relay * [QOL-4495] use Brisbane timezone * [QOL-4531] remove Java 7 to ensure we use 8 * [QOL-4570] fix AWS query expression when an alias actually exists * [QOL-4570] silence output from cron jobs so they don't email us * [QOL-4570] drop unnecessary and broken cron job to manage admins * [QOL-4570][QOL-4571] remove obsolete cron job * [QOL-4659] only run CKAN cron jobs on one instance - instances will coordinate by logging their name and instance ID on the EFS while they are running. Each instance can then examine the list of active instances and use a consistent algorithm to pick a job server, eg the instance with the highest number. * [QOL-4495] ensure lowercase domain names when consulting Route 53 * [QOL-4639] ship Apache and Nginx logs to S3 daily or when instance is shut down * [QOL-4531] bind the AWS SMTP relay to a different local address - this allows the default MTA to continue handling system emails like cron * [QOL-4570][QOL-4571] fix Solr user initialisation - use numeric GID instead of string to ensure no ambiguity * [QOL-4570][QOL-4571] update the Route53 zone ID on every startup since it can theoretically change * [QOL-4570][QOL-4571] update the hostname list on every startup since it can theoretically change * [QOL-4570][QOL-4571] update the DataPusher hostname list on every startup since it can theoretically change * [QOL-4694] patch DataPusher to use day-first date parsing * [QOL-4697] increase DataPusher file size limit to 200MB * [QOL-4639] clean up logging - point DataPusher log at the Apache directory since it seems to be a partial copy of the Apache log - move Apache log configuration into its own recipes so CKAN and DataPusher can share them - move some low-side-effect blocks into 'configure' recipes so they can be re-run more easily * [QOL-4639] archive Solr logs to S3 daily and on server shutdown * [QOL-4639] oops install packages before assigning EFS folder ownership - the 'httpd' package creates the 'apache' account, which we need * [QOL-4749] customise Apache to match Emantra config - Except trying to prevent Fanstatic from XSS, since it's no longer vulnerable * [QOL-4639] oops update Apache log directory ownership so non-root processes can browse and write to it - the DataPusher fails otherwise * [QOL-4790] update CKAN extensions to the selected revision on each deployment * [QOL-4694] more comprehensive date parsing patch - when setting the dayfirst flag, we need to specifically handle ISO dates * [CHG-4659] include stack params in DataPusher recipes even when split from other recipes * [QOL-4818] refactor recipes to better accommodate deployments - extract Nginx config files into their own recipe file - move CKAN maintenance jobs and Apache config files from configure to deploy - move tracking and search index rebuilds to configure so we don't have to run them on every deploy - update to selected CKAN and DataPusher revisions on deployment - move user, group, and most directory creation to setup * [QOL-4639] archive system logs to S3 - also improve S3 prefixes to be more human-readable * [QOL-4818] deploy theme if app definition is present * [QOL-4818] update DB schema if needed - this task is quick and idempotent, so it's worth running on each deployment. This should make upgrades easier. * [QOL-4818] fix group ownership for filestore directories * [QOL-4867] add prefixes for Salsa CKAN extensions * [QOL-4818] refactor recipes to make deployments easier - DNS updates and config files typically go in 'configure' recipes - Tracking and Solr index rebuilds should not be run automatically * [QOL-4818] oops fix recipes - Solr configure recipe needs the service name to be specified - DataPusher deployment needs to set up the standard stack parameters * [QOL-4415] push stats to enable CloudWatch monitoring
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The
ckanweb_deployrecipe checks for the presence of a file from the CKAN source repository, and uses that to govern a large portion of the CKAN installation process; requirements installation, CKAN egg installation, INI setup, database initialisation, datastore initialisation. This means that failed installations typically cannot be re-run without wiping the disk.The text was updated successfully, but these errors were encountered: