Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rundeck Docker: MySQL fail to start #132

Open
happicamper opened this issue Jun 19, 2018 · 13 comments
Open

Rundeck Docker: MySQL fail to start #132

happicamper opened this issue Jun 19, 2018 · 13 comments

Comments

@happicamper
Copy link

Greetings!
I'm unable to start my docker service since the upgrade of the image. I've saw an upgrade notes which has to alter a table in Mysql but I'm only using Rundeck Docker image and run it using an embedded mysql inside. I'm trying to alter that table but mysql fail to start and exits the container/service.
Any help will be greatly appreciated, thanks!
@jjethwa
Copy link
Owner

jjethwa commented Jun 19, 2018

Hi @happicamper

Sorry to hear that. What version are you upgrading from? 😞

@happicamper
Copy link
Author

It's also my fault that my script will fetch the latest image whenever the service is down. If I'm not mistaken it's from 2.11.3 upgrade to 2.11.4 broke my setup.

@jjethwa
Copy link
Owner

jjethwa commented Jun 19, 2018

Hi @happicamper

I don't think there is a schema update for 2.11.3 -> 2.11.4 (See: http://rundeck.org/docs/upgrading/index.html )

When starting the container, does MariaDB start properly?

@happicamper
Copy link
Author

@jjethwa Hello, yea it seems I'm not in that version. I've tried backward versions but still the docker service mysql isn't working. Where can I dig logs for this mysql failure? Thankful as always on your responses.

@jjethwa
Copy link
Owner

jjethwa commented Jun 20, 2018

Hi @happicamper

You should be able to see the logs under /var/logs/mysql inside the container. Also, the container log should show the supervisor output.

sudo docker logs <CONTAINER_NAME>

Hoping that gives us a clue 😃

@happicamper
Copy link
Author

@jjethwa unfortunately, this container only displays:

=>Installing plugins from /opt/rundeck-plugins
=>Initializing rundeck - This may take a few minutes
=>/etc/rundeck check OK
=>Initializing local MySQL...
[....] Starting MySQL database server: mysqld . . . . . . .[FAIL. . . . . . . . . . . . . . . . . . . . . failed!

And whenever I get inside the docker, cause its failing, it will force me to log out which prevents me to investigate inside the mysql. Though I have a volume of mysql.
If I may, Let me post my run command here:

docker service create \
    --name=$RUNDECK \
    -t \
    --env EXTERNAL_SERVER_URL=https://lan-rundeck.toroserver.com \
    --env RUNDECK_WITH_SSL=true \
    --env TZ=Asia/Manila \
    --publish 7443:4443 \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/etc/rundeck,destination=/etc/rundeck \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/rundeck,destination=/var/rundeck \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/rundeck/,destination=/var/lib/rundeck/.ssh \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/mysql/,destination=/var/lib/mysql \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/var/log/rundeck,destination=/var/log/rundeck \
    --mount type=bind,source=/datastore/lan-resources/volumes/rundeck/opt/rundeck-plugins,destination=/opt/rundeck-plugins \
    --mount type=bind,source=type=bind,source=/datastore/lan-resources/volumes/rundeck/var/lib/rundeck/var/storage,destination=/var/lib/rundeck/var/storage \
    --mount type=bind,source=/etc/localtime,destination=/etc/localtime:ro \
    --reserve-cpu 1 \
    --reserve-memory 1gb \
    --limit-cpu 1 \
    --limit-memory 1gb \
    jordan/rundeck:latest

@jjethwa
Copy link
Owner

jjethwa commented Jun 20, 2018

Hi @happicamper

Do you mind adding a volume for /var/logs/mysql ?

@happicamper
Copy link
Author

Hello @jjethwa, upon checking the container, I got this under /var/log/mysql:

root@7c104115e030:/var/log/mysql# cat error.log
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2018-06-08  8:51:15 140506259280448 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2018-06-08  8:51:16 140506259280448 [Warning] InnoDB: New log files created, LSN=45883
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Doublewrite buffer not found: creating new
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Doublewrite buffer created
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:16 140506259280448 [Warning] InnoDB: Creating foreign key constraint system tables.
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Foreign key constraint system tables created
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Creating tablespace and datafile system tables.
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Tablespace and datafile system tables created.
2018-06-08  8:51:16 140506259280448 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:16 140506259280448 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 0
2018-06-08  8:51:16 140505622996736 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:19 139826199306816 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:19 139826199306816 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616799
2018-06-08  8:51:19 139825562740480 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:22 139806483702336 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:22 139806483702336 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616809
2018-06-08  8:51:22 139805849511680 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: The InnoDB memory heap is disabled
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Using Linux native AIO
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Using SSE crc32 instructions
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Completed initialization of buffer pool
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Highest supported file format is Barracuda.
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: 128 rollback segment(s) are active.
2018-06-08  8:51:25 139986440979008 [Note] InnoDB: Waiting for purge to start
2018-06-08  8:51:25 139986440979008 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 1616819
2018-06-08  8:51:25 139986440979008 [Note] Plugin 'FEEDBACK' is disabled.
2018-06-08  8:51:25 139985806124800 [Note] InnoDB: Dumping buffer pool(s) not yet started

@jjethwa
Copy link
Owner

jjethwa commented Jun 20, 2018

Hmm, that's out of my expertise, but I'll see what I can find. Are you able to start a fresh container without problems? (Try with no volumes)

@happicamper
Copy link
Author

@jjethwa It seems, it's running well without any volumes which means it's on my setup right? Is possible that I can migrate my jobs into a new container without the mysql files?

@jjethwa
Copy link
Owner

jjethwa commented Jun 20, 2018

@happicamper That depends on your setup. By default, jobs are stored in the database as well. There are export/import configs for SCM like git. You could also export the jobs using the CLI tool or export projects the UI.

I still need to comb through the mysql logs to see if I can find the smoking gun, but I would try to clone your volumes and try running with an older version of the container first or try some repair tools. Perhaps it's just some corruption that can easily be fixed.

@happicamper
Copy link
Author

@jjethwa Sorry for bugging you around 😆. I'm planning to move my Rundeck server into a standalone server instead of docker because of resources constraints and concerns. I've tried exporting projects through XMLs and get back to you whatever results will be.

@jjethwa
Copy link
Owner

jjethwa commented Jun 20, 2018

No problem at all, @happicamper The XML format will work. You'll be able to import via the UI or CLI tool 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jjethwa @happicamper