FSND-P5_Linux-Server-Configuration

A baseline installation of Ubuntu Linux on a virtual machine to host a Flask web application. This includes the installation of updates, securing the system from a number of attack vectors and installing/configuring web and database servers.

Note: The below step-by-step walkthrough is the solution to project 5 of the Udacity Full Stack Web Developer Nanodegree and deploys the solution of project 3 on the virtual machine. The solution is graded with "Exceeds Specifications".

Issues

• The web application is not reachable under the AWS-server instance anymore. Should be shifted to my own server.

Step by Step Walkthrough

In the detailed guide below I tried to briefly document the steps I executed to get to the projects solution. The numbering and headlines should be congruent to the projects tasks 1-11. Please read through the linked sources to fully understand what is done in those steps. The walkthrough is surely far from perfect and I'm thankful for any feedback, reported errors, advice, etc. I get on this.

The parts which are not necessarily needed are marked with a *, the ones which only need to be done for extra credit ('suceeds expectations') are marked with **. I used vim as text editor in this guide, but you can surely use another one like nano. Have fun!

1 & 2 - Create Development Environment: Launch Virtual Machine and SSH into the server

Source: Udacity

1. Create new development environment.
2. Download private keys and write down your public IP address.
3. Move the private key file into the folder ~/.ssh:
   $ mv ~/Downloads/udacity_key.rsa ~/.ssh/
4. Set file rights (only owner can write and read.):
   $ chmod 600 ~/.ssh/udacity_key.rsa
5. SSH into the instance:
   <pre>$ ssh -i ~/.ssh/udacity_key.rsa root@PUPLIC-IP-ADDRESS

3 & 4 - User Management: Create a new user and give user the permission to sudo

Source: DigitalOcean

1. Create a new user:
   $ adduser NEWUSER
2. Give new user the permission to sudo
3. Open the sudo configuration:
   $ visudo
4. Add the following line below root ALL...:
   NEWUSER ALL=(ALL:ALL) ALL
5. *List all users (Source: Ask Ubuntu):
   $ cut -d: -f1 /etc/passwd

5 - Update and upgrade all currently installed packages

Source: Ask Ubuntu

1. Update the list of available packages and their versions:
   $ sudo apt-get update
2. Install newer vesions of packages you have:
   $ sudo sudo apt-get upgrade

5** - Include cron scripts to automatically manage package updates

Source: Ubuntu documentation

1. Install the unattended-upgrades package:
   $ sudo apt-get install unattended-upgrades
2. Enable the unattended-upgrades package:
   $ sudo dpkg-reconfigure -plow unattended-upgrades

6 - Change the SSH port from 22 to 2200 and configure SSH access

Source: Ask Ubuntu

1. Change ssh config file:

2. Open the config file:
   $ vim /etc/ssh/sshd_config

3. Change to Port 2200.

4. Change PermitRootLogin from without-password to no.

5. • To get more detailed logging messasges, open /var/log/auth.log and change LogLevel from INFO to VERBOSE.

6. Temporalily change PasswordAuthentication from no to yes.

7. Append UseDNS no.

8. Append AllowUsers NEWUSER.
   Note: All options on UNIXhelp

9. Restart SSH Service:
   $ /etc/init.d/ssh restart or # service sshd restart

10. Create SSH Keys:
    Source: DigitalOcean

11. Generate a SSH key pair on the local machine:
    $ ssh-keygen

12. Copy the public id to the server:
    $ ssh-copy-id username@remote_host -p**_PORTNUMBER_**

13. Login with the new user:
    $ ssh -v grader@PUBLIC-IP-ADDRESS -p2200

14. Open SSHD config:
    $ sudo vim /etc/ssh/sshd_config

15. Change PasswordAuthentication back from yes to no.

16. *Get rid of the warning message sudo: unable to resolve host ... when sudo is executed:
    Source: Ask Ubuntu

17. Open $ vim /etc/hostname.

18. Copy the hostname.

19. Append the hostname to the first line:
    $ sudo sudonano /etc/hosts

20. • Simplify SSH login:

21. Logout of the SSH instance:
    $ exit

22. Open the SSH config file on your local machine:
    $ sudo vim .ssh/config

23. Add the following lines:
    Host NEWHOSTNAME HostName PUPLIC-IP-ADDRESS Port 2200 User NEWUSER

24. Now, you can login into the server more quickly:
    $ ssh NEWHOSTNAME

25. *Handle the message System restart required after login:
    Source: Super User

26. List all packages which cause the reboot:
    $ cat /var/run/reboot-required.pkgs

27. List everything with high security issues:
    $ xargs aptitude changelog < /var/run/reboot-required.pkgs | grep urgency=high

28. If wanted or needed, reboot the system:
    $ sudo shutdown -r now
    Note: More info on rebooting on Ask Ubuntu.

7 - Configure the Uncomplicated Firewall (UFW) to only allow incoming connections for SSH (port 2200), HTTP (port 80), and NTP (port 123)

Source: Ubuntu documentation

1. Turn UFW on with the default set of rules:
   $ sudo ufw enable
2. *Check the status of UFW:
   $ sudo ufw status verbose
3. Allow incoming TCP packets on port 2200 (SSH):
   $ sudo ufw allow 2200/tcp
4. Allow incoming TCP packets on port 80 (HTTP):
   $ sudo ufw allow 80/tcp
5. Allow incoming UDP packets on port 123 (NTP):
   $ sudo ufw allow 123/udp

7** - Configure Firewall to monitor for repeated unsuccessful login attempts and ban attackers

Source: DigitalOcean

1. Install Fail2ban:
   $ sudo apt-get install fail2ban
2. Copy the default config file:
   $ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
3. Check and change the default parameters:
   1. Open the local config file:
      $ sudo vim /etc/fail2ban/jail.local
   2. Set the following Parameters:

     set bantime  = 1800  
     destemail = YOURNAME@DOMAIN  
     action = %(action_mwl)s  
     under [ssh] change port = 2220  
   

Note: In the next three steps iptables is installed. However, the before installed UFW is actually a frontend for iptables and is set up already. So configuring iptables separately (as I did by just following the guide at DigitalOcean) would be a redundant step. So just install sendmail and go on with step 7.

4. Install needed software for our configuration:
   $ sudo apt-get install sendmail iptables-persistent
5. Set up a basic firewall only allowing connections from the above ports:
   $ sudo iptables -A INPUT -i lo -j ACCEPT
$ sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
$ sudo iptables -A INPUT -p tcp --dport 2200 -j ACCEPT
$ sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
$ sudo iptables -A INPUT -p udp --dport 123 -j ACCEPT
$ sudo iptables -A INPUT -j DROP
6. *Check the current firewall rules:
   $ sudo iptables -S
7. Stop the service:
   $ sudo service fail2ban stop
8. Start it again:
   $ sudo service fail2ban start

8 - Configure the local timezone to UTC

Source: Ubuntu documentation

1. Open the timezone selection dialog:
   $ sudo dpkg-reconfigure tzdata
2. Then chose 'None of the above', then UTC.
3. *Setup the ntp daemon ntpd for regular and improving time sync:
   $ sudo apt-get install ntp
4. *Chose closer NTP time servers:
5. Open the NTP configuration file:
   $ sudo vim /etc/ntp.conf
6. Open http://www.pool.ntp.org/en/ and choose the pool zone closest to you and replace the given servers with the new server list.

9 - Install and configure Apache to serve a Python mod_wsgi application

Source: Udacity

1. Install Apache web server:
   $ sudo apt-get install apache2
2. Open a browser and open your public ip address, e.g. http://52.25.0.41/ - It should say 'It works!' on the top of the page.
3. Install mod_wsgi for serving Python apps from Apache and the helper package python-setuptools:
   $ sudo apt-get install python-setuptools libapache2-mod-wsgi
4. Restart the Apache server for mod_wsgi to load:
   $ sudo service apache2 restart
5. *Get rid of the message "Could not reliably determine the servers's fully qualified domain name" after restart Source: Ask Ubuntu
6. Create an empty Apache config file with the hostname:
   $ echo "ServerName HOSTNAME" | sudo tee /etc/apache2/conf-available/fqdn.conf
7. Enable the new config file:
   $ sudo a2enconf fqdn

11 - Install git, clone and setup your Catalog App project

As this is by far the biggest project task, it is split in several parts.

11.1 - Install and configure git

Source: GitHub

1. Install Git:
   $ sudo apt-get install git
2. Set your name, e.g. for the commits:
   $ git config --global user.name "YOUR NAME"
3. Set up your email address to connect your commits to your account:
   $ git config --global user.email "YOUR EMAIL ADDRESS"

11.2 - Setup for deploying a Flask Application on Ubuntu VPS

Source: DigitalOcean

1. Extend Python with additional packages that enable Apache to serve Flask applications:
   $ sudo apt-get install libapache2-mod-wsgi python-dev
2. Enable mod_wsgi (if not already enabled):
   $ sudo a2enmod wsgi
3. Create a Flask app:
4. Move to the www directory:
   $ cd /var/www
5. Setup a directory for the app, e.g. catalog:
   1. $ sudo mkdir catalog
   2. $ cd catalog and $ sudo mkdir catalog
   3. $ cd catalog and $ sudo mkdir static templates
   4. Create the file that will contain the flask application logic:
   $ sudo nano __init__.py 5. Paste in the following code:
   python from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Veni vidi vici!!" if __name__ == "__main__": app.run()
6. Install Flask
7. Install pip installer:
   $ sudo apt-get install python-pip
8. Install virtualenv:
   $ sudo pip install virtualenv
9. Set virtual environment to name 'venv':
   $ sudo virtualenv venv
10. Enable all permissions for the new virtual environment (no sudo should be used within):
    Source: Stackoverflow
    $ sudo chmod -R 777 venv
11. Activate the virtual environment:
    $ source venv/bin/activate
12. Install Flask inside the virtual environment:
    $ pip install Flask
13. Run the app:
    $ python __init__.py
14. Deactivate the environment:
    $ deactivate
15. Configure and Enable a New Virtual Host#
16. Create a virtual host config file
    $ sudo nano /etc/apache2/sites-available/catalog.conf
17. Paste in the following lines of code and change names and addresses regarding your application:

  <VirtualHost *:80>
      ServerName PUBLIC-IP-ADDRESS
      ServerAdmin admin@PUBLIC-IP-ADDRESS
      WSGIScriptAlias / /var/www/catalog/catalog.wsgi
      <Directory /var/www/catalog/catalog/>
          Order allow,deny
          Allow from all
      </Directory>
      Alias /static /var/www/catalog/catalog/static
      <Directory /var/www/catalog/catalog/static/>
          Order allow,deny
          Allow from all
      </Directory>
      ErrorLog ${APACHE_LOG_DIR}/error.log
      LogLevel warn
      CustomLog ${APACHE_LOG_DIR}/access.log combined
  </VirtualHost>

3. Enable the virtual host:
   $ sudo a2ensite catalog
4. Create the .wsgi File and Restart Apache
5. Create wsgi file:
   $ cd /var/www/catalog and $ sudo vim catalog.wsgi
6. Paste in the following lines of code:

  #!/usr/bin/python
  import sys
  import logging
  logging.basicConfig(stream=sys.stderr)
  sys.path.insert(0,"/var/www/catalog/")
  
  from catalog import app as application
  application.secret_key = 'Add your secret key'

7. Restart Apache:
   $ sudo service apache2 restart

11.3 - Clone GitHub repository and make it web inaccessible

1. Clone project 3 solution repository on GitHub:
   $ git clone https://github.com/stueken/FSND-P3_Music-Catalog-Web-App.git
2. Move all content of created FSND-P3_Music-Catalog-Web-App directory to /var/www/catalog/catalog/-directory and delete the leftover empty directory.
3. Make the GitHub repository inaccessible:
   Source: Stackoverflow
4. Create and open .htaccess file:
   $ cd /var/www/catalog/ and $ sudo vim .htaccess
5. Paste in the following:
   RedirectMatch 404 /\.git

11.4 - Install needed modules & packages

1. Activate virtual environment:
   $ source venv/bin/activate
2. Install httplib2 module in venv:
   $ pip install httplib2
3. Install requests module in venv:
   $ pip install requests
4. *Install flask.ext.seasurf (only seems to work when installed globally):
   $ *sudo pip install flask-seasurf
5. Install oauth2client.client:
   $ sudo pip install --upgrade oauth2client
6. Install SQLAlchemy:
   $ sudo pip install sqlalchemy
7. Install the Python PostgreSQL adapter psycopg:
   $ sudo apt-get install python-psycopg2

10 - Install and configure PostgreSQL

Source: DigitalOcean (alternatively, nice short guide on Kill The Yak as well)

1. Install PostgreSQL:
   $ sudo apt-get install postgresql postgresql-contrib
2. Check that no remote connections are allowed (default):
   $ sudo vim /etc/postgresql/9.3/main/pg_hba.conf
3. Open the database setup file:
   $ sudo vim database_setup.py
4. Change the line starting with "engine" to (fill in a password):
   python engine = create_engine('postgresql://catalog:PW-FOR-DB@localhost/catalog')
5. Change the same line in application.py respectively
6. Rename application.py:
   $ mv application.py __init__.py
7. Create needed linux user for psql:
   $ sudo adduser catalog (choose a password)
8. Change to default user postgres:
   $ sudo su - postgre
9. Connect to the system:
   $ psql
10. Add postgre user with password:
    Sources: Trackets Blog and Super User
11. Create user with LOGIN role and set a password:
    # CREATE USER catalog WITH PASSWORD 'PW-FOR-DB'; (# stands for the command prompt in psql)
12. Allow the user to create database tables:
    # ALTER USER catalog CREATEDB;
13. *List current roles and their attributes: # \du
14. Create database:
    # CREATE DATABASE catalog WITH OWNER catalog;
15. Connect to the database catalog # \c catalog
16. Revoke all rights:
    # REVOKE ALL ON SCHEMA public FROM public;
17. Grant only access to the catalog role:
    # GRANT ALL ON SCHEMA public TO catalog;
18. Exit out of PostgreSQl and the postgres user:
    # \q, then $ exit
19. Create postgreSQL database schema:
    $ python database_setup.py

11.5 - Run application

1. Restart Apache:
   $ sudo service apache2 restart
2. Open a browser and put in your public ip-address as url, e.g. 52.25.0.41 - if everything works, the application should come up
3. *If getting an internal server error, check the Apache error files:
   Source: A2 Hosting
4. View the last 20 lines in the error log: $ sudo tail -20 /var/log/apache2/error.log
5. *If a file like 'g_client_secrets.json' couldn't been found:
   Source: Stackoverflow

11.6 - Get OAuth-Logins Working

Source: Udacity and Apache

1. Open http://www.hcidata.info/host2ip.cgi and receive the Host name for your public IP-address, e.g. for 52.25.0.41, its ec2-52-25-0-41.us-west-2.compute.amazonaws.com
2. Open the Apache configuration files for the web app: $ sudo vim /etc/apache2/sites-available/catalog.conf
3. Paste in the following line below ServerAdmin:
   ServerAlias HOSTNAME, e.g. ec2-52-25-0-41.us-west-2.compute.amazonaws.com
4. Enable the virtual host:
   $ sudo a2ensite catalog
5. To get the Google+ authorization working:
6. Go to the project on the Developer Console: https://console.developers.google.com/project
7. Navigate to APIs & auth > Credentials > Edit Settings
8. add your host name and public IP-address to your Authorized JavaScript origins and your host name + oauth2callback to Authorized redirect URIs, e.g. http://ec2-52-25-0-41.us-west-2.compute.amazonaws.com/oauth2callback
9. To get the Facebook authorization working:
10. Go on the Facebook Developers Site to My Apps https://developers.facebook.com/apps/
11. Click on your App, go to Settings and fill in your public IP-Address including prefixed hhtp:// in the Site URL field
12. To leave the development mode, so others can login as well, also fill in a contact email address in the respective field, "Save Changes", click on 'Status & Review'

11.7** - Install Monitor application Glances

Sources: Glances and Web Host Bug

1. $ sudo apt-get install python-pip build-essential python-dev
2. $ sudo pip install Glances
3. $ sudo pip install PySensors