Exercise_01_RPrimer.Rmd

---
title: 'Lab 1 - R Primer'
author: "EE509"
output: html_document
---

## Objectives

The objective of today's hands-on activity is to provide a basic overview of R.  For each topic covered the first part will be directed – you will follow the prescribed sequence of R commands in order to familiarize yourself with what they do.  The second part of each topic will ask you to apply these R commands. 



# Assignment

For this activity you will turn in a Rmd file that shows your work.

**Always check that the Rmd file "knits" before submitting!**

All Rmd documents should start with a section that looks something like this:

```
---
title: "My Lab"
author: "My Name"
output: html_document
---
```
Make sure to give your file a title and put ***your name*** in the author field. Notice that the title and author need to be in quotes. 


### Knitting Rmarkdown documents

To be able to compile Rmd into HTML or any other format you will need to install the "knitr" library. 

If you do not already see a button at the top of the editor window that says "Knit" with an icon of a blue ball of yarn next to it, you will need to install this package. 

![](img/Knit_button.png){width=100px}

- From the "Packages" tab in the bottom right pane click on "Install"
- Enter knitr as the package name, and then click Install
- From the main menu bar, select Session > Restart R
- The Knit button should now be visible

If you do have the Knit button, now you should click it. This will create a new file, that has the same name as the Rmd file, except it ends in `.html`. Rstudio should automatically preview this file for you. 
You will see that it contains exactly the same content as the Rmd file, but formatted nicely in to an east to read document. You are free to keep reading the assignment in either the Rmd file or the html file, but **you can only edit the Rmd file**. When you edit the Rmd file, your changes will not show up in the html file until you select Knit. **Kitting is the best way to check that all your code runs.**

*NOTE: this document will not knit until after Question 10, when the variable `met` is defined *

###  Code Answers

All your code should be placed into what we call "R chunks."

In the editor pane you can recognize them because: 

They start with: ` ```{r} ` (the `{r}` is very important because it's what makes it specifically an R chunk, as opposed to a different type of code chunk)

They end with ` ``` `

R code chunks also have in the top right hand corner three important icons (from left to right)
![](img/R_chunk.png){width=100px}

1. Optional Settings. Once you get more comfortable using R, they can be useful.
2. Run the code in all chunks up to this one.
3. Run the code in this chunk.

Below is an R chunk. Try hitting the play button. 
You will see that the output appears below the code chunk. 
```{r}
1 + 2
```

### Written Answers 

Any verbal answers should be written in the space between code chunks. You can also include some of your written responses as commented code within the R chunk. Comments start with `#` and are not evaluated by R. Putting comments in code is very useful for remembering what you did when you come back to a file later.

There are multiple ways to format written answers and Rmarkdown is very versatile. What you do comes down to personal preference. 

**Always check that the Rmd file "knits" before submitting!**

### Example of a properly answered question 

1. **If there are 7 cats and 5 dogs, how many animals are there in total?**

```{r}
cats = 7 # there are 7 cats
dogs = 5 # there are 5 dogs
animals = cats + dogs # the sum of cats and dogs gives us the total animals

animals # this will print out the number of animals
```
From the calculations above, we can see that there are 12 animals in total. 

You can also use `` `r` `` to reference calculations within your text , which has the advantage that you don't need to update your text if specific numbers change. For example:

We can see that there are `r animals` animals in total.

# R Basics

## About R

The R software we are using this semester is open-source statistical software that has gained rapid popularity because of its power and flexibility.  In addition, there are a large number of “packages” for R that have been written by users and are freely downloadable from [CRAN](http://cran.us.r-project.org/) (Comprehensive R Archive Network).  Individual packages do everything from allowing R to interface with supercomputers to solving sudoku puzzles.  They contain most every classical statistical test you're likely to come across as well as interfaces that allow R to interact with a large number of other programs and software libraries.  Unlike many pieces of software you may be familiar with, R is a scripting language.  Usually you will be using R “interactively” which means that the basic mode of operation is to type commands at a command prompt and have it spit back a result, which you'll often want to cut-and-paste elsewhere.  R can also be run in “batch” mode, whereby a file containing a list of R commands is run all at once.  This mode is particularly useful for large analyses that take a long time to run because batch jobs can be submitted to computer clusters.

## Getting this exercise

This exercise is available off of http://github.com/mdietze and assumes that you will be working from within the RStudio editor with git installed. This requires a few steps:

0. Before installing RStudio, you'll want to make sure you have [installed R](http://cran.us.r-project.org/)
1. RStudio can be downloaded for free from http://www.rstudio.com/
2. You will need to install Git, which you can do by following the instructions at [RStudio Support - Version Control with Git and SVN](https://support.rstudio.com/hc/en-us/articles/200532077-Version-Control-with-Git-and-SVN).
3. You will need to make sure RStudio knows where git is installed. 
  + In RStudio click on Tools > Global Options > Git/SVN
  + Make sure the "Git executable" path is set to where git was installed 
  + Make sure "Enable version control interface" is checked
4. You will need to introduce yourself to git.
  + From RStudio click Tools > Shell
  + To set your name, enter the following & hit Return: 
```
git config -–global user.name “your name”
```
  + To set your email, enter the following & hit Return: 
```
git config -–global user.email “your.email@gmail.com”
```
  + You can now close this Shell window
  
The bestest way to get a copy of this exercise, and all the other course exercises, is to click on the project pull-down menu in the top-right corner of RStudio and select "New Project...".  

Next, click "Version Control" and then "Git".

Enter the following address in the Repository URL: https://github.com/mdietze/EE509.git

You can optionally choose to name the folder something different than EE509 (though be aware that many activities will assume this is the name of the folder) or have the folder saved somewhere other than your Home directory.

When you hit "Create Project" this will clone a copy of the git project from the github.com website.

If you anticipate making changes to a git project on GitHub then, instead of cloning from https://github.com/mdietze/EE509 you should go to that website and click on the "Fork" button in the top right corner. This requires that you have an account on GitHub, which is free and easy to set up. Once you fork the repository, this will make a copy to https://github.com/username/EE509 where _username_ is your GitHub username, and you can use that as your Repository URL.

If you just can't get Git to work, you can download this activity from the same website, https://github.com/mdietze/EE509, by clicking on the "Download ZIP" button in the bottom right.

Next, to be able to compile this document into HTML or any other format you will need to install the "knitr" library. From the "Packages" tab in the bottom right window click on "Install", enter knitr as the package name, and then click Install

Finally, go to File > Open File and open this file, "Exercise_01_Rprimer.Rmd" and then click "Knit HTML" in top left window's menu bar.

## What R has to offer

Through your web-browser go to http://www.r-project.org.  

* Please briefly look over the “What is R?” section.

*	Next, go to the “Manuals” section.  This section gives an overview of some of the on-line documentation you may want to use from time to time to gain a greater understanding of how R works and to solve problems you come across.  Some of today's activity is borrowed from the “Introduction to R”.  You are encouraged to read this section later on your own, especially if you have no previous familiarity with R or any other programming language.
*	Next, go to the “Search” section and in the “Google” box type in “mantel test.”  You'll find this gives you a list of different R packages that have different forms of mantel tests (a test of correlation between two matrices).  Press the back button and this time click on “Searchable mail archives” and again type “mantel test” in the query box.  This time you will see any discussions from the R email list about Mantel's tests.  These searches are often very useful if you're looking for an example or have run into a problem (because often someone else has had the same problem)
*	Next, go to the “CRAN” section, and select one of the sites – it doesn't matter which one since they are identical “mirrors” of each other.
*	Click on “Task Views” and then select one of the “tasks” that you find interesting.  You will then be presented with a brief overview of major subtopics within that area and the R packages that are useful for those types of problems.  These summaries are often an efficient way to familiarize yourself with what R has to offer for a specific type of analysis but are not exhaustive because new packages are constantly being added to R and not all types of analysis have a “Task View”.
*	Click on “Packages” and then “Table of available packages” and you will see a long list of all of the submitted R packages.  Look around a bit and then click on one you find interesting.  Here you will see basic info on the package including the “Depends” which is the list of packages that you need in order to use this package.  Click on the PDF “Reference manual”.  You'll find that the R documentation for a package describes the inputs and outputs of all the functions in the package and often provides examples of what code calling this function might look like.  However, most packages don't describe how a test works, its assumptions, or why you might want to use it – you'll usually want to look up info about a test rather than apply it blindly.



## R basics

Lets see how R does some basic arithmetic. Note that in the examples below that a pound sign (#) indicates a comment in the code. Everything after a # is not read by R and is just there for the benefit of the person reading the code (so you don’t have to type it all in). Within the Console window (bottom left) type the following at the command prompt symbol '>':

```
3+12  		    ## addition
5 - pi			  ## subtraction
2*8			      ## multiplication
14/5			    ## division
14 %/% 5		  ## integer division
14 %% 5		    ## modulus (a.k.a. remainder)
sqrt(25)		  ## square root
z = 5			    ## assignment to a variable
z			        ## value of a variable
y = sqrt(36)	## square root
y
```
 
The window should show

```{r, echo=TRUE}
3+12    	    ## addition
5 - pi			  ## subtraction
2*8			      ## multiplication
14/5			    ## division
14 %/% 5		  ## integer division
14 %% 5		    ## modulus (a.k.a. remainder)
sqrt(25)		  ## square root
z = 5			    ## assignment to a variable
z			        ## value of a variable
y = sqrt(36)	## square root
y
```
  
The number [1] before the answers just means that this item is the first element of a vector (vectors can be thought of as a collection of related values, such as a column in a data table). From these examples we can see a few things about R. First is that it knows the value of common constants (e.g. π) and can use them like numbers. Second, that we can assign both constants and the results of functions to variables and we can see the value of a variable by entering it at the command prompt. This also demonstrates “sqrt”, the square root function, and that values are passed to functions within parenthesis.  Typically, values passed to function are referred to as the arguments of the functions. Make sure you understand what each operator (+, -, *, /, %%, %/%) does before you proceed.  Some of the other common mathematical operators in R that you should try running are:

```
10^2  			        ## power
exp(1)				      ## exponential
log(10)			        ## natural logarithm (i.e. ln)
log10(10)			      ## log base 10
log2(10)            ## log base 2
sin(pi/2)			      ## sine
cos(pi)			        ## cosine
tan(pi/4)			      ## tangent
asin(0.5)			      ## arc-sine
acos(0.5)			      ## arc-cosine
atan(1)*180/pi			## arc-tangent
atan2(-1,-1)*180/pi	## arc-tangent (alternate version)
factorial(10)			  ## factorial
``` 
Note that the atan2 function takes TWO arguments. In R there are many functions that can take more than one argument, and these arguments are always separated by commas. To get information on a function in R you can precede the function name with a ?.  So if we wanted to look up what the two arguments of the atan2 function are, we could type

```
?atan2
```
 
The rest of this activity will often list the names of functions you might find useful, and ? can be used to find out about the syntax of these functions.
If on the other hand you are looking for a command (e.g. because you've forgotten its name) you can use help.search. For example:

```
help.search("mantel")
```
 
This will return a list of relevant functions with the parenthesis after the name indicating what package it is in.  help.search will only search packages you have already installed, not all the ones that exist -- you'll want to use CRAN to find new packages.  

Within RStudio you can also search for help with commands in the **Help** tab in the bottom-right window. This search window combines the functionality of both ? and help.search.

To get new functions in packages that you do not currently have installed, you will have to look up the package you want to use (e.g., on CRAN) and then install it. Packages are installed using the ‘Install Packages’ button under the “Packages” tab or by using 'install.packages' function on the command line. Packages are loaded by clicking the check box next to a packages name or by using the 'library' function at the command line.  Packages only need to be installed once but need to be loaded every time you start R (hint: you'll probably want to list library commands near the top of your script files [described below])

### Questions:

```
1.	Evaluate the following:
   a.	ln(1)
   b.	ln(0)
   c.	ln(e)
   d.	ln(-5)
   e.	-ln(5)
   f.	ln(1/5)
   g.	How does R represent when the output of a function is not a number?
   
2.	Looking back at the Triginometry section, hy are we multiplying the results of the atan and atan2 functions by 180/pi? 

3.	What is the difference between log and log10? (hint: use `help()`)

4.	Given a right triangle with sides x = 5 and y = 13, calculate the length of the hypotenuse (show code and use variables)

5.	Subtract the month you were born from the remainder left after you divide the year you were born by the day you were born (show code)
```


## R Scripts

Now click on File>New File>R Script to open up a new script window.  It is often useful to work on R from a script window (or a new Rmd) because it provides a record of what you did in your analysis and can be reused for similar analyses.  It is particularly essential for more complicated analyses. From the File > New File menu you'll also see that R Studio supports a wide range of R file formats (R scripts, Rmd, R notebooks, Quarto) as well as a wide range of other programing languages.

In the script window type
 
```
x = 1:10
x
#y
```
 
Unlike at the command prompt nothing happens when you hit return at the end of a line. Highlight the code and hit the Run button.  At the command line you should see 
 
```{r}
x = 1:10
x
#y
```
 
Here the result, a vector of ten values from 1 to 10, demonstrates the basic R syntax for creating a sequence of numbers.  This example also demonstrates that the comment character '#' also works in scripts.  Putting comments in scripts is very useful for remembering what you did when you come back to a file later.  For the rest of this activity I'll use boxes to indicate text to be typed in and run, and will use > to indicate that it should be typed on the command prompt and no prompt to indicate that you probably want to type it in a script or Rmd instead.
	There are a number of other ways of generating useful sequences besides the ':' syntax
 
```{r}
seq(1,10,by=0.5)
seq(1,by=0.5,length=10)
rep(1,10)
x=seq(0,3,by=0.01)
```
 
In all cases you need to provide the first value in a sequence (the first argument to seq and rep), and after that you need to provide some combination of step size (by), length, and finishing value. 
Most functions in R can be applied to vectors of data, not just individual data points.  Indeed, many only make sense when applied to vectors, such as the following that calculate sums, first differences, and cumulative sums.
 
```{r}
sum(1:10)  	## sum up all values in a vector
diff(1:10)		## calculate the differences between adjacent values in a vector 
cumsum(1:10)		## cumulative sum of values in a vector
prod(1:10)		## product of values in a vector
```
 
```
Questions:
6.	Describe the difference in output between sum and cumsum.
7.	Generate a sequence of even numbers from -6 to 6
8.	Generate a sequence of values from -4.8 to -3.43 that is length 8 (show code)
  - What is the difference between values in this sequence?
9. What is the sum of the exponential of the sequence in question 8?
```

## Loading and Saving Data

There are a number of ways to get information into and out of R, but the most simple is in ASCII text formats, such as tab-delimited (.txt) or comma-separated (.csv).  It’s usually straightforward to export data in one of these formats from most any program (e.g. Excel).
Lets begin by opening the “Lab1_frogs.txt” file in the "data" folder. This data and some of the examples below come from Ben Bolker's handy book "Ecological Models and Data in R". 

Note that if you just click on the file from within the **Files** tab, or try to open the file from File > "Open File..."" that this opens the file as a text file, but it it doesn't load the data into R in any way we can use it. Instead, we'll want to run the following command

```{r}
dat = read.table("data/Lab1_frogs.txt",header=TRUE)
```
 
The second variable “header=TRUE” informs R that your data file has column headers that should be read rather than treated as just another line of data.  For the above command to work **R has to be looking at the correct folder**.  You can find out what folder R is currently looking at (its “working directory”) using “getwd” and you can change that directory using “setwd” or within the “Files” window tab under **More > Set As Working Directory**.  
 
<table border="1" bgcolor="yellow"><tr><td>
<h3>AUTO-COMPLETE</h3>
<p>Be aware that RStudio has the capacity to auto-complete function names, function arguments, and file names. So, for example, if you type ‘read.t’ and then hit TAB, RStudio will finish typing read.table and it would also show what information you can specify for the read.table function.  If you type read.table( and then hit TAB, RStudio will allow you to select the function argument that you want to fill in. If you type read.table(“ and then hit TAB, RStudio will show you the files in your current working directory and allow you to select one. If there are a lot of files in the directory, you can start typing the file name you want and then hit TAB again and RStudio will limit what it shows to just those files that match what you’ve typed so far.  The same ‘search’ functionality applies to file names as well. So, if you type just ‘re’ and then hit TAB, then RStudio will show you all function that begin with re. </p>
</td></tr></table>

      
For saving ASCII data there is an equivalent command “write.table”. Type ?read.table and ?write.table to learn more about these functions. While read.table and write.table can read and write CSV (comma separated value) files by just specifying the ‘sep’ argument as sep = “,” (i.e. a comma in quotes), there are also predefined functions `read.csv` and `write.csv`.
 
```{r}
write.table(dat,"my_frogs.csv",row.names=FALSE,sep=",")   	## save as CSV
```
 
	R also has a built-in binary data format that is good for saving results for later use in R and can store any number of data types of different shapes and sizes (note just single tables).  The function “save” is used to save data

```{r}
save(dat,x,y,z,file="Lab1.RData")
```
 
This command saves any number of data objects, separated by commas, that come before the ‘file=’ argument, which tells the function the name of the file you want to write to. This data can then be reloaded at a later time, or on a different computer, using “load”
 
```{r}
load("Lab1.RData")
```
 
There is also a “save.image” function that saves every variable you have defined so far

```{r}
save.image("Lab1_all.RData")
```

These commands will be very helpful if you don’t finish a activity by the end of the period and want to take your whole R desktop home, for saving work in progress, or archiving results of analyses. When you quit R you will be asked if you want to save your desktop and if you answer ‘y’, then save.image is called by default and will save to a file simply named “.Rdata” which is automatically loaded the next time you start R.  While this is convenient, I actually recommend against it and suggest using save or save.image explicitly instead because otherwise it is very easy to accidentally use variables and data sets defined in previous analyses, or to be unsure which version of an analysis you’re working with.
	You can always see what variables you currently have defined in the **Environment** window or by using the command
  
```{r}
ls()
```
 
Within the Environment window, clicking on a variable will show you the contents in a spreadsheet-like format in the script window.
	Finally, while we won’t use these explicitly in this class, there are a large number of other options for getting data in and out of R in specific formats (e.g. GIS data, image data, etc) and ways to connect R to data sources more dynamically (e.g. SQL databases) that can be particularly useful when dealing with large data sets. The **R Data Import/Export** manual on the R website is a place to start to learn more about moving data in and out of R.

```
Question:

10.	Use R commands (e.g., do not click "import dataset" and use the dialog there) to read in the file `met_hourly.csv` and assign it the variable name `met`

11. Save the vector `x` to a new csv file 
```


## Data types and dimensions

One of the first things you’ll do with any data set when you first load it up is some basic checks to see what you are dealing with.  Typing the variable name will show you its contents, but if you just loaded up something with a million entries then you’ll sit for a long time as R lists every number on the screen. The function class will tell you the type of data you’ve just loaded.  
 
```{r}
class(dat)
```
 
In this case the data is in a “data.frame”, which is like a matrix but can also contain non-numeric data.  The basic (or atomic) data types in R are integer, numeric (decimal), logical (TRUE/FALSE), factors, and character.  Character data in R is usually displayed in double quotes to indicate that it is character data (e.g. the character “1” rather than the number 1). When writing character data in R (e.g. file names in read.data) it is necessary to use double quotes as well so that R can tell the difference between character data and the names of variables and functions. By contrast, R usually reads character data from files correctly even if the data isn’t in quotes. In addition to the basic data types in R, there are a wide variety of derived data types built up from these basic types that are used for a wide range of purposes. A common example is the Date type, which can be useful for analyzing and plotting data through time, and which you can learn more about by looking at the help for **as.Date** and **strptime**.

At the most basic level R organizes data into vectors of data of a given data type and each column of a data.frame consists of a vector.  R also has a matrix data type, which must contain data of a single basic data type (usually numeric).  It is important to be aware of data types because certain operations can only be applied to certain data types (e.g. you can multiply two matrices but not two data frames).
 
```{r}
str(dat)
```
 
will tell you the basic structure of the data.  From these we learn that there are four columns of data named “frogs”, “tadpoles”, “color”, “spots” and that there are 20 rows of data, and that the data is numeric for the first two, a factor for the third, and logical for the fourth.  If we didn’t need all this information 
 
```{r}
names(dat)
```
 
will tell you the names of the columns in you data frame.
```{r}
dim(dat)
```
**dim** will tell you the dimensions of the data, in this case [1] 20 4 which means we 20 rows and 4 columns.  Each of these pieces of information is accessible individually using **nrow** and **ncol**. 
```{r}
nrow(dat)
ncol(dat)
```
Note that dim will not work on a single vector (e.g. dim(x)) but that **length(x)** will tell you the length of a vector. 
```{r}
head(dat)
tail(dat)
```
The functions **head** and **tail** show just the first and last few lines of a data set, respectively
```{r}
summary(dat)
```
will give you basic summary statistics on a data set
	Data within vectors, matrices, and data frames can be accessed using [ ] notation.  Subsets of data can also be accessed by specifying just rows, just columns, or ranges within either.  These are often referred to as subscripts or indices and the first is the row number while the second is the column.
 
```{r}
x[5]  	   # select the 5th element only
dat		     # select the entire data frame
dat[5,1]	 # select the entry in the 5th row, 1st column
dat[,2]	 	 # select all rows of the second column
dat[1:5,]	 # select rows 1 through 5, all columns
dat[6:10,2:3] 	 # select rows 6 through 10, columns 2 and 3
```
 
We can also refer to specific columns of data by name using the $ syntax

```{r}
dat$frogs  	# show just the ‘frogs’ column
dat$color[6:10]		# show the 6th though 10th elements of the color column
```

In general, it is better to **access data by name**, rather than using the row and column numbers, because this makes your code easier to understand and debug, making the process of coding less error prone. It also makes it much easier to adapt your code to new situations or data sets, where the columns might not come in the same order, or the data might not have the same number of rows or columns. This highlights a more general point, that you should use variables to represent names and numbers, especially if those names and numbers are reused, rather than ‘hard coding’ numeric values into the code.
Finally, there are functions for converting data from one data type to another
 
```{r}
as.character(dat$color)
as.numeric(dat$spots)
as.logical(0:1)
as.matrix(dat)
```

```
Questions:
12.	Show just the spots data as characters
13.	Show the 3rd through 8th rows of the 1st  though 3rd columns
14.	Show just the odd numbered rows in the frog data. Write this code for the GENERAL CASE (i.e. don’t just type c(1,3,5,…) but use functions that you learned in previous sections to set up the sequence.
```

## Combining vectors

There is a simple function **c( )** in R that “combines” vectors or numbers into a single vector. You use it like this:
 
```{r}
x=c(1,7)
x
y=c(10:15,3,9)
y
c(x,y)
```

Vectors can also be used for indexing other vectors. For example:
 
```{r}
y[x]  		## return the 1st and 7th element of y
```
 
We can also combine vectors to build up data frames by “binding” them together either are rows or as columns

```{r}
p = 1:10
q = 10:1
cbind(p,q)  	# bind as columns
rbind(q,p)		# bind as rows
```

**cbind** and **rbind** can also be applied to existing data frames, for example to add another column to an existing data frame or to take two data sets with the same columns and bind them together by row to make a larger data set.

   
```
Question:
15.	Create a character vector that contains the names of 4 people you admire.
```

## Logical operators and indexing

R can perform standard logical comparisons, which can be very useful for comparing and selecting data. It's important to know the syntax for the different logical operators, some of which are odd:

		>	greater than
		<	less than
		>=	greater than or equal to
		<=	less than or equal to
		==	equal to (TWO equals signs...you were very close!)
		!=	not equal
As a simple example you could compare individual numbers:
 
```{r}
1 > 3
5 < 7
4 >= 4
-11 <= pi
log(1) == 0
exp(0) != 1
```

You can also combine multiple logical operators using the symbols for ‘and’ (&) and ‘or’ ( | )

```{r}
w = 4
w > 0 & w < 10
w < 0 | w > 10
```

You can also apply logical operators to vectors and matrices. When you type a "logical" expression like "y > x" in R you get a TRUE/FALSE answer of the same shape as the inputs. e.g.:

```{r}
z = y>13
z
```

You will notice that by default logical operations are performed element-by-element. If you want to apply a logical test to a whole vector at a time you can use the function **any** to test if any of the values are true and **all** to test if all values are true

```{r}
any(y>13)
all(y>13)
```

You also need to know that logical vectors like "z" above can be used as indices for other vectors of the same length. Commonly, you'll use them as indices to one of the vectors that produced them. e.g.:

```{r}
y[z]
```

Or, skipping the middleman "z":

```{r}
y[y>13]
```

These simple comparisons can provide a powerful means for subsetting data. These comparisons can also be used in matrices and data frames the same way we were using sequences of row or column numbers above. For example, if you just wanted the rows where there were 3 or more frogs, you could type  

```{r}
dat[dat$frogs >= 3,]
```


R also has a built in function **subset** for doing this sort of subsetting that takes the data set as the first argument and the condition used for subsetting as the second argument. So the above could also be rewritten as
 
```{r}
subset(dat,frogs >= 3)
```

**subset** also has an optional 3rd argument for just returning specific columns. So if you wanted to run the previous subset but only needed the columns tadpoles and spots you could run

```{r}
subset(dat,frogs >= 3,c("tadpoles","spots"))
```

Note that when your data is characters you'll need double-quotes in your comparison. e.g. 

```{r}
a=c("north","south","east","west")
a == "east"
```

Two more things about logical vectors. First, sometimes it's easier or necessary to have a list of the indices to the TRUE values only (e.g. when the list includes NA values). The R function "which" is just for this purpose:

```{r}
which(y==3)
# The seventh element of y equals 3
```

And finally, sometimes TRUE/FALSE behave just like 0/1, which can be very useful. For example, this handy syntax:

```{r}
sum(y>13)
# two elements of y are >13
```

Logicals don't work exactly like 0/1 in some situations, so be careful. You can always convert them explicitly with as.numeric( ) too if need be:
 
```{r}
as.numeric(y>13)
```

## dplyr: The filter and select functions

In addition to data manipulation functions provided within the base R libraries, there are a lot of external libraries developed to make data management more efficient. One of the most popular such libraries is `dplyr`.

Here we load the library dplyr. If you haven't installed dplyr yet this line of code will throw an error, which you can correct by either using `install.packages("dplyr")` or the Packages > Install menu in the bottom right window of RStudio
```{r}
library(dplyr)
```

dplyr includes a function `filter` that also does the sort of subsetting we saw in the previous tasks. I takes the data set as the first argument and the condition used for subsetting as the second argument. So the previous example could also be rewritten as
 
```{r}
hotData = dplyr::filter(met, AirTemp > 25)
```

`dplyr` also has an function `select` for just selecting specific columns. So if you wanted to subset just the columns `ShortWave` and `Rain` you could run

```{r}
foo = dplyr::select(met, c("ShortWave","Rain"))
head(foo)
```

We'll see more dplyr commands below, but a quick "cheat sheet" can be found [here](https://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf)

## Pipes

When cleaning and organizing data it is common to have to string together multiple functions sequentially. For example, if we had data set `x` we might need to first run x through `function1()` and then run the output through `function2()`. In the examples above we did this either by using temporary variables,

```
y = function1(x)
z = function2(y)
```

or by embedding one function call within another

```
z = function2(function1(x))
```

Many consider the first both options cumbersome, as the first creates a lot of temporary variables we don't need and the latter can be confusing, particularly when multiple functions are involved or the functions have additional arguments, because one ends up reading the functions in the reverse order that they are called.

Pipes, `|>`, aim to address this issue by creating a way to pass the output of one function to another in order without creating a temporary variable.

```
z = x |> function1() |> function2()
```

As another example, if we wanted to both filter the met data by temperature AND select a subset of columns, we could pipe together multiple dplyr commands

```{r}
foo = met |> filter(AirTemp > 25) |> select(c("ShortWave","Rain"))
head(foo)
```

In this instance, when used alone both functions took the dataset as the first arguement and then additional information as the second argument. Here, the first argument is implicit and one only needs to use specify the second. An important thing to remember about pipes is that they will _always_ pass in the *first* argument in a function.

```
Question
16.	**Using the frog data set and either base or dplyr functions:** 
*  a.	display just the rows where frogs have spots
*  b.	display just the rows where frogs are blue
*  c.	create a new object containing just the rows where there are between 3 and 5 tadpoles
*  d.	display just the rows where there are less than 2.5 red frogs
*  e.	display where either frogs do not have spots or there are more than 5 frogs
```

## Plots, tables, and exploratory analysis

Often understanding our data requires more than just being able to subset the raw data, but also the ability to summarize and visualize data. The **table** command can do basic tabulation and cross tabulation of data

```{r}
table(dat$color)
table(dat$color,dat$spots)
```

There are also a number of commands for calculating basic statistical measures

```{r}
mean(dat$frogs)
median(dat$tadpoles)
var(dat$frogs)  			## variance
sd(dat$frogs)				## standard deviation
cov(met$AirTemp,met$LongWave)		## covariance
cor(met$AirTemp,met$LongWave)		## correllation
quantile(dat$tadpoles,c(0.05,0.90))		## 5% and 95% quantiles
min(dat$frogs)				## smallest value
max(dat$frogs)				## largest value
```

## Apply

R also has a set of apply functions for applying any function to sets of values within a data structure.

```{r}
apply(dat[,1:2],1,sum)  	        # calculate sum of frogs & tadpoles by row (1st dimension)
apply(dat[,1:2],2,sum) 		        # calculate sum of frogs & tadpoles by column (2nd dimension)
```

The function *apply* will apply a function to either every row (dimension 1) or every column (dimension 2) of a matrix or data.frame. In this example the commands apply the “sum” function to the first two columns of the data (frogs & tadpoles) first calculated by row (the total number of individuals in each population) and second by column (the total number of frogs and tadpoles)

![](img/apply_1.png){width=600px}

![](img/apply_2.png){width=600px}

```{r}
tapply(dat$frogs,dat$color,mean)          # calculate mean of frogs by color
tapply(dat$frogs,dat[,c("color","spots")],mean)  # calculate mean of frogs by color & spots
```

![](img/tapply.png){width=200px}

The function **tapply** will apply a function to an R data object, grouping data according to a second variable or set of variables. The first example applies the “mean” function to frogs grouping them by color. The second shows that tapply can be used to apply a function over multiple groups, in this case color X spots. 

`dplyr` also provide an alternative to `tapply` using the `group_by` and `summarize` commands. For example, the previous example could be written as

```{r}

dat |> group_by(color,spots) |> summarise(avg = mean(frogs))

```

Note that the `summarize` function creates a new column to store the output of the operation being performed. In this way `summarize` also makes it easy to calculate multiple summary statistics at once for the same groups. For example:

```{r}
dat |> group_by(color,spots) |> 
  summarise(avgFrogs = mean(frogs), 
            sdFrogs = sd(frogs), 
            avgTadpoles = mean(tadpoles),
            sdTadpoles = sd(tadpoles))
```

Another handy dplyr function is `mutate` which provides the ability to add new columns to a dataframe, for example based on computations involving other columns. For example, if we wanted to add a column to `met` that converted the datetime column `time` to just the calendar date we could do this as

```{r}
met = met |> 
  mutate(time = as.POSIXct(time,tz="GMT")) |> ## convert from character to datetime
  mutate(date = as.Date(time))                    ## convert from datetime to date
head(met)
```

```
Question
17. **Use dplyr to calculate the *daily* mean air temperature**
```

# Plotting & visualization 

There are a lot of options for plotting data in R.  The simplest of these include

```{r}
plot(dat$frogs,dat$tadpoles)  		## x-y scatter plot
abline(a=0,b=1)					## add a 1:1 line (intercept=0, slope=1)

hist(dat$tadpoles)				## histogram
abline(v=mean(dat$tadpoles))			## add a vertical line at the mean

pairs(met[,2:6])					## pairwise scatter plots for selected columns

barplot(tapply(dat$frogs,dat$color,mean))		## barplot of frogs by color
abline(h=3)					## add a horizontal line at 3
```

The functions **lines** and **points** are also frequently used to add additional lines and points (respectively) to an existing plot.

Within the Plot window, graphs can be cut-and-pasted into other documents or saved to file fairly simply by using Export. If you want to automate the process of exporting graphics, for example when you generate a whole bunch of figures at once and don’t want to Export each one by hand, you'll want to use the graphical functions such as 'postscript', 'pdf', or 'tiff'. For all of these plot functions there are numerous additional (optional) arguments that control the formatting of the plots. The help for par (i.e. ?par) gives a fairly detailed list of these options, some of which you will see in further examples below.

```
Questions
  18.	Plot a histogram of blue frogs
  19. Plot shortwave (solar) radiation against time
  20. Plot shortwave (solar) radiation against relative humidity
```

## Tidyverse

Visualization also brings us to introduce the concept of the `tidyverse`. Over the last decade Hadley Wickham and colleagues have introduces a set of packages for data manipulation and visualization that "share an underlying design philosophy, grammar, and data structures" around what they call "tidy" data. The `dplyr` package, which we saw above, is part of the tidyverse and as we saw provides a number of alternative functions for subsetting and summarizing data (see, for example, the data wrangling chapters in Wickham's [R for Data Science](https://r4ds.had.co.nz/)). Another popular part of the tidyverse are its data visualization tools, which are anchored around the `ggplot2` package, which numerous other packages build upon (see the Reverse Dependencies listing for the package https://cran.r-project.org/web/packages/ggplot2/index.html)

The syntax for ggplot is a bit less intuitive than `plot`, but the following tool `esquisse` provides a graphical interface around ggplot that then returns the underlying code used to generate the plot.

Outside of knitr, run the following code to install and launch `esquisse`
```
install.packages("esquisse")  ## only needs to be run once
esquisse::esquisser()         ## launch tool
```
Once the tool launches use it to answer the following question. Note that you won't be able to type here while the tool is running, so you may want to copy the questions, and your answers, to another doc temporarily.


```
Question

21. Histogram
* Select “met” as the data.frame
* Click “validate imported data”
* Drag AirTemp into the Y box (which will initially cause the tool to draw a histogram)
* Click on “Labels and Title” to add axes labels, units, and title
* Click on “Plot options” to change the color, theme, and number of bins
* Click on “Data” and filter the dates down to just the summer (approximately)
* Click on Export & code and hit “insert code into script” to add the code for this figure
* When you do this you’ll see a lot of new syntax. First, the `%>%` is known as the _pipe operator_ and is used string together multiple functions. So here we see that we start with the met data, then we filter it based upon time, then we call ggplot(). After we call ggplot we see that the different parts of the figure are added together using a `+`. First, we use `aes` (aesthetic) to select the data being used. Then `geom_histogram` is called to plot a histogram, with additional arguments for the number of bins and the fill color (in this case specified using a hexadecimal color code). Next, we see axis labels being set using the `labs` command. Finally we see the theme that’s been applied.
* Turn in the code for drawing your histogram

22. Line/scatter plot
* Returning to the ggplot builder, next drag “time” into the X box, which should turn the plot into a time series line plot.
* As before you can play with label, plot options, data filtering, etc
* Export this code to your Rmd as well and briefly explain (in text) what has changed in your code from the histogram plot
```

## Classical tests

Since R was initially developed for statistical programming, it can easily perform a wide variety of statistical tests and analyses. In R linear regression is done with the function “lm” (linear models)
 
```{r}
reg1 = lm(tadpoles ~ frogs,data=dat)   #model syntax: y ~ x
reg1  			     # default return from lm
summary(reg1)			     # detailed summary of results
anova(reg1)			     # ANOVA table of results
plot(reg1)			     # diagnostic plots (4 panels)
plot(residuals(reg1))		# residuals by row
coef(reg1)			# parameter coefficients
vcov(reg1)			# parameter covariance matrix
plot(dat$frogs,dat$tadpoles)
abline(reg1)			# adding regression line to the scatter plot
```


The “equation” syntax for models in R often confuses people because while the order of the data is that you would use for writing down the equation [y = f(x)], it is the opposite order from the scatterplot (x,y). The equation syntax allows one to add additional variables to the regression model (e.g. y ~ x1 + x2). This syntax also makes it easy to specify interaction terms (y ~ x1 + x2 + x1*x2). 
Note that the linear model is returning an object and that all the other functions are acting on this object. You can use all the functions you used to explore data objects (e.g. class, names, str, summary) to explore the objects returned by functions. Similar to 'lm', ANOVA models are done with 'aov'

```{r}
anov1 = aov(tadpoles ~ color + spots + color*spots,data=dat)
summary(anov1)
```


Finally, we can get a bit more sophisticated in our graphs to display these results. Note that R doesn’t care about white space—you can add spaces, tabs, and/or carriage returns wherever you want to make your code more readable.

```{r}
plot(dat$frogs,dat$tadpoles,
     cex=1.5,  			# increase the symbol size
     col=as.character(dat$color),	# change the symbol color by name
     pch=dat$spots+1,			# change the symbol (by number)
     cex.axis=1.3,			# increase the font size on the axis 
     xlab="Frog Density",		# label the x axis
     ylab="Tadpole Density",		# label the y axis
     cex.lab=1.3,			# increase the axis label font size
     main="Frog Reproductive Effort", # title
     cex.main=2			# increase title font size
     )
abline(reg1,col="green",		# add the regression line
     ,lwd=3)				# increase the line width
legend("topleft",
       c("Red no spot","Blue no spot","Red spots","Blue Spots"),
       pch=c(1,1,2,2),
       col=c("red","blue","red","blue"),cex=1.3
       )
```

```
Question
23.	Using the frog data
  a.	Fit a linear model of tadpoles as a function of frogs for just the RED individuals and report the summary data of the fit.  
  b.	Make a scatter plot of this data that includes the regression line
```

## IF statements

Logical operators are not just used for subsetting data, but can be used to control the flow of an analysis and make decisions. The idea is that we want to tell the computer a set of rules, such as “if X happens, then do Y, otherwise do Z”.  The syntax for this in R is 

```
if(condition){
  ## Do Y
} else {
	## Do Z
}
```
s
The “condition” part of this syntax is always a logical comparison, which does the first part (Y) if the condition is TRUE and the second part if it is FALSE. It should also be noted that the “else{ }” part of the syntax is optional, which would correspond to telling the computer, “if X do Y, otherwise just keep going”. For example, if we wanted to do integer division on integers but normal division otherwise we could write 

```{r}
if(is.integer(x) & is.integer(y)){
  z = x %/% y 	## Do Integer division
} else {
	z = x/y          	## Do normal division
}
z
```


It is also possible to string together multiple if statements sequentially to deal with multiple possible cases and outcomes. For example, we might want the above code to give us a warning if we try to do division on non-numeric data rather than failing with an error

```{r}
if(!is.numeric(x) | !is.numeric(y)){
  warning("Cannot perform division on non-numeric data")
}else if(is.integer(x) & is.integer(y)){
	z = x %/% y 	## Do Integer division
} else {
	z = x/y          	## Do normal division
}
z
```


For cases where the outcomes are simple, or when we want to apply an ‘if’ to every element in a vector, then the ifelse function can be an efficient alternative. ifelse takes three arguments, the condition, what to do if its true, and what to do if its false. For example, the following checks the sign on the frog data before taking a log. 
```{r}
ifelse(dat$frogs>0, log(dat$frogs),log(-dat$frogs))
```

Aside: As is very common in R, as we learn more we often find more efficient ways of solving problems . For example, the above is equivalent to log(abs(dat$frogs))

```
Question
24.	Write an if statement that makes a scatter plot of x if all the values are positive, and plots a histogram otherwise.
```

## Defining custom functions

One of the powerful things about computer languages is that they allow us to encapsulate repetitive tasks into functions, making it easier to abstract a problem. In R you are not limited to the pre-defined functions but you can define your own functions as well. For example, if we found that we were repeating the previous block of ‘if’ code multiple places in our code, we might want to convert it to a function so that we could save on retyping the code again and again. Putting the code in one place also means that if we change the code we only need to change it once and it applies everywhere. At the extreme, its often argued that anything you do more than once in a piece of code should be converted to a function.  So how do we define a function in R?  

```
name = function(arguments){
  # do some calculations
	return(z)
}
```

We need to give it a name, for example we could call the previous if statement ‘my.division’, and we need to define the arguments to the function. We also need to be explicit in defining what data we want the function to return, since in many cases the outside user doesn’t need to know everything that goes on inside the function but is only interested in the result. Putting these together would give us the following

```{r}
my.division = function(x,y){
  if(!is.numeric(x) | !is.numeric(y)){
		warning("Cannot perform division on non-numeric data")
	}else if(is.integer(x) & is.integer(y)){
	z = x %/% y 	## Do Integer division
} else {
	z = x/y          	## Do normal division
}
	return(x)
}

my.division(x,y)
my.division(y,x)
my.division(x,"5")
```

 
```
Question
25.	Convert the more complicated graphing example at the end of  “Classic Tests” into a function that will make the same plots for any data set. Show how you would call the function passing it just the subset of data where there are 5 or more tadpoles.
```

## For loops

Another powerful aspect of computers is there ability to easily repeat the same task time and time again. In fact, one of the major reasons many people learn to code is that they’ve figured out how to do some analysis once, but they want to apply the same analysis hundreds or thousands of times to different data sets, sites, individuals, pictures, etc. Doing so by clicking through a typical graphical user interface thousands of times if at best mind-numbing, if not outright impossible. Loops allow us to easily repeat an analysis over and over. The most common type of loop we will encounter is the for loop, which will repeat a chunk of code one time for each values specified by some sequence
 
```
for( variable in sequence){
  ## do something
}
```

As a very simple example, we might want to print the numbers 1:10

```{r}
for( i in 1:10){
  print(i)
}
```

A more complicated, but common, example might be to loop over all rows in a data set, or to loop over all files in a directory. We also commonly use for loops to do simple simulations. For example, if we want to simulate logistic growth, we might code it as follows:

```{r}
NT = 100  		## number of time steps
N0 = 1				## initial population size
r = 0.2				## population growth rate
K = 10				## carrying capacity
N = rep(N0,NT)
for(t in 2:NT){
	N[t] = N[t-1] + r*N[t-1]*(1-N[t-1]/K)    ## discrete logistic growth
}
plot(N)
```

```
Question
26.	Starting with a vector x = 1:10, write a for loop that adds 5 to each value in the vector. Note that it is permissible to assign an element in a vector to itself (e.g. x[i] = x[i] + 5)
```

## Vector and Matrix Math

While loops are a powerful tool, there are many places where a calculation we want to perform can be done so directly on a vector or a matrix as a whole rather than having to loop through each value. For example, in question 26 we had a vector x = 1:10 and we wanted to add 5 to every value. We could have done this more simply as
 
```{r}
x + 5
```

which adds 5 to each element. We can also multiply, divide, and subtract with vectors, and the operations are applied to each element

```{r}
5*x
x/5
x-5
```

Furthermore, if we have two or more vectors, and we perform mathematical operations on them, the operations are performed element by element

```{r}
y = 10:1
x*y
x-y
x/y
dat$frogs + dat$tadpoles
```

R also allows us to define matrices and perform element-wise math on them as well

```{r}
z = matrix(1:25,5,5)
z
z + 5
z*5
z - 1:5
```


The last example shows that we can also perform element-wise math between a matrix and a vector. Similarly, matrix-to-matrix math is also allowed, though for these cases you need to pay a lot of attention to the dimensions of the matrices and the order that vectors are being applied.
	In addition to element-wise math, R can also perform standard matrix math. If you have not seen this math before you don’t need to worry about what it is doing at this point, any matrix operations will be explained on a case-by-case later in the semester.
 
```{r}
t(z)  		## transpose
diag(z)			## diagonal of an existing matrix
diag(1,5)   ## create a diagonal matrix
w = z + diag(1,5) ## matrix addition
w
solve(w)		## inversion
crossprod(z,z)		## cross product
z %*% z 		## matrix multiplication
```

## Not covered…

While this summary is really just the tip of the iceberg relative to the full depth of available functions and packages in R, there are a few major topics that will be introduced in later exercises but which you would be encouraged to look ahead to. These include how R treats probability distributions and random numbers, missing values (NA), and a very flexible data structure called a list. As the course progresses you will also be exposed to more examples of R code, the structure of R packages, command line execution, how to read a write a larger suite of data types, and how R interfaces with a few other software systems.

## Synopsis

Most of this document is meant to familiarize you with R, so that when you encounter a problem you need to solve you will have a vague memory of something that might work and can use this as a reference. Over time you will come to memorize a larger fraction of this and through exploration you will find many additional functions and coding techniques that are useful. However, here are some parting thoughts on what you should take home at the end of the day.

If you have never programmed before:

*	Focus first on the core concepts that define almost all programming languages
 + Mathematical operators
 + Logical operators
 + Indexing of vectors, matrices, and data frames
 + If statements
 + For loops
 + Functions
*	Programming, at its essence, is a process of breaking down a complicated problem into a series of very simple steps, and then translating those steps into a symbolic language (code). This document is mostly about the vocabulary and syntax of one such language (R), the fun and creative process of using this language for problem solving will come through examples and experience.

If you know another language and are picking up R:

*	The hardest transition will be SYNTAX

Everyone:

*	Save code early and often
  + Even better is to commit your code in a version control system like git
  + Commit code frequently, typically one new feature per commit
*	Keep code well documented
*	Use meaningful variable names
*	Develop a habit of actively searching and exploring R. 
 +	Read the help documents for a function
 +	Search for new functions and techniques
 +	Scour the web when debugging. 
* The key to good programming is being able to teach yourself