Exploratory Data Analysis I

Quantitative Attribute

• Discrete Data: We speak of discrete data if its values are distinct and separate. In other words: We speak of discrete data if the data can only take on certain values. This type of data can not be measured but it can be counted. For example, Number of defective item in a box, Number of children in a household.

• Continuous Data: Continuous Data represents measurements and therefore their values cannot be counted but they can be measured. For example, temperature, height, or weight.

• Interval Data Interval values represent ordered units that have the same difference. Therefore we speak of interval data when we have a variable that contains numeric values that are ordered and where we know the exact differences between the values. It can be either discrete or continious.

The problem about the interval data is zero have no real meaning. That’s why a lot of descriptive and inferential statistics cannot be applied. For example, Temperature.

• Ratio Data : Ratio values are also ordered units that have the same difference. Ratio values are the same as interval values, with the difference that they do have an absolute zero. In other words, zero has its real meaning. For example, age, distance

Qualitative Attribute

• Nominal Data: Nominal values represent discrete units and are used to label variables, that have no quantitative value. Just think of them as labels. Note that nominal data that has no order. For example, Gender: Male, Female

• Binary Data:

  Ordinal Data: Ordinal values represent discrete and ordered units. As you would guess from it’s name, order have an importance. For example,

• 1- Totally disagree.

• 2- Disagree.

• 3- Neither agree nor disagree.

• 4- Agree.

• 5- Totally Agree.

We can also categorized data set types into several groups, For example,

• Cross-Sectional: It is a collection of observations (behaviour) for multiple subjects(entities) at single point in time.

• Time Series: It is a collection of observations (behaviour) for a single subject(entity) at different time intervals (generally equally spaced)

• Panel Data: It is usually called as Cross-sectional Time-series data as it a combination of above mentioned types, i.e., collection of observations for multiple subjects at different time points.

• Circular Data:

Kahoot Time

Categorical Variables

Frequency Table:Frequency refers to the number of times an event or a value occurs. A frequency table is a table that lists items and shows the number of times the items occur. It is generally applied on categorical variables.

table(diamonds$cut) #creates frequency table 

## 
##      Fair      Good     Ideal   Premium Very Good 
##      1610      4906     21551     13791     12082

Interpretation is above.

prop.table(table(diamonds$cut)) #shows proportions

## 
##       Fair       Good      Ideal    Premium  Very Good 
## 0.02984798 0.09095291 0.39953652 0.25567297 0.22398962

Bar Plot: A bar chart represents data in rectangular bars with length of the bar proportional to the value of the variable. It is used for the visualization of the categorical variable, i.e displaying the distrbution of categorical variable.

To create bar plot in R, you need to create the frequency table of the categorical variable at first.

Research Question 2

What is the frequency distribution of CUT?

c = table(diamonds$cut)
barplot(c)

barplot(c,col=c("red","yellow","black","blue","orange"),main="Bar Plot of Cut")
text(c,labels=c,col="white",pos=1)

# col argument fills the bar
# main argument adds title
#ylim argument arranges the y-axis
#names.arg argument changes the bar names
#text argument shows the frequencies on the plot.  
#chas is the name of the frequency table

Note: Bar plots are sometimes are used in the illustration of numerical variable.

Numerical Variables

As a numerical EDA, the summary statistics including mean, median, standard deviation etc. are considered. You can find how to do in R above.

Histogram: A histogram represents the frequencies of values of a variable bucketed into ranges. Histogram is similar to bar chat but the difference is it groups the values into continuous ranges. Each bar in histogram represents the height of the number of values present in that range.

Research Question 3

What is the distribution of carat?

hist(diamonds$carat) #hist function is used to draw histogram

hist(diamonds$carat,col="red",main="Histogram of Carat",xlab="Frequency of Carat")

#xlab changes the x axis name. 

Same Histogram with different bin

Bin: The bar in the histogram is called bin.

hist(diamonds$carat,col="red",main="Histogram of Carat",xlab="Frequency of Carat",breaks = 20)

#xlab changes the x axis name. 
#breaks sets the number of bin in the histogram

It is seen that the carat of diamonds has right skewed distributions.

Density Plot: A density plot is a representation of the distribution of a numeric variable. It uses a kernel density estimate to show the probability density function of the variable see more. It is a smoothed version of the histogram and is used in the same concept

d = density(diamonds$carat)
print(d)

## 
## Call:
##  density.default(x = diamonds$carat)
## 
## Data: diamonds$carat (53940 obs.);   Bandwidth 'bw' = 0.04827
## 
##        x                y            
##  Min.   :0.0552   Min.   :0.0000000  
##  1st Qu.:1.3301   1st Qu.:0.0001509  
##  Median :2.6050   Median :0.0037733  
##  Mean   :2.6050   Mean   :0.1959014  
##  3rd Qu.:3.8799   3rd Qu.:0.1916445  
##  Max.   :5.1548   Max.   :1.7672776

#Bandwidth is a parameter used for arranging the degree of smoothness.

plot(d,col="red",main="Density Plot of Carat",type="l")

Interpretation

It has multimodal distribution.

Histogram with density plot

If you would like to draw a histogram with density plot, you should use prob=T argument.

hist(diamonds$carat,col="red",main="Histogram of Carat",xlab="Frequency of Carat",breaks = 20,prob=T)
lines(density(diamonds$carat),col="blue",main="Density Plot of Carat",type="l")

Box Plot: It is created based on Tukeys Five Number Summary including minimum, maximum, median, first and third quartile. The box plot can be used for two main purposes,

• To investigate the distribution of a univariate numerical ariable and checking the existence of outliers.

• To see the relationship between numerical variable and categorical variable with levels or compare a numerical variable with respect to a categorical variable.

Therefore, it is suitable for both univariate and multivariate EDA.

Anatomy of Box Plot

Anatomy of Box Plot

Research Question 3

boxplot(diamonds$carat)

boxplot(diamonds$carat,main="Box Plot of Carat",xlab="Carat",col="red")

Interpretation

It is seen that the interested variable has right skewed distribution and have many outliers. The median of the data is between 0 and 1.

Box Plot on the top of the Histogram

# Draw the boxplot and the histogram 
layout(mat = matrix(c(1,2),2,1, byrow=TRUE),  height = c(1,8))
par(mar=c(0, 3.1, 1.1, 2.1))
boxplot(diamonds$carat,main="Box Plot and Histogram of Carat ",col="red",horizontal=TRUE,frame=F)
par(mar=c(1, 3.1, 1.1, 2.1))
hist(diamonds$carat,col="red",xlab="Carat",main="")

#xlab changes the x axis name. 

Mid-Exercise 1

What is the variation of Price?

Type a R code to draw a histogram and denstiy plot. Explain what you get.

Categorical Variables

Research Question 4

How does the color of diamonds distribute over cut type?

If you are analyzing two categorical variables, we can still create a frequency table, called contingency table.

Contingency Table : In statistics, a contingency table, also known as a cross tabulation or crosstab, is a type of table in a matrix format that displays the multivariate frequency distribution of the variables. They are heavily used in survey research, business intelligence, engineering and scientific research. -Vikipedia :)

You can create contingency table by using table() in R.

table(diamonds$color,diamonds$cut)

##    
##     Fair Good Ideal Premium Very Good
##   D  163  662  2834    1603      1513
##   E  224  933  3903    2337      2400
##   F  312  909  3826    2331      2164
##   G  314  871  4884    2924      2299
##   H  303  702  3115    2360      1824
##   I  175  522  2093    1428      1204
##   J  119  307   896     808       678

Grouped Bar Plot: It is an efficient tool to display the association between two categorical variables.

counts = table(diamonds$color,diamonds$cut)
barplot(counts, main="Color Distribution by Cut Type",
  xlab="Median value of Income", col=c("grey","darkred","orange","black","maroon","gold1","darkgreen"),
  legend = rownames(counts),beside=TRUE)

Interpretation

It can be said that the frequency of color J increases as the quality of diamond cut increases. Also, the plot shows that most of the diamonds have color G.

What else?

Stacked Bar Plot: A stacked barplot is very similar to the grouped barplot above. The subgroups are just displayed on top of each other, not beside. The stacked barchart is the default option of the barplot() function in base R, so you do not need to use the beside argument.

counts = table(diamonds$color,diamonds$cut)
barplot(counts, main="Color Distribution by Cut Type",
  xlab="Cut Type", col=colors()[c(22,32,42,52,62,72,82)],
  legend = rownames(counts))

The interpretation is in the previous plot.

Continuous Variables

Research Question 5

What is the association between carat and price?

Scatter Plot: Scatter plot is a graphical way used to display the relationship between two numeric variables by using dots. Each dot represents an observation. Their position on the X (horizontal) and Y (vertical) axis represents the values of the 2 variables.

plot(diamonds$carat,diamonds$price)

#logic-> plot(independent variable, dependent variable)

plot(diamonds$carat,diamonds$price,main="The Association btw Carat and Price",xlab="Carat",ylab="Price",col="darkred",pch=12,cex=1)

#cex=size of dots
#pch=point type

It is seen that when the carat increases, the prices increases. (Positive relationship.)

Scatter Plot with Polynomial Curve

plot(diamonds$carat,diamonds$price,main="The Association btw Carat and Price",xlab="Carat",ylab="Price",col=rgb(0.4,0.4,0.8,0.6),pch=16,cex=1,ylim=c(-3500,17000))
model <- lm(diamonds$price~diamonds$carat)
 
# I can get the features of this model :
#summary(model)
#model$coefficients
#summary(model)$adj.r.squared
 
# For each value of x, I can get the value of y estimated by the model, and add it to the current plot !
myPredict <- predict( model ) 
ix <- sort(diamonds$carat,index.return=T)$ix
lines(diamonds$carat[ix], myPredict[ix], col=2, lwd=0.5 )  

# I add the features of the model to the plot
coeff <- round(model$coefficients , 2)
text(3, -2000 , paste("Model : ",coeff[1] , " + " , coeff[2] , "*x"  , "\n\n" , "P-value adjusted = ",round(summary(model)$adj.r.squared,2)))

You can customize your scatter plot using the following arguments in the plot function.

• cex = shape size

• lwd = line width

• col = control colors

• lty = line type

• pch = marker shape

• type = link between dots

Categorical and Numerical Variables

Research Question 6

How diamonds prices distribute over cut type?

Box Plot It is probably the most commonly used chart type to compare distribution of several groups.

boxplot(diamonds$price~diamonds$cut,main="The Boxplot of Diamond Prices by Cut",col=colors()[c(25,35,45,55,65)],xlab="Cut Type",ylab="Price",ylim=c(0,25000))

legend("topleft", legend = levels(diamonds$cut) , 
    col=colors()[c(25,35,45,55,65)] , bty = "n", pch=20 , pt.cex = 1, cex = 1, horiz = T, inset = c(0.01, 0.01))

#inset set the position of legend
#horiz = add legend horizontally

Interpretation

In a short way, the median price for each cut type are close to each other. All of them have outlier observations. According to plot, all of them have right skewed distribution but more visual technique should be considered.

Mid-Exercise 2

How diamonds prices distribute over color type?

Research Question 7

How diamonds prices distribute over cut type and depth?

Here, we need a transformation because we aim to analze a numerical variable(price) by two categorical variables(cut and depth).

diamonds$depth_factor = ifelse(diamonds$depth<mean(diamonds$depth),"Below Average","Above Average")

# I make the boxplot, asking to use the 2 factors : depth factor and cut:
par(mar=c(3,4,3,1))
myplot <-
boxplot(price ~ depth_factor*cut , data=diamonds,
        boxwex=0.4 , ylab="Price",
        main="The Box Plot of Price by Cut and Depth" , 
        col=c("slateblue1" , "tomato") ,  
        xaxt="n",ylim=c(0,30000))

my_names <- sapply(strsplit(myplot$names , '\\.') , function(x) x[[2]] )
my_names <- my_names[seq(1 , length(my_names) , 2)]

axis(1, at =seq(0.5 , 10 , 2),labels = my_names , 
     tick=FALSE , cex=0.3)

# Add the grey vertical lines
for(i in seq(0.5 ,10 , 2)){ 
  abline(v=i,lty=1, col="grey")
  }
 

# Add a legend
legend("topright", legend = c("Above", "Below"), 
       col=c("slateblue1" , "tomato"),
       pch = 15, bty = "n", pt.cex = 3, cex = 1.2,  horiz = F, inset = c(0.1, 0.1))

Interpretation

The price of diamonds does not show a visual difference by their depth for each cut type.

As you know, R has many number of packages for visualization, better than Python. We will cover some of those in this lab.

Lattice Plots

It is a powerful and elegant high-level data visualization system, with an emphasis on multivariate data. It generates a plot splitted into the level of a categorical variable.

What it has?

Research Question 8

What is the association between price and carats by cut type?

Scatter Plot in Lattice lattice library offers the xyplot() function. It builds a scatterplot for each levels of a factor automatically.

library(lattice)

## Warning: package 'lattice' was built under R version 3.6.3

xyplot(price ~ carat | cut , data=diamonds , pch=20 , cex=0.5 , col=rgb(0.2,0.4,0.8,0.5) )

Interpretation Carat and price have positive relationship in each cut type as we expected.

You can also make your scatter plot colorful by the level of one factor variable using xyplot().

xyplot(price ~ carat , data=diamonds,group = cut,auto.key = TRUE)

Research Question 9?

How does the price of diamonds distribute by color?

There are several ways to answer this quesiton, and see the possible solutions below.

Histogram

histogram(~ price | color, data = diamonds, breaks = 20)

We can say that price has multimodal distribution for color H, I and J and right skewed distribution for the rest of the colors.

Density Plot

densityplot(~ price | color, data = diamonds)

densityplot(~price ,group=color, data = diamonds,auto.key = TRUE)

Box and Violin Plot

bwplot(~ price | color, data = diamonds)

Violin Plot: Violin plots are similar to box plots, except that they also show the kernel probability density of the data at different values.

bwplot(~ price | color, data = diamonds,panel = panel.violin)

Recall Research Question 6

bwplot(price ~  cut, data = diamonds)

Recall Research Question 4

Heat Map

df<-table(diamonds$color,diamonds$cut)
levelplot(df)

References

• https://r4ds.had.co.nz/exploratory-data-analysis.html#covariation

• https://www.r-graph-gallery.com/

• https://www.guide2research.com/research/how-to-write-a-research-question