# Day 29 : Statistical Analysis in R

**Meta Description:** Learn statistical analysis in R with hands-on examples. Understand hypothesis testing (t-test, ANOVA, chi-square), p-values, correlation, and simple and multiple linear regression using the built in mtcars dataset.

## Introduction

Imagine a food delivery app that proudly claims: *“Your order arrives in 30 minutes on average.”*

You have ordered from it many times, and it feels like it always takes longer. But is your feeling real, or did you just get unlucky a few times? Your friends say the same thing, but a few complaints do not prove anything. To settle this fairly, you would need a method:

1. Start by **assuming the company is telling the truth** (average is 30 minutes)
2. Collect real delivery times from many orders
3. Ask: *“If the company’s claim were true, how surprising would my data be?”*
4. If the data would be **extremely surprising**, you reject the claim. If not, you cannot say it is false

That is the heart of **hypothesis testing**. The company’s claim is the **null hypothesis**, your orders are the **sample**, and “how surprising is my data” is the **p-value**.

Now suppose you also notice that the farther the restaurant is, the longer the delivery takes. You would love to say something precise like: *“Every extra kilometer adds about 3 minutes.”* Putting an exact number on a relationship like that is what **regression** does.

Today, we use R to do both: test claims with data, and measure relationships between variables. These are the tools that turn raw numbers into defensible conclusions, which is exactly what we need for tomorrow’s case study.

---

Theory and Brief History

> Hypothesis testing helps you decide whether a pattern in your sample is real or just chance, and regression helps you measure how one variable changes when another changes.

- **1900:** Karl Pearson introduced the **chi-square test** for comparing categorical data.
- **1908:** William Sealy Gosset, a brewer at Guinness who needed to test small batches of beer, published the **t-test** under the pen name **“Student”**, because his employer did not allow staff to publish under their names. That is why it is called Student’s t-test.
- **1920s:** Ronald Fisher popularized the **p-value** and developed **ANOVA** (Analysis of Variance).
- **1886:** Francis Galton coined the term **“regression”** after noticing that tall parents tend to have children closer to average height, which he called “regression toward the mean.” The mathematical method behind it, **least squares**, goes back to Legendre and Gauss in the early 1800s.

---

## Hypothesis Testing: Key Vocabulary

![](https://cdn.hashnode.com/uploads/covers/632da7dc3a049a0d9db73ba7/c553ec17-b96f-4cf0-8a6b-1beed8fa57e3.png align="center")

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p>A p-value is <strong>not</strong> the probability that the null hypothesis is true. It is the probability of seeing data this extreme <strong>assuming</strong> the null hypothesis is true. This is one of the most commonly misunderstood ideas in all of statistics, and interviewers love to test it.</p></div>
</div>

### Two Kinds of Mistakes

- **Type I error (false positive):** rejecting H0 when it is actually true. Its probability is alpha.
- **Type II error (false negative):** failing to reject H0 when it is actually false. Its probability is beta, and **power = 1 - beta**.

Which test to use?

![](https://cdn.hashnode.com/uploads/covers/632da7dc3a049a0d9db73ba7/ee6417ac-3c12-406d-95a0-cc53b396cca6.png align="center")

If data is clearly not normal, non parametric alternatives exist, such as `wilcox.test()` and `kruskal.test()`. In Python, the same tools live in `scipy.stats` and `statsmodels`.

## Regression in a nutshell

**Simple linear regression** fits a straight line:

```python
y = intercept + slope * x
```

- **Intercept:** the predicted y when x is 0
- **Slope:** how much y changes for each one unit increase in x
- **Multiple regression** uses several predictors: `y = b0 + b1*x1 + b2*x2 + ...`, and each coefficient is interpreted **holding the other variables constant**

**Reading the model summary:**

- **Estimate:** the fitted coefficient
- **Std. Error, t value, Pr(>|t|):** tell you whether that coefficient is significantly different from zero
- **R-squared:** the share of the variation in y explained by the model (0 to 1)
- **Adjusted R-squared:** R-squared corrected for the number of predictors, so adding useless variables does not look like an improvement
- **Residual standard error:** the typical size of prediction errors, in the units of y

**Assumptions of linear regression (remember the word LINE):**

- **L**inearity: the relationship between x and y is linear
- **I**ndependence: observations do not depend on each other
- **N**ormality: residuals are roughly normally distributed
- **E**qual variance (homoscedasticity): residuals have constant spread

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p>Correlation does not imply causation. Cars with larger engines may have lower mileage, but regression alone cannot prove the engine size is the cause.</p></div>
</div>

---

## Practical

As in Days 27 and 28, use **Posit Cloud**, a **Colab notebook with an R runtime**, or a local R setup. We use the built in `mtcars` dataset (32 cars, from a 1974 magazine road test), so no file download is needed. Key columns: `mpg` (miles per gallon), `wt` (weight in 1000 lbs), `hp` (horsepower), `cyl` (cylinders), and `am` (transmission: 0 = automatic, 1 = manual).

Expected outputs are shown under each cell. Tiny differences in formatting between R versions are normal.

Cell 1: Explore the data

```python
head(mtcars, 3)
```

```python
               mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4     21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710    22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
```

Cell 2: Descriptive statistics for mileage

```python
mean(mtcars$mpg)
sd(mtcars$mpg)
summary(mtcars$mpg)
```

```python
[1] 20.09062
[1] 6.026948
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  10.40   15.43   19.20   20.09   22.80   33.90
```

Always describe your data before testing it. The average car here gets about 20 mpg.

Cell 3: Compare average mileage by transmission

```python
tapply(mtcars$mpg, mtcars$am, mean)
```

```python
       0        1 
17.14737 24.39231
```

Automatic cars (0) average about 17.1 mpg, while manual cars (1) average about 24.4 mpg. Is this gap real, or just luck from a small sample? Hypothesis tests will tell us.\

Cell 4: Check normality with the Shapiro-Wilk test

```python
shapiro.test(mtcars$mpg)
```

```python
	Shapiro-Wilk normality test

data:  mtcars$mpg
W = 0.94756, p-value = 0.1229
```

H0 here is “the data is normal.” Since p = 0.1229 is above 0.05, we **do not reject** H0, so treating mileage as roughly normal is reasonable.

Cell 5: One-sample t-test

```python
t.test(mtcars$mpg, mu = 20)
```

```python
	One Sample t-test

data:  mtcars$mpg
t = 0.08506, df = 31, p-value = 0.9328
alternative hypothesis: true mean is not equal to 20
95 percent confidence interval:
 17.91768 22.26357
sample estimates:
mean of x 
 20.09062
```

H0: the true mean mileage is 20. The p-value of 0.93 is huge, so we have no evidence against it. Notice the 95% confidence interval (17.9 to 22.3) contains 20, which agrees with this conclusion.

Cell 6: Two-sample t-test (automatic vs manual)

```python
t.test(mpg ~ am, data = mtcars)
```

```python
	Welch Two Sample t-test

data:  mpg by am
t = -3.7671, df = 18.332, p-value = 0.001374
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
 -11.280194  -3.209684
sample estimates:
mean in group 0 mean in group 1 
       17.14737        24.39231
```

H0: both transmission types have the same average mileage. With p = 0.0014 (below 0.05), we **reject H0**. The confidence interval for the difference does not include 0, so manual cars in this dataset are significantly more fuel efficient. R uses **Welch’s** version by default, which does not assume equal variances in both groups.

Cell 7: Chi-square test for two categorical variables

```python
tbl <- table(mtcars$cyl, mtcars$am)
tbl
chisq.test(tbl)
```

```python
    0  1
  4  3  8
  6  4  3
  8 12  2

	Pearson's Chi-squared test

data:  tbl
X-squared = 8.7407, df = 2, p-value = 0.01265

Warning message:
In chisq.test(tbl) : Chi-squared approximation may be incorrect
```

Rows are cylinders (4, 6, 8) and columns are transmission (0, 1). H0: cylinders and transmission type are independent. With p = 0.013, we reject H0, since 8 cylinder cars are mostly automatic. The **warning** appears because some expected counts are below 5. In practice, `fisher.test(tbl)` is a safer choice here.

Cell 8: ANOVA across three groups

```python
anova_model <- aov(mpg ~ factor(cyl), data = mtcars)
summary(anova_model)
```

```python
            Df Sum Sq Mean Sq F value   Pr(>F)    
factor(cyl)  2  824.8   412.4    39.7 4.98e-09 ***
Residuals   29  301.3    10.4                     
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
```

H0: average mileage is the same for 4, 6, and 8 cylinder cars. F = 39.7 and p is tiny, so at least one group differs. ANOVA does not say **which** groups differ. For that you follow up with `TukeyHSD(anova_model)`.

Cell 9: Correlation

```python
cor(mtcars$mpg, mtcars$wt)
cor.test(mtcars$wt, mtcars$mpg)
```

```python
[1] -0.8676594

	Pearson's product-moment correlation

data:  mtcars$wt and mtcars$mpg
t = -9.559, df = 30, p-value = 1.294e-10
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.9338264 -0.7440872
sample estimates:
       cor 
-0.8676594
```

A correlation of -0.87 is strong and negative: heavier cars get fewer miles per gallon. The p-value confirms it is very unlikely to be zero in the wider population.

Cell 10: Simple linear regression

```python
model <- lm(mpg ~ wt, data = mtcars)
model
```

```python
Call:
lm(formula = mpg ~ wt, data = mtcars)

Coefficients:
(Intercept)           wt  
     37.285       -5.344
```

The fitted line is: **mpg = 37.285 - 5.344 x wt**. Every extra 1000 lbs of weight lowers mileage by about 5.3 mpg on average.

Cell 11: Read the full regression summary

```python
summary(model)
```

```python
Call:
lm(formula = mpg ~ wt, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.5432 -2.3647 -0.1252  1.4096  6.8727 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
wt           -5.3445     0.5591  -9.559 1.29e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.046 on 30 degrees of freedom
Multiple R-squared:  0.7528,	Adjusted R-squared:  0.7446 
F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10
```

Key reading: weight is a highly significant predictor (p about 1e-10), and **weight alone explains about 75% of the variation in mileage** (R-squared = 0.7528). Typical prediction errors are about 3 mpg.

Cell 12: Confidence intervals and prediction

```python
confint(model)
predict(model, newdata = data.frame(wt = 3))
```

```python
                2.5 %    97.5 %
(Intercept) 33.450500 41.119753
wt          -6.486308 -4.202635

       1 
21.25171
```

*We are 95% confident the true slope is between -6.49 and -4.20. A car weighing 3000 lbs (wt = 3) is predicted to get about 21.25 mpg.*

Cell 13: Multiple linear regression

```python
model2 <- lm(mpg ~ wt + hp, data = mtcars)
summary(model2)
```

```python
Call:
lm(formula = mpg ~ wt + hp, data = mtcars)

Residuals:
   Min     1Q Median     3Q    Max 
-3.941 -1.600 -0.182  1.050  5.854 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 37.22727    1.59879  23.285  < 2e-16 ***
wt          -3.87783    0.63273  -6.129 1.12e-06 ***
hp          -0.03177    0.00903  -3.519  0.00145 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.593 on 29 degrees of freedom
Multiple R-squared:  0.8268,	Adjusted R-squared:  0.8148 
F-statistic: 69.21 on 2 and 29 DF,  p-value: 9.109e-12
```

Adding horsepower raises R-squared from 0.75 to 0.83. Holding horsepower constant, each extra 1000 lbs lowers mileage by about 3.9 mpg. Notice the weight coefficient changed from -5.34 to -3.88 once horsepower was included, which is a classic sign that predictors are related to each other.

Cell 14: Is the bigger model really better?

```python
anova(model, model2)
```

```python
Analysis of Variance Table

Model 1: mpg ~ wt
Model 2: mpg ~ wt + hp
  Res.Df    RSS Df Sum of Sq      F   Pr(>F)   
1     30 278.32                                
2     29 195.05  1    83.274 12.381 0.001451 **
```

H0: the extra predictor (hp) adds nothing. With p = 0.0015, we reject H0, so including horsepower gives a statistically significant improvement.

Cell 15: Check regression assumptions with diagnostic plots

```python
par(mfrow = c(2, 2))
plot(model2)
```

**What you should see:** four plots arranged in a 2 by 2 grid.

- **Residuals vs Fitted:** points scattered around the zero line. A clear curve would suggest non linearity.
- **Normal Q-Q:** points lying roughly along the diagonal line. Big departures at the ends suggest non normal residuals.
- **Scale-Location:** a roughly flat spread. A funnel shape would suggest unequal variance.
- **Residuals vs Leverage:** helps spot influential cars that pull the fitted line strongly.

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p><em>Take a screenshot of these four plots. They are exactly what an interviewer wants you to explain when asked “how do you check your regression assumptions?”</em></p></div>
</div>

---

## Key Takeaways

- Hypothesis testing starts by assuming **H0**, then asks how surprising your data is under that assumption (the **p-value**)
- If **p &lt; alpha** (usually 0.05), reject H0. If not, you simply have insufficient evidence, which is not the same as proving H0 true
- Type I error = false positive (probability alpha), Type II error = false negative (power = 1 - beta)
- Choose the test by question: **t-test** (means of 1 or 2 groups), **ANOVA** (3+ groups), **chi-square** (two categorical variables), **cor.test** (linear relationship)
- Use `shapiro.test()` to check normality, and non parametric tests when assumptions fail
- **Linear regression** fits `y = b0 + b1*x1 + ...`, and `summary(lm(...))` gives coefficients, p-values, R-squared, and residual error
- **Adjusted R-squared** penalizes useless predictors, and `anova(model1, model2)` compares nested models
- Check the **LINE** assumptions with diagnostic plots (`plot(model)`)
- Correlation is not causation, and statistical significance is not the same as practical importance

---

## Interview Questions to Look Out For

1. What is hypothesis testing, and what are the null and alternative hypotheses? Hypothesis testing is a statistical method for deciding whether sample data provides enough evidence to reject a default assumption. The null hypothesis (H0) is the default claim of no effect or no difference, and the alternative hypothesis (H1) is what you suspect is true instead.
2. What is a p-value, and what is a common misinterpretation? A p-value is the probability of observing data at least as extreme as what you saw, assuming the null hypothesis is true. A common mistake is saying it is the probability that the null hypothesis is true, or that a small p-value proves the effect is large or important. It only measures how incompatible the data is with H0.
3. What are Type I and Type II errors? A Type I error is rejecting a true null hypothesis (a false positive), and its probability is the significance level alpha. A Type II error is failing to reject a false null hypothesis (a false negative), with probability beta. Statistical power, equal to 1 minus beta, is the chance of correctly detecting a real effect.
4. When would you use a t-test, ANOVA, and a chi-square test? Use a t-test to compare means of one or two groups, ANOVA to compare means across three or more groups (followed by a post hoc test like Tukey to find which groups differ), and a chi-square test to check whether two categorical variables are associated.
5. What is Welch’s t-test, and why is it the default in R? Welch’s t-test compares two group means without assuming that both groups have equal variances. R uses it by default because equal variance is often unrealistic, and Welch’s version performs well even when variances happen to be equal.
6. What are the assumptions of linear regression? Linearity (the relationship is linear), independence of observations, normally distributed residuals, and equal variance of residuals (homoscedasticity). Together these are remembered as LINE. In multiple regression, predictors should also not be highly collinear.
7. How do you interpret R-squared and adjusted R-squared? R-squared is the proportion of variation in the response explained by the model. It never decreases when predictors are added, even useless ones. Adjusted R-squared penalizes extra predictors and only improves when a new variable adds real explanatory value, so it is better for comparing models of different sizes.
8. How do you interpret a regression coefficient? In simple regression, the slope is the average change in the response for a one unit increase in the predictor. In multiple regression, each coefficient is the average change in the response for a one unit increase in that predictor while holding all other predictors constant.
9. What is the difference between correlation and regression, and does correlation imply causation? Correlation measures the strength and direction of a linear relationship between two variables, symmetrically, as a number between -1 and 1. Regression models one variable as a function of others, so it can be used to predict and to quantify effect sizes. Neither proves causation, since a hidden third variable or reverse causality could explain the relationship.
10. What is multicollinearity, how do you detect it, and why does it matter? Multicollinearity occurs when predictors are strongly correlated with each other. It makes coefficient estimates unstable and hard to interpret, inflating their standard errors, even if overall predictions remain fine. It can be detected with correlation matrices or the Variance Inflation Factor (VIF), where values above about 5 to 10 are a warning sign. Remedies include removing or combining variables, or using regularization.

---

## Conclusion

Today we moved from describing data to drawing conclusions from it. We learned the logic of hypothesis testing (null hypothesis, p-values, significance level, and the two types of error) and applied one sample and two sample t-tests, chi-square, and ANOVA in R. We then measured relationships using correlation, simple linear regression, and multiple regression, learned how to read the model summary, compared models, and checked assumptions with diagnostic plots. These skills turn “it looks like” into “the evidence shows,” which is exactly what we need for tomorrow’s end to end case study that brings Big Data and R together.

**Next up:** Day 30: Case Study: End-to-End Big Data + R Project
