8. Assignment 8: Regression#

8.1. Quick Facts#

8.3. Assessment#

task

skill

fit a linear regression model

regression (2)

evaluate fit of linear regression

evaluate (2)

use multiple metrics evaluate performance

evaluate (2)

interpret how decisions (test/train size, model parameters) impact model performance

evaluate (2)

interpret the model performance in the context of the dataset

process (2)

analyze the impact of model parameters on model performance

process (2)

use loops and lists effectively

python (2)

use EDA techniques to examine the experimental results

summarize (2), visualize (2)

create a dataset by combining data from multiple sources

construct (2)

8.4. Instructions#

Find a dataset suitable for regression. We recommend a dataset from the UCI repository. Complete the following in a single notebook.

8.4.1. Linear Regression Basics#

Fit a linear regression model, measure the fit with two metrics, and make a plot that helps visualize the result.

  1. Include a basic description of the data(what the features are)

  2. Write your own description of what the regression task is and why a linear model is a reasonable model to try for this data.

  3. Fit a linear model with 75% training data

  4. Test it on 25% held out test data and measure the fit with two metrics and one plot

  5. Inspect the model to answer:

    • Does this model make sense?

    • What to the coefficients tell you?

    • What to the residuals tell you?

  6. Repeat the split, train, and test steps 5 times.

    • Is the performance consistent enough you trust it?

  7. Interpret the model and its performance in terms of the application. Some questions you might want to answer in order to do this include:

  • do you think this model is good enough to use for real?

  • is this a model you would trust?

  • do you think that a more complex model should be used?

  • do you think that maybe this task cannot be done with machine learning?

  1. Try fitting the model only on one feature. Justify your choice of feature based on the results above. Plot this result.

8.4.2. Part 2: Exploring Evaluation#

Note

If you have the relevant level 2 achievements (evaluation, summarize, visualize) you can skip this part, but it might still be interesting.

Do an experiment to compare test set size vs performance:

  1. Re-fit your regression model using 10%, 30%, … , 90% of the data for training. Save the results of both test and train r2 and MSE for each size training data in a DataFrame with columns [‘train_pct’,‘n_train_samples’,‘n_test_samples’,‘train_r2’,‘test_r2’,‘train_mse’,‘test_mse’]

  2. Plot the metrics vs training percentage in a line graph.

  3. Interpret these results. How does training vs test size impact the model?

Thinking Ahead

  1. Try these experiments with a different type of regression.

  2. How do your evaluation experiment results compare in regression vs classification?