10. Assignment 10: Tuning Model Parameters#

10.1. Quick Facts#

10.3. Assessment#

Eligible skills: (links to checklists)

  • first chance optimization 1 and 2

  • clustering 1 and 2

  • regression 1 and 2

  • classification 1 and 2

  • evaluate (must use extra metrics to earn this here) 1 and 2

  • summarize 1 and 2

  • visualize 1 and 2

10.4. Instructions#

summary Extend the work you did in assignment 7,8, or 9, by optimizing the model parameters.

  1. Choose your dataset, task, and a model. It can be any model we have used in class so far. Include a breif description of the task(this can be copied from 7,8, or 9 if applicable but please include it for completeness).

  2. Fit the model with default parameters and check the score. Interpret the score in context. (again can be reused)

  3. Choose reasonable model parameter values for your parameter grid by assessing how well the model fit with the default values.

  4. Use grid search to find the best performing model parameters.

  5. Examine the best fit model, how different is it from the default? Score the best fit model on a held out test set.

  6. Examine and interpret the cross validation results. How do they vary in terms of time? Is the performance meaningfully different or just a little?

  7. Try varying the cross validation parameters (eg the number of folds and/or type of cross validation). Does this change your conclusions?

Tip

this is best for regression or classification, but if you use clustering use the scoring parameter to pass better metrics than the default of the score method.

Hint

Assignment 11 will be to optimize two models and then compare two models on the same task

Thinking Ahead

What other tradeoffs might you want to make in choosing a model? How could you present these results using your EDA skills?