2. Assignment 2: Practicing Python and Accessing Data#

Quick Facts

2.1. Objective & Evaluation#

This assignment is an opportunity to earn level 1 and 2 achievements in python and access and begin working toward level 1 for summarize. You can also earn level 1 for process.

Eligible skills: (links to checklists)

  • first chance access 1 and 2

  • python 1 and 2

  • summarize 1

  • process 1

This assignment is an opportunity to earn level 1 and 2 achievements in python and access and begin working toward level 1 for summarize. You can also earn level 1 for process.

In this assignment, you’ll practice/ review python skills by manipulating datasets and extracting basic information about them.

2.3. Setting#

Next week, we are going to learn about summarizing data. In this assignment, you are going to build a small dataset about datasets. In class next week, we will combine all of your datasets about datasets together in order to be able to answer questions like:

  • how much total data did you all load

  • how many students picked the same dataset?

  • how many total rows of data did each student load?

2.4. Find Datasets#

Find 3 datasets of interest to you that are provided in at least two different file formats. Choose datasets that are not too big, so that they do not take more than a few second to load. At least one dataset, must have non numerical (eg string or boolean) data in at least 1 column.

In your notebook, create a markdown cell for each dataset that includes:

  • heading of the dataset’s name

  • a 1-2 sentence summary of what the dataset contains and why it was collected

  • a “more info” link to where someone can learn about the dataset

  • 1-2 questions you would like to answer with that dataset.

2.5. Store info about data for loading#

Create a list of dictionaries in datasets.py, so that there is one dictionary for each dataset. Each dictionary should have the following keys:

Table 2.1 Meta data of the dictionaries#

url

the full url of the dataset

short_name

a short name

load_function

(the actual function handle) what function should be used to load the data into a pandas.DataFrame.

Hint

See below for how you will use the dictionary as help for how you should construct it

2.6. Make a dataset about your datasets#

In a notebook called dataset_of_datasets.ipynb, import the list of dictionaries from the datasets module you created in the step above. Then iterate over the list of dictionaries, and:

  1. load each dataset like dataset_dict['load_function'](dataset_dict['url'])

  2. save it to a local csv using the short name you provided for the dataset as the file name, without writing the index column to the file.

  3. record attributes about the dataset as in the table below in a list or dictionary of lists

  4. Use that to create a DataFrame with columns that match the rows of the following table.

Table 2.2 Meta Data Description of the DataFrame to build#

name

a short name for the dataset

source

a url to where you found the data

num_rows

number of rows in the dataset

num_columns

number of columns in the dataset

num_numerical

number of numerical variables in the dataset

2.7. Explore Your Datasets#

In a second notebook file called exploration.ipynb:

For one dataset that includes nonnumerical data:

  • read it in from your local csv using a relative path

  • display the heading and the first 6 rows

  • make a numpy array of only the numerical data and save it to a new variable (select these programmatically)

  • was the format that the data was provided in a good format? why or why not?

For any other dataset:

  • read it in from your local csv using a relative path

  • display the heading with the last seven rows

  • display the datatype for each column

  • Are there any variables where pandas may have read in the data as a datatype that’s not what you expect (eg a numerical column mistaken for strings)? If so, investigate and try to figure out why.

For the third dataset:

  • read it in from your local csv using a relative path

  • save every fifth row (5,10,15 ,…) of the data for two columns of your choice into a new DataFrame and display that

2.8. Exploring data files#

There are two files in the data folder, both can be read in with read_csv but need some options or fixing.

  • try to read in the german.data file, what happens with the default settings? What option do you need to use to make it look right?

  • try to read in the .csv file that’s included in the template repository, use the error messages you get to try to fix the file manually (any text editor, including jupyter can edit a .csv), making notes about what changes you made in a markdown cell.

Hint

For the csv file in the template’s data folder, in Jupyter Lab, it will not let you edit a .csv file, but you can chagne the file name to txt (in your code too) and then it will work.

2.9. Submission#

This time you have to separately submit from posting your code to make grading easier.

  1. Go to the actions tab

  2. Click the action called “Prepare & Submit” in the left hand sidebar

  3. click the run workflow button on the right hand side.

  4. Cilck run workflow

Hint

see the github docs for screenshots of how to do these steps.

2.10. Thinking ahead#

Important

This section is not required, but is intended to help you get started thinking about ideas for your portfolio. If you complete it, we’ll give your feedback to help shape your ideas to get to level 3 achievements. If you want to focus only on level 2 at this moment in time, feel free to skip this part. You could also think about these after submitting the assignment. If you want, you could discuss these ideas in office hours.

  1. When might you prefer one datatype over another?

  2. How does PEP 8 standard code help you be collaborative?

  3. Learn about Datasheets for Datasets and find some examples, (eg this google scholar result ) How could something like this impact your work as a data scientist?