{ "cells": [ { "cell_type": "markdown", "id": "d3713984", "metadata": {}, "source": [ "# Class 5: Accessing Data, continued\n", "\n", "Today's agenda:\n", "\n", "- warm up/ review\n", "- announcements\n", "- working with dataframes\n", "- the power of functions as objects\n", "- (maybe) exploratory data analysis" ] }, { "cell_type": "markdown", "id": "b8034373", "metadata": {}, "source": [ "```{admonition} Try it out -->\n", "Read the tables off of the [syllabus course map]('https://rhodyprog4ds.github.io/BrownFall20/syllabus/course_map.html) page with `read_html` and make a list of the shapes of all of the tables on the page. Save the output to a variable and paste the *value* of that variable as your answer to the question.\n", "\n", "```" ] }, { "cell_type": "code", "execution_count": 1, "id": "7bda5359", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[(14, 3), (15, 5), (15, 15), (15, 6)]" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "[df.shape for df in pd.read_html('https://rhodyprog4ds.github.io/BrownFall20/syllabus/course_map.html')]" ] }, { "cell_type": "markdown", "id": "4942501a", "metadata": {}, "source": [ "## Announcements\n", "\n", "- annotated notes are up\n", "- beginning portfolio prompts and instructions are up\n", "- Assignment due Sunday,\n", "- office hours will remain Fridays\n", "- TA office hours posted.\n", "\n", "## More Pandas\n", "\n", "We'll go back to the SAFI dataset from Wednesday." ] }, { "cell_type": "code", "execution_count": 2, "id": "ec38b0f3", "metadata": {}, "outputs": [], "source": [ "safi_df = pd.read_csv('https://raw.githubusercontent.com/brownsarahm/python-socialsci-files/master/data/SAFI_clean.csv')" ] }, { "cell_type": "markdown", "id": "38634c9f", "metadata": {}, "source": [ "We downloaded the data into memory, but we can also write it to disk." ] }, { "cell_type": "code", "execution_count": 3, "id": "6d0ae13a", "metadata": {}, "outputs": [], "source": [ "safi_df.to_csv('safi_clean.csv')" ] }, { "cell_type": "markdown", "id": "bd86bedb", "metadata": {}, "source": [ "It will go to the same folder as the notebook, but we can also use a relative path. If we make a `data` folder in the folder where we've saved the notebook, we can write the file there instead.\n", "\n", "````{margin}\n", "```{tip}\n", "In class we used the Jupyter GUI to create a new folder. You could also use your computer's default file management tool (Windows Explorer, Mac Finder, etc). Here, since the notebooks have to run completely automatically for this website, we use a ipython [`bash` magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#cellmagic-bash) cell to make the folder. Jupyter notebooks use an ipython kernel.\n", "```\n", "````" ] }, { "cell_type": "code", "execution_count": 4, "id": "2faf9878", "metadata": {}, "outputs": [], "source": [ "safi_df.to_csv('data/safi_clean.csv')" ] }, { "cell_type": "markdown", "id": "6ef2b9f7", "metadata": {}, "source": [ "Now we can read it in using the same path" ] }, { "cell_type": "code", "execution_count": 5, "id": "954d5480", "metadata": {}, "outputs": [], "source": [ "safi_df2= pd.read_csv('data/safi_clean.csv')" ] }, { "cell_type": "markdown", "id": "47e666ee", "metadata": {}, "source": [ "Note that now it has an extra column" ] }, { "cell_type": "code", "execution_count": 6, "id": "8d1ce949", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
001God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
111God2016-11-17T00:00:00Z79muddaub1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...2Jan;Sept;Oct;Nov;Decuuid:099de9c9-3e5e-427b-8452-26250e840d6e
\n", "
" ], "text/plain": [ " Unnamed: 0 key_ID village interview_date no_membrs years_liv \\\n", "0 0 1 God 2016-11-17T00:00:00Z 3 4 \n", "1 1 1 God 2016-11-17T00:00:00Z 7 9 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "0 muddaub 1 NaN NaN 1 \n", "1 muddaub 1 yes once 3 \n", "\n", " items_owned no_meals \\\n", "0 bicycle;television;solar_panel;table 2 \n", "1 cow_cart;bicycle;radio;cow_plough;solar_panel;... 2 \n", "\n", " months_lack_food instanceID \n", "0 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "1 Jan;Sept;Oct;Nov;Dec uuid:099de9c9-3e5e-427b-8452-26250e840d6e " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df2.head(2)" ] }, { "cell_type": "code", "execution_count": 7, "id": "b48a754a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
01God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
11God2016-11-17T00:00:00Z79muddaub1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...2Jan;Sept;Oct;Nov;Decuuid:099de9c9-3e5e-427b-8452-26250e840d6e
\n", "
" ], "text/plain": [ " key_ID village interview_date no_membrs years_liv \\\n", "0 1 God 2016-11-17T00:00:00Z 3 4 \n", "1 1 God 2016-11-17T00:00:00Z 7 9 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "0 muddaub 1 NaN NaN 1 \n", "1 muddaub 1 yes once 3 \n", "\n", " items_owned no_meals \\\n", "0 bicycle;television;solar_panel;table 2 \n", "1 cow_cart;bicycle;radio;cow_plough;solar_panel;... 2 \n", "\n", " months_lack_food instanceID \n", "0 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "1 Jan;Sept;Oct;Nov;Dec uuid:099de9c9-3e5e-427b-8452-26250e840d6e " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.head(2)" ] }, { "cell_type": "markdown", "id": "f4000b7b", "metadata": {}, "source": [ "We can prevent this by writing it out with the `index` parameter set to `False`\n", "````{margin}\n", "```{tip}\n", "False must be with a capital letter to be a boolean variable in python, as with True. You'll know if you did it right in your jupyter notebook, if the word terms bold and green.\n", "```\n", "````" ] }, { "cell_type": "code", "execution_count": 8, "id": "a29263d1", "metadata": {}, "outputs": [], "source": [ "safi_df.to_csv('data/safi_clean.csv',index=False)" ] }, { "cell_type": "markdown", "id": "58abe62c", "metadata": {}, "source": [ "Now when we read it in, there's no extra column." ] }, { "cell_type": "code", "execution_count": 9, "id": "e0f9c961", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
01God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
11God2016-11-17T00:00:00Z79muddaub1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...2Jan;Sept;Oct;Nov;Decuuid:099de9c9-3e5e-427b-8452-26250e840d6e
23God2016-11-17T00:00:00Z1015burntbricks1NaNNaN1solar_torch2Jan;Feb;Mar;Oct;Nov;Decuuid:193d7daf-9582-409b-bf09-027dd36f9007
\n", "
" ], "text/plain": [ " key_ID village interview_date no_membrs years_liv \\\n", "0 1 God 2016-11-17T00:00:00Z 3 4 \n", "1 1 God 2016-11-17T00:00:00Z 7 9 \n", "2 3 God 2016-11-17T00:00:00Z 10 15 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "0 muddaub 1 NaN NaN 1 \n", "1 muddaub 1 yes once 3 \n", "2 burntbricks 1 NaN NaN 1 \n", "\n", " items_owned no_meals \\\n", "0 bicycle;television;solar_panel;table 2 \n", "1 cow_cart;bicycle;radio;cow_plough;solar_panel;... 2 \n", "2 solar_torch 2 \n", "\n", " months_lack_food instanceID \n", "0 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "1 Jan;Sept;Oct;Nov;Dec uuid:099de9c9-3e5e-427b-8452-26250e840d6e \n", "2 Jan;Feb;Mar;Oct;Nov;Dec uuid:193d7daf-9582-409b-bf09-027dd36f9007 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df3 = pd.read_csv('data/safi_clean.csv')\n", "safi_df3.head(3)" ] }, { "cell_type": "markdown", "id": "e57c0f2e", "metadata": {}, "source": [ "Recall, we indexed a column with the name in square brackets" ] }, { "cell_type": "code", "execution_count": 10, "id": "1a3eca2e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 God\n", "1 God\n", "2 God\n", "3 God\n", "4 God\n", " ... \n", "126 Ruaca\n", "127 Ruaca\n", "128 Ruaca\n", "129 Chirodzo\n", "130 Chirodzo\n", "Name: village, Length: 131, dtype: object" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df['village']" ] }, { "cell_type": "markdown", "id": "189e728c", "metadata": {}, "source": [ "To index rows, we can use `loc`" ] }, { "cell_type": "code", "execution_count": 11, "id": "a12ab9b4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "key_ID 4\n", "village God\n", "interview_date 2016-11-17T00:00:00Z\n", "no_membrs 7\n", "years_liv 6\n", "respondent_wall_type burntbricks\n", "rooms 1\n", "memb_assoc NaN\n", "affect_conflicts NaN\n", "liv_count 2\n", "items_owned bicycle;radio;cow_plough;solar_panel;mobile_phone\n", "no_meals 2\n", "months_lack_food Sept;Oct;Nov;Dec\n", "instanceID uuid:148d1105-778a-4755-aa71-281eadd4a973\n", "Name: 3, dtype: object" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.loc[3]" ] }, { "cell_type": "markdown", "id": "83ea6e0f", "metadata": {}, "source": [ "To select a range, use `:`" ] }, { "cell_type": "code", "execution_count": 12, "id": "53c8e411", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
34God2016-11-17T00:00:00Z76burntbricks1NaNNaN2bicycle;radio;cow_plough;solar_panel;mobile_phone2Sept;Oct;Nov;Decuuid:148d1105-778a-4755-aa71-281eadd4a973
45God2016-11-17T00:00:00Z740burntbricks1NaNNaN4motorcyle;radio;cow_plough;mobile_phone2Aug;Sept;Oct;Novuuid:2c867811-9696-4966-9866-f35c3e97d02d
56God2016-11-17T00:00:00Z33muddaub1NaNNaN1NaN2Aug;Sept;Octuuid:daa56c91-c8e3-44c3-a663-af6a49a2ca70
\n", "
" ], "text/plain": [ " key_ID village interview_date no_membrs years_liv \\\n", "3 4 God 2016-11-17T00:00:00Z 7 6 \n", "4 5 God 2016-11-17T00:00:00Z 7 40 \n", "5 6 God 2016-11-17T00:00:00Z 3 3 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "3 burntbricks 1 NaN NaN 2 \n", "4 burntbricks 1 NaN NaN 4 \n", "5 muddaub 1 NaN NaN 1 \n", "\n", " items_owned no_meals \\\n", "3 bicycle;radio;cow_plough;solar_panel;mobile_phone 2 \n", "4 motorcyle;radio;cow_plough;mobile_phone 2 \n", "5 NaN 2 \n", "\n", " months_lack_food instanceID \n", "3 Sept;Oct;Nov;Dec uuid:148d1105-778a-4755-aa71-281eadd4a973 \n", "4 Aug;Sept;Oct;Nov uuid:2c867811-9696-4966-9866-f35c3e97d02d \n", "5 Aug;Sept;Oct uuid:daa56c91-c8e3-44c3-a663-af6a49a2ca70 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.loc[3:5]" ] }, { "cell_type": "markdown", "id": "93377fa3", "metadata": {}, "source": [ "You only have to have a number on one side of the colon, it will go from the beginnig up to that number like this:" ] }, { "cell_type": "code", "execution_count": 13, "id": "6c79020d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
01God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
11God2016-11-17T00:00:00Z79muddaub1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...2Jan;Sept;Oct;Nov;Decuuid:099de9c9-3e5e-427b-8452-26250e840d6e
23God2016-11-17T00:00:00Z1015burntbricks1NaNNaN1solar_torch2Jan;Feb;Mar;Oct;Nov;Decuuid:193d7daf-9582-409b-bf09-027dd36f9007
34God2016-11-17T00:00:00Z76burntbricks1NaNNaN2bicycle;radio;cow_plough;solar_panel;mobile_phone2Sept;Oct;Nov;Decuuid:148d1105-778a-4755-aa71-281eadd4a973
45God2016-11-17T00:00:00Z740burntbricks1NaNNaN4motorcyle;radio;cow_plough;mobile_phone2Aug;Sept;Oct;Novuuid:2c867811-9696-4966-9866-f35c3e97d02d
\n", "
" ], "text/plain": [ " key_ID village interview_date no_membrs years_liv \\\n", "0 1 God 2016-11-17T00:00:00Z 3 4 \n", "1 1 God 2016-11-17T00:00:00Z 7 9 \n", "2 3 God 2016-11-17T00:00:00Z 10 15 \n", "3 4 God 2016-11-17T00:00:00Z 7 6 \n", "4 5 God 2016-11-17T00:00:00Z 7 40 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "0 muddaub 1 NaN NaN 1 \n", "1 muddaub 1 yes once 3 \n", "2 burntbricks 1 NaN NaN 1 \n", "3 burntbricks 1 NaN NaN 2 \n", "4 burntbricks 1 NaN NaN 4 \n", "\n", " items_owned no_meals \\\n", "0 bicycle;television;solar_panel;table 2 \n", "1 cow_cart;bicycle;radio;cow_plough;solar_panel;... 2 \n", "2 solar_torch 2 \n", "3 bicycle;radio;cow_plough;solar_panel;mobile_phone 2 \n", "4 motorcyle;radio;cow_plough;mobile_phone 2 \n", "\n", " months_lack_food instanceID \n", "0 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "1 Jan;Sept;Oct;Nov;Dec uuid:099de9c9-3e5e-427b-8452-26250e840d6e \n", "2 Jan;Feb;Mar;Oct;Nov;Dec uuid:193d7daf-9582-409b-bf09-027dd36f9007 \n", "3 Sept;Oct;Nov;Dec uuid:148d1105-778a-4755-aa71-281eadd4a973 \n", "4 Aug;Sept;Oct;Nov uuid:2c867811-9696-4966-9866-f35c3e97d02d " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.loc[:4]" ] }, { "cell_type": "markdown", "id": "3f9d2982", "metadata": {}, "source": [ "```{admonition}\n", "How do you think you'd get the last 3 rows with `loc`?\n", "```\n", "\n", "With two `::` we can also set an increment" ] }, { "cell_type": "code", "execution_count": 14, "id": "9df3137e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
01God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
56God2016-11-17T00:00:00Z33muddaub1NaNNaN1NaN2Aug;Sept;Octuuid:daa56c91-c8e3-44c3-a663-af6a49a2ca70
1011God2016-11-21T00:00:00Z620sunbricks1NaNNaN2radio;cow_plough2Oct;Novuuid:d29b44e3-3348-4afc-aa4d-9eb34c89d483
1516God2016-11-24T00:00:00Z647muddaub1NaNNaN4radio;cow_plough;solar_panel;solar_torch3Jan;Febuuid:d17db52f-4b87-4768-b534-ea8f9704c565
2021God2016-11-21T00:00:00Z820burntbricks1nonever3NaN2Jan;Feb;Mar;Oct;Nov;Decuuid:6570a7d0-6a0b-452c-aa2e-922500e35749
2526Ruaca2016-11-21T00:00:00Z320burntbricks2nonever2radio;cow_plough;table;mobile_phone2noneuuid:1c54ee24-22c4-4ee9-b1ad-42d483c08e2e
3031Ruaca2016-11-21T00:00:00Z32muddaub1NaNNaN1NaN3noneuuid:cb06eb49-dd39-4150-8bbe-a599e074afe8
3536Chirodzo2016-11-17T00:00:00Z623sunbricks1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...3noneuuid:c90eade0-1148-4a12-8c0e-6387a36f45b1
4041God2016-11-17T00:00:00Z722muddaub1NaNNaN2motorcyle;bicycle;radio;cow_plough;table3Oct;Novuuid:b3ba34d8-eea1-453d-bc73-c141bcbbc5e5
4546Chirodzo2016-11-17T00:00:00Z1042burntbricks2noonce2motorcyle;computer;television;sterio;solar_pan...2Sept;Oct;Novuuid:35f297e0-aa5d-4149-9b7b-4965004cfc37
5051Chirodzo2016-11-16T00:00:00Z530muddaub1NaNNaN1radio3Oct;Novuuid:18ac8e77-bdaf-47ab-85a2-e4c947c9d3ce
5556Chirodzo2016-11-16T00:00:00Z1223burntbricks2yesnever2motorcyle;bicycle;mobile_phone3noneuuid:973c4ac6-f887-48e7-aeaf-4476f2cfab76
6061Chirodzo2016-11-16T00:00:00Z1014muddaub1yesmore_once3cow_cart;motorcyle;bicycle;television;radio;co...3Jan;Feb;Decuuid:2401cf50-8859-44d9-bd14-1bf9128766f2
6566Chirodzo2016-11-16T00:00:00Z1037burntbricks3yesfrequently4cow_cart;motorcyle;bicycle;television;radio;co...3noneuuid:a457eab8-971b-4417-a971-2e55b8702816
7071Ruaca2016-11-18T00:00:00Z614burntbricks1yesmore_once3radio;cow_plough;mobile_phone2Aug;Sept;Oct;Novuuid:761f9c49-ec93-4932-ba4c-cc7b78dfcef1
75155God2016-11-24T00:00:00Z44burntbricks1NaNNaN1electricity2Jan;Sept;Oct;Nov;Decuuid:77b3021b-a9d6-4276-aaeb-5bfcfd413852
80182God2016-11-25T00:00:00Z721muddaub3nomore_once2solar_panel3Jan;Feb;Nov;Decuuid:394033e8-a6e2-4e39-bfac-458753a1ed78
85197God2016-11-28T00:00:00Z519burntbricks2nomore_once3bicycle;television;radio;cow_plough;solar_torc...2Novuuid:85c99fd2-775f-40c9-8654-68223f59d091
9073Ruaca2017-04-26T00:00:00Z79burntbricks2yesmore_once3cow_cart;motorcyle;bicycle;television;radio;co...3Jan;Sept;Octuuid:ac3da862-9e6c-4962-94b6-f4c31624f207
95101God2017-04-27T00:00:00Z34muddaub1nonever1bicycle;solar_torch3Sept;Oct;Novuuid:3c174acd-e431-4523-9ad6-eb14cddca805
100104Ruaca2017-04-28T00:00:00Z1452sunbricks1yesnever4cow_cart;bicycle;cow_plough3Jan;Feb;Decuuid:bb2bb365-7d7d-4fe9-9353-b21269676119
105113Ruaca2017-05-03T00:00:00Z1126burntbricks3nonever4cow_cart;motorcyle;bicycle;radio;cow_plough;so...3noneuuid:01210861-aba1-4268-98d0-0260e05f5155
110108God2017-05-11T00:00:00Z1522burntbricks2nonever4cow_cart;bicycle;radio;cow_plough;solar_panel;...3Aug;Sept;Oct;Novuuid:e4f4d6ba-e698-45a5-947f-ba6da88cc22b
115150Ruaca2017-05-18T00:00:00Z78muddaub1nonever1mobile_phone3Sept;Oct;Novuuid:92613d0d-e7b1-4d62-8ea4-451d7cd0a982
120167Ruaca2017-06-03T00:00:00Z824muddaub1nonever3motorcyle;radio;cow_plough;solar_panel;solar_t...2Jan;Nov;Decuuid:a9d1a013-043b-475d-a71b-77ed80abe970
125192Chirodzo2017-06-03T00:00:00Z920burntbricks1noonce1bicycle;television;radio;sterio;solar_panel;so...3Jan;Nov;Decuuid:f94409a6-e461-4e4c-a6fb-0072d3d58b00
130200Chirodzo2017-06-04T00:00:00Z820burntbricks2NaNNaN3radio;cow_plough;solar_panel;solar_torch;table...3Oct;Novuuid:aa77a0d7-7142-41c8-b494-483a5b68d8a7
\n", "
" ], "text/plain": [ " key_ID village interview_date no_membrs years_liv \\\n", "0 1 God 2016-11-17T00:00:00Z 3 4 \n", "5 6 God 2016-11-17T00:00:00Z 3 3 \n", "10 11 God 2016-11-21T00:00:00Z 6 20 \n", "15 16 God 2016-11-24T00:00:00Z 6 47 \n", "20 21 God 2016-11-21T00:00:00Z 8 20 \n", "25 26 Ruaca 2016-11-21T00:00:00Z 3 20 \n", "30 31 Ruaca 2016-11-21T00:00:00Z 3 2 \n", "35 36 Chirodzo 2016-11-17T00:00:00Z 6 23 \n", "40 41 God 2016-11-17T00:00:00Z 7 22 \n", "45 46 Chirodzo 2016-11-17T00:00:00Z 10 42 \n", "50 51 Chirodzo 2016-11-16T00:00:00Z 5 30 \n", "55 56 Chirodzo 2016-11-16T00:00:00Z 12 23 \n", "60 61 Chirodzo 2016-11-16T00:00:00Z 10 14 \n", "65 66 Chirodzo 2016-11-16T00:00:00Z 10 37 \n", "70 71 Ruaca 2016-11-18T00:00:00Z 6 14 \n", "75 155 God 2016-11-24T00:00:00Z 4 4 \n", "80 182 God 2016-11-25T00:00:00Z 7 21 \n", "85 197 God 2016-11-28T00:00:00Z 5 19 \n", "90 73 Ruaca 2017-04-26T00:00:00Z 7 9 \n", "95 101 God 2017-04-27T00:00:00Z 3 4 \n", "100 104 Ruaca 2017-04-28T00:00:00Z 14 52 \n", "105 113 Ruaca 2017-05-03T00:00:00Z 11 26 \n", "110 108 God 2017-05-11T00:00:00Z 15 22 \n", "115 150 Ruaca 2017-05-18T00:00:00Z 7 8 \n", "120 167 Ruaca 2017-06-03T00:00:00Z 8 24 \n", "125 192 Chirodzo 2017-06-03T00:00:00Z 9 20 \n", "130 200 Chirodzo 2017-06-04T00:00:00Z 8 20 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "0 muddaub 1 NaN NaN 1 \n", "5 muddaub 1 NaN NaN 1 \n", "10 sunbricks 1 NaN NaN 2 \n", "15 muddaub 1 NaN NaN 4 \n", "20 burntbricks 1 no never 3 \n", "25 burntbricks 2 no never 2 \n", "30 muddaub 1 NaN NaN 1 \n", "35 sunbricks 1 yes once 3 \n", "40 muddaub 1 NaN NaN 2 \n", "45 burntbricks 2 no once 2 \n", "50 muddaub 1 NaN NaN 1 \n", "55 burntbricks 2 yes never 2 \n", "60 muddaub 1 yes more_once 3 \n", "65 burntbricks 3 yes frequently 4 \n", "70 burntbricks 1 yes more_once 3 \n", "75 burntbricks 1 NaN NaN 1 \n", "80 muddaub 3 no more_once 2 \n", "85 burntbricks 2 no more_once 3 \n", "90 burntbricks 2 yes more_once 3 \n", "95 muddaub 1 no never 1 \n", "100 sunbricks 1 yes never 4 \n", "105 burntbricks 3 no never 4 \n", "110 burntbricks 2 no never 4 \n", "115 muddaub 1 no never 1 \n", "120 muddaub 1 no never 3 \n", "125 burntbricks 1 no once 1 \n", "130 burntbricks 2 NaN NaN 3 \n", "\n", " items_owned no_meals \\\n", "0 bicycle;television;solar_panel;table 2 \n", "5 NaN 2 \n", "10 radio;cow_plough 2 \n", "15 radio;cow_plough;solar_panel;solar_torch 3 \n", "20 NaN 2 \n", "25 radio;cow_plough;table;mobile_phone 2 \n", "30 NaN 3 \n", "35 cow_cart;bicycle;radio;cow_plough;solar_panel;... 3 \n", "40 motorcyle;bicycle;radio;cow_plough;table 3 \n", "45 motorcyle;computer;television;sterio;solar_pan... 2 \n", "50 radio 3 \n", "55 motorcyle;bicycle;mobile_phone 3 \n", "60 cow_cart;motorcyle;bicycle;television;radio;co... 3 \n", "65 cow_cart;motorcyle;bicycle;television;radio;co... 3 \n", "70 radio;cow_plough;mobile_phone 2 \n", "75 electricity 2 \n", "80 solar_panel 3 \n", "85 bicycle;television;radio;cow_plough;solar_torc... 2 \n", "90 cow_cart;motorcyle;bicycle;television;radio;co... 3 \n", "95 bicycle;solar_torch 3 \n", "100 cow_cart;bicycle;cow_plough 3 \n", "105 cow_cart;motorcyle;bicycle;radio;cow_plough;so... 3 \n", "110 cow_cart;bicycle;radio;cow_plough;solar_panel;... 3 \n", "115 mobile_phone 3 \n", "120 motorcyle;radio;cow_plough;solar_panel;solar_t... 2 \n", "125 bicycle;television;radio;sterio;solar_panel;so... 3 \n", "130 radio;cow_plough;solar_panel;solar_torch;table... 3 \n", "\n", " months_lack_food instanceID \n", "0 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "5 Aug;Sept;Oct uuid:daa56c91-c8e3-44c3-a663-af6a49a2ca70 \n", "10 Oct;Nov uuid:d29b44e3-3348-4afc-aa4d-9eb34c89d483 \n", "15 Jan;Feb uuid:d17db52f-4b87-4768-b534-ea8f9704c565 \n", "20 Jan;Feb;Mar;Oct;Nov;Dec uuid:6570a7d0-6a0b-452c-aa2e-922500e35749 \n", "25 none uuid:1c54ee24-22c4-4ee9-b1ad-42d483c08e2e \n", "30 none uuid:cb06eb49-dd39-4150-8bbe-a599e074afe8 \n", "35 none uuid:c90eade0-1148-4a12-8c0e-6387a36f45b1 \n", "40 Oct;Nov uuid:b3ba34d8-eea1-453d-bc73-c141bcbbc5e5 \n", "45 Sept;Oct;Nov uuid:35f297e0-aa5d-4149-9b7b-4965004cfc37 \n", "50 Oct;Nov uuid:18ac8e77-bdaf-47ab-85a2-e4c947c9d3ce \n", "55 none uuid:973c4ac6-f887-48e7-aeaf-4476f2cfab76 \n", "60 Jan;Feb;Dec uuid:2401cf50-8859-44d9-bd14-1bf9128766f2 \n", "65 none uuid:a457eab8-971b-4417-a971-2e55b8702816 \n", "70 Aug;Sept;Oct;Nov uuid:761f9c49-ec93-4932-ba4c-cc7b78dfcef1 \n", "75 Jan;Sept;Oct;Nov;Dec uuid:77b3021b-a9d6-4276-aaeb-5bfcfd413852 \n", "80 Jan;Feb;Nov;Dec uuid:394033e8-a6e2-4e39-bfac-458753a1ed78 \n", "85 Nov uuid:85c99fd2-775f-40c9-8654-68223f59d091 \n", "90 Jan;Sept;Oct uuid:ac3da862-9e6c-4962-94b6-f4c31624f207 \n", "95 Sept;Oct;Nov uuid:3c174acd-e431-4523-9ad6-eb14cddca805 \n", "100 Jan;Feb;Dec uuid:bb2bb365-7d7d-4fe9-9353-b21269676119 \n", "105 none uuid:01210861-aba1-4268-98d0-0260e05f5155 \n", "110 Aug;Sept;Oct;Nov uuid:e4f4d6ba-e698-45a5-947f-ba6da88cc22b \n", "115 Sept;Oct;Nov uuid:92613d0d-e7b1-4d62-8ea4-451d7cd0a982 \n", "120 Jan;Nov;Dec uuid:a9d1a013-043b-475d-a71b-77ed80abe970 \n", "125 Jan;Nov;Dec uuid:f94409a6-e461-4e4c-a6fb-0072d3d58b00 \n", "130 Oct;Nov uuid:aa77a0d7-7142-41c8-b494-483a5b68d8a7 " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.loc[::5]" ] }, { "cell_type": "markdown", "id": "0f42446a", "metadata": {}, "source": [ "These can be combined to index a subset at an increment.\n", "\n", "We can index columns in two ways, as we did on Wednesday" ] }, { "cell_type": "code", "execution_count": 15, "id": "9a324c7b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 God\n", "1 God\n", "Name: village, dtype: object" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df['village'].head(2)" ] }, { "cell_type": "markdown", "id": "5991bbe5", "metadata": {}, "source": [ "Or using a `.`" ] }, { "cell_type": "code", "execution_count": 16, "id": "401b67f2", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 God\n", "1 God\n", "Name: village, dtype: object" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.village.head(2)" ] }, { "cell_type": "markdown", "id": "c24ba984", "metadata": {}, "source": [ "We can select multiple columns, using a `list` of column names. We can define the list inline." ] }, { "cell_type": "code", "execution_count": 17, "id": "66df0c75", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
villageno_membrsyears_liv
0God34
1God79
\n", "
" ], "text/plain": [ " village no_membrs years_liv\n", "0 God 3 4\n", "1 God 7 9" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df[['village','no_membrs','years_liv']].head(2)" ] }, { "cell_type": "markdown", "id": "2ca0b103", "metadata": {}, "source": [ "or in a separate variable" ] }, { "cell_type": "code", "execution_count": 18, "id": "01f7930a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
villageno_membrsyears_liv
0God34
1God79
\n", "
" ], "text/plain": [ " village no_membrs years_liv\n", "0 God 3 4\n", "1 God 7 9" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "columns_of_interest = ['village','no_membrs','years_liv']\n", "safi_df[columns_of_interest].head(2)" ] }, { "cell_type": "markdown", "id": "9feb80f5", "metadata": {}, "source": [ "## Functions are objects" ] }, { "cell_type": "code", "execution_count": 19, "id": "a8004d6a", "metadata": {}, "outputs": [], "source": [ "syllabus_df_list = pd.read_html('https://rhodyprog4ds.github.io/BrownFall20/syllabus/course_map.html')" ] }, { "cell_type": "markdown", "id": "d56be257", "metadata": {}, "source": [ "And we can put them in a dictionary. `lambda` functions are special functions defined in a single line." ] }, { "cell_type": "code", "execution_count": 20, "id": "de4d20ee", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'hello sarah'" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "greetingl = lambda name: 'hello ' + name\n", "greetingl('sarah')" ] }, { "cell_type": "markdown", "id": "8757fe8d", "metadata": {}, "source": [ "is the same as" ] }, { "cell_type": "code", "execution_count": 21, "id": "3b53765f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'hello sarah'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def greetingf(name):\n", " return 'hello ' + name\n", "greetingf('sarah')" ] }, { "cell_type": "markdown", "id": "93a1173f", "metadata": {}, "source": [ "So, we can define a function in a dictionary like this:\n", "````{margin}\n", "```{tip}\n", "this is how to do the equivalent of a switch or case in other languages in python\n", "```\n", "````" ] }, { "cell_type": "code", "execution_count": 22, "id": "d473151b", "metadata": {}, "outputs": [], "source": [ "view_rows = {0: lambda df: print(df.head()),\n", " 1: lambda df: print(df.tail())}" ] }, { "cell_type": "markdown", "id": "1083a84e", "metadata": {}, "source": [ "The `len` function works on all iterables" ] }, { "cell_type": "code", "execution_count": 23, "id": "b6622e45", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Unnamed: 0_level_0 topics \\\n", " week Unnamed: 1_level_1 \n", "0 1 [admin, python review] \n", "1 2 Loading data, Python review \n", "2 3 Exploratory Data Analysis \n", "3 4 Data Cleaning \n", "4 5 Databases, Merging DataFrames \n", "\n", " skills \n", " Unnamed: 2_level_1 \n", "0 process \n", "1 [access, prepare, summarize] \n", "2 [summarize, visualize] \n", "3 [prepare, summarize, visualize] \n", "4 [access, construct, summarize] \n", " Unnamed: 0_level_0 skill \\\n", " keyword Unnamed: 1_level_1 \n", "10 evaluate Evaluate model performance \n", "11 optimize Optimize model parameters \n", "12 compare compare models \n", "13 unstructured model unstructured data \n", "14 workflow use industry standard data science tools and w... \n", "\n", " Level 1 \\\n", " Unnamed: 2_level_1 \n", "10 Explain basic performance metrics for differen... \n", "11 Identify when model parameters need to be opti... \n", "12 Qualitatively compare model classes \n", "13 Identify options for representing text data an... \n", "14 Solve well strucutred problems with a single t... \n", "\n", " Level 2 \\\n", " Unnamed: 3_level_1 \n", "10 Apply basic model evaluation metrics to a held... \n", "11 Manually optimize basic model parameters such ... \n", "12 Compare model classes in specific terms and fi... \n", "13 Apply at least one representation to transform... \n", "14 Solve semi-strucutred, completely specified pr... \n", "\n", " Level 3 \n", " Unnamed: 4_level_1 \n", "10 Evaluate a model with multiple metrics and cro... \n", "11 Select optimal parameters based of mutiple qua... \n", "12 Evaluate tradeoffs between different model com... \n", "13 apply multiple representations and compare and... \n", "14 Scope, choose an appropriate tool pipeline and... \n", " Unnamed: 0_level_0 A1 A2 \\\n", " keyword Unnamed: 1_level_1 Unnamed: 2_level_1 \n", "10 evaluate 0 0 \n", "11 optimize 0 0 \n", "12 compare 0 0 \n", "13 unstructured 0 0 \n", "14 workflow 0 0 \n", "\n", " A3 A4 A5 \\\n", " Unnamed: 3_level_1 Unnamed: 4_level_1 Unnamed: 5_level_1 \n", "10 0 0 0 \n", "11 0 0 0 \n", "12 0 0 0 \n", "13 0 0 0 \n", "14 0 0 0 \n", "\n", " A6 A7 A8 \\\n", " Unnamed: 6_level_1 Unnamed: 7_level_1 Unnamed: 8_level_1 \n", "10 0 0 0 \n", "11 0 0 0 \n", "12 0 0 0 \n", "13 0 0 0 \n", "14 0 0 0 \n", "\n", " A9 A10 A11 \\\n", " Unnamed: 9_level_1 Unnamed: 10_level_1 Unnamed: 11_level_1 \n", "10 0 1 1 \n", "11 0 1 1 \n", "12 0 0 1 \n", "13 0 0 0 \n", "14 0 1 1 \n", "\n", " A12 A13 # Assignments \n", " Unnamed: 12_level_1 Unnamed: 13_level_1 Unnamed: 14_level_1 \n", "10 0 0 2 \n", "11 0 0 2 \n", "12 0 1 2 \n", "13 1 1 2 \n", "14 1 1 4 \n", " Unnamed: 0_level_0 Level 3 \\\n", " keyword Unnamed: 1_level_1 \n", "10 evaluate Evaluate a model with multiple metrics and cro... \n", "11 optimize Select optimal parameters based of mutiple qua... \n", "12 compare Evaluate tradeoffs between different model com... \n", "13 unstructured apply multiple representations and compare and... \n", "14 workflow Scope, choose an appropriate tool pipeline and... \n", "\n", " P1 P2 P3 P4 \n", " Unnamed: 2_level_1 Unnamed: 3_level_1 Unnamed: 4_level_1 Unnamed: 5_level_1 \n", "10 0 1 1 0 \n", "11 0 0 1 1 \n", "12 0 0 1 1 \n", "13 0 0 1 1 \n", "14 0 0 1 1 \n" ] } ], "source": [ "for df in syllabus_df_list:\n", " num_row = len(df)\n", " view_rows[num_row%2](df)\n" ] }, { "cell_type": "markdown", "id": "ff34d7dc", "metadata": {}, "source": [ "## The beginning of Exploratory Data Analysis\n", "\n", "Pandas will give us descriptive statistics" ] }, { "cell_type": "code", "execution_count": 24, "id": "f4083fc4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDno_membrsyears_livroomsliv_countno_meals
count131.000000131.00000131.000000131.000000131.000000131.000000
mean85.4732827.1908423.0534351.7404582.3664122.603053
std63.1516283.1722716.9130411.0925471.0827750.491143
min1.0000002.000001.0000001.0000001.0000002.000000
25%32.5000005.0000012.0000001.0000001.0000002.000000
50%66.0000007.0000020.0000001.0000002.0000003.000000
75%138.0000009.0000027.5000002.0000003.0000003.000000
max202.00000019.0000096.0000008.0000005.0000003.000000
\n", "
" ], "text/plain": [ " key_ID no_membrs years_liv rooms liv_count no_meals\n", "count 131.000000 131.00000 131.000000 131.000000 131.000000 131.000000\n", "mean 85.473282 7.19084 23.053435 1.740458 2.366412 2.603053\n", "std 63.151628 3.17227 16.913041 1.092547 1.082775 0.491143\n", "min 1.000000 2.00000 1.000000 1.000000 1.000000 2.000000\n", "25% 32.500000 5.00000 12.000000 1.000000 1.000000 2.000000\n", "50% 66.000000 7.00000 20.000000 1.000000 2.000000 3.000000\n", "75% 138.000000 9.00000 27.500000 2.000000 3.000000 3.000000\n", "max 202.000000 19.00000 96.000000 8.000000 5.000000 3.000000" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.describe()" ] }, { "cell_type": "markdown", "id": "8ed8bd4d", "metadata": {}, "source": [ "The statistics of the `key_ID` column don't make a lot of sense. We can avoid that by making it the index" ] }, { "cell_type": "code", "execution_count": 25, "id": "26f87216", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
key_IDvillageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
01God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
11God2016-11-17T00:00:00Z79muddaub1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...2Jan;Sept;Oct;Nov;Decuuid:099de9c9-3e5e-427b-8452-26250e840d6e
23God2016-11-17T00:00:00Z1015burntbricks1NaNNaN1solar_torch2Jan;Feb;Mar;Oct;Nov;Decuuid:193d7daf-9582-409b-bf09-027dd36f9007
34God2016-11-17T00:00:00Z76burntbricks1NaNNaN2bicycle;radio;cow_plough;solar_panel;mobile_phone2Sept;Oct;Nov;Decuuid:148d1105-778a-4755-aa71-281eadd4a973
45God2016-11-17T00:00:00Z740burntbricks1NaNNaN4motorcyle;radio;cow_plough;mobile_phone2Aug;Sept;Oct;Novuuid:2c867811-9696-4966-9866-f35c3e97d02d
\n", "
" ], "text/plain": [ " key_ID village interview_date no_membrs years_liv \\\n", "0 1 God 2016-11-17T00:00:00Z 3 4 \n", "1 1 God 2016-11-17T00:00:00Z 7 9 \n", "2 3 God 2016-11-17T00:00:00Z 10 15 \n", "3 4 God 2016-11-17T00:00:00Z 7 6 \n", "4 5 God 2016-11-17T00:00:00Z 7 40 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "0 muddaub 1 NaN NaN 1 \n", "1 muddaub 1 yes once 3 \n", "2 burntbricks 1 NaN NaN 1 \n", "3 burntbricks 1 NaN NaN 2 \n", "4 burntbricks 1 NaN NaN 4 \n", "\n", " items_owned no_meals \\\n", "0 bicycle;television;solar_panel;table 2 \n", "1 cow_cart;bicycle;radio;cow_plough;solar_panel;... 2 \n", "2 solar_torch 2 \n", "3 bicycle;radio;cow_plough;solar_panel;mobile_phone 2 \n", "4 motorcyle;radio;cow_plough;mobile_phone 2 \n", "\n", " months_lack_food instanceID \n", "0 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "1 Jan;Sept;Oct;Nov;Dec uuid:099de9c9-3e5e-427b-8452-26250e840d6e \n", "2 Jan;Feb;Mar;Oct;Nov;Dec uuid:193d7daf-9582-409b-bf09-027dd36f9007 \n", "3 Sept;Oct;Nov;Dec uuid:148d1105-778a-4755-aa71-281eadd4a973 \n", "4 Aug;Sept;Oct;Nov uuid:2c867811-9696-4966-9866-f35c3e97d02d " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.head()" ] }, { "cell_type": "markdown", "id": "11bdf830", "metadata": {}, "source": [ "the `inplace` parameter of a `pandas` functions applies the operation to the DataFrame in memory, but then the function returns nothing, but if we display after that, we see that now the `key_ID` column is now the index." ] }, { "cell_type": "code", "execution_count": 26, "id": "2dccb6cd", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
villageinterview_dateno_membrsyears_livrespondent_wall_typeroomsmemb_assocaffect_conflictsliv_countitems_ownedno_mealsmonths_lack_foodinstanceID
key_ID
1God2016-11-17T00:00:00Z34muddaub1NaNNaN1bicycle;television;solar_panel;table2Januuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef
1God2016-11-17T00:00:00Z79muddaub1yesonce3cow_cart;bicycle;radio;cow_plough;solar_panel;...2Jan;Sept;Oct;Nov;Decuuid:099de9c9-3e5e-427b-8452-26250e840d6e
\n", "
" ], "text/plain": [ " village interview_date no_membrs years_liv \\\n", "key_ID \n", "1 God 2016-11-17T00:00:00Z 3 4 \n", "1 God 2016-11-17T00:00:00Z 7 9 \n", "\n", " respondent_wall_type rooms memb_assoc affect_conflicts liv_count \\\n", "key_ID \n", "1 muddaub 1 NaN NaN 1 \n", "1 muddaub 1 yes once 3 \n", "\n", " items_owned no_meals \\\n", "key_ID \n", "1 bicycle;television;solar_panel;table 2 \n", "1 cow_cart;bicycle;radio;cow_plough;solar_panel;... 2 \n", "\n", " months_lack_food instanceID \n", "key_ID \n", "1 Jan uuid:ec241f2c-0609-46ed-b5e8-fe575f6cefef \n", "1 Jan;Sept;Oct;Nov;Dec uuid:099de9c9-3e5e-427b-8452-26250e840d6e " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.set_index('key_ID',inplace=True)\n", "safi_df.head(2)" ] }, { "cell_type": "markdown", "id": "6a01ba91", "metadata": {}, "source": [ "and if we describe again, we see it doesn't compute on that column" ] }, { "cell_type": "code", "execution_count": 27, "id": "e86faa32", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
no_membrsyears_livroomsliv_countno_meals
count131.00000131.000000131.000000131.000000131.000000
mean7.1908423.0534351.7404582.3664122.603053
std3.1722716.9130411.0925471.0827750.491143
min2.000001.0000001.0000001.0000002.000000
25%5.0000012.0000001.0000001.0000002.000000
50%7.0000020.0000001.0000002.0000003.000000
75%9.0000027.5000002.0000003.0000003.000000
max19.0000096.0000008.0000005.0000003.000000
\n", "
" ], "text/plain": [ " no_membrs years_liv rooms liv_count no_meals\n", "count 131.00000 131.000000 131.000000 131.000000 131.000000\n", "mean 7.19084 23.053435 1.740458 2.366412 2.603053\n", "std 3.17227 16.913041 1.092547 1.082775 0.491143\n", "min 2.00000 1.000000 1.000000 1.000000 2.000000\n", "25% 5.00000 12.000000 1.000000 1.000000 2.000000\n", "50% 7.00000 20.000000 1.000000 2.000000 3.000000\n", "75% 9.00000 27.500000 2.000000 3.000000 3.000000\n", "max 19.00000 96.000000 8.000000 5.000000 3.000000" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df.describe()" ] }, { "cell_type": "markdown", "id": "6797a49d", "metadata": {}, "source": [ "We can also call any of those on one column or one statistic." ] }, { "cell_type": "code", "execution_count": 28, "id": "bcd328f9", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1.7404580152671756" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "safi_df['rooms'].mean()" ] }, { "cell_type": "markdown", "id": "58d3c93f", "metadata": {}, "source": [ "Pandas also has some built in plotting functions." ] }, { "cell_type": "code", "execution_count": 29, "id": "7affa4a3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEHCAYAAACk6V2yAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcOElEQVR4nO3df3Ac5Z3n8fd3kJAV22Ah64zXgrVZO1QIKA434fhlF4HFIVnWcOskm9SyhOQoNlXZXMLVHmaXFOSo5PbWeweX5HIbHLIJhFw2GwT2hvzAhEAIy8+xYwtjh1jBBMuHhazIYHGykJnv/TFtMxIz8rTVPT8efV5VU555pvvpbz9ufdTq6ek2d0dERMKTqXUBIiKSDgW8iEigFPAiIoFSwIuIBEoBLyISqKZaF1Bs7ty5vnDhwlqXISLSMDZu3LjX3TtKvVdXAb9w4UJyuVytyxARaRhm9tty7+kQjYhIoBTwIiKBUsCLiARKAS8iEigFvIhIoFINeDO71syeNbOtZvZdM5uR5vLqxeDwKFt27WNweLTWpYjINJbaaZJmtgD4j8Bp7j5iZv8MfAT4VlrLrAfrN+9mdXcPzZkMY/k8a1Z1sXLpglqXJSLTUNqHaJqAVjNrAt4G/N+Ul1dTg8OjrO7u4cBYnv2jBzkwlue67h7tyYtITaQW8O6+G/jvwIvAS8Ar7r5h4nRmdo2Z5cwsNzAwkFY5VdE3NEJzZvyQNmcy9A2N1KgiEZnOUgt4M2sDLgMWAb8HzDSzKyZO5+5r3T3r7tmOjpLftm0YnW2tjOXz49rG8nk621prVJGITGdpHqL5Q2Cnuw+4+xhwD3BuisurufZZLaxZ1cWM5gyzW5qY0Zxhzaou2me11Lo0EZmG0rwWzYvA2Wb2NmAEuAgI/kIzK5cu4LzFc+kbGqGzrVXhLiI1k1rAu/uTZnY3sAk4CPwSWJvW8upJ+6wWBbuI1FyqV5N095uAm9JchoiIlKZvsoqIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigUgt4MzvVzDYXPV41s8+mtTwRERkvtZtuu/tzwFIAMzsG2A3cm9byRERkvGodorkI+I27/7ZKyxMRmfaqFfAfAb5bpWWJiAhVCHgzOxZYCXy/zPvXmFnOzHIDAwNplyMiMm1UYw/+/cAmd+8v9aa7r3X3rLtnOzo6qlCOiMj0UI2A/yg6PCMiUnWpBryZzQQuBu5JczkiIvJWqZ0mCeDurwHtaS5DRERK0zdZRUQCpYAXEQmUAl5EJFAKeBGRQCngRUQCpYAXEQmUAl5EJFAKeBGRQCngRUQCpYAXEQmUAl5EJFAKeBGRQCngRUQCpYAXEQmUAl5EJFAKeBGRQCngRUQCpYAXEQmUAl5EJFAKeBGRQKUa8GY2x8zuNrNfmdl2MzsnzeWJiMibmlLu/0vAT9z9g2Z2LPC2lJcnIiKR1ALezI4HlgNXAbj768DraS1PRETGS/MQzSJgAPimmf3SzG43s5kTJzKza8wsZ2a5gYGBFMsREZle0gz4JuBM4B/c/d3Aa8D1Eydy97XunnX3bEdHR4rliIhML2kGfB/Q5+5PRq/vphD4IiJSBakFvLvvAXaZ2alR00XAtrSWJyIi46V9Fs2nge9EZ9A8D3w85eWJiEgk1YB3981ANs1liIhIafomq4hIoBTwIiKBUsCLiARKAS8iEigFvIhIoBTwIiKBUsCLiARKAS8iEigFvIhIoBTwIiKBUsCLiARKAS8iEigFvIhIoBTwIiKBUsCLiARKAS8iEigFvIhIoBTwIiKBUsCLiAQq1XuymtkLwH7gDeCgu+v+rCKTGBwepW9ohM62VtpntdS6nMSEul71LtWAj7zX3fdWYTkiDW395t2s7u6hOZNhLJ9nzaouVi5dUOuypizU9WoEOkQjUgcGh0dZ3d3DgbE8+0cPcmAsz3XdPQwOj9a6tCkJdb0aRdoB78AGM9toZteUmsDMrjGznJnlBgYGUi5HpD71DY3QnBn/49icydA3NFKjipIR6no1itgBb2ZtZtZV4eTnu/uZwPuBT5nZ8okTuPtad8+6e7ajoyNuOSJB6GxrZSyfH9c2ls/T2dZao4qSEep6NYqKAt7MHjaz48zsBGAT8HUzu+VI87n77ujfl4F7gbOmUqxIqNpntbBmVRczmjPMbmliRnOGNau6Gv4DyVDXq1FU+iHr8e7+qpldDdzp7jeZWc9kM5jZTCDj7vuj5yuAm6dYr0iwVi5dwHmL5wZ3tkmo69UIKg34JjObD3wYuKHCeeYB95rZoeX8H3f/SfwSRaaP9lktQQZgqOtV7yoN+JuB+4FH3f1pMzsF2DHZDO7+PPCuKdYnIiJHqaKAd/fvA98vev08sCqtokREZOoqCngzWwR8GlhYPI+7r0ynLBERmapKD9GsA74B/ADITz6piIjUg0oD/oC7fznVSkREJFGVBvyXzOwmYANw+DvG7r4plapERGTKKg34M4A/By7kzUM0Hr0WEZE6VGnAfwg4xd1fT7MYERFJTqXXotkKzEmxDhERSVile/BzgF+Z2dOMPwav0yRFROpUpQF/U6pViIhI4ir9JuvPzWwe8J6o6anoCpEiIlKnKr1c8IeBpyh82Pph4Ekz+2CahYmIyNRUeojmBuA9h/bazawD+Clwd1qFiYjI1FR6Fk1mwiGZwRjziohIDVS6B/8TM7sf+G70+k+BH6VTkoiIJKHSD1n/s5n9CXB+1LTW3e9NrywREZmqSvfgAf4VGKNwiYKn0ilHRESSEvcsmg+is2hERBqCzqIREQlU6mfRmNkxZvZLM7svdnVSNwaHR9myax+Dw6NHnriB5HYOcsuG58jtHKx1KUC44yy1ccQ9eDMz4OkpnEXzGWA7cNxRVSg1t37zblZ399CcyTCWz7NmVRcrly6odVlTdsXtT/BobyHYv/yzXpYtbufbV59ds3pCHWepnSPuhbu7A2cBtwFd0WOtu68+0rxm1gn8EXD7FOuUGhkcHmV1dw8HxvLsHz3IgbE813X3NPweZm7n4OFwP+QXvYM125MPdZyltio9RLMR2OXu/yl6VHqK5P8ErmOS+7ia2TVmljOz3MDAQIXdSrX0DY3QnBm/mTRnMvQNjdSoomQ8smNvrPa0hTrOUluVBvy/Ax43s9+YWc+hx2QzmNmlwMvuvnGy6dx9rbtn3T3b0dFRYTlSLZ1trYzlx/9+Hsvn6WxrrVFFyVi+ZG6s9rSFOs5SW5UG/PuAP6Bwi74/LnpM5jxgpZm9APwTcKGZ3XWUdUqNtM9qYc2qLmY0Z5jd0sSM5gxrVnXRPqul1qVNSXZRO8sWt49rW7a4neyi9jJzpCvUcZbassIh9pQXYnYB8Ffufulk02WzWc/lcqnXI/ENDo/SNzRCZ1trUKGT2znIIzv2snzJ3JqFe7FQx1nSY2Yb3T1b6r0432SVaax9VkuQgZNdVLu99lJCHWepjaoEvLs/DDxcjWWJiEiBLvkrIhIoBbyISKAU8CIigVLAi4gESgEvIhIoBbyISKAU8CIigVLAi4gESgEvIhIoBbyISKAU8CIigVLAi4gESgEvIhIoBbyISKAU8CIigVLAi4gESgEvIhIoBbyISKAU8CIigUot4M1shpk9ZWZbzOxZM/svaS1LRETeKs09+FHgQnd/F7AUuMTMzk5jQYPDo2zZtY/B4dE0uq+ZJNYrqbHJ7Rzklg3Pkds5OKV+Hty2h9V3b+HBbXum1E9Sevv3c3duF739+6fUT1LrFeq2LLXRlFbH7u7AcPSyOXp40stZv3k3q7t7aM5kGMvnWbOqi5VLFyS9mKpLYr2SGpsrbn+CR3sLwf7ln/WybHE73746/u/qFbc+zK/7XwPge7k+Tp03k/uvvSB2P0m5cd0z3PnEi4dfX3nOydx82Rmx+0lqvULdlqV2Uj0Gb2bHmNlm4GXgAXd/Msn+B4dHWd3dw4GxPPtHD3JgLM913T0Nv/eTxHolNTa5nYOHw/2QX/QOxt6Tf3DbnsMheMhz/a/VbE++t3//uHAHuPPxF2PvySe1XqFuy1JbqQa8u7/h7kuBTuAsMzt94jRmdo2Z5cwsNzAwEKv/vqERmjPjV6E5k6FvaGQKVddeEuuV1Ng8smNvrPZyNmzrj9Wets279sVqLyep9Qp1W5baqspZNO6+D3gIuKTEe2vdPevu2Y6Ojlj9dra1MpbPj2sby+fpbGudQrW1l8R6JTU2y5fMjdVezorT5sVqT9vSk+bEai8nqfUKdVuW2krzLJoOM5sTPW8FLgZ+leQy2me1sGZVFzOaM8xuaWJGc4Y1q7pon9WS5GKqLon1SmpssovaWba4fVzbssXtZBe1l5mjtItOO5FT580c13bqvJlcdNqJsfpJyuJ5s7nynJPHtV15zsksnjc7Vj9JrVeo27LUlhU+C02hY7Mu4A7gGAq/SP7Z3W+ebJ5sNuu5XC72sgaHR+kbGqGzrTWoH4gk1iupscntHOSRHXtZvmRu7HAv9uC2PWzY1s+K0+bVLNyL9fbvZ/OufSw9aU7scC+W1HqFui1Lesxso7tnS76XVsAfjaMNeBGR6WqygNc3WUVEAqWAFxEJlAJeRCRQCngRkUAp4EVEAqWAFxEJlAJeRCRQCngRkUAp4EVEAqWAFxEJlAJeRCRQCngRkUAp4EVEAqWAFxEJlAJeRCRQCngRkUAp4EVEAqWAFxEJlAJeRCRQCngRkUClFvBmdpKZPWRm28zsWTP7TFrLSsrg8Chbdu1jcHh0Sv309u/n7twuevv3T6mf2x7awfu/9Ai3PbQjiFoAcjsHuWXDc+R2Dk6pn6TWK6l+ktp2khqfepPU+IRWC6Rbj7l74p0CmNl8YL67bzKz2cBG4HJ331Zunmw267lcLpV6jmT95t2s7u6hOZNhLJ9nzaouVi5dELufG9c9w51PvHj49ZXnnMzNl50Ru593fO5HjBx88/+mtcnY/oUPNGwtAFfc/gSP9r4ZXMsWt/Ptq8+O3U9S65VUP0ltO0mNT71JanxCqyWpesxso7tnS72X2h68u7/k7pui5/uB7UDtRnISg8OjrO7u4cBYnv2jBzkwlue67p7Yv1F7+/ePCwyAOx9/Mfbe4W0P7RgXqAAjBz3W3nM91QKFPdPi8AL4Re9g7D3VpNYrqX6S2naSGp96k9T4hFZLteqpyjF4M1sIvBt4ssR715hZzsxyAwMD1SjnLfqGRmjOjB+K5kyGvqGRWP1s3rUvVns563peitVe77UAPLJjb6z2cpJar6T6SWrbSWp86k1S4xNaLdWqJ/WAN7NZQDfwWXd/deL77r7W3bPunu3o6Ei7nJI621oZy+fHtY3l83S2tcbqZ+lJc2K1l3N51/xY7fVeC8DyJXNjtZeT1Hol1U9S205S41Nvkhqf0GqpVj2pBryZNVMI9++4+z1pLmsq2me1sGZVFzOaM8xuaWJGc4Y1q7pon9USq5/F82Zz5Tknj2u78pyTWTxvdqx+/uK9S2htsnFtrU3GX7x3SUPWApBd1M6yxe3j2pYtbie7qL3MHKUltV5J9ZPUtpPU+NSbpMYntFqqVU+aH7IacAfwO3f/bCXz1PJDVigcE+sbGqGzrXVKg9zbv5/Nu/ax9KQ5sQOj2G0P7WBdz0tc3jU/dqDWYy1QONb8yI69LF8yd0rhldR6JdVPUttOUuNTb5Ian9BqSaKeyT5kTTPgzwd+ATwDHPo75G/c/Ufl5ql1wIuINJrJAr4prYW6+6OAHXFCERFJhb7JKiISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBLyISKAW8iEigFPAiIoFKLeDN7B/N7GUz25rWMg7J7Rzklg3Pkds5OKV+br1/Oxf8/UPcev/2KfXzxR9s5Zy//Slf/MHUVj2Jfq66/XHefsMPuer2x6dUy6fvepp33vhjPn3X01Pq567HdvKhrz3GXY/tnFI/D27bw+q7t/Dgtj1T6mfdpl1cfcfTrNu0a0r9DA6PsmXXPgaHR6fUT71Jar1CHZ96Z+6eTsdmy4Fh4E53P72SebLZrOdyuVjLueL2J3i0981gX7a4nW9ffXasPgCW/PUPGSsaimaDHX/7R7H7OeX6H5Ivep0Bnv9vteln4fU/fEvbC0dRS1L9vOvzP+GVA28cfn38jGPY8vlLYvez4taH+XX/a4dfnzpvJvdfe0Hsfs7+rw+w59XXD7+ef9yxPP43F8fuZ/3m3azu7qE5k2Esn2fNqi5WLl0Qu596k9R6hTo+9cLMNrp7ttR7qe3Bu/sjwO/S6h8Ke+7F4Q7wi97B2Hvyt96/fVy4A4w5sffkv/iDreNCGSAftVe7n3J77HH35Mvtscfdk7/rsZ3jwh3glQNvxN6Tf3DbnnHhDvBc/2ux9+TXbdo1LtwBXnr19dh78oPDo6zu7uHAWJ79owc5MJbnuu6eht9TTWq9Qh2fRlHzY/Bmdo2Z5cwsNzAwEGveR3bsjdVezvqe0uFQrr2c+7aWnr5ce5r9PLaz9O/Wcu3l/OzXpceyXHs563teitVezoZt/bHay7nvmTJjXKa9nL6hEZoz43+MmjMZ+oZGYvVTb5Jar1DHp1HUPODdfa27Z90929HREWve5Uvmxmov57KuE2O1l3Pp6aWnL9eeZj/nLjohVns5F7699FiWay/nsq75sdrLWXHavFjt5Vx6RpkxLtNeTmdbK2P58X9vjeXzdLa1xuqn3iS1XqGOT6OoecBPRXZRO8sWt49rW7a4neyi9jJzlHbt+95Bs41va7ZCexw3/PHpbxnQTNRe7X6+dfU5sdrL+coV74nVXs4V5y7i+BnHjGs7fsYxXHHuolj9XHTaiZw6b+a4tlPnzeSi0+IF8+VnnsT8444d1zb/uGO5/MyTYvXTPquFNau6mNGcYXZLEzOaM6xZ1UX7rJZY/dSbpNYr1PFpFKl9yApgZguB+9L8kBUKx+If2bGX5Uvmxg73Yrfev531PXu4rOvE2OFe7Is/2Mp9W/dw6eknxg73pPu56vbHeWzn7zh30Qmxw73Yp+96mp/9ei8Xvn1u7HAvdtdjO1nf8xKXdc2PHe7FHty2hw3b+llx2rzY4V5s3aZd3PfMHi4948TY4V5scHiUvqEROttagwqvpNYr1PGpB5N9yJrmWTTfBS4A5gL9wE3u/o3J5jnagBcRma4mC/imtBbq7h9Nq28RETmyhj4GLyIi5SngRUQCpYAXEQmUAl5EJFCpniYZl5kNAL9NcRFzgXhfway9Rqu50eoF1VwtjVZzo9T7++5e8luidRXwaTOzXLnTiepVo9XcaPWCaq6WRqu50eotRYdoREQCpYAXEQnUdAv4tbUu4Cg0Ws2NVi+o5mpptJobrd63mFbH4EVEppPptgcvIjJtKOBFRAIVVMCb2Ulm9pCZbTOzZ83sMyWmucDMXjGzzdHjxlrUOqGmF8zsmaiet1xO0wq+bGa9ZtZjZmfWos6iek4tGr/NZvaqmX12wjQ1H+dSN343sxPM7AEz2xH921Zm3o9F0+wws4/VuOa/N7NfRf/395rZnDLzTrodVbnmz5vZ7qL//w+UmfcSM3su2ravr2G93yuq9QUz21xm3pqM8VFz92AewHzgzOj5bODXwGkTprmAwjXqa15vUU0vAHMnef8DwI8BA84Gnqx1zUW1HQPsofBli7oaZ2A5cCawtahtDXB99Px64O9KzHcC8Hz0b1v0vK2GNa8AmqLnf1eq5kq2oyrX/HngryrYdn4DnAIcC2yZ+PNarXonvP8/gBvraYyP9hHUHry7v+Tum6Ln+4HtQAi3b78MuNMLngDmmFm8e92l5yLgN+6e5jeQj4qXvvH7ZcAd0fM7gMtLzPo+4AF3/527DwEPAJekVWexUjW7+wZ3Pxi9fALorEYtlSozzpU4C+h19+fd/XXgnyj8/6RqsnrNzIAPA99Nu45qCCrgi0V3k3o38GSJt88xsy1m9mMze2d1KyvJgQ1mttHMrinx/gJgV9HrPurnF9dHKP/DUG/jDDDP3Q/d6XsPUOpmrvU83p+g8NdcKUfajqrtL6PDSv9Y5lBYPY7zMqDf3XeUeb/exnhSQQa8mc0CuoHPuvurE97eROFwwruArwDrqlxeKee7+5nA+4FPmdnyWhdUCTM7FlgJfL/E2/U4zuN44W/uhjlP2MxuAA4C3ykzST1tR/8A/AGwFHiJwmGPRvBRJt97r6cxPqLgAt7MmimE+3fc/Z6J77v7q+4+HD3/EdBsZnOrXObEmnZH/74M3EvhT9diu4HiG4Z2Rm219n5gk7v3T3yjHsc50n/o8Fb078slpqm78Tazq4BLgT+LfjG9RQXbUdW4e7+7v+HueeDrZWqpq3E2sybgT4DvlZumnsa4EkEFfHT87BvAdne/pcw0J0bTYWZnURiDwepV+ZZ6ZprZ7EPPKXygtnXCZP8CXBmdTXM28ErRYYZaKru3U2/jXORfgENnxXwMWF9imvuBFWbWFh1aWBG11YSZXQJcB6x09/9XZppKtqOqmfAZ0b8vU8vTwBIzWxT9NfgRCv8/tfKHwK/cva/Um/U2xhWp9ae8ST6A8yn8yd0DbI4eHwA+CXwymuYvgWcpfGL/BHBujWs+JaplS1TXDVF7cc0GfJXCGQfPANk6GOuZFAL7+KK2uhpnCr98XgLGKBzf/Q9AO/AgsAP4KXBCNG0WuL1o3k8AvdHj4zWuuZfCsepD2/TXoml/D/jRZNtRDWv+drSt9lAI7fkTa45ef4DC2W6/qVbNpeqN2r91aPstmrYuxvhoH7pUgYhIoII6RCMiIm9SwIuIBEoBLyISKAW8iEigFPAiIoFSwIuIBEoBL5IiM7vKzP5XreuQ6UkBL1Jj0TeU9bMoidNGJQ3JzBaa2XYz+7oVbu6ywcxazWypmT1RdHOMkjf0iPp42MxuNbNc1Nd7zOye6CYfXyia7gozeyq6ycNtZnZM1D5shZtxPGtmPzWzs6I+nzezlUWLOilq32FmNxXV/5yZ3Unh6+4nmdm3zGxrdEOJa1MaOplGFPDSyJYAX3X3dwL7gFXAncBqd++i8FX5m47Qx+vungW+RuG6NJ8CTgeuMrN2M3sH8KfAee6+FHgD+LNo3pnAz6Ll7we+AFxM4dorNxct46yoti7gQ2aWLar/f0fzzwUWuPvp7n4G8M2jGA+RcZpqXYDIFOx0983R840ULk87x91/HrXdQelLGRc7dHGrZ4BnPbqIm5k9T+FKh+cD/xZ4Orp2WitvXoHydeAnRfOPuvuYmT0DLCxaxgPuPhj1e0/U5zrgt164gQsU7hp1ipl9BfghsKGC9ReZlAJeGtlo0fM3gDlT6CM/ob88hZ8PA+5w978uMe+Yv3kxp8Pzu3s+uvTsIRMv+HTo9WuHG9yHzOxdFO4m9UkKdxX6RPzVEXmTDtFISF4BhsxsWfT6z4GfTzJ9JR4EPmhm/wYO37T792P2cXE0XyuFWwT+68QJomvlZ9y9G/gchXuGikyJ9uAlNB8DvmZmb6Nw2OPjU+nM3beZ2eco3KYtQ+ESs58C4tyD9ikKN6HpBO5y95wVbilZbAHwzaKzaUr9xSASiy4XLCISKB2iEREJlA7RSPDM7KvAeROav+TuOhVRgqZDNCIigdIhGhGRQCngRUQCpYAXEQmUAl5EJFD/H2Xm4PUXN6ddAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "filenames": { "image/png": "/home/runner/work/BrownFall20/BrownFall20/_build/jupyter_execute/notes/2020-09-18_57_1.png" }, "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "safi_df.plot.scatter('no_membrs','rooms')" ] }, { "cell_type": "markdown", "id": "b9f5e24c", "metadata": {}, "source": [ "## After Class Questions\n", "\n", "````{dropdown} How can we clean data? what other topics will we cover in this class?\n", "\n", "Great Question! We'll get to data cleaning soon. See the [Schedule](schedule).\n", "````\n", "\n", "````{dropdown} How does the syntax on the question from prismia today work?\n", "\n", "The question was:\n", "\n", "> Read the tables off of the [syllabus course map]('https://rhodyprog4ds.github.io/BrownFall20/syllabus/course_map.html) page with `read_html` and make a list of the shapes of all of the tables on the page. Save the output to a variable and paste the *value* of that variable as your answer to the question.\n", "\n", "And the solution was:\n", "```{code-cell} ipython3\n", "import pandas as pd\n", "[df.shape for df in pd.read_html('https://rhodyprog4ds.github.io/BrownFall20/syllabus/course_map.html')]\n", "```\n", "\n", "This uses something called a [list comprehension](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions), they are a way to make lists, that look a lot like putting a for loop in a list. There are two ways to use them under the hood, as in the linked documentation, the one above is considered more pythonic, because it is more concise.\n", "\n", "The `[]` define a list, we can do that with anything, for example my zoom meetings today were:\n", "\n", "\n", "```{code-cell} ipython3\n", "todays_meetings = ['writing QEM','CSC310', 'office hours']\n", "todays_meetings\n", "```\n", "Note that the last line is there to display what happened, but it would work without.\n", "\n", "We could get the same output using [`append`](https://docs.python.org/2/tutorial/datastructures.html#more-on-lists) and a regular `for` loop. That syntax would look like:\n", "\n", "```{code-cell} ipython3\n", "shape_list = []\n", "\n", "for df in pd.read_html('https://rhodyprog4ds.github.io/BrownFall20/syllabus/course_map.html'):\n", " shape_list.append(df.shape)\n", "\n", "shape_list\n", "```\n", "\n", "````\n", "\n", "\n", "````{dropdown} Where I can go to find a list of all the syntax.\n", "\n", "A *list* of all the syntax might be hard to find, but the [course textbook](https://jakevdp.github.io/PythonDataScienceHandbook/) is free online with a lot of reference material in it.\n", "\n", "A lot of what we did today is in [Data Indexing and Selection](https://jakevdp.github.io/PythonDataScienceHandbook/03.02-data-indexing-and-selection.html)\n", "\n", "Also, the class notes are listed after each class on this website.\n", "\n", "Also, you can get help from within a notebook and the [pandas User guide](https://pandas.pydata.org/docs/user_guide/index.html) has full details.\n", "\n", "\n", "````\n", "\n", "## More Practice\n", "\n", "These additional questions are for if you want more practice with things we've done this week, before class next week.\n", "\n", "\n", "````{dropdown} Which of the following is a dictionary?\n", "```{code-cell} ipython3\n", "opt1 = [char for char in 'abcde']\n", "opt2 = {char:i for i, char in enumerate('abcde')}\n", "opt3 = ('a','b','c','d','e')\n", "opt4 = 'a b c d e'.split(' ')\n", "```\n", "\n", "Check using the `type` function. We can go further and build a list of them to display the one that's a dictionary.\n", "\n", "First, we can look at them\n", "```{code-cell} ipython3\n", "options = [opt1, opt2, opt3, opt4]\n", "\n", "for op in options:\n", " print(op)\n", "```\n", "\n", "Then we can check the types and find the dictionary\n", "```{code-cell} ipython3\n", "[op for op in options if type(op)==dict]\n", "```\n", "\n", "````\n", "\n", "\n", "````{dropdown} What type is the shape of a pandas DataFrame?\n", "\n", "We can find that by using the `type` function:\n", "```{code-cell} ipython3\n", "type(safi_df.shape)\n", "```\n", "\n", "`tuple` is another iterable type, so we can index them\n", "\n", "```{code-cell} ipython3\n", "safi_shape = safi_df.shape\n", "safi_shape[0]\n", "```\n", "\n", "Also, in python, we can assign to multiple values when it returns a tuple. We saw this before, when we used the `items()` method on a dictionary.\n", "\n", "```{code-cell} ipython3\n", "ex_dict = {char:i for i, char in enumerate('abcde')}\n", "\n", "for k,v in ex_dict.items():\n", " print(k, ' is the ',v,'th letter')\n", "\n", "```\n", "\n", "we can see that with the type function\n", "```{code-cell} ipython3\n", "[type(d) for ex_dict in d1.items()]\n", "```\n", "\n", "So, with the shape property we could also return it to two values\n", "\n", "```{code-cell} ipython3\n", "n_rows, n_cols = safi_df.shape\n", "print('there are ',n_rows, ' rows and ', n_cols, ' columns')\n", "```\n", "\n", "````\n", "\n", "````{dropdown} What does indexing with -1 do?\n", "\n", "It returns the last value, here's an example, using the `todays_meetings` variable defined above.\n", "\n", "```{code-cell} ipython3\n", "todays_meetings\n", "```\n", "\n", "\n", "```{code-cell} ipython3\n", "todays_meetings[-1]\n", "```\n", "\n", "## Further Reading\n", "\n", "\n", "\n", "If you've made it this far, [let me know](https://forms.gle/PDATAmuRS5tAHiiZ8) how you found these notes." ] } ], "metadata": { "jupytext": { "text_representation": { "extension": ".md", "format_name": "myst", "format_version": 0.12, "jupytext_version": "1.6.0" } }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" }, "source_map": [ 12, 24, 32, 36, 50, 52, 55, 57, 68, 70, 74, 76, 79, 83, 85, 95, 97, 100, 103, 106, 108, 112, 114, 118, 120, 123, 125, 133, 135, 141, 143, 147, 149, 153, 155, 158, 161, 165, 167, 171, 174, 178, 182, 191, 194, 198, 203, 208, 210, 214, 216, 220, 223, 227, 229, 233, 235, 239, 241 ] }, "nbformat": 4, "nbformat_minor": 5 }