daily pastebin goal
28%
SHARE
TWEET

Untitled

a guest May 16th, 2018 96 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. Assignment 3 - More Pandas
  2. This assignment requires more individual learning then the last one did - you are encouraged to check out the pandas documentation to find functions or methods you might not have used yet, or ask questions on Stack Overflow and tag them as pandas and python related. And of course, the discussion forums are open for interaction with your peers and the course staff.
  3.  
  4. Question 1 (20%)
  5. Load the energy data from the file Energy Indicators.xls, which is a list of indicators of energy supply and renewable electricity production from the United Nations for the year 2013, and should be put into a DataFrame with the variable name of energy.
  6.  
  7. Keep in mind that this is an Excel file, and not a comma separated values file. Also, make sure to exclude the footer and header information from the datafile. The first two columns are unneccessary, so you should get rid of them, and you should change the column labels so that the columns are:
  8.  
  9. ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']
  10.  
  11. Convert Energy Supply to gigajoules (there are 1,000,000 gigajoules in a petajoule). For all countries which have missing data (e.g. data with "...") make sure this is reflected as np.NaN values.
  12.  
  13. Rename the following list of countries (for use in later questions):
  14.  
  15. "Republic of Korea": "South Korea",
  16. "United States of America": "United States",
  17. "United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
  18. "China, Hong Kong Special Administrative Region": "Hong Kong"
  19.  
  20. There are also several countries with numbers and/or parenthesis in their name. Be sure to remove these,
  21.  
  22. e.g.
  23.  
  24. 'Bolivia (Plurinational State of)' should be 'Bolivia',
  25.  
  26. 'Switzerland17' should be 'Switzerland'.
  27.  
  28.  
  29.  
  30. Next, load the GDP data from the file world_bank.csv, which is a csv containing countries' GDP from 1960 to 2015 from World Bank. Call this DataFrame GDP.
  31.  
  32. Make sure to skip the header, and rename the following list of countries:
  33.  
  34. "Korea, Rep.": "South Korea",
  35. "Iran, Islamic Rep.": "Iran",
  36. "Hong Kong SAR, China": "Hong Kong"
  37.  
  38.  
  39.  
  40. Finally, load the Sciamgo Journal and Country Rank data for Energy Engineering and Power Technology from the file scimagojr-3.xlsx, which ranks countries based on their journal contributions in the aforementioned area. Call this DataFrame ScimEn.
  41.  
  42. Join the three datasets: GDP, Energy, and ScimEn into a new dataset (using the intersection of country names). Use only the last 10 years (2006-2015) of GDP data and only the top 15 countries by Scimagojr 'Rank' (Rank 1 through 15).
  43.  
  44. The index of this DataFrame should be the name of the country, and the columns should be ['Rank', 'Documents', 'Citable documents', 'Citations', 'Self-citations', 'Citations per document', 'H index', 'Energy Supply', 'Energy Supply per Capita', '% Renewable', '2006', '2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015'].
  45.  
  46. This function should return a DataFrame with 20 columns and 15 entries.
  47.  
  48. import pandas as pd
  49. import numpy as np
  50. def answer_one():
  51.     #Load xls file
  52.     xls_file = pd.ExcelFile('Energy Indicators.xls')
  53.     energy = xls_file.parse(skiprows=17,skip_footer=(38))
  54.     energy = energy[['Unnamed: 1','Petajoules','Gigajoules','%']]
  55.     # Rename the columns
  56.     energy.columns = ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']
  57.     # Convert "..." to np.NaN
  58.     energy[['Energy Supply', 'Energy Supply per Capita', '% Renewable']] =  energy[['Energy Supply', 'Energy Supply per Capita', '% Renewable']].replace('...',np.NaN).apply(pd.to_numeric)
  59.     #Conevrt Energy Supply to gigajoules (there are 1,000,000 gigajoules in a petajoule)
  60.     energy['Energy Supply'] = energy['Energy Supply']*1000000
  61.     #Rename Countries
  62.     energy['Country'] = energy['Country'].replace({"Republic of Korea": "South Korea", "United States of America": "United States",
  63.                                                    "United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
  64.                                                    "China, Hong Kong Special Administrative Region": "Hong Kong"})
  65.     #Remove parantheses
  66.     energy['Country'] = energy['Country'].str.replace(r" \(.*\)","")
  67.    
  68.     #GDP read_csv & Skip header & only include 10 last years
  69.     GDP = pd.read_csv('world_bank.csv', skiprows=4)
  70.     #Rename Countries
  71.     GDP = GDP[['Country Name','2006','2007','2008','2009','2010','2011','2012','2013','2014','2015']]
  72.     GDP['Country Name'] = GDP['Country Name'].replace({"Korea, Rep.": "South Korea", "Iran, Islamic Rep.": "Iran",
  73.                                              "Hong Kong SAR, China": "Hong Kong"})
  74.     #Only inlcude 10 last years of GDP
  75.     GDP.columns = ['Country','2006','2007','2008','2009','2010','2011','2012','2013','2014','2015']
  76.     #Load scimagojr-3.xlsx
  77.     ScimEn = pd.read_excel(io='scimagojr-3.xlsx')
  78.     ScimEn_top = ScimEn[:15]
  79.    
  80.     #Join the three DataFrames on Country
  81.     ScimEn_Energy = pd.merge(ScimEn_top, energy, how='inner', left_on='Country', right_on='Country')
  82.     df_joint = pd.merge(ScimEn_Energy, GDP, how='inner', left_on='Country', right_on='Country')
  83.     df_joint = df_joint.set_index('Country')
  84.     return df_joint
  85.  
  86. answer_one()
  87.  
  88. Question 2 (6.6%)
  89. The previous question joined three datasets then reduced this to just the top 15 entries. When you joined the datasets, but before you reduced this to the top 15 items, how many entries did you lose?
  90.  
  91. This function should return a single number.
  92.  
  93.  
  94. %%HTML
  95. <svg width="800" height="300">
  96.   <circle cx="150" cy="180" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="blue" />
  97.   <circle cx="200" cy="100" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="red" />
  98.   <circle cx="100" cy="100" r="80" fill-opacity="0.2" stroke="black" stroke-width="2" fill="green" />
  99.   <line x1="150" y1="125" x2="300" y2="150" stroke="black" stroke-width="2" fill="black" stroke-dasharray="5,3"/>
  100.   <text  x="300" y="165" font-family="Verdana" font-size="35">Everything but this!</text>
  101. </svg>
  102. Everything but this!
  103.  
  104. -
  105. def answer_two():
  106.         #Load xls file
  107.     xls_file = pd.ExcelFile('Energy Indicators.xls')
  108.     energy = xls_file.parse(skiprows=17,skip_footer=(38))
  109.     energy = energy[['Unnamed: 1','Petajoules','Gigajoules','%']]
  110.     # Rename the columns
  111.     energy.columns = ['Country', 'Energy Supply', 'Energy Supply per Capita', '% Renewable']
  112.     # Convert "..." to np.NaN
  113.     energy[['Energy Supply', 'Energy Supply per Capita', '% Renewable']] =  energy[['Energy Supply', 'Energy Supply per Capita', '% Renewable']].replace('...',np.NaN).apply(pd.to_numeric)
  114.     #Conevrt Energy Supply to gigajoules (there are 1,000,000 gigajoules in a petajoule)
  115.     energy['Energy Supply'] = energy['Energy Supply']*1000000
  116.     #Rename Countries
  117.     energy['Country'] = energy['Country'].replace({"Republic of Korea": "South Korea", "United States of America": "United States",
  118.                                                    "United Kingdom of Great Britain and Northern Ireland": "United Kingdom",
  119.                                                    "China, Hong Kong Special Administrative Region": "Hong Kong"})
  120.     #Remove parantheses
  121.     energy['Country'] = energy['Country'].str.replace(r" \(.*\)","")
  122.    
  123.     #GDP read_csv & Skip header & only include 10 last years
  124.     GDP = pd.read_csv('world_bank.csv', skiprows=4)
  125.     #Rename Countries
  126.     GDP = GDP[['Country Name','2006','2007','2008','2009','2010','2011','2012','2013','2014','2015']]
  127.     GDP['Country Name'] = GDP['Country Name'].replace({"Korea, Rep.": "South Korea", "Iran, Islamic Rep.": "Iran",
  128.                                              "Hong Kong SAR, China": "Hong Kong"})
  129.     #Only inlcude 10 last years of GDP
  130.     GDP.columns = ['Country','2006','2007','2008','2009','2010','2011','2012','2013','2014','2015']
  131.     #Load scimagojr-3.xlsx
  132.     ScimEn = pd.read_excel(io='scimagojr-3.xlsx')
  133.        
  134.     #Join the three DataFrames on Country
  135.     ScimEn_Energy = pd.merge(ScimEn, energy, how='inner', left_on='Country', right_on='Country')
  136.     df_joint = pd.merge(ScimEn_Energy, GDP, how='inner', left_on='Country', right_on='Country')
  137.     df_joint = df_joint.set_index('Country')
  138.     df_joint_top = df_joint[:15]
  139.    
  140.     #The Actual answer is this, but (I guess) since the df_joint and df_joint_top were defined in the function answer_one() they
  141.     # were not defined here so I repeated all the code lines, But I think there should be something like 'global' to use the
  142.     # variables defined in other functions
  143.     return len(df_joint.index) - len(df_joint_top.index)
  144. answer_two()
  145. 147
  146.  
  147. Question 3 (6.6%)
  148. What is the average GDP over the last 10 years for each country? (exclude missing values from this calculation.)
  149.  
  150. This function should return a Series named avgGDP with 15 countries and their average GDP sorted in descending order.
  151. def answer_three():
  152.     Top15 = answer_one()
  153.     avgGDP = Top15[['2006','2007','2008','2009','2010','2011','2012','2013','2014','2015']].mean(axis=1).rename('avgGDP').sort_values(ascending=False)
  154.     return avgGDP
  155. answer_three()
  156. Country
  157. United States         1.536434e+13
  158. China                 6.348609e+12
  159. Japan                 5.542208e+12
  160. Germany               3.493025e+12
  161. France                2.681725e+12
  162. United Kingdom        2.487907e+12
  163. Brazil                2.189794e+12
  164. Italy                 2.120175e+12
  165. India                 1.769297e+12
  166. Canada                1.660647e+12
  167. Russian Federation    1.565459e+12
  168. Spain                 1.418078e+12
  169. Australia             1.164043e+12
  170. South Korea           1.106715e+12
  171. Iran                  4.441558e+11
  172. Name: avgGDP, dtype: float64
  173.  
  174.  
  175. Question 4 (6.6%)
  176. By how much had the GDP changed over the 10 year span for the country with the 6th largest average GDP?
  177.  
  178. This function should return a single number.
  179.  
  180.  
  181. import pandas as pd
  182. import numpy as np
  183. def answer_four():
  184.     Top15 = answer_one()
  185.     avgGDP = answer_three()
  186.     sixth_avg_GDP = avgGDP[avgGDP == avgGDP[5]]
  187.     sixth_avg_GDP = sixth_avg_GDP.reset_index()
  188.     Sixth_avgGDP_name = sixth_avg_GDP.loc[0][0]
  189.     Top15.reset_index(inplace=True)
  190.     diff = (Top15[Top15['Country'] == Sixth_avgGDP_name]['2015']-Top15[Top15['Country'] == Sixth_avgGDP_name]['2006']).astype(float)
  191.     return diff.loc[3]
  192. answer_four()
  193.  
  194. import pandas as pd
  195. import numpy as np
  196. def answer_four():
  197.     Top15 = answer_one()
  198.     avgGDP = answer_three()
  199.     sixth_avg_GDP = avgGDP[avgGDP == avgGDP[5]]
  200.     sixth_avg_GDP = sixth_avg_GDP.reset_index()
  201.     Sixth_avgGDP_name = sixth_avg_GDP.loc[0][0]
  202.     Top15.reset_index(inplace=True)
  203.     diff = (Top15[Top15['Country'] == Sixth_avgGDP_name]['2015']-Top15[Top15['Country'] == Sixth_avgGDP_name]['2006']).astype(float)
  204.     return diff.loc[3]
  205. answer_four()
  206. 246702696075.3999
  207.  
  208.  
  209. Question 5 (6.6%)
  210. What is the mean Energy Supply per Capita?
  211.  
  212. This function should return a single number.
  213.  
  214.  
  215. def answer_five():
  216.     Top15 = answer_one()
  217.     mean = Top15["Energy Supply per Capita"].mean()
  218.     return mean
  219. answer_five()
  220. 157.6
  221.  
  222. Question 6 (6.6%)
  223. What country has the maximum % Renewable and what is the percentage?
  224.  
  225. This function should return a tuple with the name of the country and the percentage.
  226.  
  227. def answer_six():
  228.     Top15 = answer_one()
  229.     Max_Renewable = Top15[Top15['% Renewable'] == Top15['% Renewable'].max()]
  230.     return (Max_Renewable.index.tolist()[0],Max_Renewable['% Renewable'].tolist()[0])
  231. answer_six()
  232. ('Brazil', 69.64803)
  233.  
  234. Question 7 (6.6%)
  235. Create a new column that is the ratio of Self-Citations to Total Citations. What is the maximum value for this new column, and what country has the highest ratio?
  236.  
  237. This function should return a tuple with the name of the country and the ratio.
  238.  
  239. def answer_seven():
  240.     Top15 = answer_one()
  241.     Top15['Ratio_Self_Citation'] = Top15['Self-citations']/Top15['Citations']
  242.     Max_Ratio_Self_Citation = Top15[Top15['Ratio_Self_Citation'] == Top15['Ratio_Self_Citation'].max()]
  243.     return (Max_Ratio_Self_Citation.index.tolist()[0],Max_Ratio_Self_Citation['Ratio_Self_Citation'].tolist()[0])
  244. answer_seven()
  245. ('China', 0.6893126179389422)
  246.  
  247. Question 8 (6.6%)
  248. Create a column that estimates the population using Energy Supply and Energy Supply per capita. What is the third most populous country according to this estimate?
  249.  
  250. This function should return a single string value.
  251.  
  252. def answer_eight():
  253.     Top15 = answer_one()
  254.     Top15['Population_Estimate'] = Top15['Energy Supply']/Top15['Energy Supply per Capita']
  255.     df_Population = Top15['Population_Estimate'].sort_values(ascending=False).reset_index()
  256.     Third_Most_Populous = df_Population['Country'][2]
  257.     return Third_Most_Populous
  258.  
  259. answer_eight()
  260. 'United States'
  261.  
  262. Question 9 (6.6%)
  263. Create a column that estimates the number of citable documents per person. What is the correlation between the number of citable documents per capita and the energy supply per capita? Use the .corr() method, (Pearson's correlation).
  264.  
  265. This function should return a single number.
  266.  
  267. (Optional: Use the built-in function plot9() to visualize the relationship between Energy Supply per Capita vs. Citable docs per Capita)
  268.  
  269. def answer_nine():
  270.     Top15 = answer_one()
  271.     Top15['Population_Estimate'] = Top15['Energy Supply']/Top15['Energy Supply per Capita']
  272.     Top15['Citable_Docs'] = Top15['Citable documents']/Top15['Population_Estimate']
  273.     #Top15[['Country','Population_Estimate', 'Citable_Docs']]
  274.     correlation = Top15['Citable_Docs'].corr(Top15['Energy Supply per Capita'])
  275.     return correlation
  276. answer_nine()
  277. 0.79400104354429424
  278.  
  279. Question 10 (6.6%)
  280. Create a new column with a 1 if the country's % Renewable value is at or above the median for all countries in the top 15, and a 0 if the country's % Renewable value is below the median.
  281.  
  282. This function should return a series named HighRenew whose index is the country name sorted in ascending order of rank.
  283.  
  284. def answer_ten():
  285.     Top15 = answer_one()
  286.     median = Top15['% Renewable'].median()
  287.     Top15['HighRenew'] = (Top15['% Renewable'] >= median).astype(int)
  288.     Sorted = Top15.sort('Rank')
  289.     return Sorted.HighRenew
  290. answer_ten()
  291. Country
  292. China                 1
  293. United States         0
  294. Japan                 0
  295. United Kingdom        0
  296. Russian Federation    1
  297. Canada                1
  298. Germany               1
  299. India                 0
  300. France                1
  301. South Korea           0
  302. Italy                 1
  303. Spain                 1
  304. Iran                  0
  305. Australia             0
  306. Brazil                1
  307. Name: HighRenew, dtype: int64
  308.  
  309.  
  310. Question 11 (6.6%)
  311. Use the following dictionary to group the Countries by Continent, then create a dateframe that displays the sample size (the number of countries in each continent bin), and the sum, mean, and std deviation for the estimated population of each country.
  312.  
  313. ContinentDict  = {'China':'Asia',
  314.                   'United States':'North America',
  315.                   'Japan':'Asia',
  316.                   'United Kingdom':'Europe',
  317.                   'Russian Federation':'Europe',
  318.                   'Canada':'North America',
  319.                   'Germany':'Europe',
  320.                   'India':'Asia',
  321.                   'France':'Europe',
  322.                   'South Korea':'Asia',
  323.                   'Italy':'Europe',
  324.                   'Spain':'Europe',
  325.                   'Iran':'Asia',
  326.                   'Australia':'Australia',
  327.                   'Brazil':'South America'}
  328. This function should return a DataFrame with index named Continent ['Asia', 'Australia', 'Europe', 'North America', 'South America'] and columns ['size', 'sum', 'mean', 'std']
  329.  
  330. def answer_eleven():
  331.     Top15 = answer_one()
  332.     ContinentDict  = {'China':'Asia',
  333.                   'United States':'North America',
  334.                   'Japan':'Asia',
  335.                   'United Kingdom':'Europe',
  336.                   'Russian Federation':'Europe',
  337.                   'Canada':'North America',
  338.                   'Germany':'Europe',
  339.                   'India':'Asia',
  340.                   'France':'Europe',
  341.                   'South Korea':'Asia',
  342.                   'Italy':'Europe',
  343.                   'Spain':'Europe',
  344.                   'Iran':'Asia',
  345.                   'Australia':'Australia',
  346.                   'Brazil':'South America'}
  347.     Top15['Population_Estimate'] = (Top15['Energy Supply']/Top15['Energy Supply per Capita']).astype(float)
  348.     Top15.reset_index(inplace=True)
  349.     Top15['Continent'] = [ContinentDict[country] for country in Top15['Country']]
  350.     df = Top15.set_index('Continent').groupby(level=0)['Population_Estimate'].agg({'size': np.size, 'sum': np.sum, 'mean': np.mean,'std': np.std})
  351.     return df
  352.    
  353. answer_eleven()
  354.  
  355.  
  356. Question 12 (6.6%)
  357. Cut % Renewable into 5 bins. Group Top15 by the Continent, as well as these new % Renewable bins. How many countries are in each of these groups?
  358.  
  359. This function should return a Series with a MultiIndex of Continent, then the bins for % Renewable. Do not include groups with no countries.
  360.  
  361. def answer_twelve():
  362.     Top15 = answer_one()
  363.     ContinentDict  = {'China':'Asia',
  364.                   'United States':'North America',
  365.                   'Japan':'Asia',
  366.                   'United Kingdom':'Europe',
  367.                   'Russian Federation':'Europe',
  368.                   'Canada':'North America',
  369.                   'Germany':'Europe',
  370.                   'India':'Asia',
  371.                   'France':'Europe',
  372.                   'South Korea':'Asia',
  373.                   'Italy':'Europe',
  374.                   'Spain':'Europe',
  375.                   'Iran':'Asia',
  376.                   'Australia':'Australia',
  377.                   'Brazil':'South America'}
  378.     Top15.reset_index(inplace=True)
  379.     Top15['Continent'] = [ContinentDict[country] for country in Top15['Country']]
  380.     Top15['Renew_bins'] = pd.cut(Top15['% Renewable'],5)
  381.     df = Top15.groupby(['Continent','Renew_bins']).size()
  382.     return df
  383. answer_twelve()
  384.  
  385. Continent      Renew_bins      
  386. Asia           (2.212, 15.753]     4
  387.                (15.753, 29.227]    1
  388. Australia      (2.212, 15.753]     1
  389. Europe         (2.212, 15.753]     1
  390.                (15.753, 29.227]    3
  391.                (29.227, 42.701]    2
  392. North America  (2.212, 15.753]     1
  393.                (56.174, 69.648]    1
  394. South America  (56.174, 69.648]    1
  395. dtype: int64
  396.  
  397.  
  398. Question 13 (6.6%)
  399. Convert the Population Estimate series to a string with thousands separator (using commas). Do not round the results.
  400.  
  401. e.g. 317615384.61538464 -> 317,615,384.61538464
  402.  
  403. This function should return a Series PopEst whose index is the country name and whose values are the population estimate string.
  404.  
  405. def answer_thirteen():
  406.     Top15 = answer_one()
  407.     Top15['Population_Estimate'] = (Top15['Energy Supply']/Top15['Energy Supply per Capita']).astype(float)
  408.     ans = Top15['Population_Estimate'].apply(lambda x : '{:,}'.format(x))
  409.     return ans
  410. answer_thirteen()
  411. Country
  412. China                 1,367,645,161.2903225
  413. United States          317,615,384.61538464
  414. Japan                  127,409,395.97315437
  415. United Kingdom         63,870,967.741935484
  416. Russian Federation            143,500,000.0
  417. Canada                  35,239,864.86486486
  418. Germany                 80,369,696.96969697
  419. India                 1,276,730,769.2307692
  420. France                  63,837,349.39759036
  421. South Korea            49,805,429.864253394
  422. Italy                  59,908,256.880733944
  423. Spain                    46,443,396.2264151
  424. Iran                    77,075,630.25210084
  425. Australia              23,316,017.316017315
  426. Brazil                 205,915,254.23728815
  427. Name: Population_Estimate, dtype: object
  428.  
  429.  
  430. Optional
  431. Use the built in function plot_optional() to see an example visualization.
  432.  
  433.  
  434. def plot_optional():
  435.     import matplotlib as plt
  436.     %matplotlib inline
  437.     Top15 = answer_one()
  438.     ax = Top15.plot(x='Rank', y='% Renewable', kind='scatter',
  439.                     c=['#e41a1c','#377eb8','#e41a1c','#4daf4a','#4daf4a','#377eb8','#4daf4a','#e41a1c',
  440.                        '#4daf4a','#e41a1c','#4daf4a','#4daf4a','#e41a1c','#dede00','#ff7f00'],
  441.                     xticks=range(1,16), s=6*Top15['2014']/10**10, alpha=.75, figsize=[16,6]);
  442.     for i, txt in enumerate(Top15.index):
  443.         ax.annotate(txt, [Top15['Rank'][i], Top15['% Renewable'][i]], ha='center')
  444.     print("This is an example of a visualization that can be created to help understand the data. \
  445. This is a bubble chart showing % Renewable vs. Rank. The size of the bubble corresponds to the countries' \
  446. 2014 GDP, and the color corresponds to the continent.")
  447.  
  448. plot_optional() # Be sure to comment out plot_optional() before submitting the assignment!
  449. This is an example of a visualization that can be created to help understand the data. This is a bubble chart showing % Renewable vs. Rank. The size of the bubble corresponds to the countries' 2014 GDP, and the color corresponds to the continent.
RAW Paste Data
Top