Project 2: Holiday weather

by Rob Garnsey, 6 November 2015

This is the project notebook for Week 2 of The Open University's Learn to code for Data Analysis course.

In this project I am going to use the historic weather data from the Weather Underground for Sydney, Australia, as a guide for choosing two weeks in which to holiday this coming summer (December, January, February).

Getting the data

In the southern hemisphere summer takes place from December to February inclusive - in other words it straddles two consecutive years. I chose to download twelve months of data between 1 October 2014 to 30 September 2015 for this exercise.

I discovered (to my cost) that it is really important to use the following steps exactly as described for downloading and renaming the CSV file. Be careful not to open and save your file with Excel because it seems to cause PANDAS problems with interpreting dates.

At the http://www.wunderground.com/history site type the required city in the 'Location' input box and confirm that it has found the correct town by selecting it and clicking 'Submit'.

When the next page opens click on the 'Custom' tab and selet the time period for the data you want to download and then click on 'Get History'. The data for that year should then be displayed. Scroll to the end of the data and then right click on the blue link labelled 'Comma Delimited File':

• if you are using the Safari Browser choose Download Linked File As ...
• if you are using the Chrome Browser choose Save Link As ...

then, in the File dialogue that appears save the file with its default name of 'CustomHistory' to the folder you created for this course and where this notebook is located. Once the file has been downloaded rename it from 'CustomHistory.html' to 'City_name_2014.csv'.

Now load the CSV file into a dataframe making sure that any extra spaces are skipped:

from pandas import *
sydney = read_csv('Sydney_2014-15.csv')

sydney.head(365)

	AEDT	Max TemperatureC	Mean TemperatureC	Min TemperatureC	Dew PointC	MeanDew PointC	Min DewpointC	Max Humidity	Mean Humidity	Min Humidity	...	Max VisibilityKm	Mean VisibilityKm	Min VisibilitykM	Max Wind SpeedKm/h	Mean Wind SpeedKm/h	Max Gust SpeedKm/h	Precipitationmm	CloudCover	Events	WindDirDegrees<br />
0	2014-10-1	34	23	13	6	1	-8	55	29	10	...	31	21	10	50	27	60	0.00	1	NaN	230<br />
1	2014-10-2	26	18	10	14	5	-4	88	45	12	...	31	20	10	39	19	58	0.00	1	NaN	276<br />
2	2014-10-3	20	17	13	14	11	10	88	69	46	...	31	13	10	27	11	NaN	0.00	4	NaN	131<br />
3	2014-10-4	23	19	16	17	14	11	83	68	48	...	31	12	10	34	14	NaN	0.00	4	Rain	21<br />
4	2014-10-5	34	24	14	16	10	-1	94	56	5	...	31	13	8	53	18	72	0.00	4	NaN	309<br />
5	2014-10-6	26	21	17	17	14	12	83	64	34	...	31	13	10	42	24	NaN	0.00	5	NaN	26<br />
6	2014-10-7	27	22	18	15	9	4	78	42	15	...	31	15	10	42	21	53	0.25	4	Rain	84<br />
7	2014-10-8	20	18	15	16	12	8	94	69	38	...	31	11	5	39	21	NaN	0.25	5	Rain	142<br />
8	2014-10-9	22	18	14	12	11	9	82	59	34	...	31	12	10	34	13	NaN	0.00	3	NaN	67<br />
9	2014-10-10	24	19	14	15	12	9	78	58	29	...	31	19	10	35	16	NaN	0.00	2	NaN	7<br />
10	2014-10-11	28	21	14	18	13	7	88	60	18	...	31	26	26	29	13	NaN	0.00	NaN	NaN	5<br />
11	2014-10-12	28	22	16	17	14	5	88	62	20	...	31	20	10	50	18	63	0.00	1	NaN	21<br />
12	2014-10-13	26	20	15	17	14	12	94	70	31	...	31	11	2	53	18	74	10.92	5	Rain-Thunderstorm	360<br />
13	2014-10-14	18	14	10	14	9	3	100	68	27	...	31	9	0	93	27	106	83.06	6	Rain-Thunderstorm	237<br />
14	2014-10-15	15	12	9	12	8	5	100	71	42	...	31	10	3	69	52	101	0.25	6	Rain-Thunderstorm	211<br />
15	2014-10-16	22	16	9	10	6	1	82	52	19	...	26	17	10	29	14	NaN	0.00	1	NaN	356<br />
16	2014-10-17	18	14	12	12	9	6	88	65	40	...	31	12	10	42	21	50	0.00	3	Rain	185<br />
17	2014-10-18	21	17	13	9	8	6	72	53	28	...	26	12	10	26	13	NaN	0.00	5	NaN	34<br />
18	2014-10-19	25	19	13	13	10	7	77	53	25	...	31	22	10	35	16	NaN	0.00	1	NaN	4<br />
19	2014-10-20	19	17	15	15	12	9	88	68	48	...	31	12	9	61	26	77	0.00	6	Rain	175<br />
20	2014-10-21	18	17	15	11	7	5	77	50	32	...	31	12	10	24	16	NaN	0.00	6	Rain	88<br />
21	2014-10-22	22	18	14	14	10	6	73	53	29	...	31	14	10	40	19	NaN	0.00	5	NaN	15<br />
22	2014-10-23	27	21	15	18	14	12	94	66	31	...	31	15	10	24	11	NaN	0.00	4	NaN	67<br />
23	2014-10-24	28	23	18	18	17	14	94	75	36	...	19	11	10	40	19	53	0.25	6	Rain	31<br />
24	2014-10-25	25	21	18	19	18	16	100	85	49	...	26	8	0	32	16	NaN	0.51	4	Fog-Rain-Thunderstorm	209<br />
25	2014-10-26	31	24	18	19	17	16	94	77	31	...	23	10	6	39	19	NaN	0.25	5	Rain-Thunderstorm	213<br />
26	2014-10-27	33	26	18	17	12	0	83	52	11	...	26	14	10	58	19	77	0.00	5	NaN	3<br />
27	2014-10-28	29	23	16	15	4	-6	83	38	4	...	31	23	10	35	19	47	0.00	NaN	NaN	277<br />
28	2014-10-29	23	19	16	15	10	3	73	50	21	...	31	13	10	32	18	NaN	0.00	3	NaN	103<br />
29	2014-10-30	31	23	16	14	10	0	82	48	8	...	31	19	10	29	16	NaN	0.00	1	NaN	348<br />
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
335	2015-9-1	17	12	8	9	4	2	82	54	25	...	31	19	10	27	14	NaN	0.00	1	NaN	230<br />
336	2015-9-2	20	13	7	9	6	3	76	52	27	...	31	25	10	29	13	NaN	2.03	6	Rain	352<br />
337	2015-9-3	20	16	11	11	8	2	94	61	19	...	31	12	10	42	19	60	3.05	5	Rain	264<br />
338	2015-9-4	17	14	12	13	11	8	100	73	49	...	26	10	4	35	29	48	0.25	5	Rain	198<br />
339	2015-9-5	19	15	11	12	10	8	88	69	39	...	31	13	10	27	14	NaN	2.03	4	Rain	223<br />
340	2015-9-6	17	14	12	13	12	10	90	78	54	...	26	12	7	23	13	NaN	3.05	6	Rain	253<br />
341	2015-9-7	24	17	9	12	7	-1	100	56	18	...	31	11	5	40	16	47	0.25	1	Fog	309<br />
342	2015-9-8	20	14	9	7	1	-2	63	38	15	...	10	10	10	35	24	52	0.00	1	NaN	268<br />
343	2015-9-9	17	13	10	10	6	3	68	53	32	...	31	13	10	32	21	NaN	0.00	1	NaN	218<br />
344	2015-9-10	18	14	11	11	8	6	82	58	34	...	31	13	10	32	16	NaN	0.00	4	NaN	188<br />
345	2015-9-11	23	18	14	10	8	6	72	51	28	...	31	16	10	34	14	NaN	0.00	5	NaN	348<br />
346	2015-9-12	25	18	11	14	10	6	88	61	23	...	31	27	19	24	10	NaN	0.00	NaN	NaN	340<br />
347	2015-9-13	23	18	12	16	12	8	88	65	29	...	31	23	10	27	14	NaN	0.00	1	NaN	12<br />
348	2015-9-14	24	19	14	15	13	11	88	64	40	...	31	29	26	39	14	NaN	0.00	NaN	NaN	18<br />
349	2015-9-15	31	22	14	13	7	2	94	41	9	...	31	28	19	50	19	64	0.00	NaN	NaN	306<br />
350	2015-9-16	19	16	12	12	8	4	68	55	32	...	31	13	10	27	18	NaN	0.00	2	NaN	174<br />
351	2015-9-17	18	15	12	14	11	8	94	70	51	...	31	12	9	37	16	53	1.02	3	Rain-Thunderstorm	188<br />
352	2015-9-18	18	16	13	12	10	8	95	67	44	...	31	12	5	42	29	50	9.91	4	Rain	181<br />
353	2015-9-19	19	16	12	13	11	7	100	72	41	...	31	11	5	29	16	NaN	0.00	6	Rain	143<br />
354	2015-9-20	18	14	12	14	11	8	94	73	48	...	31	12	5	21	11	NaN	0.51	4	Rain	83<br />
355	2015-9-21	23	18	12	14	12	8	94	68	34	...	31	12	8	29	13	26	0.25	2	Fog-Rain	342<br />
356	2015-9-22	18	16	13	12	8	4	77	56	29	...	31	13	10	48	26	66	0.00	4	Rain	215<br />
357	2015-9-23	14	12	11	9	6	2	88	63	45	...	27	11	5	58	42	76	3.05	4	Rain	213<br />
358	2015-9-24	14	11	9	12	8	2	100	76	51	...	31	9	3	63	39	82	13.97	5	Rain	209<br />
359	2015-9-25	16	13	10	12	9	7	94	73	46	...	26	10	5	47	37	66	0.51	6	Rain	187<br />
360	2015-9-26	18	16	12	12	9	4	94	66	29	...	31	12	6	29	24	50	2.03	5	Rain	167<br />
361	2015-9-27	17	14	12	12	9	8	94	69	47	...	31	12	5	42	26	52	0.00	5	Rain	210<br />
362	2015-9-28	20	16	11	10	8	5	77	58	29	...	31	16	10	21	14	NaN	0.00	2	NaN	178<br />
363	2015-9-29	25	18	11	12	9	4	82	52	18	...	31	19	10	32	13	NaN	0.00	2	NaN	343<br />
364	2015-9-30	20	18	17	16	12	9	88	67	47	...	31	12	10	32	21	NaN	0.00	5	Rain	145<br />

365 rows × 23 columns

Cleaning the data

First we need to clean up the data. I'm not going to make use of 'WindDirDegrees' in my analysis, but you might in yours so we'll rename 'WindDirDegrees< br />' to 'WindDirDegrees'.

sydney = sydney.rename(columns={'WindDirDegrees<br />' : 'WindDirDegrees'})

remove the < br /> html line breaks from the values in the 'WindDirDegrees' column.

sydney['WindDirDegrees'] = sydney['WindDirDegrees'].str.rstrip('<br />')

and change the values in the 'WindDirDegrees' column to float64:

sydney['WindDirDegrees'] = sydney['WindDirDegrees'].astype('float64')   

We definitely need to change the values in the timezone column into values of the datetime64 date type.

sydney['AEDT'] = to_datetime(sydney['AEDT'])

We also need to change the index from the default to the datetime64 values in the timezone column so that it is easier to pull out rows between particular dates and display more meaningful graphs:

sydney.index = sydney['AEDT']

Finding a summer break

According to meteorologists, summer extends for the whole months of June, July, and August in the northern hemisphere and the whole months of December, January, and February in the southern hemisphere. As I'm in the southern hemisphere I'm going to create a dataframe that holds just December, January and February using the datetime index, like this:

summer = sydney.ix[datetime(2014,12,1) : datetime(2015,2,28)]
summer.head(50)

	AEDT	Max TemperatureC	Mean TemperatureC	Min TemperatureC	Dew PointC	MeanDew PointC	Min DewpointC	Max Humidity	Mean Humidity	Min Humidity	...	Max VisibilityKm	Mean VisibilityKm	Min VisibilitykM	Max Wind SpeedKm/h	Mean Wind SpeedKm/h	Max Gust SpeedKm/h	Precipitationmm	CloudCover	Events	WindDirDegrees
AEDT
2014-12-01	2014-12-01	29	24	19	20	17	16	88	65	37	...	31	13	10	40	18	NaN	0.00	5	Rain	26
2014-12-02	2014-12-02	31	24	19	21	18	16	100	68	36	...	26	11	6	47	31	61	0.76	3	Rain-Thunderstorm	27
2014-12-03	2014-12-03	33	27	21	21	19	18	94	72	30	...	31	11	3	52	23	72	8.89	4	Rain-Thunderstorm	99
2014-12-04	2014-12-04	29	24	20	21	19	18	94	73	44	...	26	12	4	37	16	58	0.25	6	Rain-Thunderstorm	13
2014-12-05	2014-12-05	29	24	19	21	17	13	94	68	30	...	31	13	10	34	18	NaN	0.25	5	Rain-Thunderstorm	83
2014-12-06	2014-12-06	28	24	19	21	18	14	94	73	32	...	26	11	2	27	13	NaN	16.00	6	Rain-Thunderstorm	92
2014-12-07	2014-12-07	29	23	18	21	18	17	100	77	44	...	26	12	2	37	14	NaN	23.88	3	Rain-Thunderstorm	21
2014-12-08	2014-12-08	33	26	18	21	17	10	90	64	21	...	31	13	6	58	18	76	1.02	3	Rain-Thunderstorm	302
2014-12-09	2014-12-09	24	22	21	19	17	16	88	74	58	...	31	12	10	27	21	NaN	0.00	6	NaN	134
2014-12-10	2014-12-10	25	22	19	21	18	16	94	75	51	...	31	12	6	34	14	NaN	6.10	6	Rain-Thunderstorm	65
2014-12-11	2014-12-11	23	20	17	21	18	15	94	85	67	...	19	9	4	58	24	80	4.06	-161	Rain-Thunderstorm	205
2014-12-12	2014-12-12	21	19	18	15	12	9	83	61	43	...	31	12	10	50	39	69	0.00	4	Rain	162
2014-12-13	2014-12-13	23	20	17	12	11	8	68	51	32	...	31	13	10	35	27	NaN	0.00	4	NaN	131
2014-12-14	2014-12-14	25	20	16	14	12	9	82	54	31	...	31	14	10	34	14	NaN	0.00	3	NaN	27
2014-12-15	2014-12-15	27	22	17	15	13	11	83	54	30	...	31	15	10	35	14	NaN	0.00	1	NaN	30
2014-12-16	2014-12-16	26	23	21	19	17	14	88	64	46	...	31	18	10	58	27	76	0.00	1	Rain	31
2014-12-17	2014-12-17	22	20	19	19	14	11	88	64	38	...	19	11	7	50	23	66	0.00	5	NaN	158
2014-12-18	2014-12-18	27	23	18	21	17	12	88	69	48	...	26	11	8	29	13	NaN	0.00	5	Rain	57
2014-12-19	2014-12-19	23	21	18	20	13	8	94	61	35	...	31	12	9	48	24	69	0.00	4	NaN	174
2014-12-20	2014-12-20	23	21	18	13	11	9	68	52	33	...	31	14	10	27	16	NaN	0.00	5	NaN	118
2014-12-21	2014-12-21	27	22	17	18	16	13	82	61	36	...	31	13	10	42	16	60	0.00	2	NaN	20
2014-12-22	2014-12-22	28	24	21	19	18	17	83	66	39	...	31	12	10	52	26	61	0.00	4	Rain	34
2014-12-23	2014-12-23	27	24	21	23	18	17	94	69	46	...	31	12	8	35	26	NaN	2.03	5	Rain-Thunderstorm	28
2014-12-24	2014-12-24	23	22	20	21	18	17	94	84	72	...	26	11	10	42	19	55	0.00	6	Rain	176
2014-12-25	2014-12-25	29	24	19	21	19	17	100	79	49	...	23	10	2	37	14	NaN	7.87	5	Rain-Thunderstorm	87
2014-12-26	2014-12-26	27	23	19	20	16	9	100	70	33	...	31	12	10	45	18	NaN	0.25	5	Rain	207
2014-12-27	2014-12-27	22	20	19	13	11	8	64	49	33	...	31	12	10	27	19	NaN	0.00	6	NaN	128
2014-12-28	2014-12-28	26	22	18	17	14	12	83	62	39	...	31	13	10	27	14	NaN	0.00	6	Rain-Snow	6
2014-12-29	2014-12-29	28	23	18	20	17	15	83	64	41	...	31	14	6	39	19	52	0.00	1	Rain-Thunderstorm	16
2014-12-30	2014-12-30	34	27	19	19	14	-1	83	53	5	...	31	16	10	42	16	50	0.00	1	NaN	143
2014-12-31	2014-12-31	26	22	19	19	16	14	78	62	41	...	26	12	10	37	26	NaN	0.25	3	Rain	94
2015-01-01	2015-01-01	30	26	22	22	19	18	89	67	44	...	23	11	4	39	18	48	0.00	3	Rain	35
2015-01-02	2015-01-02	26	23	21	22	21	18	100	85	68	...	23	10	8	35	23	48	0.00	6	Rain	175
2015-01-03	2015-01-03	31	27	23	22	21	18	94	71	41	...	31	12	10	48	19	61	0.00	2	NaN	35
2015-01-04	2015-01-04	33	27	22	21	20	18	88	68	39	...	31	13	10	53	26	69	0.00	3	NaN	51
2015-01-05	2015-01-05	25	23	21	21	19	17	94	78	56	...	26	11	10	35	26	NaN	0.00	6	NaN	165
2015-01-06	2015-01-06	28	24	21	21	20	17	94	77	42	...	31	12	10	24	16	NaN	0.00	4	NaN	144
2015-01-07	2015-01-07	31	27	22	21	18	12	94	61	22	...	31	13	10	42	14	NaN	0.00	3	NaN	50
2015-01-08	2015-01-08	31	26	21	19	17	14	73	58	32	...	31	16	10	52	23	71	0.00	2	NaN	25
2015-01-09	2015-01-09	32	27	22	21	19	18	83	64	36	...	26	16	10	42	21	58	0.25	1	Rain-Thunderstorm	40
2015-01-10	2015-01-10	28	24	21	22	21	19	100	80	51	...	31	11	5	47	16	63	5.08	6	Rain-Thunderstorm	120
2015-01-11	2015-01-11	22	21	20	21	20	19	94	92	84	...	26	10	6	34	26	48	3.05	6	Rain	174
2015-01-12	2015-01-12	23	21	20	20	19	17	94	82	69	...	26	11	8	40	32	61	1.02	6	Rain	184
2015-01-13	2015-01-13	27	23	19	21	19	18	100	80	54	...	26	11	4	40	21	60	0.25	6	Rain	21
2015-01-14	2015-01-14	34	28	23	23	17	13	94	54	19	...	31	14	10	45	23	53	0.25	4	Rain	300
2015-01-15	2015-01-15	26	23	22	19	16	13	73	56	40	...	31	13	10	32	23	NaN	0.00	2	NaN	178
2015-01-16	2015-01-16	28	24	21	19	18	17	88	64	41	...	31	14	10	27	14	NaN	0.00	4	NaN	98
2015-01-17	2015-01-17	35	28	21	21	12	5	88	40	9	...	31	19	10	34	16	50	0.00	1	NaN	295
2015-01-18	2015-01-18	27	24	21	21	18	17	88	70	49	...	26	11	10	29	24	NaN	0.00	4	NaN	163
2015-01-19	2015-01-19	23	21	20	18	15	12	83	65	45	...	26	11	8	35	27	NaN	0.00	6	Rain	159

50 rows × 23 columns

I now look for the days with warm temperatures.

summer[summer['Mean TemperatureC'] >= 25]

	AEDT	Max TemperatureC	Mean TemperatureC	Min TemperatureC	Dew PointC	MeanDew PointC	Min DewpointC	Max Humidity	Mean Humidity	Min Humidity	...	Max VisibilityKm	Mean VisibilityKm	Min VisibilitykM	Max Wind SpeedKm/h	Mean Wind SpeedKm/h	Max Gust SpeedKm/h	Precipitationmm	CloudCover	Events	WindDirDegrees
AEDT
2014-12-03	2014-12-03	33	27	21	21	19	18	94	72	30	...	31	11	3	52	23	72	8.89	4	Rain-Thunderstorm	99
2014-12-08	2014-12-08	33	26	18	21	17	10	90	64	21	...	31	13	6	58	18	76	1.02	3	Rain-Thunderstorm	302
2014-12-30	2014-12-30	34	27	19	19	14	-1	83	53	5	...	31	16	10	42	16	50	0.00	1	NaN	143
2015-01-01	2015-01-01	30	26	22	22	19	18	89	67	44	...	23	11	4	39	18	48	0.00	3	Rain	35
2015-01-03	2015-01-03	31	27	23	22	21	18	94	71	41	...	31	12	10	48	19	61	0.00	2	NaN	35
2015-01-04	2015-01-04	33	27	22	21	20	18	88	68	39	...	31	13	10	53	26	69	0.00	3	NaN	51
2015-01-07	2015-01-07	31	27	22	21	18	12	94	61	22	...	31	13	10	42	14	NaN	0.00	3	NaN	50
2015-01-08	2015-01-08	31	26	21	19	17	14	73	58	32	...	31	16	10	52	23	71	0.00	2	NaN	25
2015-01-09	2015-01-09	32	27	22	21	19	18	83	64	36	...	26	16	10	42	21	58	0.25	1	Rain-Thunderstorm	40
2015-01-14	2015-01-14	34	28	23	23	17	13	94	54	19	...	31	14	10	45	23	53	0.25	4	Rain	300
2015-01-17	2015-01-17	35	28	21	21	12	5	88	40	9	...	31	19	10	34	16	50	0.00	1	NaN	295
2015-01-22	2015-01-22	28	26	22	21	19	17	88	66	43	...	26	12	10	35	16	NaN	0.00	2	NaN	47
2015-01-23	2015-01-23	31	27	23	21	20	19	83	66	41	...	31	13	10	47	21	55	0.00	3	NaN	27
2015-01-24	2015-01-24	31	27	23	22	19	16	89	62	30	...	31	14	10	39	26	NaN	0.00	2	Rain-Thunderstorm	31
2015-01-25	2015-01-25	35	28	21	22	19	15	89	64	23	...	31	15	10	50	18	69	0.00	2	Rain	304
2015-02-08	2015-02-08	33	26	19	23	17	13	94	59	26	...	31	20	10	23	13	NaN	0.00	1	NaN	4
2015-02-11	2015-02-11	30	26	22	20	18	15	88	62	31	...	31	13	10	35	18	NaN	0.00	2	Rain	42
2015-02-12	2015-02-12	29	26	22	20	19	16	88	67	38	...	31	12	6	29	16	NaN	0.00	2	Rain-Thunderstorm	87
2015-02-16	2015-02-16	29	26	22	19	17	15	73	58	35	...	31	13	10	37	24	NaN	0.00	4	NaN	36
2015-02-17	2015-02-17	29	26	22	19	17	11	78	58	24	...	31	13	10	42	23	NaN	0.00	2	NaN	36
2015-02-18	2015-02-18	29	26	23	19	17	16	73	58	34	...	31	12	10	37	23	NaN	0.00	1	NaN	43

21 rows × 23 columns

No shortage of warm days over the summer of 2014-14 as can be seen by the following graph.

First tell Jupyter to display any graph that I create inside this notebook:

%matplotlib inline

Now plot the 'Mean TemperatureC' for the summer:

summer['Mean TemperatureC'].plot(grid=True, figsize=(10,5))

<matplotlib.axes._subplots.AxesSubplot at 0x93367b8>

Plenty of mean temperatures over 20 degrees C in the Sydney summer but how did we go with wet days?

summer[['Mean TemperatureC', 'Precipitationmm']].plot(grid=True, figsize=(10,5))

<matplotlib.axes._subplots.AxesSubplot at 0x945be48>

January and February have the odd patch of rain in January and February but nothing too dramatic or long lasting. Let's take a close look at these two months.

july = summer.ix[datetime(2015,1,1) : datetime(2015,2,28)]
july[['Mean TemperatureC', 'Precipitationmm']].plot(grid=True, figsize=(10,5))

<matplotlib.axes._subplots.AxesSubplot at 0x9bf9400>

Assuming history repeats February seems to be the most reliable month in terms of temperatures around the mid 20s with only a couple of days of light rain.

Conclusions

On the basis of one year's record, February seems like it could be a good month for beach weather. However, we need to see if that conclusion is supported over a number of years of data.