BW #26: Hot weather
Is it hot where you live? Temperatures are rising all over the globe. This week, we'll find out where it has been hottest, and whether temperatures have been rising over the last few years.
I live in Israel, and since I moved here in 1995, people have always complained about how hot the summers are. But over the last few years, people have begun to realize that it has been hotter than usual during the summers, with many days in a row in the mid-30s Celsius (i.e., mid-90s Fahrenheit). Earlier this month, the electric company had to cut power to my city for an hour or two, becuse the demand for air conditioning had maxed out their power grid.
I was just in Prague last week for the Euro Python conference, and while temperatures there were cooler than Israel, they were relatively hot for central Europe. My wife and I even spent a day in Bohemian Switzerland, a large national park... where we saw the remains of trees that had burned down in huge wildfires.
If you liked the above photo, I also have a short video:
And of course, temperatures have been high in North America, as well. Phoenix, Arizona, where my father grew up, was always hot — but even residents of Phoenix have been experiencing some awfully high temperatures over the last few weeks.
Bottom line: No matter where you live, summers are hotter than they've ever been before. (My apologies to readers of Bamboo Weekly from south of the equator, who might want to wait six months for this issue to be relevant.) Earlier this month, it was even reported that the planet experienced its hottest-ever day.
This week, we'll go through some of that hot data, looking at just how hot it is, and how different that is from previous years. Along the way, we'll play with fixed-width fields and trimming large data sets.
Data and questions
The data this week comes from the National Centers for Environmental Information (NCEI), part of the National Oceanic and Atmospheric Administration (NOAA), part of the US Department of Commerce. If you're wondering why the Department of Commerce, which sounds like it has to do with business, is in charge of such things, then I suggest you read "The Fifth Risk," a great book by Michael Lewis about the US government and what it does. There, he says that the Department of Commerce should really be called the "Department of Data," because its job is really about collecting data for the public.
Warning: There is a *ton* of data here. I'm only scratching the surface of it, and I'm sure I'll look at it more in the coming weeks and months.
First: you'll want to at least look through the README for the data, at
https://www.ncei.noaa.gov/pub/data/ghcn/daily/readme.txt
Once you've done that, and have gotten your bearings at least a bit, here are the 7 questions and tasks for this week. Creating the data frame (question 3) will be tricky, and will likely take a long time -- but it's good practice working with large data sets, and cutting them down to size.
The learning goals include working with large data sets, working with fixed-width field files, transforming data with Python functions, grouping, joining, and plotting.
- Download the list of weather stations (https://www.ncei.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt) and turn it into a data frame. Use the specifications for the file, as described in the README. You'll want to set your own names for the column headers. Make the `id` column into the index.
- Download the GHCND-ALL data (https://www.ncei.noaa.gov/pub/data/ghcn/daily/ghcnd_all.tar.gz). NOTE: This file is 3.4 GB in size, so it might take a while to download to your computer. Follow the directions in the README to un-tar the file. This will result in about 30 GB of files being created under the `ghcnd_all` directory.
