Category: Week 3: Databases (Back End)

Posted on

3B: Databases, Classification and Metadata

In our continuing quest to explore what goes on “under the hood” of digital humanities projects, this week we are moving from the front-end client-side user experience to the database “back end” on the server side, where all the data storage and information retrieval magic happens.  In order to perform analysis, or present the results of our research to the public on the web, we first need to collect, categorize and store our data in a way that will give us the best combination of structure and flexibility.

We’ve seen a few examples of how you can use a simple flat spreadsheet to store enough data to power some pretty impressive applications using JavaScript alone, like using the  Google Maps API or the beautiful TimelineJS framework.

Last time, we created individual timelines using Timeline JS, and entered the data in spreadsheets. We will explore the concept of metadata and data cleaning by trying to combine our different year bands into one group timeline.

Go to the Aggregate Google Sheet below, and add your group’s data from the sheet you created last time.

Class Aggregate TimelineJS Template

  • Are the data compatible?
  • Are they “clean”?
  • What problems arise?

This works great for the timeline, but what if we wanted to do different things with the same data?  What if we wanted to reorder our data by something other than chronology, or extract all the people or buildings, or add spatial locations? And what if we wanted to model the relationships between those elements?  Our spreadsheet is just not flexible enough for this.  In order to store complex data sets, we need a more sophisticated way to store it; enter the relational database.

There is a vast amount of literature out there on database design theory and practice, but the articles we read for this week provide a good starting point into the general characteristics of relational databases, and the raging debates over how to move beyond them in the brave new world of ‘big data‘ in humanities research.

The key takeaway from these debates is that “data” are not value free and neutral pieces of information.  Any time we break information down and classify it into categories, we are imposing our human world view and experiences on the information, whether consciously or not.  This is unavoidable, but the best way to deal with it honestly is to acknowledge our biases, document our decisions and explain our thinking at each step of the process.  The resulting metadata (data about the data) are critical for successful scholarly projects, and we will discuss their importance throughout the course.

 

In Class

lampstack-1For today, we are interested primarily in exploring how relational databases work in a typical DH project, which often shares a lot of similarities with how web applications work in general.  So we are going to stick with what we already know and get to know databases by exploring the backend of a WordPress site.

If you were going to do this the old fashioned way, you would need some space on a server running the LAMP stack (Linux, Apache, MySQL, and PHP) to install and run a fully customizable WordPress site, but we are going to using our cPanel in Reclaim Hosting which takes care of all the system administration work for us.

WordPress database model
The WordPress database model

Most web applications and DH projects consist of two main components: files and a database. The main WordPress files you’ll interact with are the PHP files in the theme layer, which change the look and feel of your site, and the plugins in the plugins directory, which add functionality.  Check out the Resources section below for more on how to customize these.

The database can be accessed via phpMyAdmin, a super helpful tool that lets you interrogate and take actions on the database without having to type SQL commands directly into a shell prompt.

  • Explore your WordPress db, consulting the diagram at right,
  • See if you can figure out how the data and metadata of a typical post, page and comment are broken up and stored in the db.
  • Add a new plugin and a new theme to your site.
    • Did either change the database?  Which one?  Why?

Continue to explore the guts of WordPress and ask yourself: how are the data are structured, stored, and ultimately rendered in the browser? Do you understand all the component parts?

 

Assignment

 

  1. Continue exploring the server environment on your own, and try to see if you can understand how the different pieces fit together within the LAMP stack itself and within the MySQL relational databases.
  2. Continue to explore the WordPress backend and think about what this structure allows.
  3. Try to reproduce and populate some of the database that Stephen Ramsay describes in this article on your local host by either executing the SQL statements or using the phpMyAdmin graphical user interface that comes on your host.
  4. Finally, write a blog post that discusses the benefits and drawbacks of flat data structures like spreadsheets vs. relational databases.
    • What are the pros and cons of each?
    • What issues with data collection and metadata must be considered and solved before you get too far?

Resources

 

Posted on

3A: Big Data and Digital Humanities

Zooniverse: People powered research

In-Class Project

Last week we tore furiously through the front-end of web development — HTML, CSS and JavaScript.  But we are not learning coding here, we are doing Digital Humanities. Today let’s look at how we can put those skills into practice for humanities data to tell a story.

One of the longest-running types of applications is an interactive timeline.  We will do a class project to convert a flat timeline into an interactive one using an easy to use application, but there are other tools that require more coding and show you how these work.  Take a look at the SIMILE timeline tool for an example of one of those.

Today, we are going to use the beautiful TimelineJS framework to give the college archive’s Timeline of Carleton History a web 2.0 overhaul.  This timeline was created in 1991 for the college’s 125th anniversary celebration, and while the content is still great, the presentation could use an update as we near the 150th anniversary in 2016.  Archivist Tom Lamb has given us permission to use the timeline as our dataset, and we are going to build a new, dynamic JSified instance of it as our first group project.

The timeline is broken into five date ranges between 1866 and 2002.  I have set up the first one as an example, and it is each group’s task to replicate this work for your own date range by doing the following:

  1. Go to the TimelineJS page and follow the 4 step instructions to Make a New Timeline
    • In Step 2, whoever downloads the Google template should share it with the other group members so that all may editScreen Shot 2015-01-15 at 12.17.03 PM
    • In Step 4, copy the embed code and paste it into a new jsBin.  This is where you will work on your own date ranges for now, and we will combine them all together next week.
  2. Once you are setup, delete the template data and move over your group’s data from the Timeline of Carleton History. The dates and captions should come over with an easy copy/paste, but then you’ll probably need to finesse the rest of the data a bit.
    • You might need to change the number format of the Date columns to a different date display or even Number > Plain Text to get them to display and order properly
    • All entries should have a brief headline that summarizes the text on that date’s card, which you’ll need to write
    • Where there are images on your page, click them to bring up the full resolution version in the fancybox viewer, then use your DevTools knowledge to find the image URL to paste in the appropriate Media column in the sheet
    • Where there are no images, see if you can insert a Google Map if appropriate.  Or search the Carleton Archives Digital Collections to find other appropriate photos or scanned documents
      • NB: All Media should have a Media Credit, which will usually be “Archives, Carleton College Gould Library“
    • Finally, explore what happens to the timeline when you use tags to categorize events.  I used buildings and people as two basic categories on my example

If you’re in doubt or stuck, post a comment, ask a question on your blog or (as a last resort) send an email and we’ll try to help each other out.

Big Data

Big Data generally refers to extremely large datasets that require demanding computational analysis to reveal patterns and trends, such as the map below generated from the data in millions of Twitter posts. We are producing reams of this data in the 21st century, but how do we analyze it from a humanities perspective?  How do we perform these sorts of analyses if we are interested in periods before regular digital record keeping?

World travel and communications recorded on Twitter
World travel and communications recorded on Twitter

Enter digitization and citizen science initiatives.  One of the major trends in Digital Humanities work is the digitization of old records or print books that are then made searchable and available online for analysis.  Google Books is the most well-known project of this type, and we also read Tim Hitchcock’s article about his pioneering historical projects in this arena, e.g. the Old Bailey Online and London Lives.  These projects took years to build and required the dedicated paid labor of a team of scholars and professionals.  But there’s another model out there that relies on the unpaid labor of thousands of non-expert volunteers who collectively are able to do this work faster and more accurately than our current computers: crowdsourcing.

Zooniverse is a crowdsourcing initiative that bills itself as “the world’s largest and most popular platform for people-powered research.”  This platform takes advantage of the fact that people can distinguish detailed differences between images that regularly trip up computers, and empowers non-experts to contribute to serious research by reducing complex problems to relatively straightforward decisions:

  • is this galaxy a spiral or an ellipse?
  • is this a lion or a zebra?
  • is this the Greek letter tau or epsilon?

The project that Evan and his team just launched, Measuring the Anzacs, seeks to study demographic and health trends in the early 20th century by transcribing 4.5 million pages worth of service records from the Australian and New Zealand Army Corps during WWI.  This data would take countless years to process with a small team of researchers, but as Evan told us, they hope to speed up this process tremendously by taking advantage of the fact that there are lots of people who have access to a computer, speak English and can read handwriting.

Tim Hitchcock ended his piece with a conundrum:

How to turn big data in to good history?  How do we preserve the democratic and accessible character of the web, while using the tools of a technocratic science model in which popular engagement is generally an afterthought rather than the point.

The Zooniverse model has taken a major step towards resolving this tension and turning formerly restricted research practices into consciously public digital humanities work.

 

Assignment

Explore the Measuring the Anzacs project and work your way through at least one document, marking and transcribing the text.

Screen Shot 2015-09-30 at 6.24.24 AM

When you’re done, post a brief comment below giving some feedback on the process.  Were the instructions easy to follow?  Was the text easy to transcribe?  Did you feel like you were making a real contribution to the project?  What did you get out of the project, from a humanities perspective?  Did you come away with a greater understanding of either the research process or the lived experience of the individual people whose records you were working with?

css.php