The Interactive Archivist

Jeremy Heil
Frank Huntley
June 30, 2009

Introduction

Context of Case Study

This case study focuses on the integration of Google Maps within the Stones website.

Funded by the government of Canada through the Canadian Culture Online Programme Partnership Fund, Stones is a guide to the social history of Kingston, Ontario, and incorporates a variety of primary and secondary resources including textual records, photographs, contemporary news articles, sound recordings, and video.

Using a set of social history tours that were designed for and run through Queen’s University Archives in 2005 as a starting point, Stones will provide access to information on the history of Kingston, its people, events, and landmarks, through an online exhibit and virtual tours. The tours include: “The Jewish Experience in Kingston,” “A Prisoner’s Life in Kingston,” “The Gay and Lesbian History of Kingston,” “A Black History of Kingston,” “The Chinese Community of Kingston” and “L’histoire de la francophonie à Kingston.” The tours will also become available as a downloadable audio file and enhanced audio file with photographs, which can be accessed by a user’s cell phone, or transferred to a portable audio player or portable video player, directing users to their destination, and describing the historical significance of each location in sequence along the tour routes. Each tour is woven into the greater context of Kingston society, and demonstrates the importance of each community in the development of the city.

The Stones Project is a partnership of Queen’s University Archives, the Kingston Frontenac Public Library, Kingston Area Network (KANnet), the City of Kingston, the Penitentiary Museum, L’Association canadienne-française de l’Ontario (ACFO) Milles-Îles, and the Cataraqui Archaeological Research Foundation.

According to Shu-Wai Chow, a PHP programmer, “Web mashups are exactly what they sound like—web applications that merge data from one or more sources and present them in new ways. Very often, the data owners encourage and facilitate third parties to use the data. In many cases, this facilitation is made possible by the data owners providing application programming interfaces (API) to their data. These APIs follow standard web service protocols and can be implemented quickly and easily in a variety of programming languages, including PHP.”¹ In other words, a mashup takes data that supports one particular purpose and “mashes” it with one or more different data sources, each of which is used in its own stand-alone application, and produces a new and richer user experience.

Although a few distinct technologies are being used in Stones, this case study will focus on the development of a mashup—in particular, the use of the Google Maps API as the basis for presenting six virtual and drivable tours of Kingston, Ontario, integrated with tour data.

Description of a Mashup

For the most part, a mashup is simply a combination of any Web 2.0 technology into one interface. Blogs, Flickr, RSS feeds, tag clouds, and wikis can all be mashed from various sources.

In the archival world, the potential uses for mashup technology are exponential. The limiting factors are only the imagination of the creator. Datasets and Web 2.0 technology can be combined to produce new and unique products that enhance the user or researcher experience, or even further enlighten by cross-referencing datasets.

There are numerous examples of mashups using fixed datasets such as those found within archival resources or archival description. One such mashup is the Census Dashboard.² In the developer’s own words, this is “[a] mashup using two StrikeIron APIs, Zip Code Information and Population Demographics By ZIP Code, plus Yahoo Maps to give detailed information for a specific U.S. zip code.”³ The mashup effectively combines these elements to provide graphs and charts on the age distribution, ethnic distribution, population statistics, household statistics, housing units, and vacancy rates of each zip code in the United States. The interface intertwines Yahoo Maps, graphs made available through Flash, and, although slightly irrelevant to the purpose of the website, the current weather conditions.

How to Create a Mashup

Many technologies and data sources can be integrated to produce a new mashup service. The knowledge and skill required to produce an effective mashup can vary widely from application to application.

A seemingly obvious starting point would be applying familiar tools to new paradigms, such as an editor. A number of editors intended to assist the user in creating a mashup based on one product as a starting point are available for download or online use. Some of the more well-known editors include the Google Mashup Editor, Microsoft Popfly, and Yahoo! Pipes-Beta.

For the amateur or developing programmer, Popfly is perhaps the best place to start. This is a visual (and pretty fun) way to get started with mashups. There are a wide number of “blocks” (application, data feeds, etc.) that a mashup designer can choose from to combine into a new tool. Perhaps the most interesting aspect about this editor is how it allows the user to rip other user-contributed mashups to see how they work, tweak them, or otherwise play around with them. One of the most useful features available to the new masher is the Block Creator/Editor. This tool allows the user to examine and edit pre-existing blocks at the code level, and get a feel for how the JavaScript is written to allow interaction with other blocks.

There are multiple guides published on how to create a mashup for specific APIs, including a series through Wiley Publishing that includes guides on Yahoo! Maps, Flickr, and Delicious, among others.⁴ These list step-by-step instructions on developing mashups using those particular programs, which is something I will not endeavor to duplicate here.

Implementation

In the case of Stones, none of the aforementioned editors were found to be suitable, or they had been deprecated (e.g., Google Maps Editor Mashup). A more comprehensive approach was needed, since mashups are not just about data, they are also about functionality.

You have to integrate both data and functionality to create an effective mashup. That eliminates approaches drawing more heavily on animation (e.g., a flash-based approach), or at least restricts them to supplementing more comprehensive approaches. The more comprehensive approach is also more effective for long-term application and data evolution, and it is anticipated that it would be easier to maintain.

The first step was to establish the minimum required and primary feature set components. Two key technical paths were needed:

1. Data mashup foundation: Because this was a system for touring locations, a base layer includes the core operations of locate (to find a specific location) and zoom (to find out more about a specific location). While there are multiple options here, such as NASA’s World Wind or Mapquest, Google Maps has become a standard data framework.⁵ Furthermore, it supports the ability to react to user actions, which it demonstrates with the simple, well-known (and limited) speech bubble graphics that are typically found on a Google Map.

Multiple forms of data were integrated or are expected to be integrated in the future. Data structures can be defined for integrating multiple approaches to data—XML, SQL, and KML—and in the future, the system could be evolved to incorporate an Application Programming Interface (API) to a data abstraction layer, to allow other layers to be part of the mashup. The following items were considered:

• the need to develop a multi-faceted data structure for most flexibility, with a clean architecture for most flexibility
• the need to put early effort into XML, which acts as a Rosetta Stone to translate to speak to disparate systems in a mashup and interprets the data
• Google Maps—KML (Keyhole Markup Language—XML specific for use in geographically- and geodetically-based applications). KML uses 3D geographic coordinates: longitude, latitude and altitude, in that order.⁶ We wanted to integrate Google Maps with large databases
• SQL for scale and speed

We also needed to accommodate non- or less-structured data, such as text and multimedia.

2. Functionality mashup foundation: We needed a programming framework that would support the addition of a judicious selection of available components, custom programming, and “data abstraction” (i.e., a simple, standardized way to integrate some forms of data). Available components are tools already created by others, such as an online editor, wiki, or discussion forum. Normally, these tools would be open-source in situations where maintenance and evolution support is uncertain, so that future developers will not be stuck with abandoned technology they can do nothing with. While there are many frameworks available (e.g., Drupal), each framework has its own strengths and weaknesses. We chose one that was focused on supporting programming extensions, and that had core functionality built in for security, audit, and content management. Projects should not be focused on re-developing these areas.

Key to maximizing development within a framework is eliminating redeveloping the wheel, as one part of the required “functionality mashup.” For example, developers would typically not need to write a security system. Caution is required in choosing applications that are well supported, so all elements of the functionality mashup will integrate now and in the future with minimal maintenance and effort. Selecting a framework that supports modular development allows the developers to take advantage of other work that could not be completed within the time and resources of a project with a finite time line.

At least as important, however, were the strategies employed for development. Our experience relied extensively on a rapid-iteration agile approach to coding, where a small sample of users are provided the options to repeatedly provide feedback and define features, comment on features, and articulate data management needs. This can create expectation management challenges, because changes made in the course of development spark new ideas, and there are no large transitions, which can lead to a sense of not completing enough, and iterate repeatedly through a loop of “feedback, modify, and review.” The non-technical members of the team need to be both educated and teachable on the fact that their involvement is necessary and ongoing to create a successful mashup. With the above two technologies in place, the domain experts need to be shown the shell of the system and encouraged to explore and extrapolate. We found that a four-week cycle of review and feedback, redefine technology priorities, and development worked best for making significant progress.

To improve the user experience, we have been working towards overlaying historic maps of Kingston onto the basic Google Maps interface. As each tour moves from the earliest to latest chronological points, the maps will expand to a version contemporary to the time period represented in the next tour stop.

Assessment

The success of the project is based upon how much the site and its materials are used, how much it evolves, and how much it fosters new content, engages new co-operative behavior, and extends mashups or straight-forward applications.

Technical Assessment

Technical assessment was ongoing, and will continue as the mashups continue to evolve beyond the current project. The project used a rapid iteration approach, where priorities are re-evaluated every two to three weeks, based on an assessment of how technologies are meeting the highest priority needs. This assessment approach led, for example, to a higher focus on AJAX-based technologies, and identified needs for more context-based algorithms in the future (e.g., automatically centering and scaling the map so a given percentage of tour locations are visible). It allows for ongoing consideration of the technology used and its effectiveness.

Usability Assessment

One-on-one reviews of the website are done with novice and experienced users, including those with extensive historical information professionally (archivists) and informally (amateur genealogists) and members of the general public. These have begun, and have identified preferences for different User Interface elements (e.g., a book metaphor for new information, where the screen is always laid out like a book with frames, as opposed to the Google Maps pop-up approach).

Satisfaction Measurement

All locations have a ranking JavaScript, where information can be assessed per location on a one-to-five point scale. We anticipate expanding this approach, to provide more specific and detailed assessments, and to provide users with tools that will make their perspectives and evaluations more directly visible (e.g., let users construct their own highlight tours or add their own points). This will lead to enhanced measurement and subsequent fostering of more cooperative knowledge management and distribution.

As of the end of June, 2009, testing for Stones was still under way. A complete assessment of the use of mashups in Stones will not be available for several months.

Analytics Evaluation

We currently track all traffic to the site, including standard analytics such as referring sites and number of hits per page. Perhaps ironically, we do not use Google Analytics, because their current contract means all your traffic data is theirs. We are reviewing other options, particularly open-source ones.

Lessons Learned

• YouTube—One option our Web developers explored was tying video content to the site by uploading them to YouTube. While this proved to be a simple solution that covered problems of bandwidth and host server space, and made the content available to a wider audience, there were some concerns we immediately foresaw. First and foremost were the issues of copyright and control. The copyright for the videos we wanted to make available was still held by the individual producers. Making these videos available through YouTube would have breached the terms of use by making them more widely available beyond the limits of the Stones website.
• Map overlays—We realized early on that customizing Google Maps is not a task to be taken lightly. One of the first maps we had digitized was an artist’s-rendered bird’s-eye view of Kingston from 1875. While visually appealing, the bird’s eye view would have had to be modified extensively to match the satellite view of the streets. As a result, the historic maps that could best be used for overlays needed to be as precise as possible. Even then, we began with the idea that we could change the maps as users progressed through the virtual tours. This concept also proved to be unworkable, as users found the change in background to be slightly jarring. Instead, using the tried-and-true Google Maps model, users will be able to choose which map they wish to use, starting with a map specifically designed for this website. This will also allow users to compare and contrast with earlier and later maps of the city to see how the city’s boundaries have grown over time.
• Scope creep—Web development is nothing if not fun (all deadlines aside!). Still, it is so easy to get sidetracked trying to add interesting features that may or may not end up in the final product. As mentioned earlier in this case study, mashups really do have the ability to combine all manner of functions and data, and allow for a lot of custom work in the interface. For the purpose of a finite project, it is very important to stay focused on the core objectives of the site being developed, and avoid the interesting distractions of adding new features that are not essential to the overall usability of the site. That said, no one should ever be content with developing a mashup that will never again be tweaked or enhanced in future reincarnations!
• Customizing Google Maps—It is not so much lessons learned with Google as lessons reinforced.
1. Data is not perfect—when points were added to the site, sometimes they would just not appear on the website because addresses can be ambiguous (e.g., addresses in a park). This means that some liberties need to be taken with address-based information, giving approximate locations instead of actual locations. This problem will, hopefully, be corrected in the future with the addition of proper GIS coordinates.
2. Data is not perfectly aligned; merging multiple sources of data of different types (e.g., points, block addresses, maps) could lead to jarring displays;
3. Usability is part preference—traveling through a virtual tour, with modified Google pop-ups as you moved from location to location, was not well-received by some users. They preferred an interactive book metaphor, where information changed in a window on the side.
4. Single technologies are not enough to take advantage of data-heavy environments such as Google and associated mashups—we used AJAX, PHP, XML and mySQL.
5. Don’t reinvent the wheel—Security was left to the development framework we were working in, as log-in, user, and role management were already handled. That left programming resources to focus on the Google-based mashups, and time to respond to the lessons about usability and the new ideas that occurred even within the relatively short time frame for this project.

Future Plans

Applying the mashup to the archival realm can open innumerable opportunities, combining elements of archival process in ways never thought of before. Queen’s Archives is in the earliest stages of testing ICA-AtoM as its new archival description database.⁷ The flexibility of an open-source system like ICA-AtoM will allow us to further integrate archival description dynamically with digitized content. The benefit in the case of developing Web exhibits would be added context to the images that is otherwise removed when repurposing the images. Further information from other institution’s descriptive databases (where fonds (record groups) are split) could likewise be merged in real time, reflecting any changes to the descriptive information due to accruals.

To make better use of the mashup experience, and to improve services to actual tourists coming to Kingston, we are looking towards connecting the City of Kingston’s roadwork database with the Stones map. This could serve to provide automatic extensions of the site, such as new roads in routes or automatically change tour routes for tourists to compensate for road closures over the summer months.

Additional funding is being sought to expand Stones to include the history of other Kingston buildings and sites in the context of other social and cultural community histories, such as a maritime perspective and a medical history, and to make the site more of an interactive experience for its users (e.g., provide feedback on sites, or add their own family history). Further development on the first phase may also include the ability for users to add their own reminiscences of particular sites or events in a wiki or forum.

Resources

Hoffman, Harrison. “The Battle of the Mashup Editors.” The Web Services Report. http://news.cnet.com/8301-13515_3-9747138-26.html?tag=mncol (accessed May 21, 2009).

ProgrammableWeb. “Census Dashboard—ProgrammableWeb Mashup Detail.” ProgrammableWeb. http://www.programmableweb.com/mashup/census-dashboard (accessed May 21, 2009).

Google Code. “Google Code FAQ—Building an Extensive Maps Mashup,” Google. http://code.google.com/support/bin/answer.py?answer=73041&topic=12044 (accessed May 21, 2009).

Google Code. “Google Code FAQ—Using KML in Google Mashup Editor.” Google. http://code.google.com/support/bin/answer.py?answer=78649&topic=12044&ctx=sibling (accessed May 21, 2009).

Microsoft. “Microsoft Popfly.” Microsoft. http://www.popfly.com/ (accessed May 21, 2009).

Yahoo Pipes. “Pipes: Rewire the Web.” Yahoo. http://pipes.yahoo.com/pipes/ (accessed May 21, 2009).

Chow, Shu-Wai. PHP Web 2.0 Mashup Projects. Olton, Birmingham: Packt Pub, 2007. http://proquestcombo.safaribooksonline.com/9781847190888 (accessed June 30, 2009).

Freedman, Charles. Yahoo! Maps Mashups. Indianapolis, Ind.: Wiley Pub., 2007.

Kelley, Martin. Web 2.0 Mashups and Niche Aggregators.. Sebastopol, CA: O’Reilly, 2008.

Kroski, Ellyssa. Web 2.0 for Librarians and Information Professionals. New York: Neal-Schuman Publishers, 2008.

O’Connor, Brett. Del.icio.us Mashups. Indianapolis, Ind.: Wiley Pub., 2007.

Wilkinson, David. Flickr Mashups. Indianapolis, Ind.: Wiley Pub., 2007.

Yee, Raymond. Pro Web 2.0 Mashups. Berkeley, CA; New York: Apress, 2008.

Young, Michael. Google Maps Mashups: With Google Mapplets. Berkeley, CA; New York: Apress, 2008.

Notes

1. Shu-Wai Chow, PHP Web 2.0 Mashup Projects (Olton, Birmingham: Packt Pub., 2007), http://proquestcombo.safaribooksonline.com/9781847190888 (accessed June 30, 2009).

2. Cynergy, “Zip Code Census Distribution,” Cynergy, http://www.cynergysystems.com/blogs/blogs/andrew.trice/census/# (accessed June 30, 2009).

3. ProgrammableWeb, “Census Dashboard—ProgrammableWeb Mashup Detail,” ProgrammableWeb, http://www.programmableweb.com/mashup/census-dashboard (accessed May 21, 2009).

4. Brett O’Connor, Del.icio.us Mashups (Indianapolis, Ind.: Wiley Pub., 2007); David A. Wilkinson, Flickr Mashups (Programmer to Programmer) (Indianapolis, Ind.: Wiley Pub., 2007); and Charles Freedman, Yahoo! Maps Mashups (Wrox Mashup Books) (Indianapolis, Ind.: Wiley Pub., 2007).

5. National Aeronautics and Space Administration, “World Wind,” National Aeronautics and Space Administration, http://worldwind.arc.nasa.gov/ (accessed June 30, 2009); Mapquest, Inc., “Mapquest,” Mapquest Inc., http://www.mapquest.com (accessed June 30, 2009).

6.Wikipedia, “Keyhole Markup Language,” Wikimedia Foundation, http://en.wikipedia.org/wiki/Keyhole_Markup_Language (accessed June 30, 2009).

7. ICA-AtoM is available on the Web at International Council on Archives, “ICA-AtoM: Open Source Archival Description Software,” International Council on Archives, http://ica-atom.org/ (accessed June 30, 2009).