Magazine

SPECIAL FOCUS: DATA AND INFORMATION COLLECTION ON THE NET

One Perspective on Collecting Data on the Web

by Jennifer Gold and Denise Ethier

Introduction

With so many people having access to the Internet, more and more surveys and data collection tools are being used to monitor surfers' use of the Web. Web developers collect data for many reasons. Data can be used to identify personal information about users, general interests, entertainment preferences, product purchasing patterns, content and opinions expressed in online discussions, and exposure to advertising. For non-profit and research and development organizations that are funded by outside sources, data collection is critical for other reasons. These reasons include: (1) ensuring that the organization's goals are being met, (2) providing data to their funder that substantiates their existence, and (3) informing and furthering the field.

In this article, we describe how and why one non-profit organization collects data electronically, the various data collection tools that were used, and what was learned through the data collection process.

Why Collect Data Electronically

The National Center to Improve Practice (NCIP), located at Education Development Center, Inc. in Newton, MA, is funded by the U.S. Department of Education, Office for Special Education Programs (OSEP). Its goal is to improve practice around the use of technology, media, and materials for students in pre-K through grade 12 who have disabilities. Over the past five years, NCIP has developed an innovative and productive approach to professional development and knowledge utilization to help educators to:

• expand their knowledge about the use of technology by students who have disabilities

• positively change their beliefs and attitudes about ways in which technology, media and materials (TMM) can help students

• improve their professional practices so that students with disabilities demonstrate improved outcomes, and

• share their knowledge, experience and questions about assistive and instructional technologies with others

In order to achieve this goal, NCIP relies heavily on the World Wide Web (WWW) as its main venue for providing a full range of professional development opportunities. In trying to be responsive to our audience's varying learning styles, interests, and comfort levels with technology, NCIP offers a range of opportunities that include online discussions, workshops, guided tours and tutorials, a library of resources, and links to other sites on the Web.

What NCIP Wanted to Collect

As a federally-funded project, it is important for NCIP to constantly evaluate its work to meet its goals. In order to focus our data collection, we designed the following set of research questions. In particular, we wanted to know:

• Did we reach our audience?

• How did the professional development opportunities we provided on the Web impact our audience's knowledge and practice?

• What were the advantages and disadvantages of using the Web as our primary vehicle?

As described by Harasim, Hiltz, Teles, and Turoff (1996), assessment can include both "top-down accountability approaches" and an analysis for "bottom-up instructional improvement." This coincides with NCIP's assessment, evaluating data both for reports submitted to the U.S. Department of Education and to help visitors gain the most from our Web site. NCIP selected its data collection tools in order to conduct a formative evaluation to identify sources of success and weaknesses for future improvements.

Data Collection Tools

Evaluation of data should take into consideration a multiplicity of evidence gathered through numerous data collection methods and incorporate both quantitative and qualitative methods (Patton, 1987). One of the advantages of collecting data from electronic sources is that it allows researchers to collect data that may not be available in a non-telecommunications environment. Furthermore, since technology maintains a permanent record of exchanges that occur in computer-mediated communication, data that is recorded automatically can be stored for future analysis.

While realizing that there is an enormous amount of data available through electronic means, NCIP recognized that no single tool could meet the needs of all of our research questions and that some tools are better than others for collecting specific kinds of information. NCIP wanted to incorporate methods which would measure both the reactions of a large population to a set of questions (quantitative data) and more in-depth and detailed descriptions of attitudes and behaviors (qualitative data). In thinking through this process, NCIP carefully selected and analyzed numerous data collection tools and methods to coincide with their research questions, providing NCIP with the quantitative and qualitative data we needed.

Based on these considerations, NCIP selected four electronic data gathering tools in order to help us in our evaluation: (1) data gathered from the server, (2) Web-based surveys mounted at the site, (3) electronic mail surveys sent to individual email addresses, and (4) spontaneous electronic mail messages.

1. Data gathered from the server.

   Web statistics gathered through the server are one data collection tool which enables us to gather quantitative data. A monthly report provides daily, weekly, and monthly usage of our Web site, i.e. which pages got hit, how many hits per page, and number of visits by individual hosts per page (identified by "IP addresses"). Thus, we are able to see general trends in terms of usage of the entire site and its components over time.

2. Web-based surveys mounted at the site.

   In order to identify our audience and collect qualitative data about the usefulness and impact of our resources, we designed fill-in forms for mounting on the Web site. Using the hypertext markup language (HTML) and common-gateway- interface (cgi) programming, we have used Web-based surveys in two ways.

   First, we developed a mailing list form. This Web-based fill-in form includes fields for name and email address, as well as questions regarding job role, area of interest, and location, (e.g., school, clinic, university, and so on). The information from the mailing list is then imported into a database that allows for tracking of mailings and for use as part of our ongoing evaluation. You can view mailing list form.

   Second, we used Web-based surveys to analyze one component of our site. This survey includes eight questions, four in multiple choice format and four seeking short answers about the content and design of this particular component. In preparation for possible follow-up, NCIP asked that respondents give their name and email address if they were willing to be contacted about their use of this component. You can view the Early Childhood Tour - Evaluation Form.

3. Electronic mail surveys sent to individual email addresses.

   Similar to paper surveys, electronic mail can be used to deliver surveys to individuals' mailboxes. NCIP took advantage of this method, administering surveys by using Decisive Software, a software program that is used to send surveys, track response rates, and sort and compile responses.

   Electronic surveys were used as part of an analysis of the entire site to determine the effectiveness of both the content and the formats we offered. NCIP sent surveys to everyone on its mailing list. This mailing included electronic mail surveys, as well as a version sent via postal mail to those for whom we did not have email addresses. From this sample, we were able to determine which components were used most often and how the information found at the site was used. NCIP then used a method of purposeful sampling to send a second survey to those people who had visited the site frequently. This second survey was used to gather specific information about the impact of our site.

   Email surveys were also used in an analysis of NCIP's Web-based workshops. As in face-to-face workshops where participants may be asked to assess the quality of a course, participants were asked to respond to questions prior to and at the completion of one of our Web-based courses. Data from these surveys in conjunction with usage data and postings to discussions provided information- rich cases for analysis.

4. Spontaneous electronic mail messages.

   The last electronic data collection tool NCIP used was in the form of unsolicited messages. Unsolicited messages include email messages sent to NCIP's mailbox and the technical assistance provider, messages posted to Web- based discussion conferences, and comments from the mailing list form. For example, one workshop participant emailed the NCIP HelpDesk saying that she was a "visual" learner and she was having a hard time finding her way around the workshop. NCIP added a graphical map to the workshop that helped participants get oriented quickly. In addition to helping staff make ongoing improvements to the site, these messages are stored for future analysis. Over time we have collected over a thousand such messages.

What We Are Learning About Collecting Data Electronically

While gathering information about collecting data electronically, we found that little scholarly research has been done in this area. However, our emerging findings about the process of collecting electronic data are consistent with the research we did find.

NCIP is learning that data collected through unsolicited methods are very effective, as Harasim, Hiltz, Teles, Turoff (1996) suggest. Telecommunications allows researchers to use unobtrusive measurements such as usage statistics and spontaneous comments whenever possible. Data collected in this way does not interfere with what visitors are doing at the site and comments can be stored and analyzed at a later date. For example, NCIP received an overwhelming number of registrations for our mailing list through our online mailing list form posted on our site. This form proved to be highly successful in helping us identify a group of people who are very interested in what we have to offer. Moreover, NCIP found that unsolicited comments visitors made to the NCIP mailbox also helped in pointing out individuals who were most apt to find our site useful.

Unlike the mailing list form, the survey we posted on our Web site to one of the components has not been as successful. While Web-based surveys seemed to be convenient for visitors to our site to fill out, there is no precise and accurate way of determining response rate (Smith, 1997). For example, approximately 25 people have completed this survey since mounting it on our site close to a year ago. Although survey responses have provided us with valuable information we needed in order to assess the quality of the individual component, it remains unclear what percentage of the total number of visitors this small sample represents or how representative their comments are of the field.

Likewise, statistics gathered through the server provided us with important information about the use of our site without intruding upon visitors' use of the site. However, the unobtrusive nature of the data meant that we were unable to collect as much information as we would have liked. Since Internet users are assigned IP addresses dynamically, this makes their DNS identifiers completely unique and different each time they access a page. What this means for our data collection is that, although individual visits can be tracked, we are unable to identify whether these hits represent repeat visitors or the same visitors to different pages.

In terms of electronic mail surveys, NCIP found issues of time and space to be eliminated, enabling us to reach a wider audience with a quicker rate of return over those we sent via postal mail. While the bulk of the email surveys were returned within the first 24 hours, hardcopy versions of the survey were received two to three weeks later. This coincides with research by Mehta and Sivadas (1995) and Parker (1992) who found email response rate to be quicker than postal mail. NCIP learned that Web-based and electronic mail surveys provide significant cost reduction over mail and telephone surveys, not only in terms of telephone and postal mail charges, but also in the time it takes project staff to assemble mailings, contact individuals and execute the survey, and in entering and organizing the data once it is gathered.

NCIP also found that we had a higher response rate from participants who received the surveys via email over those who received it via postal mail. The response rate for the email survey was approximately 41%, while the response rate for the postal mail survey was 20%. One possible disadvantage of electronic surveys is that it is possible that technological prowess of respondents may skew data when information is delivered inaccurately as a result of not following directions. On the other hand, factors that may contribute to a higher response rate may be the ability to facilitate interaction between surveyors and respondents if there are any questions, and that replying to surveys is often more convenient for participants. This is consistent with findings described in "Casting the Net: Surveying an Internet Population" (Smith, 1997)

Similar to paper surveys, NCIP found that a portion of the surveys were returned due to undeliverable email and incorrect email addresses and that reminders were critical to response rate. However, NCIP found electronic surveys to be advantageous in the end, in that they were easier to manage in terms of keeping track of undeliverable surveys and individuals who required a reminder message.

Lastly, NCIP learned to think hard about the number of surveys sent electronically in an era of spamming and unwanted emails, even when participants had requested to be put on our mailing list. Throughout the data collection process, NCIP was cognizant of how often they solicited information from participants, so as not to flood visitors' emailboxes with unwanted messages. Likewise, NCIP paid careful attention to respecting visitors' anonymity and confidentiality of their responses.

What this Means for Non-Profits Collecting Data on the Web

In this article, we describe why it is important for non-profit organizations to collect data electronically; collecting data about a Web site can help organizations ensure that its goals are being met and that resources are being well-utilized. Findings can be included in reports to funders and can inform and further the field.

For NCIP, electronic data significantly contributed to our understanding of our audience and what they value at our Web site. It has also provided us with feedback from our audience so that we can build on our initial successes in delivering professional development opportunities that are useful to the field. For a non-profit, collecting data electronically means that information can be gathered more efficiently and cost-effectively. Finally, NCIP believes it is possible to find a middle ground between gathering sufficient data to meet project's goals and further the field, and remaining unobtrusive to its users.

References

• Harasim, L., Hiltz, S.R., Teles, L., Turoff, M. (1996). Learning networks: A field guide to teaching and learning online. Cambridge, MA: The MIT Press.

• Mehta, R., & Silvadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of the Market Research Society, 37(4), 429-439.

• Parker, L. (1992, July). Collecting data the e-mail way. Training and Development, 52-54.

• Patton, M.Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.

• Smith, C.B. (1997). Casting the net: Surveying an Internet population. Journal of Computer Mediated Communication, 3(1). http://www.usc.edu/dept/annenberg/vol3/issue1/smith.html

Jennifer Gold (JGold@edc.org) is Research Assistant and Denise Ethier is Telecommunications Coordinator for the National Center to Improve Practice, located at Education Development Center, Inc. in Newton, MA. Denise Ethier can be reached at DeniseE@edc.org.

Copyright © 1997 by Jennifer Gold and Denise Ethier. All Rights Reserved.