One Perspective on Collecting Data on the Web
by Jennifer Gold and Denise Ethier
With so many people having access to the Internet, more and more surveys and data collection tools are being used to monitor surfers' use of the Web. Web developers collect data for many reasons. Data can be used to identify personal information about users, general interests, entertainment preferences, product purchasing patterns, content and opinions expressed in online discussions, and exposure to advertising. For non-profit and research and development organizations that are funded by outside sources, data collection is critical for other reasons. These reasons include: (1) ensuring that the organization's goals are being met, (2) providing data to their funder that substantiates their existence, and (3) informing and furthering the field.
In this article, we describe how and why one non-profit organization collects data electronically, the various data collection tools that were used, and what was learned through the data collection process.
Why Collect Data Electronically
The National Center to Improve Practice (NCIP), located at Education Development Center, Inc. in Newton, MA, is funded by the U.S. Department of Education, Office for Special Education Programs (OSEP). Its goal is to improve practice around the use of technology, media, and materials for students in pre-K through grade 12 who have disabilities. Over the past five years, NCIP has developed an innovative and productive approach to professional development and knowledge utilization to help educators to:
In order to achieve this goal, NCIP relies heavily on the World Wide Web (WWW) as its main venue for providing a full range of professional development opportunities. In trying to be responsive to our audience's varying learning styles, interests, and comfort levels with technology, NCIP offers a range of opportunities that include online discussions, workshops, guided tours and tutorials, a library of resources, and links to other sites on the Web.
What NCIP Wanted to Collect
As a federally-funded project, it is important for NCIP to constantly evaluate its work to meet its goals. In order to focus our data collection, we designed the following set of research questions. In particular, we wanted to know:
As described by Harasim, Hiltz, Teles, and Turoff (1996), assessment can include both "top-down accountability approaches" and an analysis for "bottom-up instructional improvement." This coincides with NCIP's assessment, evaluating data both for reports submitted to the U.S. Department of Education and to help visitors gain the most from our Web site. NCIP selected its data collection tools in order to conduct a formative evaluation to identify sources of success and weaknesses for future improvements.
Data Collection Tools
Evaluation of data should take into consideration a multiplicity of evidence gathered through numerous data collection methods and incorporate both quantitative and qualitative methods (Patton, 1987). One of the advantages of collecting data from electronic sources is that it allows researchers to collect data that may not be available in a non-telecommunications environment. Furthermore, since technology maintains a permanent record of exchanges that occur in computer-mediated communication, data that is recorded automatically can be stored for future analysis.
While realizing that there is an enormous amount of data available through electronic means, NCIP recognized that no single tool could meet the needs of all of our research questions and that some tools are better than others for collecting specific kinds of information. NCIP wanted to incorporate methods which would measure both the reactions of a large population to a set of questions (quantitative data) and more in-depth and detailed descriptions of attitudes and behaviors (qualitative data). In thinking through this process, NCIP carefully selected and analyzed numerous data collection tools and methods to coincide with their research questions, providing NCIP with the quantitative and qualitative data we needed.
Based on these considerations, NCIP selected four electronic data gathering tools in order to help us in our evaluation: (1) data gathered from the server, (2) Web-based surveys mounted at the site, (3) electronic mail surveys sent to individual email addresses, and (4) spontaneous electronic mail messages.
What We Are Learning About Collecting Data Electronically
While gathering information about collecting data electronically, we found that little scholarly research has been done in this area. However, our emerging findings about the process of collecting electronic data are consistent with the research we did find.
NCIP is learning that data collected through unsolicited methods are very effective, as Harasim, Hiltz, Teles, Turoff (1996) suggest. Telecommunications allows researchers to use unobtrusive measurements such as usage statistics and spontaneous comments whenever possible. Data collected in this way does not interfere with what visitors are doing at the site and comments can be stored and analyzed at a later date. For example, NCIP received an overwhelming number of registrations for our mailing list through our online mailing list form posted on our site. This form proved to be highly successful in helping us identify a group of people who are very interested in what we have to offer. Moreover, NCIP found that unsolicited comments visitors made to the NCIP mailbox also helped in pointing out individuals who were most apt to find our site useful.
Unlike the mailing list form, the survey we posted on our Web site to one of the components has not been as successful. While Web-based surveys seemed to be convenient for visitors to our site to fill out, there is no precise and accurate way of determining response rate (Smith, 1997). For example, approximately 25 people have completed this survey since mounting it on our site close to a year ago. Although survey responses have provided us with valuable information we needed in order to assess the quality of the individual component, it remains unclear what percentage of the total number of visitors this small sample represents or how representative their comments are of the field.
Likewise, statistics gathered through the server provided us with important information about the use of our site without intruding upon visitors' use of the site. However, the unobtrusive nature of the data meant that we were unable to collect as much information as we would have liked. Since Internet users are assigned IP addresses dynamically, this makes their DNS identifiers completely unique and different each time they access a page. What this means for our data collection is that, although individual visits can be tracked, we are unable to identify whether these hits represent repeat visitors or the same visitors to different pages.
In terms of electronic mail surveys, NCIP found issues of time and space to be eliminated, enabling us to reach a wider audience with a quicker rate of return over those we sent via postal mail. While the bulk of the email surveys were returned within the first 24 hours, hardcopy versions of the survey were received two to three weeks later. This coincides with research by Mehta and Sivadas (1995) and Parker (1992) who found email response rate to be quicker than postal mail. NCIP learned that Web-based and electronic mail surveys provide significant cost reduction over mail and telephone surveys, not only in terms of telephone and postal mail charges, but also in the time it takes project staff to assemble mailings, contact individuals and execute the survey, and in entering and organizing the data once it is gathered.
NCIP also found that we had a higher response rate from participants who received the surveys via email over those who received it via postal mail. The response rate for the email survey was approximately 41%, while the response rate for the postal mail survey was 20%. One possible disadvantage of electronic surveys is that it is possible that technological prowess of respondents may skew data when information is delivered inaccurately as a result of not following directions. On the other hand, factors that may contribute to a higher response rate may be the ability to facilitate interaction between surveyors and respondents if there are any questions, and that replying to surveys is often more convenient for participants. This is consistent with findings described in "Casting the Net: Surveying an Internet Population" (Smith, 1997)
Similar to paper surveys, NCIP found that a portion of the surveys were returned due to undeliverable email and incorrect email addresses and that reminders were critical to response rate. However, NCIP found electronic surveys to be advantageous in the end, in that they were easier to manage in terms of keeping track of undeliverable surveys and individuals who required a reminder message.
Lastly, NCIP learned to think hard about the number of surveys sent electronically in an era of spamming and unwanted emails, even when participants had requested to be put on our mailing list. Throughout the data collection process, NCIP was cognizant of how often they solicited information from participants, so as not to flood visitors' emailboxes with unwanted messages. Likewise, NCIP paid careful attention to respecting visitors' anonymity and confidentiality of their responses.
What this Means for Non-Profits Collecting Data on the Web
In this article, we describe why it is important for non-profit organizations to collect data electronically; collecting data about a Web site can help organizations ensure that its goals are being met and that resources are being well-utilized. Findings can be included in reports to funders and can inform and further the field.
For NCIP, electronic data significantly contributed to our understanding of our audience and what they value at our Web site. It has also provided us with feedback from our audience so that we can build on our initial successes in delivering professional development opportunities that are useful to the field. For a non-profit, collecting data electronically means that information can be gathered more efficiently and cost-effectively. Finally, NCIP believes it is possible to find a middle ground between gathering sufficient data to meet project's goals and further the field, and remaining unobtrusive to its users.
Jennifer Gold (JGold@edc.org) is Research Assistant and Denise Ethier is Telecommunications Coordinator for the National Center to Improve Practice, located at Education Development Center, Inc. in Newton, MA. Denise Ethier can be reached at DeniseE@edc.org.
Copyright © 1997 by Jennifer Gold and Denise Ethier. All Rights Reserved.