Graduate Students:<br><br>Something to think about concerning research.<br><br>Dr. Franklin<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Rick Reis</b> <span dir="ltr"><<a href="mailto:reis@stanford.edu">reis@stanford.edu</a>></span><br>
Date: Tue, Oct 2, 2012 at 12:22 PM<br>Subject: TP Msg. #1199 Quantitative and Qualitative and Assessment Methods<br>To: tomorrows-professor <<a href="mailto:tomorrows-professor@lists.stanford.edu">tomorrows-professor@lists.stanford.edu</a>><br>
<br><br><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div><font face="Helvetica, Arial, sans-serif" size="3">Each primary type of qualitative data contributes unique and valuable perspectives about student learning to the outcomes-based assessment process. When used in combination, a more complete or holistic picture of student learning is created.</font><br>
<div style="text-align:center;font-style:normal;font-size:12pt;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-weight:normal">-----------------------------------------------------------------------------------------------------------------------------------</div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">TOMORROW'S PROFESSOR(sm) eMAIL NEWSLETTER</span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><a href="http://cgi.stanford.edu/%7Edept-ctl/cgi-bin/tomprof/postings.php" target="_blank">http://cgi.stanford.edu/~dept-ctl/cgi-bin/tomprof/postings.php</a></span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Archives of all past postings can be found at:</span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><a href="http://cgi.stanford.edu/%7Edept-ctl/cgi-bin/tomprof/postings.php" target="_blank">http://cgi.stanford.edu/~dept-ctl/cgi-bin/tomprof/postings.php</a></span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Sponsored by</span></div><div style="text-align:center">
<span style="font-family:Helvetica,Arial,sans-serif">Stanford Center for Teaching and Learning</span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><a href="http://ctl.stanford.edu" target="_blank">http://ctl.stanford.edu</a></span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Check out the Tomorrow's Professor Blog at:</span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><a href="http://derekbruff.org/blogs/tomprof/" target="_blank">http://derekbruff.org/blogs/tomprof/</a></span></div><br><font face="Helvetica, Arial, sans-serif" size="3">Folks: </font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">The posting below describes the differences between quantitative and qualitative research and the appropriate uses of each of them. It is from Chapter 4, Assessment Methods in the book: Demonstrating Student Success, A Practical Guide to Outcomes-Based Assessment of Learning and Development in Student Affairs, by Marilee J. Bresciani, Megan Moore Gardner, and Jessica Hickmott. Published by Stylus Publishing , LLC 2283 Quicksilver Drive, Sterling, Virginia 20166-2102.[<a href="http://www.styluspub.com" target="_blank">http://www.styluspub.com</a>] © Copyright 2009 by Stylus Publishing, LLC. All rights reserved. Reprinted with permission.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Regards,</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Rick Reis</font><br><font face="Helvetica, Arial, sans-serif" size="3"><a href="mailto:reis@stanford.edu" target="_blank">reis@stanford.edu</a></font><br>
<font face="Helvetica, Arial, sans-serif" size="3">UP NEXT: What Mentors Do </font><br><br><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Tomorrow's Research </span></div><div style="text-align:center">
<span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">---------------------------------------------------------- 1,700 words --------------------------------------------------------- </span></div>
<div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Quantitative and Qualitative and Assessment Methods</span></div>
<br><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Quantitative Assessment Methods</span></div><br><font face="Helvetica, Arial, sans-serif" size="3">Quantitative methods use numbers for interpreting data (Maki, 2004) and “are distinguished by emphasis on numbers, measurement, experimental design, and statistical analysis” (Palomba & Banta 1999). Large numbers of cases may be analyzed using quantitative design, and this type of design is deductive in nature, often stemming from a preconceived hypothesis (Patton, 2002). The potential to generalize results to a broader audience and situations make this type of research/assessment design popular with many. Although assessment can be carried out with the rigor of traditional research, including a hypothesis and results that are statistically significant, this is not a necessary component of programmatic outcomes-based assessment. It is not essential to have a certain sample size unless the scope of your assessment is on the institutional level.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">A traditionally favored type of research design that has influenced outcomes-based assessment methodology is quantitative assessment. Quantitative assessment offers a myriad of data collection tools including structured interviews, questionnaires, and tests. In the higher education setting, this type of design is found in many nationally employed assessment tools (e.g., National Survey of Student engagement, Community College survey of Student Engagement, and the CORE Institute Alcohol and Drug Survey) but can also be locally developed and used to assess more specific campus needs and student learning outcomes. It is important when engaging in quantitative methodological design, sampling, analysis, and interpretation to ensure that those individuals involved are knowledgeable about, as well as comfortable with, engaging in quantitative design (Palomba & Banta, 1999).</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">At Colorado State University, two primary quantitative assessment methods are used to examine apartment life on campus. “The Apartment Life Exit Survey is given to residents as they begin the 'vacate' process from their apartment. Results are tabulated twice each year, once at the end of fall semester and once in the summer” (Bresciani et al., in press).</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Administrators at Pennsylvania State University originally measured the success of their newspaper readership program based on satisfaction and use. The quantitative survey they were using was later revised “to include more detailed information on students' readership behavior (e.g., how frequently they are reading a paper, how long, and which sections), students' engagement on campus and in the community, and their self-reported gains in various outcomes (e.g., developing an understanding of current issues, expanding their vocabulary, articulating their views on issues, increasing their reading comprehension)” (Bresiani et al., 2009). This revision allowed them to use survey methodology while still measuring the impact of the program on student learning.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">CSUS underwent a similar revision process of a locally developed quantitative survey looking at its new student orientation program. Originally, only student and parent satisfaction were measured. This was later revised to include a true/false component in the orientation evaluation that used a form of indirect assessment. In the final revision, a pre-and post-test were administered to those students attending orientation to measure the knowledge gained in the orientation session (Bresciani et al., 2009).</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">In addition, a great deal of data already contained in student transactional systems can be used to assist in the evaluation of programs. Data such as facility usage, service usage, adviser notations, participation in student organizations, leadership role held, and length of community service can all help in explaining why outcomes may have been met. For instance, staff at an institution's counseling service desire for all students who are treated for sexually transmitted diseases to be able to identify the steps and strategies to avoid contracting them before leaving the 45-minute office appointment. However, when they evaluated this, they learned that only 70% of the students were able to do this, but they also examined their office appointment log and realized that because of the high volume of patients, they were only able to spend 27 minutes with each student on average. The decreased intended time to teach students about their well-being may explain why the counseling staff's results were lower than they would have desired.</font><br>
<br><div style="text-align:center"><span style="font-family:Helvetica,Arial,sans-serif">Qualitative Assessment Methods</span></div><br><font face="Helvetica, Arial, sans-serif" size="3">According to Denzin and Lincoln (2004), qualitative research is “multimethod in focus, involving an interpretive, naturalistic approach to its subject matter” (p. 2). Upcraft and Schuh (1996) expand this definition by stating, “Qualitative methodology is the detailed description of situations, events, people, interactions, and observed behaviors, the use of direct quotations from people about their experiences, attitudes, beliefs, and thoughts” (p. 21). Qualitative assessment is focused on understanding how people make meaning of and experience their environment or world (Patton, 2002). It is narrow in scope, applicable to specific situations and experiences, and is not intended for generalization to broad situations. Different from quantitative research, qualitative research employs the researcher as the primary means of data collection (e.g., interviews, focus groups, and observations. Also unlike quantitative research, the qualitative approach is inductive in nature, leading to the development or creation of a theory rather than the testing of a preconceived theory of hypothesis (Patton). It is important to note then that when applying qualitative methodology to outcomes-based assessment, you are not fully using an inductive approach because you are using the methodology to determine whether an intended outcome has been identified. However, the application of the methods themselves can yield very rich findings for outcomes-based assessment. </font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Data for qualitative analysis generally result from fieldwork. According to Patton (2002), during fieldwork a researcher spends a significant amount of time in the setting that is being investigated or examined. Generally multimethod in focus, three types of findings often result from the qualitative fieldwork experience; interviews, observations, and documents.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Each primary type of qualitative data contributes unique and valuable perspectives about student learning to the outcomes-based assessment process. When used in combination, a more complete or holistic picture of student learning is created.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Interviews</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Interviews comprise a number of open-ended, questions that result in responses that yield information “about people's experiences, perceptions, opinions, feelings, and knowledge” (Patton, 2002, p.4). It is common to engage in face-to face verbal interviews with one individual; however, interviews may also be conducted with a group and administered via mail, telephone, or the Web (Upcraft & Schuh, 1996). Though questions and format may differ, an essential component of any interview is the “trust and rapport to be built with respondents” (Upcraft & schuh, p. 32). Open-ended questions can also be given to students at the conclusion of a program or an event to receive quick and immediate feedback. At Widener University, “questions presented before, during, and after the {student health services} presentations allowed for an interactive experience and a means to monitor learning progress” (Bresciani et al., in press.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Observations</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Observations, on the other hand, do not require direct contact with a study participant or group. Rather, this type of data collection involves a researcher providing information-rich descriptions of behavior, conversations, interactions, organizational processes, or any other type of human experience obtained through observation. Such observation may be either participant, in which the researcher is actually involved in the activities, conversations, or organizational processes, or nonparticipant, in which the research remains on the outside of the activity, conversation, or organizational process in scope (Creswell, 1998; Denzin & Lincoln, 2000; Patton, 2002). In keeping a record of observations, many methods can be used. One way is to take notes during the observation; another method commonly employed is to create a checklist or rubric to use during the observation. The checklist or rubric not only gives the observer a set of criteria to observe, but it also allows the observer to show student progress over time and to correlate a number with a qualitative process. At North Carolina State University, for example, </font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3"> a total of 259 students that were found guilty of a violation of the {Student} Code {of Conduct} were assigned a paper with questions </font><br><font face="Helvetica, Arial, sans-serif" size="3"> specifically written to correspond with the criteria for the development of insight and impact on life issues, as identified in the learning </font><br>
<font face="Helvetica, Arial, sans-serif" size="3"> outcome. A rubric was used to review the papers. The rubric was created based on a theory of insight by Mary M. Murray (1995). In her book </font><br><font face="Helvetica, Arial, sans-serif" size="3"> Artwork of the Mind, Murray describes how to determine the development of insight through writing. Initially 20 papers were drawn randomly to </font><br>
<font face="Helvetica, Arial, sans-serif" size="3"> test the rubric. The rubric originally had a scale with three categories; beginning, developing and achieved and six dimensions based on the</font><br><font face="Helvetica, Arial, sans-serif" size="3"> theory and practice. In total, 22 papers were drawn and reviewed based on the rubric. (Bresciani eit al., 2009).</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Isothermal Community College (ICC) incorporated the qualitative assessment method of using portfolios for professionals completing the assessment process. Although this particular example focuses on staff and departments using portfolios, this method of assessment is commonly used with students as well. At ICC </font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3"> each year staff set aside time to reflect on what has been learned through assessment, compile related documents into a portfolio, and summarize</font><br><font face="Helvetica, Arial, sans-serif" size="3"> major areas of learning into what we refer to as “reflective narratives.” The process is systematic and ongoing with portfolios and narratives </font><br>
<font face="Helvetica, Arial, sans-serif" size="3"> submitted for review by various administrators in June of each year. (Bresciani et al., 2009.</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Documents</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Finally, documents include “written materials and other documents from organizational, clinical, or programs records; memoranda and correspondence; official publications and reports; personal diaries, letters, artistic works, photographs, and memorabilia; and written responses to open-ended surveys” (Patton, 2002, p. 4). Public records and personal documents are the two primary categories of documents one might use when doing outcomes-based assessment or research (Upcraft & Schuh, 1996). Newspaper and magazine excerpts, enrollment and retention records, and judicial records are examples of personal records. Both types of documents can enhance the overall data collected in an assessment project. It is important to note, however, that the authenticity of documents must be determined prior to using them for assessment (Creswell, 1998; Patton, 2992; Upcraft & Schuh, 1996).</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">In addition to the aforementioned documents, many student affairs professionals also use portfolios, student reflections, reports, or other forms of classroom-type documents for outcomes-based assessment data collection. Again, criteria checklists or rubrics can be used in the analysis of documents to identify whether outcomes are met. Keep in mind that whenever criteria are used with a qualitative method, the process of inductive discovery is diminished and therefore is the true nature of the qualitative methodology. Nonetheless, documents are a rich source of information and provide a great starting point for any assessment project.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">REFERENCES</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Bresciani, M. J. (in press-a). Challenges in the implementation of outcome-based assessment program review in a California Community College District. Community College Journal of Research and Practice.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Bresciani, M. J. (in press-b). An introduction to outcomes-based assessment; A comparison of approaches. McClellan & J. Stringer (Eds.), Handbook for student affairs administration (3rd ed.). San Francisco; Jossey Bass.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Bresciani, M. J. (in press-c). Understanding barriers to student affairs/services professionals' engagement in outcomes-based assessment of student learning and development. College Student Journal.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Creswell, J. W. (1998). Qualitative inquiry and research design: choosing among five traditions. Thousand Oaks, CA: Sage.</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Denzin, N., & Lincoln, Y. (Eds). (2000). Handbook of qualitative research. Thousand Oaks, CA: Sage.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Maki, P. L., (2004). Assessing for learning: Building a sustainable commitment across the institution Sterling, VA: Stylus.</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Palomba, C. A. & Banta, T. W. (1999). Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Patton, M.Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.</font><br><br><font face="Helvetica, Arial, sans-serif" size="3">Schuh, J. H., Upcraft, M.L., & Associates. (2001). Assessment practice in student affairs: an application manual. San Francisco: Jossey-Bass.</font><br>
<br><font face="Helvetica, Arial, sans-serif" size="3">Upcraft, M.L. & Schuh, J.H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.</font><br><br><br><br><font face="Helvetica, Arial, sans-serif" size="3">* * * * * * * </font><br>
<font face="Helvetica, Arial, sans-serif" size="3">NOTE: Anyone can SUBSCRIBE to the Tomorrows-Professor Mailing List by going to: </font><br><font face="Helvetica, Arial, sans-serif" size="3"><a href="https://mailman.stanford.edu/mailman/listinfo/tomorrows-professor" target="_blank">https://mailman.stanford.edu/mailman/listinfo/tomorrows-professor</a> </font><br>
<font face="Helvetica, Arial, sans-serif" size="3">You can UNSUBSCRIBE by hitting "return" to this posting with the word "unsubscribe" in the subject line. </font><br><br><br><br></div><br></div></div>
<br>--++**==--++**==--++**==--++**==--++**==--++**==--++**==<br>
tomorrows-professor mailing list<br>
<a href="mailto:tomorrows-professor@lists.stanford.edu">tomorrows-professor@lists.stanford.edu</a><br>
<a href="https://mailman.stanford.edu/mailman/listinfo/tomorrows-professor" target="_blank">https://mailman.stanford.edu/mailman/listinfo/tomorrows-professor</a><br>
<br>
<br>
<br></div><br><br clear="all"><br>-- <br><font style="color:rgb(51,51,255)"><i style="font-family:arial,helvetica,sans-serif">~~~~~~The best student-centered learning </i><span style="font-family:arial,helvetica,sans-serif">experience in America~~~~</span><br style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">Dr. Teresa Franklin</span><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">Pro</span></font><font style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,255)" face="tahoma">fessor, Instructional
Technology</font><font style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,255)" face="tahoma"><br>Instructional Technology Program
Coordinator</font><font style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,255)" face="tahoma"><br>313D McCracken Hall<br>Dept. Educational
Studies</font><font style="color:rgb(51,51,255)"><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">The Gladys W. and David H. Patton College of Education</span></font><font style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,255)" face="tahoma"><br>
Ohio University</font><font style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,255)" face="tahoma"><br>Athens, OH 45701</font><font style="color:rgb(51,51,255)"><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">740-593-4561 (office)</span><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">740-541-8847 (cell)</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">740-593-0477 (fax)</span></font><font style="font-family:arial,helvetica,sans-serif;color:rgb(51,51,255)" face="tahoma"><br>also: <a href="mailto:franklinteresa@gmail.com" target="_blank">franklinteresa@gmail.com</a><br>
<br></font><font><span style="color:rgb(102,0,102);font-family:arial,helvetica,sans-serif"><span>"A teacher affects eternity; [she]he can never tell where the influence stops." - Henry Adams</span></span></font><br>
<div><p style="color:rgb(51,51,255);font-family:arial,helvetica,sans-serif"></p><p style="line-height:normal"><i><span style="color:rgb(52,65,27)"><br></span></i></p><i><span style="color:rgb(52,65,27)"></span></i><p></p>
<p><i><font><br></font></i></p></div>
<br>