First, complete the knowledge survey on knowledge surveys

of 97

Please download to get full document.

View again

All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
PDF
97 pages
0 downs
0 views
Share
Description
First, complete the knowledge survey on knowledge surveys. Plan Your Training From A to Z: Using Knowledge Surveys . Ed Nuhfer, Director of Faculty Development California State University at Channel Islands ed.nuhfer@csuci.edu. Knowledge Surveys are Assessment instruments.
Transcript
First, complete the knowledge survey on knowledge surveys Plan Your Training From A to Z: Using Knowledge Surveys Ed Nuhfer, Director of Faculty DevelopmentCalifornia State University at Channel Islandsed.nuhfer@csuci.eduKnowledge Surveys are Assessment instruments1. = I have insufficient knowledge to answer this question.2. = I have partial knowledge or know where to quickly (20 minutes or less) obtain a complete answer to this question.3. = I can fully answer this question with my present knowledge.….what knowledge surveys sampleAll learning produces complex interconnectedaffectiveandcognitive synaptic “wiring,”and knowledge surveys sample a mix of both.A Framework for Planning a KSResponsibilities published institution's goals that apply to the courseGoals-usually course instructors' goals (unpublished & general)Outcomes specifics of knowing & doing in course (usually published)How would I confirm student mastery?Items that require knowledge, skills, decisions, generating productsOrganize in order of course presentationResponsibility Example --a published descriptive statement"Goal 5: To understand how the physical sciences explain the natural world. Courses in the physical sciences that fulfill this requirement (1) examine the processes by which scientific knowledge is gained, (2) introduce the basic concepts and terminology of one or more of the physical sciences, and (3) explore how scientific knowledge influences human society." (Idaho State University Undergraduate Catalog, 2006)Example of goals that may not necessarily be a responsibility
  • I want students to achieve a life-long interest in my subject
  • I want students to develop self-assessment skills
  • I want students to understand the nature of critical thinking
  • Example of outcomesThese are action statements."Students will be able to:
  • describe the scientific method and provide an example of its application;
  • pick a single theory from the science represented by this course and explain its historical development;
  • provide two examples of testable hypotheses;
  • provide two specific examples that illustrate why it is important to the everyday life of an educated person to be able to understand science;
  • describe two current examples of the relationship between physical science and public policy…. (4 more)"
  • Making a knowledge survey (KS)Consider a skill or knowledge challenge such as might be on a test. For example, a challenge from the Geoscience Concept Inventory…Are rocks and minerals alive?(A) Yes, rocks and minerals grow(B) Yes, rocks are made up of minerals, and minerals are analogous to plant cells(C) Yes, rocks and minerals are always changing(D) No, rocks and minerals don't reproduce(E) No, rocks and minerals are not made up of atomsMake a KS item from that cognitive challenge by adding an affective root
  • I can distinguish the basis by which scientists tell whether rocks and minerals are alive.
  • (A) Yes, rocks and minerals grow
  • (B) Yes, rocks are made up of minerals, and minerals are analogous to plant cells
  • (C) Yes, rocks and minerals are always changing
  • (D) No, rocks and minerals don't reproduce
  • (E) No, rocks and minerals are not made up of atoms
  • We may choose to address the same content with several items framed in different ways. Here, the former content addressed in multiple choice format is presented again, but now in essay format.
  • I can articulate the basis by which scientists distinguish living from non-living material.
  • What kinds of items can you put onto a knowledge survey?Any challenge you can express--Any test or homework itemConvergent challenges & factual knowledgeDivergent challenges evaluated with rubricsEthical decisionsProjects outcomesAll levels of thinkingExample-content comprehensionProject: Understand development of a major theory
  • 97. In under 500 words, I can describe how the contributions of all of the following were needed to achieve the current theory of plate tectonics: 1. Thales of Miletos (600 BC), 2. Aristotle (384 - 322 B.C.), 3. Nicolaus Steno (Neils Stensen) (1669), 4. William Smith (1769-1839), 5. Baron Georges Cuvier (1769-1832), 6. James Hutton (work of 1795), 7. Charles Lyell (1797 - 1875), 8. Charles Darwin (work of 1859), 9. William Thomson (Lord Kelvin - work in 1897), 10. John Joly (work of 1908), 11. Thomas Chamberlin (work of 1899 - 1909), 12. Madame Curie (1867 - 1934), 13. Bertram B. Boltwood (work 1905 - 1909), 14. Alfred Wegener (work in 1912 - 1924), 15. Arthur Holmes (work of 1920s - 1930s) 16. Harry Hess (work in mid 1960's), 17. J. Tuzo Wilson (work in mid 1960's), 18. Fred Vine (Work in mid 1960's), 19. John Dewey (work in early 1970's), 20. Niles Eldredge and Stephen Jay Gould (1972 –2000).
  • The knowledge survey emerges when we arrange the items in the order students will encounter the challenges in the course.
  • Students respond to each challenge at least twice (starting and ending of the course) through a three-point rating scale.
  • 1. = I have insufficient knowledge to answer this question.
  • 2. = I have partial knowledge or know where to quickly (20 minutes or less) obtain a complete answer to this question.
  • 3. = I can fully answer this question with my present knowledge.
  • Calculate the class average students give to each response. (Aggregate data of class as a whole.)
  • Plot the results (next slide)
  • Results: pre-course knowledge survey
  • 1. = I have insufficient knowledge to answer this question.
  • 2. = I have partial knowledge or know where to quickly (20 minutes or less) obtain a complete answer to this question.
  • 3. = I can fully answer this question with my present knowledge.
  • At a minimum, run the same survey again at the end of the course and compare results.Best practice: refer often to the knowledge survey during the course. Why do this?For very good selfish reasonsFor AWESOME altruistic reasons!Teaching for Learning & Satisfaction(after K. Feldman, 1997; 1998)SkillsQuizzesOutcomesProjectsFactsAssignmentsFactsFactsSkillsFactsEnduringConceptsReadingsFactsPapersDiscussionsFactsFactsExamsSkillsFactsWhat we want to teach-what students see -Why not give them a map…to those destinations we want them to reach?Why make a knowledge survey?
  • Being prepared and organized
  • Make time early with text and materials
  • Ability to plan and prioritize
  • Avoid getting caught in content overload
  • Why is it worth the time up front?Knowledge surveys—what will students new to the tool think?And how do they feel when they go through the survey at the end of class?Let's look again at the pre-course knowledge survey.
  • 1. = I have insufficient knowledge to answer this question.
  • 2. = I have partial knowledge or know where to quickly (20 minutes or less) obtain a complete answer to this question.
  • 3. = I can fully answer this question with my present knowledge.
  • We can start to know our students in ways we cannot without the initial measure.Note: We asked the students to rate their present knowledge. We did NOT ask students to rate their confidence to eventually master this material. Pre-course ratings bear no relationship to what each student can or will ultimately achieve.Pre-course surveys offer useful informationabout particular class populationsConfidence Rating -->Student Learning Outcomes versus Goals
  • Student Learning Outcomes are action statements that specify that a student must demonstrate achievement through some clearly observable action or generation of product.
  • Goal statements are necessarily broad statements that describe the general traits of learning. A goal statement need not be directly assessable.
  • Examples of Outcome Statements - What we should use as KS Items24. I can define psychology.25. I can describe the evolution of psychology as defined from the 1920s through today.26. I can summarize the nature-nurture debate in psychology, and describe the principle of natural selection.Examples of Goal Statements that were twisted into KS items.These goal statements began with "Students will be able to" and were converted into knowledge surveys by replacing that phrase with the root "I can…."
  • I can think clearly and logically.
  • I can find and critically examine information.
  • I can communicate at an appropriate level in both oral and written forms.
  • Goal statements used as knowledge survey items produce no information of value. They usually stand out as aberrations in an otherwise well constructed knowledge survey.Post-coursePre-courseConstruct a full battery of items that address the high priority content of the entire course.
  • Be certain to address all chosen outcomes with knowledge survey items.
  • Revelation - a Problem with PacingWith First Use - Expect to See Learning GapsPacing Problem EliminatedNext…Varied case examplesCase - "Closing the Loop"Music Theory Course: First UseCase - Longitudinal Study- "Closing the Loop"The Value of Longitudinal Data:Using Post-Course Knowledge Survey Data from Two ClassesExample Courtesy of Dr. Dexter Perkins, U of NDUsing a Knowledge Survey in American GovernmentPre and Post Survey of 75 QuestionsSurvey is Conducted Online, Through BlackboardSurvey Questions are Organized ChronologicallyResults of Surveys are Shared with StudentsSamples of Items UsedQuestion 8: I can explain ways in which the Constitution has become more democratic since its beginning in 1789.Question 17: I can tell you what freedoms are protected by the First Amendment.Question 37: Thinking of political party membership, I know what an independent voter is.Question 56: I know the difference between a standing committee in the Congress and a conference committee.Question 69: I can identify the three levels of the federal court system.Question 72: I can explain how judges are selected in California.POLS 150-04 American Political ScienceSpring 2008POLS 150-04 American Political ScienceSpring 2008POLS 150-04 American Political ScienceSpring 2008POLS 150-04 American Political ScienceSpring 2008Case - Design Course Appropriate to Students' Thinking Abilities Bloom’s Taxonomy: Questions Reflect Level of Reasoning ChallengeBloom Considerations of Knowledge SurveysI don't know the answerI know more than 50%I know the answerMatching our plan to our studentsRate each knowledge survey item’s Bloom Level. Reflect on whether our general course plan for levels of thinking challenge matches that of the needs of students in class.First: Familiar Look at Presentation1. = I have insufficient knowledge to answer this question.2. = I have partial knowledge or know where to quickly (20 minutes or less) obtain a complete answer to this question.3. = I can fully answer this question with my present knowledge.Coding Knowledge Survey Items by Bloom LevelIs the course appropriately challenging?Once it is over, how did students do across different levels of challenge?Case: mapping knowledge survey items as a means to assess student learning outcomesPractice to avoid…Course Outcomes - Example Psychology
  • Students will be able to demonstrate knowledge and understanding of theory and research in learning, perception, and cognition.
  • Students will be able to evaluate the appropriateness of conclusions presented in disseminated research relevant to psychology.
  • Students will be able to construct examples of how psychological theories and principles relate to everyday life.
  • Students will be able to explain and defend against common thinking fallacies.
  • Students will be able to explain behavior using different cognitive and learning theories or models.
  • We don't just ask five knowledge survey items. Instead, we map numerous KS items as applicable to course outcomes.We can see the degree that students confirm that they addressed planned course content and achieved intended outcomes.Student learning outcomes assessed by pre-post averages of multiple knowledge survey items that map to each outcome.Case: Did we change students' minds?Examples of numerical correlation coefficientsand their associated scatter plotsDid we change minds?If we did, then pre-post correlations should not be very high.First-Year Seminar Combined Sections280 studentsThis pre-post correlation seems a bit high. Peaks remain peaks and valleys remain valleys. Perhaps minds were not changed that much. Lots of blue shading indicates that not a great deal of new learning took place.Medical-Surgical Nursing CoursePre-post correlation is very high and minds seemed not changed much. Learning gains (not shown here) were low. This revelation produced a course redesign.r = 0.83Slide courtesy of Judy LewisUse correlation coefficient on pre-post measures. Which course changed minds the most?Case 4: Compare instructional modalitiesHistory 151 DE vs TRAD CONCEPT ITEMSCase - Assess long term retention and effects of learning(Courtesy of Dexter Perkins)Case. larger scale applications… Assess Curricula The use of knowledge surveys has been instrumental in broadening participation and stimulating real engagement by faculty in assessment activities. The ease of carrying out the surveys, and the direct and easily understandable 'pulse-taking' results, have demonstrated to busy but dedicated teachers the kind of useful feedback that assessment practices provide and that such techniques can be practical and sustainable. We are using knowledge surveys not just for individual courses but also at the discipline and program level to help with assessment of outcomes, as well as program mapping. Prof. C. Smith – HCC ALOOverall HumanitiesCase. Assessment at Degree LevelASSOCIATE OF ARTS GRADUATE SURVEY – SPRING 2008 GENERAL EDUCATION ITEMSAA GRAD GEN ED FACULTY USER FINDINGSRATING OVERALL HCC KS PROCESSKS FINDINGS RESULTED IN TEACHING STRATEGY CHANGES CHANGED TEACHING STRATEGIES IMPROVE SLO’SOVERALL VALUE KS ASSESSING SLO’SIf Ed made it this far in allotted time…He is luckier than he has a right to be.The inevitable question: "Do knowledge surveys correlate with tests/grades?"What are we really asking?How well should knowledge surveys correlate with tests?Only very generally -- becauseLearning measures are contextual at best, the two are measures derived from two different fractal neural networks that should have some overlap;results depend on consistence of practices. If a teacher constructs a knowledge survey, teaches to something else and tests perhaps on material neither covered in the knowledge survey nor taught in the classroom, that course lacks focus. Lack of course alignment/integrity will subvert any meaningfulInstruments are limited by reliability -- the weak link seems to be our testsReliability is Fundamental
  • Cronbach's Coefficient Alpha (most used)
  • Spearman-Brown reliability (R) measure from split halves (Jacobs and Chase, 1992) (yields best graphics to show visually what is taking place.)
  • derives from the correlation coefficient (r) obtained from individuals' scores on two halves of a single test
  • and is applicable to both tests and knowledge surveys.
  • Both yield similar numerical values.Graphic Example of Correlation ValuesPoints to remember
  • Meaningful correlation of measures made by two separate instruments can only be done after knowing the reliability of both. To do otherwise is like comparing analyses from two analytical chemistry instruments without standardizing both.
  • Correlation value between two instruments is limited by the least reliable instrument of the pair.
  • On the very best test you will get this…Grade reliability derived from ten quizzesReliability of a Typical Knowledge SurveyWhat if we use a standardized test? We have done so using the Geoscience Concept Inventory of Libarkin and AndersonKnowledge Survey ReliabilityReliability measures (R) derived from split halves correlation (r) and by calculation of coefficient alpha from pre-post-knowledge surveys built from 24 GCI items that were on the from the 305 item class knowledge survey. The pre-post knowledge surveys correlate with themselves at about r = 0.9Test ReliabilityReliability measures (R) derived from split halves correlation (r) from pre-post-tests built from 24 GCI items from the 305 item class knowledge survey. The test correlates with itself at about the level of 0.6 to 0.7.Test Correlated with Knowledge SurveyCorrelation of Post-class knowledge survey with Post-class Test is at r = 0.57. Correlation is limited largely by limits of test reliability. Student class test average = 97Predicted class test average from knowledge survey is 95.Whole class knowledge surveys scaled to 100 usually give an average that is within a few points of the actual whole class average. Evaluation versus Assessment
  • Evaluation - addresses measures applied to make decisions about individuals such as tests, graded papers & projects--grades of students and summative ratings of individual professors.
  • Assessment - addresses measures of students’ success in meeting specific learning goals through performance on outcomes. Uses aggregated data to describe performance of units --courses, classes, programs, institutions--as a whole.
  • A better way to assess general course success is to look at aggregate data
  • Look at distributions of scores on pre-post tests and pre-post knowledge surveys rather than individual students' grades.
  • Do similar changes occur in both distributions?
  • Let's see what happened in the study with the Geoscience Concept Inventory…
  • Pre-Course r =.49 Post-Course r =.98Quiz - 24 items Geoscience Concept InventoryKnowledge Survey 24 items Geoscience Concept Inventory
    Related Search
    Advertisements
    Similar documents
    View more...
    Advertisements
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks