503 - Instructional Design Project 3

Transcript

F RANCHISE A SSISTANT T RAINING Teaching basic to advanced levels of grading ID Project #3 Latanya Blakley – EdTech 503-4173 5/11/2010 5/11/2010 TABLE OF CONTENTS Reflection Paper Part 1: Topic Part 1a: Stated Learning Goal Part 1b: Audience Description Part 1c: Rationale Part 2: Analysis Report Part 2a: Description of the Need Part 2b: Description of the Learning Context Part 2c: Description of the Learners Part 3: Planning Part 3a: Learning Objectives Part 3b: Bloom’s Taxonomy Part 3c: ARCS Table Part 4: Instructor Guide Part 5: Learner Content Part 5a: Learning Materials Part 5b: Summative Assessment Materials Part 5c: Technology Tool Justification Part 6: Formative Evaluation Plan Part 6a: Expert Review Part 6b: OnetoOne Evaluation Part 6c: Small Group Evaluation Part 6d: Field Trial Part 7: Formative Evaluation Report Part 7a: Evaluation Survey Part 7b: Expert Review Results Part 7c: Comments on Change Part 8: AECT Standards Grid Appendices Appendix A 2 3 3 3 3 5 5 9 10 11 11 12 13 16 19 19 19 20 22 22 22 23 23 24 24 26 26 28 34 34 1 5/11/2010 SYNTHESIS REFLECTION PAPER If you look up the word “health” in a thesaurus, you will also find terms like fitness, soundness, form, and wellness. Each of these can be used to label the general condition of an object or situation at a given point in time. An instructional design project can be very similar to devising a plan for getting healthy. First of all, an overall fitness goal should be established. Given that most people have an easy rationale for getting healthy, the next step is typically determining the best strategy to use. Let’s say for example, one decides that joining a gym would be best, it is common these days to receive a fitness assessment. In addition to using this assessment to convince you that personal training is the best solution, the gym staff will also orient you to your surroundings, equipment, services – in other words, the context. If you are to stay on track, setting small manageable objectives are best and you may even need to motivate yourself along the way. Should you opt not to use the personal training; you may decide create your own materials in the form of a workout regimen. Hopefully, as part of your plan, you have incorporated a few checkpoints to evaluate your progress so that you know when you’ve met your goals. I have increased the overall wellness of my instructional design knowledge and execution through the experiences in this class. Although I have been doing various forms of instructional design in my career, I did not always employ a wellstructured system. The use of the Smith and Ragan text as well as the readings from Gustasfon and Branch provided not only concrete systems and ways to approach instructional design, but also an assortment of tools and resources for different instructional needs and contexts of which I was not aware. I plan to reference many of the course materials in my work until it becomes second nature to systematically design courses. Even then, I’ll continue to reinforce my practice. I’m convinced that honing the foundational philosophies is necessary while I sharpen my new skills and keep abreast of trends and changes in Educational Technology and instructional design. I am proud to have an arsenal of theories and analysis techniques that will justify and support my future design decisions. For far too long, training departments have used anecdotal evidence and presumptions to create what they believe the learner needs. The initial ID project for this class helped me develop a geniune appreciation for and understanding of the importance of analysis prior to lifting a pen. This culminating project provided proof that having a strategy is the best way to achieve stated instructional goals. I am healthier in my practice because I can really walk and talk like a sound instructional designer is expected to in the field of Educational Technology. The valuable concepts I have learned will be used to take my instructional designs to the next level for the benefit of the learners. 2 5/11/2010 PART 1. TOPIC Part 1a. Stated Learning Goal Employees of a supplemental education franchise will learn basic, intermediate and advanced principles of grading student assignments. Upon completion of the instruction, employees will be able to accurately grade a full math or reading assignment. Accuracy will be determined by no more than two errors in a given grading packet. This project will cover the design of the introductory grading course and the conceptual layout for intermediate and advanced courses. Part 1b. Audience Description The audience for this course described in this instructional design document are franchise owner employees, also known as the “Assistants”. The learners will be any number of the approximately 7,000+ Assistants in our North American franchise system. Part 1c. Rationale In every franchise, there are key tasks that an Assistant must perform. Typically, Assistants are taught the basics on how to grade student work before any other task. This is often done through observation, job shadowing and simple on-the-job training. Unfortunately, this informal process of training does not result in consistent, accurate, long-term learning of grading procedures needed for high levels of job performance. Additionally, there is no way to formally measure what has been learned and what may need clarification in these informal trainings unless they are revealed through errors and sporadic questions on the job. The former can become a concerning issue because this often means the customer (Kumon student) is impacted by these mistakes. Finally, franchise owners are generally not afforded the time to conduct an indepth training for new Assistants or developmental training for Assistants wishing to advance to higher levels. In these cases, again they turn to their most senior staff to facilitate informal training sessions using the strategies mentioned earlier. The overall strategy for this course is a supplantive or mathmagenic one and this is primarily due to the nature of the learning and the business needs. Although the tasks are simple and well-defined, there is a high demand for Assistants to be competent in grading procedures. Other characteristics of the content that make it more conducive to a more supplantive strategy is the limited time and universal goals and policies that must be adhered to by such a large audience of learners. After students learn the basics, the procedures become 3 5/11/2010 more complex, therefore a high level of scaffolding and supplied materials are needed to reduce cognitive load and allow the working memory to focus on the produres and conditions for performing them. Grading worksheets will be taught to the learners using primarily procedural strategies. Why is this the most appropriate strategy? When Assistants undertake grading tasks, they must refer to the standard grading procedures and known principles that guide them to mark an answer correct or incorrect. Once these guidelines are learned, the Assistant will also have to analyze the exercises and engage in some decision making during the operation of a procedure. This supports the need to use a supplantive or expository approach as the overall strategy, as the steps are prescribed to the learners. In a few cases, declarative and conceptual knowledge will need to be taught, whereas a slightly different strategy will be used. 4 5/11/2010 P ART 2. A NALYSIS R EPORT Part 2a. Description of the Need Part 2a.1 Needs Analysis Survey Prior to this portion of the project, part of the needs analysis was done through a Vovici survey distributed to the franchise owners. This survey was conducted by a different department and therefore did not include all of the elements needed about the potential learners. However, it was established that over 75% of franchise owners believed the greatest need for training resided in the area of guiding students and grading student assignments as shown in the graph below. Another aspect of the needs assessment has to do with the nature of the business with regards to the Assistant (learner) population. As with most businesses franchise owners have two main personnel issues with which to deal, hiring and retaining staff. Training needs arise when one of four situations occurs: a) a new franchise opens with all new staff, b) an existing franchise hires new staff, c) existing staff are developed or cross-trained or d) new policies or procedures are implemented. To accommodate these situations, training courses must take into the dynamic nature of the Assistant population and the 5 5/11/2010 business environment. From the following chart, we can see that over 80% of franchise owners experience retention issues ranging from minor to critical. Through anecdotal evidence, it has been discovered that many Instructors desire a self-paced training program that doesn’t require their direct involvement all of the time and allows them to focus on other aspects of their business. Because we are unable to interview Assistants who are not yet hired, we must focus our assessment on current employees. It is our hope that they can provide some insight as to the knowledge of grading that a new Assistant needs to be successful, as well as insight into their own developmental needs. The following questions will be presented in a survey to a sample population of Assistants: 1. 2. 3. 4. 5. How long have you been a franchise Assistant? Have you ever been a franchise student (at any Center)? If yes, please indicate which subject. Up to what level of the worksheets did you complete? How were you first trained on the grading principles? (please check all that apply) a) on the job (you just began working), b) group training session, c) 6 5/11/2010 individual training session, d) computer based course, e) paper based course, f) observation (you observed someone grading) 6. Have you ever taken an online course prior to this one? 7. Which worksheet subject do you have the most experience grading? 8. What is your position at the Center? a) Assistant b) Grader c) Chief Assistant d) JK Assistant e) other 9. On a scale of 1-5, (where ‘1’ is the least comfortable and ‘5’ is the most) rate how comfortable you are with grading Reading levels 2A – CII. 10. On a scale of 1-5, rate how comfortable you are with grading Math levels 3A – D. 11. On a scale of 1-5, rate how comfortable you are with grading Reading levels DI – H. 12. On a scale of 1-5, rate how comfortable you are with grading Math levels E – H. 13. On a scale of 1-5, rate how comfortable you are with grading Reading levels I – L. 14. On a scale of 1-5, rate how comfortable you are with grading Math levels I – L. 15. On a scale of 1-5, rate how comfortable you are with grading Math levels M – O. Part 2a.2 Survey Results The survey was deployed to 30 Assistants in the company’s designated Central Region with 28 returned surveys. Of those surveyed, more than half have been employed for less than 6 months. This would indicate that although those Assistants may have some familiarity with the process of grading and the context of how things work at the franchise, they have not been employed long enough to be considered experts. About 21% of the Assistants were once students at a Center and most worked past level G, this may provide evidence that a small percentage of our audience will have a significant level of experience doing the curriculum which may help them be more proficient at grading. When asked how they were first trained, Assistants were asked to indicate all methods that applied. Here is how they responded: Method of Training On the job (you did the grading) Group training session Individual training session Computer based course Paper based course Observation (you watched someone grading) Number of Respondents 12 6 19 1 5 22 7 5/11/2010 Less than 40% of Assistants indicated that they had taken an online course prior to this one, yet I suspect more than that have experience with computers and the Internet. All but 2 of the Assistants believed they had the most experience grading Math worksheets. This is typical, because due to the more complex nature of the Reading program, most Assistants start off grading in the Math program until they have a high level of competency. Question 8 was changed to include the general Assistant position in the categories. Most respondents considered their position Assistant, with only 7 indicating Grader and no other categories were selected. The franchise owner is likely to have affected this by selecting the employees who held these two positions if they thought that the training had more relevance to their jobs. The following graph shows the averages of the responses for questions 10 – 15, regarding the ranking of Assistant comfort with grading certain worksheet levels. The more difficult the levels are, the less comfort Assistants have with grading them. This may mean that more practice will be needed in the subsequent Intermediate and Advanced grading courses. Average Grading Comfort Rating 5 4 3 2 1 0 Reading 2A-CII Math 3A-D Reading Math E - Reading Math I - Math M I- L L -O DI - H H 8 5/11/2010 Part 2b. Description of the Learning Context Part 2b.1 Learning Context The learning context description is a challenging aspect of this project, because of the variations in our over 1600 franchises in North America. However, we are able to make certain generalizations relevant to the Center environment and Assistant experience. The learners will be trained via the company’s Onling Learning Center (OLC). However, access to the online courses will be determined by the franchise business setup. While franchisees must have a computer, not all are required to have internet access in their Center. It is surmised that a large percentage of them have internet access, though it is difficult to know the exact number without 100% survey compliance or an audit of all 1600+ business locations. If we consider the typical Center with internet access, on average there is just one computer connected. Therefore, much of the prior training Assistants receive in the subject area is using pencil and paper. This does not preclude the fact that many Assistants use the computer in the Center for other tasks and because of their age ranges, are generally very knowledgable about computers and the internet. Depending on the practices of the Centers, Assistant motivation to take the courses will vary. Some franchise owners offer compensation for training, while others ask employees to study on their own time. It is our assumption that the latter is not a prevalent as the former and is becoming less popular as a practice. A franchise Center could be either in a commercial space, such as a strip mall or office building, or a shared public space, like a church or school. Over 70% of businesses are in commericial spaces, but this range of options means different square footage and availability of a quiet place to participate in training. The franchise owner and the Assistant will need to work together to ensure an environment conducive to learning is accessible for learners. Part 2b.2 Transfer Context Learners will use the knoweldge and skills they acquire about grading student worksheets immediately in the franchise environment. Because of the unique nature of the business, this is the primary context for knowledge transfer and exhibition. Some secondary concepts such as grading accuracy, proper documentation, student observation and guiding learners may be used outside the class. Other aspects of the training that are transferable to different contexts would include navigating online courses, completing interactivities, problems solving and completing online assessments. These residual skills could be useful in future development efforts at other places of employment or even in educational settings where online courses are offered. 9 5/11/2010 Part 2c. Description of the Learners The majority of Assistants are female and between the ages of 16 – 19. They tend to be high school and college students, with the next largest population being individuals with a teaching background. About 10% of all Assistants have taken some portion of Kumon curriculum before being hired. As one considers hiring, there are common characteristics that franchise owners look for in their Assistants, some of which include: • • • A strong aptitude for math and/or reading Enjoys working with children Good attention to detail and/or organization Typical length of employment is 12 – 24 months, depending on numerous factors for example, where Assistants are in their education or job. For example, a franchise owner would have better success at retaining a sophomore in high school versus a senior and of course, there are exceptions. While most would work about 6 – 8 hours per week, some Assistants may average 10 – 15 hours, depending on the duties they perform; with some working all the way up to 20 hours if they are classified as a chief Assistant. Additional information about the learners will be obtained through a direct survey of the population. The results of the survey are found in section 2a.2. 10 5/11/2010 P ART 3. P LANNING Part 3a. Learning Objectives 1. After being presented with an overview, learners will be able select from a list the purpose of grading student worksheets. 2. Learners will demonstrate knowledge of grading principles & guidelines. 2.1. Given a list, learners will identify the proper tools for conducting grading tasks. 2.2. Given a list, learners will identify the procedure for using the answer book. 3. Given a worksheet, learners will correctly use the grading scale on the a-side. 4. Given images of grading symbols, learners will be able to correctly label each. 4.1. Learners will identify the symbol for a worksheet that has no errors. 4.2. Learners will identify the symbol for a full or partial error on an exercise. 4.3. Learners will identify the symbol for a corrected error on an exercise. 5. Given a worksheet with incorrect answers, learners will be able to determine the errors and mark them incorrect. 6. Learners will be able to recognize from a list of examples and non-examples, the instances where using a triangle in the Math or Reading program is necessary. 7. Given the formula, learners will be able to accurately calculate the percentage score for a graded worksheet. 8. Given a graded packet, learners will be able to summarize the scores for recording. 9. Given the guidelines, learners will be able to discriminate exercises that need an asterisk for follow up. 10. Given a specific scenario, learners will correctly recognize when to ask for help in interpretting an answer. 11 5/11/2010 Part 3b. Matrix of Objectives, Bloom’s Taxonomy and Types of Learning Objective Bloom’s Taxonomy Number (1) Classification (2) 1.0 2.0 2.1 2.2 3.0 4.0 4.1 4.2 4.3 5.0 6.0 7.0 8.0 9.0 10.0 Comprehension Comprehension Knowledge Knowledge Application Knowledge Knowledge Knowledge Knowledge Application Analysis Application Comprehension Analysis Analysis Strategy to be employed to teach the objective (3) Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Supplantive Type of Learning (4) Declarative Conceptual Declarative Procedural Procedural Declarative Declarative Declarative Declarative Procedural Conceptual Procedural Procedural Conceptual Procedural 12 5/11/2010 Part 3c. ARCS Table Project Goal Statement: Employees of a supplemental education franchise will learn basic, intermediate and advanced principles of grading student assignments. Upon completion of the instruction, employees will be able to accurately grade a full math or reading assignment. Accuracy will be determined by no more than two errors in a given grading packet. This project will cover the design of the introductory grading course and the conceptual layout for intermediate and advanced courses. ATTENTION A.1 Perceptual Arousal >The introduction of the course will include an overview video that shows how students enter the franchise and begin to work. It then begins to incorporate the responsibilities of the Assistant in a narrative story. This will help create relevance to the Assistant’s job. A2. Inquiry Arousal >The video will continue to tell the story up until the point when the Assistant begins to grade the student’s work. At this point, we will cut the video and begin to ask a series of questions about what procedures are involved in grading student work. This will also stimulate the learner’s previous knowledge and hopefully get them thinking about what they’ve learned about grading thus far. A3. Variability >To maintain the learner’s attention, we will continue to follow the narrative to answer some of the questions posed. The learner will also encounter additional scenarios via video, interactivities, informational slides and quizzes. RELEVANCE R1. Goal orientation >First of all, by engaging students in a different format of learning than the on-thejob and hands on approach, we hope to provide a more consistent message of learning. We will ask students to think of a time when they were asked to do something, but didn’t quite understand the procedure. We will revisit the Assistant responsibilities provided in the Introductory course, and then help them understand 13 5/11/2010 the main goal of this training is to increase their level of grading competency so they can be successful on the job. R2. Motive matching >Using actual Assistants in our video footage will also create links between the learners and the material. These Assistants will provide the modeling and influences needed to provide the learners with a vision for what success looks like. As the courses are continued, learners will be able to work with their franchise owner to determine a development plan towards various certifications. R3. Familiarity >We will tie instruction to the learner’s experiences, using authentic scenarios, current procedures and lower level worksheets that each Assistant would have experienced grading within their first few days of working in the franchise. CONFIDENCE C1. Learning requirements > The course is subdivided into three levels of learning. In this course, we will inform the learner that they will gradually gain the knowledge and expertise needed to be successful. The learning expectations will be communicated early on in the course as well as suggestions for “off-line” development. C2. Success opportunities > Learners will be provided with demonstrations of the tasks and opportunities to practice. The quizzes and interactivities are truly meant to be practice and are not included in the calculation of the final course assessment. Learners are given multiple opportunities to take the course and the assessment. It is iterated that they can take the course again before attempting the assessment if necessary. C3. Personal control >The course is completely self-paced. Learners can redo the quizzes and interactivities if more practice is desired. During these learning interactivities, learners will receive immediate feedback and therefore will clearly know their success is based on their efforts and abilities. 14 5/11/2010 SATISFACTION S1. Natural consequences >Because this is job training, the meaningful opportunities to use these newly acquired skills are inherent in their position. However, as the course steps up to the next level, we will revisit some of these skills in the form of review activities. S2. Positive consequences >Much of the reinforcement will come in the form of accomplishments on the job, recognition through the certification and the chance to continue professional development and possibly job advancement. S3. Equity >Learners will be provided with a printable certificate for each course completed. They will also be given professional development credits and opportunities for further development. 15 5/11/2010 PART 4. INSTRUCTOR GUIDE The materials will be produced as an asynchronous elearning course, therefore the software will be the instructor. The following is a flow chart of how the learner will proceed through the course from beginning to end. Key elements of the Introduction, Body and Conclusion are highlighted. The accompanying materials that show this progression are provided in Appendix A. This flowchart will be revised upon the revision of materials described in the expert review. 16 5/11/2010 17 5/11/2010 18 5/11/2010 PART 5. LEARNER CONTENT Part 5a. Learning Materials The learning materials are presented in Appendix A. Because the asynchronous elearning course will be created in Articulate Presenter, the materials are provided in storyboard format using PowerPoint slides. Videos, interactivities and quizzes that will be used are indicated in the proposed flow of the course. Part 5b. Summative Assessment Materials The following questions will be a part of the summative assessment for this course. While some of the questions will be presented in multiple choice or true/false format, there are questions that involve using images of the worksheets. The format for these types of questions is listed below. 1. What is the purpose of grading worksheets? (Select all that apply) a) Students need to know their mistakes so they can learn from them b) To share with the parents all the errors their child has made c) Instructors need this information to plan the student’s upcoming work and to support them d) We need to be able to acknowledge students’ efforts e) To check it off because it is on the top of our common responsibilities list 2. Which of these tools are used for grading worksheets? a) Blue pen and an answer book b) Red pen and an answer book c) Blue pen only, answer book is not yet used d) Red pen only, answer book is not yet used 3. True or False. You should always start from the front of the assignment to grading the worksheets with the answer book. 4. Which symbol means that the entire page of work is correct? a) Triangle b) Circle c) ‘X’ d) Triangle w/ a Circle around it 5. True or False. A triangle counts as a full error in the Math Program. 6. Select all the instances where a triangle would be used in the Math Program: a) When a student’s answers are illegible 19 5/11/2010 b) When a student leaves an exercise blank or does not complete a problem c) When a student does not follow the directions d) When word problems are incomplete e) When the student gets the answer correct f) When, in Level C and higher, the student forgets to write the remainder beside the quotient g) When the student needs assistance with his or her work 7. The scores for a five page worksheet assignment are three 100%s, 70% and 80%. Please indicate how the scores should be summarized: a) 1 1 1 7 8 b) 10 10 10 7 8 c) - - - 7 8 d) 1 1 1 70 80 Questions 8 – 11. Using an interactive, the learners will be presented with worksheets on the screen. They will be asked to determine if the worksheet is all correct or has errors. If they indicate that the worksheet has errors, they must indicate which exercises are incorrect via radio buttons. Questions 12 – 15. Using an interactive, the learners will be provided with Reading grading scales on the screen. They will be asked to select the percentage associated with the type and amount of errors. 20 5/11/2010 Part 5c. Technology Tool Justification Technology Tools Laptop or Desktop Computer Rationale for Use Large scale deployment has made it necessary for this course to be created as computerbased instruction. Learners will need to access the materials via the computer to complete an asynchronous computer course. This was also necessary to allow learners to work individually and at their own pace, track usage statistics and performance data for future use in a possible Assistant certification program. Internet Access The Internet is used as a tool only to access the course materials. Learners must use a browser to navigate to the URL. Online Learning Center (OLC) This tool is used to house or “host” the course. Its primary function is to act as the current online learning management interface which stores each online course created for learners. Moodle This is the tool behind the OLC and is necessary to store the collective course data, login data and assessments. PowerPoint The storyboard for the course is created using this tool. Learners will not need to directly interface with this tool. Articulate Presenter The course is created by vendors using this tool. Learners will interface with this tool and will need to understand navigation instructions. There are several interactive elements that learners will access in the course through this interface. 21 5/11/2010 PART 6. FORMATIVE EVALUATION PLAN Part 6a. Expert Review The subject matter expert who will review this Instructional Design is Kerry Underwood, an associate in the Instruction Department. Her contact information is [email protected]. Kerry is considered an expert because she has been employed at the company for more than 10 years and has actually managed one of our company-owned sites. A survey was submitted to Kerry via Google Docs for a formative review of the materials. The results of the survey are included in the next section. Part 6b. One to One Evaluation In this stage of evaluation, two Assistants who have been employed for more than 1 month, but less than 1 year would be asked to participate. In fact, they will likely be chosen from both ends of that time continuum, giving us a very new Assistant, and an Assistant who is close to a year of employment. This way, a broad range of experience can be represented. The learners will be asked to use two techniques, the “read-think aloud” technique and an interview of the overall experience at the conclusion of the course. The first technique will be employed because the instruction is done through a self-paced, computer-based course. Learners must be given intial instructions for navigation of the course and will be asked whether those instructions are ample. As the course continues, we will observe each learner interfacing with the materials and note his or her input into interactivities, quizzes and assessments. At the conclusion of the evaluation, we will have two perspectives on a variety of questions, such as: • • • • • • Are the navigation instructions clear? Are the interactivity instructions easily understood? Did learners know what to do during the various interactive and quiz elements? Are there any terms or jargon that need to be defined or replaced? Did learners possess the entry level skills or declarative knowledge needed to be successful? Are the illustrations, videos, graphical elements helpful to the instruction objectives? Where did the learners’ encounter difficulties? What can be done to improve the overall instruction in this course? 22 5/11/2010 Part 6c. Small Group Evaluation Once revisions are ascertained and incorporated into the course, the next step of evaluation would be with a small group. Here we would choose approximately ten participants to take the revised course. The learners will likely be selected from different regions of the country and still represent a variation of employment experience with the franchise. Although the group will be handselected for this part of the evaluation, a learner analysis questionnaire will help us learn more about the audience. This group will be asked to complete the instruction from beginning to end without any assistance or supplemental study materials. We will also use this time to include a comprehensive assessment instrument to understand a complete view of the effectiveness of the instruction. Typically, the course itself would take learners no more than two hours, therefore sufficient time should be available for the detailed assessment. This assessment will include the criterion for mastery of each objective that will later be refined to include the best questions for the final learner assessment. The data from the assessments will be used to determine how well the learners’ understood the materials, if the learning objectives were mastered and if certain parts of the instruction would need further revisions. Additionally, interviews of the participants will be done via phone to gain more insight into their attitudes, motivation, opinions, and experiences with the course materials. Part 6d. Field Trial The field trials will occur after the small group revisions have been incorporated into the materials. In our company, field trials involve both the learners and the support staff. We will invite 30 participants to take the course, which will include the ten Assistants from the small group evaluation. Their input will be valuable in determining if the revisions made in the last stage of evaluations were effective. We will work to ascertain the following in this evaluation: • • • • • How much time does it actually take to complete the course and assessment? Do the questions in the assessment reflect the material presented? Are there any final issues with the course that need addressing: grammar, images, quizzes, interactivities, navigation, instructions, etc. Is the course flow conducive to learning and meeting the course objectives? How do the participants feel about the course? 23 5/11/2010 PART 7. FORMATIVE EVALUATION REPORT Part 7a. Evaluation Survey 24 5/11/2010 25 5/11/2010 Part 7b. Expert Review Results The expert was familiar with the project and the overall concept of the Basic, Intermediate and Advanced course plan. She received the storyboards, video concepts, interactivities and was asked to evaluate them. The following are the results of the survey. Q1: The materials received a rating of 4. Q2: The content of the existing materials were deemed accurate. Q3: The expert believed the instructions were clear although several demonstrations may be needed in the section regarding using the grading scale and the reading grading. Q4: We need to ensure that the term “summarizing” is clearly defined. Q5: The quizzes were still in progress, therefore a complete evaluation could note be provided. However, she agreed with the placement of quizzes and the concepts of using video and flash interactivities for variety. Q6: The expert believes the interactivity concepts will contribute to the goals and learning objectives. However, the math grading scale needs to be included. Q7: The expert believes the video concepts will contribute to the goals and learning objectives. This was one of her favorite aspect of the materials because it provides a real world feel to the materials to which learners can relate. Q8: The appropriateness of materials received a rating of 5. Q9: The expert suggested having a workbook of some sort to accompany the training course. We then discussed the details and how it would support the instruction. Part 7c. Comments on Change After reviewing the survey results, I spoke with the expert about her feedback. She mentioned that the course materials received the rating of 4 because it lacked the necessary tangible practice materials. Her suggestion is to add a printable workbook to the course that will allow new Assistants to get more hands on practice without using actual student work. She also noted that an explanation of the math grading scale should be included, even though it seems self-explanatory. I have already added more grading scale practice exercises for the reading section. I have decided to incorporate her other suggestions into the materials as well. In addition, I plan to include a few job aids that can be printed as well as a section of FAQs that can be accessed online or printed as a job aid. Lastly, I will add a slide that recaps the learning objectives and explains how they 26 5/11/2010 were met. It was beneficial to have someone else review the materials with the learning goals in mind. I have co-designed courses with this expert and she has run a company-owned franchise site. Therefore, she has a great deal of experience with Assistants and credibility with respect to the content and support needed for them to be successful. 27 5/11/2010 PART 8. AECT STANDARDS GRID Professional Standards Addressed (AECT) The following standards, developed by the Association for Educational Communications and Technology (AECT), and used in the accreditation process established by the National Council for Accreditation of Teacher Education (NCATE), are addressed to some degree in this course. The numbers of the standards correspond to the numbers next to the course tasks show on the list of assignments. Not all standards are addressed explicitly through student work. Standard 1: DESIGN 1.1 Instructional Systems Design (ISD) 1.1.1 Analyzing 1.1.2 Designing 1.1.3 Developing 1.1.4 Implementing 1.1.5 Evaluating 1.2 Message Design 1.3 Instructional Strategies 1.4 Learner Characteristics Assignments meeting standard in whole or part X X X X X X ID Projects 1 & 2 ID Projects 1 ID Projects 1 & 2 ID Projects 1 & 2 ID Project 2 Selected Discussion Forums; ID Project 2 ID Project 2 ID Project 1 X X Standard 2: DEVELOPMENT 2.0 (includes 2.0.1 to 2.0.8) 2.1 Print Technologies 2.2 Audiovisual Technologies 2.3 Computer-Based Technologies 2.4 Integrated Technologies Standard 3: UTILIZATION 3.0 (includes 3.0.1 & 3.0.2) 3.1 Media Utilization 3.2 Diffusion of Innovations 3.3 Implementation and Institutionalization 3.4 Policies and Regulations Standard 4: MANAGEMENT X X X ID Project 02 Reading Quiz; ID Projects 1 & 2 (all assignments) X (all assignments) X ID Project 2 28 5/11/2010 4.0 (includes 4.0.1 & 4.0.3) 4.1 Project Management 4.2 Resource Management 4.3 Delivery System Management 4.4 Information Management Standard 5: EVALUATION 5.1 Problem Analysis 5.2 Criterion-Referenced Measurement 5.3 Formative and Summative Evaluation 5.4 Long-Range Planning X X X ID Project 2 ID Project 2 COURSE GOALS & OBJECTIVES The overall goal for the course is for each student to consider and use the systematic process of instructional design to create an instructional product. To achieve this goal, students will engage in activities that promote reflective practice, emphasize realistic contexts, and employ a number of communications technologies. Following the course, students will be able to: 1. Discuss the historical development of the practice of instructional design with regard to factors that led to its development and the rationale for its use 2. Describe at least two reasons why instructional design models are useful 3. Identify at least six instructional design models and classify them according to their use 4. Compare and contrast the major elements of three theories of learning as they relate to instructional design 5. Define “instructional design.” 6. Define the word “systematic” as it relates to instructional design 7. Define “learning” and synthesize its definition with the practice of instructional design 8. Relate the design of instruction to the term “educational (or “instructional”) technology” 9. Describe the major components of the instructional design process and the functions of models in the design process 29 5/11/2010 10. Provide a succinct summary of various learning contexts (declarative knowledge, conceptual, declarative, principle, problem-solving, cognitive, attitudinal, and psychomotor) 11. Build an instructional design product that integrates major aspects of the systematic process and make this available on the web. a. Describe the rationale for and processes associated with needs, learner, context, goal, and task analyses i. ii. Create and conduct various aspects of a front-end analysis Identify methods and materials for communicating subject matter that are contextually relevant b. Describe the rationale for and processes associated with creating design documents (objectives, motivation, etc.) i. ii. iii. Construct clear instructional goals and objectives Develop a motivational design for a specific instructional task Develop assessments that accurately measure performance objectives c. Select and implement instructional strategies for selected learning tasks i. Select appropriate media tools that support instructional design decisions d. Describe the rationale and processes associated with the formative evaluation of instructional products i. Create a plan for formative evaluation 12. Identify and use technology resources to enable and empower learners with diverse backgrounds, characteristics, and abilities. 13. Apply state and national content standards to the development of instructional products 14. Meet selected professional standards developed by the Association for Educational Communications and Technology 15. Use various technological tools for instructional and professional communication 30 5/11/2010 AECT STANDARDS (Applicable to EDTECH 503) 1.0 Design 1.1 Instructional Systems Design 1.1.a Utilize and implement design principles which specify optimal conditions for learning. 1.1.b Identify a variety of instructional systems design models and apply at least one model. 1.1.1 Analyzing 1.1.1.a Write appropriate objectives for specific content and outcome levels. 1.1.1.b Analyze instructional tasks, content, and context. 1.1.2 Designing 1.1.2.a Create a plan for a topic of a content area (e.g., a thematic unit, a text chapter, an interdisciplinary unit) to demonstrate application of the principles of macro-level design. 1.1.2.b Create instructional plans (micro-level design) that address the needs of all learners, including appropriate accommodations for learners with special needs. 1.1.2.d Incorporate contemporary instructional technology processes in the development of interactive lessons that promote student learning. 1.1.3 Developing 1.1.3.a Produce instructional materials which require the use of multiple media (e.g., computers, video, projection). 1.1.3.b Demonstrate personal skill development with at least one: computer authoring application, video tool, or electronic communication application. 1.1.4 Implementing 1.1.4.a Use instructional plans and materials which they have produced in contextualized instructional settings (e.g., practica, field experiences, training) that address the needs of all learners, including appropriate accommodations for learners with special needs. 1.1.5 Evaluating 1.1.5.a Utilize a variety of assessment measures to determine the adequacy of learning and instruction. 1.1.5.b Demonstrate the use of formative and summative evaluation within practice and contextualized field experiences. 1.1.5.c Demonstrate congruency among goals/objectives, instructional strategies, and assessment measures. 31 5/11/2010 1.3 Instructional Strategies 1.3.a Select instructional strategies appropriate for a variety of learner characteristics and learning situations. 1.3.b Identify at least one instructional model and demonstrate appropriate contextualized application within practice and field experiences. 1.3.c Analyze their selection of instructional strategies and/or models as influenced by the learning situation, nature of the specific content, and type of learner objective. 1.3.d Select motivational strategies appropriate for the target learners, task, and learning situation. 1.4 Learner Characteristics 1.4.a Identify a broad range of observed and hypothetical learner characteristics for their particular area(s) of preparation. 1.4.b Describe and/or document specific learner characteristics which influence the selection of instructional strategies. 1.4.c Describe and/or document specific learner characteristics which influence the implementation of instructional strategies. 2.0 Development 2.0.1 Select appropriate media to produce effective learning environments using technology resources. 2.0.2 Use appropriate analog and digital productivity tools to develop instructional and professional products. 2.0.3 Apply instructional design principles to select appropriate technological tools for the development of instructional and professional products. 2.0.4 Apply appropriate learning and psychological theories to the selection of appropriate technological tools and to the development of instructional and professional products. 2.0.5 Apply appropriate evaluation strategies and techniques for assessing effectiveness of instructional and professional products. 2.0.6 Use the results of evaluation methods and techniques to revise and update instructional and professional products. 2.0.7 Contribute to a professional portfolio by developing and selecting a variety of productions for inclusion in the portfolio. 32 5/11/2010 2.1 Print Technologies 2.1.3 Use presentation application software to produce presentations and supplementary materials for instructional and professional purposes. 2.1.4 Produce instructional and professional products using various aspects of integrated application programs. 2.3 Computer-Based Technologies 2.3.2 Design, produce, and use digital information with computer-based technologies. 3.0 Utilization 3.1 Media Utilization 3.1.1 Identify key factors in selecting and using technologies appropriate for learning situations specified in the instructional design process. 3.1.2 Use educational communications and instructional technology (SMETS) resources in a variety of learning contexts. 3.3 Implementation and Institutionalization 3.3.1 Use appropriate instructional materials and strategies in various learning contexts. 3.3.2 Identify and apply techniques for integrating SMETS innovations in various learning contexts. 3.3.3 Identify strategies to maintain use after initial adoption. 4.0 Management (none specifically addressed in 503) 5.0 Evaluation 5.1 Problem Analysis 5.1.1 Identify and apply problem analysis skills in appropriate school media and educational technology (SMET) contexts (e.g., conduct needs assessments, identify and define problems, identify constraints, identify resources, define learner characteristics, define goals and objectives in instructional systems design, media development and utilization, program management, and evaluation). 5.2 Criterion-referenced Measurement 5.2.1 Develop and apply criterion-referenced measures in a variety of SMET contexts. 5.3 Formative and Summative Evaluation 5.3.1 Develop and apply formative and summative evaluation strategies in a variety of SMET contexts. SMET = School Media & Educational Technologies 33 5/11/2010 APPENDIX A: LEARNING MATERIALS 34 5/11/2010 35 5/11/2010 36 5/11/2010 37 5/11/2010 38 5/11/2010 39 5/11/2010 40 5/11/2010 41