Quantative and Qualitative Measures
Critiques, Rubrics, and Support
Questions, Journals, and Observations
Expectations and Results
Charts and Thinking Maps
Electronic Portfolio and Artifacts

It is impossible to evaluate the success of the Project without weighing its impact on student learning and teacher development. But "before you can measure success, you must first identify what elements define it," says CML President and CEO Tessa Jolls.

The experimental nature of a demonstration project such as SMARTArt makes it "difficult to determine the right questions to ask in advance because you don't know what to expect," Jolls says. "In the beginning we didn't know the answers to fundamental questions, such as, 'Is it possible to motivate teachers to teach in a new way?' 'Can the basics of the arts disciplines and media literacy be integrated into multiple curricula?' 'Is there a strong connection to language arts development?'" she explains. "As the Project progressed, we learned the answer to all of those questions was 'Yes,' but they were unknown when we began."

Because of this, the Project's common measurement tools were limited to the end-of-the-year group evaluation of the student-produced animations and written assessment of the professional development training sessions, as described in Project Evaluation.

Quantative and Qualitative Measures

It is difficult to identify student progress directly attributable to the Project because the norm-referenced, multiple-choice questions in standardized tests measure only one type of learning. In projects like SMARTArt, which use a variety of learning modalities to develop students' skills in critical thinking and creative expression, better ways to measure progress are portfolio-based assessments, teacher observations, and authentic assessments.

In the first year a Pre/Post Test, administered in videotaped interviews with students at the beginning and at the end of Project sessions, was unsuccessful. Being nonverbal/nonreaders of English, the students' responses were not enlightening about their progress. Authentic assessment "is one part that is often overlooked and difficult to undertake," Jolls notes. In SMARTArt "a certain amount of rubrics and portfolios were used, but empirical assessment wasn't done." Although artists traditionally receive training in performance-based evaluation of artifacts and portfolios, generally schoolteachers have "neither the training nor the tools to do it," she says.

The limited amount of time available to teachers during class to conduct performance-based assessments was another factor. Because of the mandated focus on standardized test scores, a great deal of classroom time is required to administer the tests and prepare children to take them. English language learners especially need coaching in order to comprehend norm-referenced test questions such as "Which one of the following does not show...?" that are expressed in syntax more easily understood by a Caucasian child.

Despite these challenges, several individuals on the SMARTArt teaching team did implement effective performance-based practices, as illustrated in the following representative examples.

Critiques, Rubrics, and Support

When artist/educator Candy Danzig evaluates a student's ability to take a story, improvise, and present it in different ways, she's looking for process, not product. "It's what we did; how they got the exercise; how they listened to and supported each other," she says. She looks at "how well students follow a basic rule in improvisation: to accept what another person creates and make it their own." In measuring progress Danzig uses both individual and group performance-based tools.

A group assessment critique called "Two Likes and a Wish" takes place at the end of each storytelling exercise or rehearsal. Students say two things they like and get to wish for something they think could be improved. Danzig models appropriate responses: "I liked the way he knew the story; I liked how the other person supported him," she says, "and l wish each person would have contributed more equally."

She also uses a standard performance rubric with five levels of proficiency, from Well Done to Less than Competent. "Going from frozen to being able to get up with a partner and participate," Danzig notes, "may score low on the rubric but it's high performance for the individual. Because often you're breaking through a history of non-participation."

"People who don't get opportunities to participate are the ones who spend most of their time in front of a computer or a television set," according to Danzig. But she's quick to point out that "students aren't going to learn how to participate without support." Danzig cites the example of giving students crayons and a piece of white paper and saying 'Draw anything you like.' "They will just sit there," she says. "You need to give them clues, like asking 'Who?' or 'What?' They need something to work with. If given parameters within which to work, students have the freedom to be creative."

"The process sets the parameters," says Danzig. "Students take it with them to create and present their project with confidence. Teachers take it with them for the rest of their career."

Questions, Journals, Observations

To uncover students' prior knowledge, teacher Elizabeth Williams uses a time-honored approach. At the beginning of a theme or unit, a "KWL Chart" prompts students to answer questions such as 'What do you know?' 'What would you like to learn?' "The children's answers let me know what they know," Williams says. "Then at the end of the unit, they write about all the things they learned."

When she first asked second-graders in her class about media literacy, "No one had any idea what it was," Williams says. "It was plastic. Students didn't relate. But when I tied it into communication, they could understand."

Each member of her class keeps a Media Journal that they write in after each media-related activity. Charting their ideas at the beginning and at the end has shown "a change in their awareness. Now they know it's OK to have and express their opinion," says Williams. "Children are very curious; they ask questions all day long," she says. So the inquiry-based media literacy activities, "help them open up and feel good about questioning and discussing what they're curious about."

Williams particularly noticed the benefits of this when the children were working on their animations about violence prevention. "When we started talking about the violence happening in the neighborhood and they began sharing how they felt about it, it helped them feel better about dealing with it on an everyday basis."

Expectations and Results

"I'm not interested in developing art critics," says visual artist/educator Alvaro Asturias, "but I want students to be articulate about what they make and what they see."

Part of a good lesson "is being really clear, setting up aesthetic valuing from the beginning," Asturias says. "There is clear criteria about what I expect students to do. I know it and they know it, and we can talk about it in those terms. We talk about whether we achieved our goals or not, where we may have fallen short, or how much farther we went than expected," he says. "And I try to get teachers to continue that dialogue after I'm gone."

He points out the importance of exhibiting the visual artifacts children create. In addition to the end-of-the-year Showcase of the student animation projects, SMARTArt artifacts "are shared in school bulletin boards or displayed in a classroom and other classes are invited in to see them," he says.

Reflecting on the value to students Asturias observes, "a lot of kids really look at media differently afterward. They continually mention it to their teachers. It's not something they forget."

Charts and Thinking Maps

Bar graphs, thinking maps, and Venn diagrams are some of the tools teacher Lorena Mendoza uses to evaluate her students' prior knowledge and progress in building new skills. Most of the second-graders in her Spanish/English "waiver" class are recent immigrants to the United States, so visual aids help them organize their thinking and remember what they learn.

"The school has standardized rubrics but we've not developed much assessment with media; there's not enough of it," says Mendoza. So she uses other tools. To get kids to consider the amount of time they spend with media, Mendoza's students make a bar graph, charting the number of hours they watch TV each day for a week. In studying the various types of media, her second-graders use "thinking maps," a graphic organizer that helps them organize their thinking about the relationships of books, radios, CDs and other media. Later they cut out magazine images of the various types of media they identified on their thinking maps and write a paragraph about them. "In the discussions and the writing activities I can see whether they've captured the essence of what we're talking about," says Mendoza.

The interlocking circles of Venn diagrams help her students organize and compare various adjectives they write down to describe two different images of Governor Arnold Schwarzenegger. One image is on the cover of a Muscle & Fitness magazine, the other is an Esquire cover published prior to the California gubernatorial election. "The children's responses, such as 'He looks smart when wearing a suit,' help me learn what their values are," Mendoza relates. (You can download this lesson plan at: http://www.medialit.org/pdf/CML_DeconstructionMags.pdf)

When her class was investigating advertising and techniques of persuasion, students cut out print ads and separated them into piles to distinguish between different persuasive techniques, then made posters with multiple examples of each technique. Displaying the posters in the classroom provides instant reference for subsequent discussions. A related homework assignment involves students in watching a TV commercial with their parents and writing answers to three media literacy questions. Mendoza designed the homework worksheet "to find out if they understood what we were talking about." Throughout the year, each member of her class compiles their media work in a print portfolio collection entitled, "My Media Literacy Journal."

Electronic Portfolios and Artifacts

At the end of the school year, each of Steve Schullo's third-graders takes home an electronic portfolio, a CD-ROM containing work they created that year.

One of the photos in the 2003 CD is of a visual art project that integrates art and media literacy with the Open Court Reading theme of urban wildlife. To elicit prior knowledge, Schullo asks questions and the children discuss what they know. Students agree that most media representations show only violent behaviors of wolves, bats, and other urban wildlife.

Accessing various forms of media including the Internet to conduct their own research, students write reports about the different animals, including what they do that is beneficial. Each member of the class contributes to creating a colorful poster with pictures and descriptions of city-dwelling wildlife. Schullo observes that throughout the project the children stayed motivated. "They focused in and stayed on task," he says. "Some of them really got into it and worked hard."

This and other SMARTArt activities have broadened Schullo's appreciation of "the value of art. In contributing to creating something it becomes yours; you own it; you work hard on it," he says. "That in itself transfers over into other activities."

< < Partnership  |  Project Evaluation > >