SMAPP Project

Enhancing Mathematics Assessment with Validated Resources

• Singapore Mathematics Assessment and Pedagogy Project • Sep 2008 – Dec 2012 • RD & I (Research, Development and Innovation) project; new assessment • Funding: Centre for Research in Pedagogy and Practice (CRPP), National Institute of Education, Nanyang Technological University

WONG Khoon Yoong (A/P)

An Institute of

Mathematics & Mathematics Education National Institute of Education Nanyang Technological University [email protected] http://math.nie.edu.sg/kywong Overview

1

Team Members

Overview

Partial participation Main study (2011)

Schools

Teachers

Students

59

Classes (S1 E) 90

9 I: 4 C: 4

I: 16 C: 14

I: 11 C: 14

I: 406 C: 527

3074

• 2 extended tasks • Everyday maths • Attitude

S1 E: Secondary 1 (Express); Grade 7; average and mixed ability 3

Overview

Wong (APEAC, 12/9/13)

Main Publication

4

Disclaimer The views expressed at this lecture are those of the author’s and do not necessarily represent the views of the • Centre for Research in Pedagogy and Practice (CRPP) • National Institute of Education (NIE) • Singapore Ministry of Education (MOE)

• Not for sale; given free to all Singapore secondary schools • E-book (reading): http://hdl.handle.net/1049 7/11492

Wong (APEAC, 12/9/13)

2

Participants

1. Wong Khoon Yoong (PI, from Nov 2010) 2. Zhao Dongsheng (Co-PI) 3. Cheang Wai Kwong 4. Fan Lianghuo (PI, 2008 to Oct 2010) 5. Lee Peng Yee Strong participation of 6. Quek Khiok Seng Mathematicians; 7. So Hyo Jeong ensure mathematical rigour 8. Teo Beng Chong 9. Teo Kok Ming 10. Yen Yeen Peng (CPDD, MOE) 11. Yvonne Ng Qiu Ting (PM) & Others who left project Wong (APEAC, 12/9/13)

Overview

Wong (APEAC, 12/9/13)

Overview

5

Wong (APEAC, 12/9/13)

Overview

6

Overview

Real-Life Contexts

1. Maths disciplinary tasks 2. IT-based assessment system 3. Attitudes toward learning maths

Wong (APEAC, 12/9/13)

• Singapore Maths Curriculum (2013): stronger emphasis on “solve real-world problems”; “connect mathematics that they have learnt to the real world” • Help students gain knowledge about the world, while honing their maths skills • Aligned with international trends

Overview

7

Wong (APEAC, 12/9/13)

OECD: Mathematical Literacy • An individual’s capacity • to identify and understand the role that mathematics plays in the world, • to make well-founded judgements and • to use and engage with mathematics in ways that • meet the needs of that individual’s life as a constructive, concerned and reflective citizen

Wong (APEAC, 12/9/13)

Overview

10

Wong (APEAC, 12/9/13)

Overview

11

Extended Tasks: Principles a) b) c) d)

Links to real life scenario Real and relevant data Curriculum connection Multiple competencies and content knowledge assessment e) Experience enriching f) Scaled levels of difficulties

a) 11 extended tasks, multiple competencies (computation, reasoning, explanation), mathematically rigorous; take about one hour to complete; learning experiences; delivered through IT system b) 10 short paper-pencil problems (Everyday Maths Items), similar to PISA; exercises or tests Overview

9

Design Framework (2009)

Two Types of SMAPP Tasks

Wong (APEAC, 12/9/13)

Overview

•

12

Zhao, D.S., Cheang, W. K., Teo, K. M., & Lee, P. Y. (2011). Some principles and guidelines for designing mathematical disciplinary tasks for Singapore schools. In J. Clark, B. Kissane, J. Mousley, T. Spencer & S. Thorton (Eds.), Mathematics: Traditions and (new) practices: Proceedings of the AAMT-MERGA conference (pp. 1107-1115). Adelaide: Australian Association of Mathematics Teachers.

Wong (APEAC, 12/9/13)

Overview

13

11 Extended Tasks: IT based No. 1 2 3 4 5 6 7 8 9* 10* 11*

Task Titles Paper Recycling Red or Black? Malacca Trip Water Water Water! Up Down Up Down!! Singapore Got Talent Money Money Money Three Rockstars on the Wall When to Retire? Which Mobile Plan? Outing to the Zoo

2 Extended Tasks: Findings Tasks Paper Recycling Red or Black?

Topics Arithmetic Arithmetic, Algebra Rate, Speed, Algebra, Inequalities Mensuration, Statistics Statistics Geometry Linear Graphs Angles, Parallel Lines Numbers, Algebra Statistics, Percentages Data handling, Algebra

* Based on teachers’ contributions in November 2010 Overview

Wong (APEAC, 12/9/13)

14

Decibel Question: Try It

Overall % 20.8 63% (364) 16.2 52% (383)

• •

Successful with routine questions Weak in unfamiliar units, multi-step questions, giving reasons, explain own ideas

•

Cheang, Teo, Zhao, http://repository.nie.edu.sg/jspui/handle/10497/8158

Wong (APEAC, 12/9/13)

Overview

16

Overview

Overview

18

a) Correct answer with working (27%); Correct answer, no working (10%); Wrong proportional reasoning (4%) b) Full mark (21%); Partial (24.5 %); Wrong (45%) Popular method: stepwise decrease • 39%: relevant to daily life (most relevant and challenging) 17

Easiest: Sale (72%)

Wong (APEAC, 12/9/13)

Everyday Maths Items: Admin

A particular item costs $6. Shop X advertises, “buy four items for the price of three”. (a) How much does a customer have to pay for 4 such items in shop X? (0.98/1) (b) What is the percentage discount for the customer who buys 4 such items from shop X? (1.61/2) (c) Another shop Y offers, “buy three at the regular price and pay 50% for the fourth item.” Shop Z offers a voucher of 10% on the total amount paid. Your parents wish to buy 4 such items. Out of these 3 shops (X, Y, and Z), which shop gives the best deal? (3.52/5) (d) Other than the amount you have to pay, what other reasons would you give to your parents to support your choice? (0.34/1) Wong (APEAC, 12/9/13)

Male 20.5 (86) 16.5 (90)

Decibel Question: Results

The loudness of sound is measured in decibels (dB). Noise from heavy traffic is about 85 dB and this can cause hearing damage if one is exposed to it for 8 hours or more. For every 3 dB over 85 dB, the exposure time before damage occurs is decreased by half. (a) If the noise is 88 dB, what is the exposure time before damage occurs? (b) John likes to listen to his music using ear-plugs at high volume of 100 dB. How long could he do this before damage occurs?

Wong (APEAC, 12/9/13)

Max Female 33 20.8 (273) 31 16.4 (287)

Overview

• 5 items administered in March 2011 as “pre” test • 5 items in Sept 2011 as “post” test • But not parallel items • Pre-post labels for identification only

19

Wong (APEAC, 12/9/13)

Overview

20

Everyday Maths Items: Findings n 814 S1E

Question 1 2

Re-arranged in order of facility

3 4 5 6 7 8 9* 10*

Context (Topic) Sale (Percentage, discount) Tourism (Interpretation of table and pie chart, rate) Kool Biscuits: Reduced fat (Percentage) Population (Interpretation of table, significant figures, rate) Types of fires (Interpretation of table, percentage change) Decibels (Four operations, rate) Earthquake (Powers of 2) Mobile plan (Rate, line graph) Hokkien char mee (Interpretation of chart, percentage) Math Olympiad (Line graph, bar graph, misuse of graphs)

Facility Index

Group

71.7 64.9

Intervention

61.8 57.4

Comparison

54.6

Overall

40.8

21

38.0 (12.1) 40.6 (12.3) 39.5 (12.3)

284

530

327 487

814

Wong (APEAC, 12/9/13)

Overview

22

• International trend: Develop IT-based assessment of construct-response items; efficiency and commercial, technical issues • Singapore: Math Explorer can grade multi-line steps; defunct • SMAPP: IT-based assessment based on pedagogy, assessment for learning 23

Assessment for Learning (AfL)

Wong (APEAC, 12/9/13)

Overview

25

IT Platform with AfL Features 1. Deliver tasks (extended) 2. Capture answers; closed & open questions 3. Automatic & Semiautomatic marking 4. Customisable feedback 5. Student responses to online feedback (*) 6. Reports by questions, students, class 7. Follow up activities (*)

• Provide informative feedback to students to enhance their performance (Sadler, Black, Wiliam, Hattie, etc.)

Feedback

37.9 (12.4) 42.6 (12.5) 40.5 (12.6)

IT-Based Assessment: Trends

Overview

Marking (Evidence)

38.0 (11.4) 37.6 (11.4) 37.7 (11.4)

Sample Size

22.2

Lack of experience with real-life contexts Inappropriate use of maths skills Amount of reading Maths beyond S1 level; more suitable for upper secondary • Student perceptions: neutral in terms of interest, relevance; a bit challenging, less confident

Teaching

Overall

26.1

• • • •

Assessment

Female

Sample Size

35.5

Everyday Maths Items: Observations

Wong (APEAC, 12/9/13)

Male

36.6

Overview

Wong (APEAC, 12/9/13)

Everyday Maths Items: Groups

(*) Not successful Wong (APEAC, 12/9/13)

Overview

26

Wong (APEAC, 12/9/13)

Overview

27

Delivery & Capture

Marking and Feedback

Semi-automatic marking

Capture of student answers

• Online calculator • Entry of mathematical symbols and expressions • Simple animations

Correct answer Question

Wong (APEAC, 12/9/13)

Overview

28

Wong (APEAC, 12/9/13)

Automatic & Semi-automatic Marking

Overview

30

Wong (APEAC, 12/9/13)

Overview

31

Students’ Responses to Feedback

• Teachers can enter own feedback comments • Comments can be saved and shared within school

Overview

29

• Every question is tagged to several feedback comments • Correct answer: Gives general praise and reiterates the correct procedure • Wrong or incomplete answer: Hints on how to proceed further

Teacher Can Add Feedback

Wong (APEAC, 12/9/13)

Overview

Customisable Feedback

• Closed questions: Automatic marking o Expedite marking; alleviate teacher marking workload o Consistency in marking • Open questions, workings: Semi-automatic marking o Teacher selects score from given scheme o Recommend teachers mark by questions to note different student solutions o Expedite marking; alleviate teacher marking workload o Consistency in marking Wong (APEAC, 12/9/13)

Customisable feedback

• After marking and entering feedback: Students asked to reflect on teacher feedback • 3 choices of student responses; encourage metacognition and reflection • Teachers to follow up

32

Wong (APEAC, 12/9/13)

Overview

33

Report System

Paper Recycling: Teacher • Responses from Teachers (n = 10).

• By students, questions, class; mean and frequency • Can be downloaded into EXCEL format

Wong (APEAC, 12/9/13)

Questions Appreciate the connections of mathematics with real life situations. It is easy to add my own feedback to the system. By focussing on open-ended questions, I know more about my students’ thinking, e.g., different methods used. I prefer this system to manual marking.

Overview

34

4.20

0.92

3.75

1.04

3.44

1.13

3.33

1.12

Overview

35

Paper Recycling: Student Interviews

• Online survey; n = 99 (2011) • 5-point Likert Scale

• Some preferred to work on paper rather than online; need to familiarise students with IT-

Questions Mean I find the scenario provided in the task realistic. 3.70 I prefer to do this task on paper rather than doing 3.49 it using the IT platform. I have no difficulty following the instructions 3.43 given. I find the scenario provided in the task interesting. 3.34 I gained some new mathematical 3.12 knowledge/skills by working on the task. The IT platform is helpful for me to do this task. 3.02

based assessment; future trend?

SD 1.00

• Some students aware of benefits of IT: lessen time, easy to search for information, do not waste paper, can erase answer easily • Knowledge about recycling: Ah…save earth, like save trees then the trees can produce oxygen for us and that it can reverse the effects of global warming

1.18 1.22 1.14 1.04 1.55

Overview

36

37

ALMQ: Development •

• Curriculum: Desirable outcome; one factor for effective problem solving (Singapore Maths Curriculum) • Learning : Positive, significant but moderate correlate (about .3) with achievement; US National Mathematics Advisory Panel (2008): “Children’s goals and beliefs about learning are related to their mathematics performance” (p. xx) • Teaching : Ways to improve attitudes • Research: Not necessarily causal with achievement; design and validation of instruments; define construct Overview

Overview

Wong (APEAC, 12/9/13)

Attitudes Toward Learning Maths

Wong (APEAC, 12/9/13)

SD

Wong (APEAC, 12/9/13)

Paper Recycling: Students

Wong (APEAC, 12/9/13)

Mean

• •

2010: 57 items. 9-point Likert; greater variation, but students may not be able to make fine distinctions Exploratory factor analysis and confirmatory factor analysis 2011: 24 items; 6 constructs, 4 items per construct: a) b) c) d) e) f)

• 39

Check solutions Confidence Enjoyment Use of IT Multiple solutions Usefulness

Cronbach’s alphas acceptable, 0.63 to 0.90

Wong (APEAC, 12/9/13)

Overview

40

ALMQ: Your Score?

ALMQ: Results

• 9 = Agree totally; 1 = Disagree totally •

• Less positive over time from March to Oct 2011

Negative items: 10 – your point

Group

Scales

Items

Check solutions

1, 7, 13*, 19

Confidence

Intervention

Sept Comparison

2, 8, 14, 20*

Enjoyment

4*, 10, 16, 22

Multiple solutions

5*, 11, 17, 23

Usefulness

6, 12, 18*, 24

Overall

Sample Size 42

Everyday Maths vs. ALMQ (Post) Attitude Overall Check solutions Confidence Enjoyment Use of IT Multiple solutions Usefulness • •

Everyday (n 800) .290 .281 .272 .231 .032 .178 .273

Paper Red? (n 350) (n 350) .296 .299 .266 .274 .260 .272 .194 .224 .103 .107 .193 .210 .291 .232

Overview

Wong (APEAC, 12/9/13)

March Sept

Male 6.37 (1.08) 5.78 (1.22) 6.28 (1.23) 5.78 (1.26) 6.31 (1.18) 5.78 (1.24) 287 296

Female 6.02 (1.04) 5.65 (1.07) 6.12 (1.00) 5.54 (1.11) 6.07 (1.02) 5.59 (1.09) 561 551

Overall 6.11 (1.06) 5.69 (1.11) 6.18 (1.10) 5.64 (1.18) 6.15 (1.08) 5.66 (1.15)

Sample Size 365 352 483 495

848 847 Overview

43

Teacher Professional Development • Important part of project 1. 8 teacher workshops for all participating schools 2. 3 mini workshops for individual schools 3. School meetings to discuss findings • Use of SMAPP IT system • Assessment literacy • Task design; teachers created 3 extended tasks • Data analysis and interpret findings

“Expected” values for 4 scales Not familiar (IT, Multiple solutions), low correlations

Wong (APEAC, 12/9/13)

March Sept

Overview

Wong (APEAC, 12/9/13)

March Sept

3, 9*, 15, 21

Use of IT

March

44

Wong (APEAC, 12/9/13)

Overview

46

Number of Trees to be saved 1. (a) The length and breadth of an A0 paper are 0.841 m and 1.189 m respectively. Which of the following expressions is

(

)

[1]

correct to obtain the area of an A0 paper? (1) (0.841 + 1.189) × 2 (2) 0.841 × 1.189 (3) 1.189 ÷ 0.841 (4) 1.189 – 0.841 (b) An A1 paper is obtained by folding an A0 paper into two equal halves lengthwise.

Similarly, an A2 paper is obtained by folding an A1 paper into two equal halves lengthwise. Table 1 shows the estimated area of the “A” series paper size and the number of sheets of paper that can be obtained from an A0 paper.

[4]

Fill in the blanks with the correct answers.

Paper Recycling : Arithmetic

Estimated Area (m2)

“A” series paper size

P a g e | 11

Table 1

Paper Size

1

Number of sheets of paper that can be obtained from an A0 paper

2

1 0.5 4

A0 A1 0.25 8

0.015625 128

0.0625

A2 A3 A4

A6 0.0078125

256

32

A7 0.00390625

A5

A8

Paper Recycling : Arithmetic

ii. how much oxygen could have been

P a g e | 12

(e) There are 1200 students in Shamila’s

[1]

produced by those 150 trees per year?

dioxide

for

every

kilometre

travelled (http://www.carpages.co.uk/co2). If

carbon

(c) On average, a car emits about 0.16 kg of

it emits the same amount of carbon

answer

as

a

whole

number.

[2]

are

[1]

litres

of

___________ litres of oil.

__________

water

[2]

[2]

and

every tonne of paper recycled can save

A wall poster at Changi Airport states that

from the poster below.

(a) Fill in the blanks based on the information

3.

Save Water!

travelled in km?

dioxide as found in (b)(i), how far has it

how many trees can be saved each year?

(70 g/m2 instead of 80 g/m2), how many

If the school uses a lighter type of paper

[3]

(Recall that 1 tonne of paper § 17 trees.)

If the school uses only recycled paper,

(for notes, test and exam papers, etc.).

uses about 3 reams of A4 paper per year

school. It is estimated that each student

(f)

your

more trees can be saved each year? Give

Benefits of using Recycled paper

Shamila wants to make a poster to present

some of the facts and figures on the benefits of

recycling paper and using recycled paper.

year?

been absorbed by those 150 trees per

i. how much carbon dioxide could have

only recycled paper,

school saves 150 trees per year by using

produce 10 kg of oxygen each year. If the

absorb 16 kg of carbon dioxide and

(b) It is estimated that each mature tree can

down 2 benefits of recycling paper. [2]

paper” on a search engine, find and write

(a) By searching for “advantages of recycling

2.

problems.

approached to help her solve the rest of the

You

Besides the number of trees that can be saved,

of

paper.

she plans to include other information on the

[2] contains ream

recycled

A5 paper, find its estimated area in

paper one

to

(c) i. Based on your answer to the area of an

of of

converts

benefits to the environment if the school [1]

[1] Looking at the packaging of the printing

paper.

that can be obtained from a sheet of A2

ii. Find the number of sheets of A5 paper

(1 m2 = 10 000 cm2).

square centimetres.

(d)

paper, Shamila sees “80 g/m2” (80 grams per square metre) printed on it.

ream

i. What is the mass of one sheet of A4 paper? ii. One

mass

500 sheets of printing paper. What is the total

A4 paper (excluding the mass of the packaging)? Give your answer in kg. [2]

Class

Name

Student Questionnaire Index Number

Singapore Mathematics Assessment and Pedagogy Project

P a g e | 162 Attitudes toward Learning Mathematics Questionnaire

School

Date

Gender

Dear students: The purpose of this survey is to find out how Secondary One students think and feel about mathematics. Your responses will help us understand students like you better. All responses will be kept strictly confidential. Please answer ALL the questions as best as you can. For each question, please tick ( ) your answer. There is no correct or wrong answer to each question. Thank you for your cooperation.

P a g e | 163 Attitudes toward Learning Mathematics Questionnaire

I often figure out different ways to solve 11. mathematics problems.

O

O

1

O

O

O

2

O

O

O

3

O

O

O

4

O

O

O

5

O

O

O

6

O

O

O

7

O

O

O

8

O

O

O

9

Agree Totally

I think mathematics is useful in solving real world problems.

O

Disagree Totally

12.

Once I have worked out an answer to a problem, I do not check my answer.

Attitudes

13.

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

14. I find mathematics easy.

Overall, I have good feelings about mathematics.

O

O

O

O

O

O

O

O

O

23.

22.

After I have solved a problem, I will look for other methods to solve it.

Mathematics software (e.g., graphing) helps me to learn mathematics.

I am not good at giving reasons in mathematics.

After I have solved a problem, I will ask 19. myself if the answer makes sense to the given problem.

18. I think mathematics is useful only for tests.

I try to understand the different solutions given by my classmates.

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

15.

O

O

O

O

4 - Disagree a little 8 - Agree a lot

O

O

O

3 - Disagree 7 - Agree

IT (Information Technology) has been 16. helpful to my mathematics learning.

O

O

1. I am good at using mathematics to solve real-life problems.

When I know I have made a mistake in solving a problem, I will try to find out why. O

O

1

O

O

2

O

O

3

O

O

4

O

O

5

O

O

6

O

O

7

O

O

8

O

O

9

O

O

O

Mathematics helps me to understand 24. reports and advertisements about prices, sale, percentages etc.

O

O

O

O

O

O

O

O

O

20.

2.

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

O

6. Mathematics is important.

O

O

O

O

After I have solved a problem, I will go 7. through the solution again and check if I have made any mistakes.

O

O

3. I enjoy doing mathematics.

I am confident in solving mathematics 8. problems.

O

Solving mathematics problems is fun to 21. me.

17.

O

O

Please take note of the following scale: 1 - Disagree totally 2 - Disagree a lot 5 - Neither Disagree nor agree 6 - Agree a little 9 - Agree totally

O

O

Agree Totally

I do not like to use the computer to learn mathematics. O

Disagree Totally

4. I do not like to think of other ways to solve the same problem.

Attitudes

5.

9. I find mathematics boring.

O

I can learn mathematics from playing computer games.

O

10.