WEBVTT

METADATA
Video-Count: 1
Video-1: youtube.com/watch?v=slPcsD-_fMA

Part: 1

1
00:00:02.159 --> 00:00:18.160
We are recording. Ready to go. Michael, >> thanks Josh. Good afternoon everyone. I'm Michael Derline, deputy executive director of policy and research with the state board. Um I have just a few comments at the very beginning of this and then you um sorry Josh Chrissy might

2
00:00:18.160 --> 00:00:32.719
need a little support. She just asked what the password was. So if somebody could reach out to her, I'd appreciate it. >> Um just a few comments in the beginning. Um otherwise I'll be quiet. We'll turn it over to a few other folks who are joining us. um for this important discussion. Just a reminder reminder,

3
00:00:32.719 --> 00:00:49.440
member learning sessions are optional sessions for our board members. Really intended to dive into specific topics of interest. Uh we record these sessions so that if members are not available, we can send it to them after the fact. Um,

4
00:00:49.440 --> 00:01:06.000
but we've had quite a few interesting conversations and this one stems from a recommendation that came before the board back in May from the Department of Education to approve alternative assessment pathways for candidates who are unable to but are close to passing

5
00:01:06.000 --> 00:01:21.200
content assessments. Um, so we wanted to take this time to really dig deep into that. Uh, give our members a chance to ask any questions they may have of the testing vendors who have joined us today. Um, and I will leave it at that and turn it over to Taylor Reed from the

6
00:01:21.200 --> 00:01:38.320
Department of Education to introduce our guests and kick off the conversation. >> Good afternoon, uh, Taylor Reed. I serve as a senior director of educator relationship with the Department of Education. Very excited and thankful to be here with you all this afternoon. Um,

7
00:01:38.320 --> 00:01:54.240
as Michael said, this uh learning session is around the alternative assessment options or assessment flexibilities as we've been calling them as a whole. The kind of origination of this item actually came from the lensure

8
00:01:54.240 --> 00:02:11.120
working group um that Michael uh leads uh with SBE and of course we participate in. Um and this was a recommendation from the relationship working group that came about actually some time ago I believe sometime in about a year or so

9
00:02:11.120 --> 00:02:26.480
ago and to ensure that we were thoughtfully going through the operations and implementation of things. We first wanted to take before the board options for expanded assessments uh which has

10
00:02:26.480 --> 00:02:43.360
recently been adopted. So we now offer a suite of both um ETS assessments and Pearson assessments. And then from there we've now brought this item on first reading before the board and there were some very thoughtful um questions and so we wanted to bring everyone together to

11
00:02:43.360 --> 00:02:59.760
have this learning session and then from here we are anticipated to bring this item back on final reading in November before the board. So did just want to make that clarification that it will go in November before the board for final read. Um, and with that, I would like to

12
00:02:59.760 --> 00:03:22.720
turn it over to my friend Lisa at ETSs to introduce herself and then um Matt from Pearson um as well. So, Lisa, if you want to start us off u with your introductions, then we'll go to Matt. And Lisa, I think you might be muted.

13
00:03:22.720 --> 00:03:39.519
I was sorry. Are you seeing the right screen? I think I have you. Okay. because it has a red box around so I got a little nervous. Hi, my name is Lisa Cologne Durham. I'm a director of educational partnerships uh for ETSs. Thank you, Taylor, for that um introduction. Um I do bring with me

14
00:03:39.519 --> 00:03:54.879
about 28 years in education. I actually served as a certification director for the state of Idaho. I also served as a middle school mathematics teacher, an elementary principal, and I actually did um served one year as a professional development director for school boards

15
00:03:54.879 --> 00:04:11.439
across the state of Idaho. That was a great experience. So, um I'm really excited to bring some of the work that we've done um at ETSs here on this innovation and flexibility. So, I am going to introduce two paths um or two innovations and flexibility to you

16
00:04:11.439 --> 00:04:28.639
today. So, please stop me if you have any questions along the way. I'm just really excited about to share this information. Um we'll just take a second to talk about ETSs. Um you most all all of you most likely know the ETSs is a global nonprofit uh leader in assessment

17
00:04:28.639 --> 00:04:45.360
and research. Uh we've served over 75 years and been at the forefront in developing assessments like practis um but we've also done GRE and TOEFL and you most likely know PRAIS as that teacher lensure assessment uh but we are really expanding um our offerings

18
00:04:45.360 --> 00:05:01.759
comprehensive researchbased solutions uh really to design to support educators all along the way and one of them that I'm very passionate about as many of you know or have heard is Prolethica which is our professional development series on ethics training uh scenario-based

19
00:05:01.759 --> 00:05:16.240
learning aligned to the model code of ethics for educators. So every time I can talk about that I do and I didn't pass up an opportunity. So but let's get into what you're really here for today and that is test innovation and flexibility. Um I'm really going to talk

20
00:05:16.240 --> 00:05:31.840
about two of them. practice bridge uh which is really that targeted learning modules for candidates who narrowly miss passing. Um rather than sitting for a full retake of the assessment, they work for uh through a professional

21
00:05:31.840 --> 00:05:47.039
development module that's focused on that individual test taker's area of greatest difficulty. The other flexible option um uh that we have worked on is what we're calling practice steps and that allows

22
00:05:47.039 --> 00:06:03.199
candidates options to how they even start or take the test. Uh they can either take it one section at a time um or they can retake just the section that they didn't pass. Um and STEPS is being introduced uh with our elementary

23
00:06:03.199 --> 00:06:23.199
education fundamentals. But let's talk about uh the bridge. So what is practice bridge? Um as I introduced before, it's specifically for candidates who narrowly miss. And it's really important uh that it is about one

24
00:06:23.199 --> 00:06:38.319
standard error measurement. So Tennessee sets the cut score and if those candidates score within one standard error measurement of Tennessee's cut score if Tennessee adopts this um then they would be eligible for bridge. It is

25
00:06:38.319 --> 00:06:56.080
based on the candidate score and we align a module specific to that candidates's area of greatest difficulty. So, we take a look at our test blueprint. We look at the subcategories. We look at that and it's really aligned to that specific text

26
00:06:56.080 --> 00:07:13.199
taker's area of greatest need. It includes content overview. It includes knowledge checks, uh, practice items to really affirm uh, their understanding of the material. Why states have really liked this is

27
00:07:13.199 --> 00:07:28.960
it's an accelerated path. um they if if the test taker doesn't pass the test, they have to wait 28 days prior to taking or retaking the test. If they qualified for bridge, they don't have to wait that time. They can start right

28
00:07:28.960 --> 00:07:46.080
away and purchase um that practice bridge module. So, it's that reduced time uh to retake. Um again, we talk about the fact that we align that area of greatest need to the module that's going to be aligned to that candidate. Um, it's seamless reporting. We do all

29
00:07:46.080 --> 00:08:04.479
the back work on our end. We send the information to the candidate. There's no nothing that the department or the board needs to do on this. Once it's implemented, it's all done uh with our systems. As a former math teacher, I love uh

30
00:08:04.479 --> 00:08:20.400
pictures and models. And so, this is just a flowchart of how that works. So, if I'm a test taker and I take the practice test, um, I score greater than one standard error of measurement, um, then I need to retake the test. I need

31
00:08:20.400 --> 00:08:37.039
to study, maybe focus, look at the score report, focus on my area of greatest need, um, and then I'll just retake that test. However, if I narrowly missed, meaning I scored within one standard error of measurement, um I have this bridge option, which again includes

32
00:08:37.039 --> 00:08:54.560
content overview, which I affectionately call retach, right? It's the ability to retach the content. Um they have practice items with annotations. Um and then there is a knowledge check. We have included a lot of tests already

33
00:08:54.560 --> 00:09:11.279
uh for this bridge eligibility. Um, and the price, we get that question a lot, is $50 uh for the module. The other thing I will point out is states can adopt a three-year retroactive. Um, this was really something that we got from our pilot

34
00:09:11.279 --> 00:09:26.640
states when we piloted this a couple years ago. wanted the ability to go back three years and say anyone that may have qualified for the bridge within those three years, we're going to give it back um or go back three years so that those

35
00:09:26.640 --> 00:09:45.760
candidates, you know, you don't miss those, oh, I just tested and now you implemented it. So, it really was this um um ability for us to go back and help those candidates, especially those that just tested. So how it works uh we actually notify

36
00:09:45.760 --> 00:10:01.440
the candidate um as soon as their scores are received uh that the practice bridge is an option for them. It also shows in their account. Once they purchase the module they have six months from the date of purchase to complete it. Um it's

37
00:10:01.440 --> 00:10:17.600
asynchronous which means they can do it on their own time. Um they can actually use do it on their phone. Um, when we know how many of our our our younger I hate to say younger because I don't want to think I'm that old, but our younger generation likes to use uh their their

38
00:10:17.600 --> 00:10:34.880
smartphones and things. Um, they do get a certificate of completion and we actually resend score reports uh to all of those entities that received the original score. For reporting purposes, we get this

39
00:10:34.880 --> 00:10:51.760
question a lot. What about title two? Um what about um our data? We actually will showcase this as what we call an alt pass. This allows the state and educator preparation programs to really look at the data and see how many of their

40
00:10:51.760 --> 00:11:08.800
candidates may be using this alt pass. The state can take a look at it. Um you can see the section on the left. Uh this image is of an actual score report for a candidate where the one on the right is uh data that would be seen at an institution level. So like one of your

41
00:11:08.800 --> 00:11:24.399
colleges or universities and also at the state level. Um so you can see here how many have passed the bridge um or passed this test under the bridge. So that data is there and is disagregated so that um uh anyone who wants to use the data for

42
00:11:24.399 --> 00:11:39.839
just to review it can look at it in that fashion. And again, for title two purposes, um an individual who uses bridge um would be a next attempt pass. So if they were trying that test and

43
00:11:39.839 --> 00:11:58.240
they qualified for bridge on the first attempt, when they did bridge, it would be considered a second attempt pass. Uh the next couple of slides I'll just go through because I think Taylor has this um presentation in the show, but I just wanted to showcase all of the

44
00:11:58.240 --> 00:12:14.720
different options or tests that we have available for Bridge and I highlighted the bold options that are currently adopted by Tennessee. So we have the elementary subjects, we have core, uh teaching, reading, elementary, uh the principles of learning and teaching,

45
00:12:14.720 --> 00:12:30.560
English language arts, special education, um most of the middle school areas, uh pair pathways, mathematics, um and we are constantly adding additional modules. I know there was some questions about what does those modules look like. So

46
00:12:30.560 --> 00:12:47.200
I'm just going to take a few minutes uh to talk about uh the modules and how they were created. Um they are designed to be high quality. They are direct alignment to the practice content skills and cognitive demands of the blueprint

47
00:12:47.200 --> 00:13:02.560
that was adopted by the state. It's selected using practice research. We use candidate performance data, subject matter expert review to ensure relevance to beginning educator practice. The module assignments, so those are the

48
00:13:02.560 --> 00:13:18.959
assignments part are tailored using the datadriven process that identifies and targets each candidates's areas for growth. So you can be rest assured that the what they're doing is completely and totally aligned to the test blueprint and aligned to that specific

49
00:13:18.959 --> 00:13:35.760
candidates's area of greatest need. Um this particular um uh slide showcases that if you were doing the elementary and I qualified for bridge for mathematics, there are three different modules that I could be assigned, but I

50
00:13:35.760 --> 00:13:52.560
would only be assigned one of those three. I'm not going to have all three. And I'm going to be assigned the one that I had the greatest area of need. So for mathematics, that could have been algebraic thinking if I didn't pass within one standard error of measurement. uh for another candidate

51
00:13:52.560 --> 00:14:09.279
that might be the numbers and operations. So um one thing I want to make sure is that they're only going to be assigned one module uh per each of these and it would be in the subcategories and identified at their um as their area of greatest need

52
00:14:09.279 --> 00:14:27.440
and they we also provide accommodations uh closed captioning etc on all the modules. So the way it land uh is is um completed is again there's a content overview then they're given practice items um that

53
00:14:27.440 --> 00:14:43.360
includes um that are aligned to the content it includes feedbacks we have glossing um I'll show a little bit of that and then we have hint and then they go to the knowledge check and this is where uh the accountability is um the

54
00:14:43.360 --> 00:14:59.920
first time they take the knowledge check they have to achieve a 75 to 80% in order to get the certificate of completion. If they don't pass it, they have to go back and they can go through the content module again, the overview,

55
00:14:59.920 --> 00:15:15.360
the practice items, but for any subsequent attempt, they have to get 100%. uh so the first time they take it they can get a lower threshold but for any other time that they take it after that it is a 100% threshold in order to

56
00:15:15.360 --> 00:15:33.839
complete uh the bridge module. The next couple of slides uh just showcase the learning management system that they would go through so you can give a look at what a candidate might experience. This particular one is for our physical science um section. Um, we

57
00:15:33.839 --> 00:15:49.279
typically have all of the different areas on the lefth hand side uh that the user or the candidate could go through. We give them course outcomes, learning objectives. Um, this is an example um of the learning course. This

58
00:15:49.279 --> 00:16:06.240
is a diagram of um the evaporation transpiration and so they can click any of these to highlight. So it's a very interactive experience on this content. science is not my area of expertise. This one is mathematics. And so, uh,

59
00:16:06.240 --> 00:16:22.079
this is an example of where they can hover or that's what we call glossing or hover where they can get the definition. Again, this would be all the practice items. This is not the knowledge check. This is them going over and being retaught the information. We also give

60
00:16:22.079 --> 00:16:39.920
them hints. We have some distractor analysis as well. um this is incorrect, this is correct that they can find out uh where maybe their thinking uh was misguided and then why this answer is correct. So

61
00:16:39.920 --> 00:16:57.920
again that whole idea of reteing the next couple of slides I did some data digging for Taylor as we talked about this um this right here is best attempt. So, if I was looking at the three years, so because there's the three-year retroactivity or retroactive

62
00:16:57.920 --> 00:17:15.600
option, we wanted to see um what that impact might be for Tennessee uh candidates. And as you can see here, if I look at best attempt for candidates, we have about 128 candidates that would have qualified potentially for bridge,

63
00:17:15.600 --> 00:17:32.880
which means they did score within one standard error of measurement um on their best attempt. And so you're looking at and some of these might be the same candidate, some of them might not be the same candidate, but these are just numbers um so that you can see that on best attempt. And then I also gave

64
00:17:32.880 --> 00:17:47.760
you a slide on first attempt um because that's where that big one is. You know, you had about 2,900 candidates within that three-year period. Of those 29 2,900 candidates on their first attempt,

65
00:17:47.760 --> 00:18:07.440
222 of them uh could have qualified for bridge and taken this option rather than a retake. Are there any questions about Bridge before I go on to our other innovation? I promise that won't be as long. Okay,

66
00:18:07.440 --> 00:18:21.280
>> I do have some questions. >> Yes. >> Okay. Um, so I'd love to get a copy of this PowerPoint. Also, uh, Lisa, I'm Anna Maxwell. I'm one of our state board members and a previous educator, retired

67
00:18:21.280 --> 00:18:36.160
happily. Um, so a couple of questions that I have. Do knowledge check questions change if the candidate does not meet the 75 to 85%

68
00:18:36.160 --> 00:18:53.679
pass the first time that they take um the re attempt, if you will, or will they be given those same questions just in a different order? >> They'll be given the same questions, but there's no hints. um or anything on

69
00:18:53.679 --> 00:19:11.039
those. So, they would have to go back, do some extra uh learning, and then it would be the same questions in a different order. >> Okay. And does it record the number of attempts that they go through before they achieve that 100%.

70
00:19:11.039 --> 00:19:28.000
>> I don't know if we collect that data or not. That is something I can follow up on. I know that they would not get a certificate of completion or say that they have moved on or got a pass unless they get the 100%. >> Okay. Has there been any thought given

71
00:19:28.000 --> 00:19:45.679
to having a different question bank related to the same skills being asked for that second attempt? >> Yes, there has been. Um, also I'm wondering

72
00:19:45.679 --> 00:20:01.600
do you have any performance data on states that have used the practice bridge like the teacher takes that they're you know meet the criteria for lensure they're hired and I'm just wondering if there's any longitudinal

73
00:20:01.600 --> 00:20:18.320
data that's been collected to show that the teachers that have had to take the practice bridge have become effective educators. That's a really great question. Most states do that on their own because they all have defined effective or high

74
00:20:18.320 --> 00:20:35.440
quality based on their evaluation systems and every state does that. ETS doesn't collect that data because we don't have any evaluation or student achievement data, that kind of stuff. But I do know the states that I have worked with um that have used that um

75
00:20:35.440 --> 00:20:52.000
data they are following that and at this time I've not seen any difference because I um one thing that I think is important is that it has to be within one standard error of measurement. Um and on my slides that I'll send uh to Taylor to forward to you all is that we

76
00:20:52.000 --> 00:21:07.440
do have an explanation of what standard error of measurement is and then this idea of false positives and false negatives and why one standard error of measurement was chosen. It was very specific. Typically that equates depending upon the standard error of

77
00:21:07.440 --> 00:21:23.360
measurement to potentially one to five questions missed on the actual exam. >> Okay. Thank you very much. >> You're very welcome. >> I have another question and I just really it's just a clarification.

78
00:21:23.360 --> 00:21:39.840
So for this bridge, if they fail the practice, they get to do a self-paced modules where there's that knowledge check. There's no fidelity. There's no accountability. They could let someone else take that and there would be no way

79
00:21:39.840 --> 00:21:55.280
of knowing that. Correct. >> That is possible. Yes. I mean it is there's no monitoring, there's no proctor, there's no uh no piece of that technically. Yes, they could do that. >> And what are the what's the percentage

80
00:21:55.280 --> 00:22:12.559
of teachers that do this that actually don't pass? Because I would I would think it would be really hard to not pass if you're giving them multiple infinite amounts of time to pass it. >> Correct. The only reason that they would not go uh pass a practice bridge on an

81
00:22:12.559 --> 00:22:30.960
alt pass is if they didn't finish it. You're correct. They would have to basically choose not to complete it. >> Thank you. >> You're welcome. >> One more question, Lisa. What types of accountability and fidelity checks are

82
00:22:30.960 --> 00:22:47.120
in place? I'll need you to kind of remind me. It's been a long time. that are in place when a student or candidate does go in to take a practice test. I'm going to show how old I am. Is it still sitebased or can they take it online?

83
00:22:47.120 --> 00:23:05.039
Can you briefly just give me what the current accountability measures are? >> Sure. So, um when they take a full test, we do have at home um options, but that is an online proctor. We go through a huge test security process. They have to

84
00:23:05.039 --> 00:23:20.080
have the camera. They have to showcase the entire room. There could be no windows mod those kind of stuff. They have to showcase the desk. Um, so if it's an at home, there is a huge security piece that they have to go through in order for that. And there is a human proctor on the other side of

85
00:23:20.080 --> 00:23:35.600
that the entire time that they take the test. Otherwise, they're taking it at a test center. And again, all of those checks probably what you remember where there's a proctor that's in the actual room. you have to leave all of your material in a locker before you can go

86
00:23:35.600 --> 00:23:52.640
in. Um there's video recording. We also do record uh the online as well. Um and we have the ability to look at all of the um uh when they access their computer if they're taking it at home, we can see any software that might be running in the background to make sure

87
00:23:52.640 --> 00:24:14.000
they're not using that. So we put a a a huge amount of protocols in that for our live assessments. I have a maybe a quick question, but I thought that was an interesting data point you just shared. Hopefully I understood it correctly. You said one

88
00:24:14.000 --> 00:24:28.880
standard error is typically between one and five questions missed. Is that right? >> Yes. Depending upon this depending upon the test and depending upon what the actual standard error of measurement is. The lower the standard error of

89
00:24:28.880 --> 00:24:45.120
measurement, it would be less um less questions. The larger the standard error of measurement, it could be a greater amount of questions. I I would assume that changes [snorts] like year to year or so is that a bit of

90
00:24:45.120 --> 00:25:02.960
a a moving target or you know how do you all I guess the broader question is how are you all using information you're now getting from these programs to help inform the tests themselves? like if you're seeing patterns in questions that

91
00:25:02.960 --> 00:25:19.200
candidates are getting wrong and they're having to come back and do the bridge program or retake. It feels like you're maybe now getting more granular data uh than you or maybe not. Maybe you always had and use this data. But just you you kind of see where I'm going with that question. >> Yeah, Michael, we've always had the psychometrics on the back end. We have

92
00:25:19.200 --> 00:25:35.520
an a huge psychometrics a part of our division at ETSS. Um and so we do uh what's called the differential item functioning analysis. Uh we are always constantly testing 20 questions that are actually not scored to ensure validity

93
00:25:35.520 --> 00:25:50.880
and reliability of our assessments. So I think that goes to the broader piece that we were talking about. Um we also update the standard error of measurement every year. Um so we put out a technical report that usually comes out in August of each year. And so you're right in

94
00:25:50.880 --> 00:26:07.600
that it could impact, but the way you calculate a standard error of measurement is you subtract that from the cut score set by the state and then you round up. So there would need to be a significant change in the standard error of measurement. Typically there

95
00:26:07.600 --> 00:26:23.919
isn't a huge change but it can happen like one might go from 7.9 to or maybe like 6.1 to 5.9. that that would have an impact um the following year um once

96
00:26:23.919 --> 00:26:40.480
that new report comes out. But that's why we don't actually put out what are the cut scores. We actually say one standard error measurement from the state's cut score and then we put a technical report out that shows all the standard error measurements. That that's another sorry one other thing you just

97
00:26:40.480 --> 00:26:56.799
made me think of is just this notion that you're using the state's cut score. Um as opposed to the nationally recommended however many years ago Tennessee moved to a place where virtually all of our assessments the cut scores are set at

98
00:26:56.799 --> 00:27:13.440
the nationally recommended cut score. Somebody else on this call who's closer to the lensure work than I am at this point. Uh probably knows how many tests we have that are not uh set at the nationally recommended. But I guess that just the point there being we've we've

99
00:27:13.440 --> 00:27:30.240
set our cut scores fairly high. Um which may not I don't know where other states are trending uh in that regard. But um are you seeing that there are still states that have lower pet scores set below the nationally recommended and you know that could obviously have an impact

100
00:27:30.240 --> 00:27:45.440
on the number of candidates that go through these programs. >> Yeah. Um in fact to your to your question is some states who have set their cut scores at a negative 1 SEM um I believe they're not my states but

101
00:27:45.440 --> 00:28:01.919
there's a couple I think Kentucky may have done it. I think it was Kentucky. Um, so don't quote me on that because it's not my state, but I think Kentucky did. Um, and I know Arkansas sets theirs as a negative -2 SCM. Neither one of those states use bridge for that

102
00:28:01.919 --> 00:28:17.840
purpose. Most states that are using Bridge really were debating on whether or not they wanted to go to a negative one standard error of measurement for cut scores. And they loved this approach rather than going and setting all the

103
00:28:17.840 --> 00:28:33.440
cut scores down by negative one because it really still met that that need of um not having these false negatives by allowing those candidates another option of showcasing uh their information or

104
00:28:33.440 --> 00:28:49.360
reteing. Uh so it was kind of it just gave those states another option than actually lowering the cut score. I know Arkansas, which is my state, is not using bridge because that was a policy decision of setting all the cuts scored to negative two and they don't want to

105
00:28:49.360 --> 00:29:12.320
go any lower than that. >> Hope that helps answer the question. >> Yes, thank you. >> I have one more question and this is probably for the department. Um, so what would keep the pass rate

106
00:29:12.320 --> 00:29:31.279
high? Like what if we change the pass rate? Would that be a department decision or is that a state board policy decision? Because what would keep us from like lowering and lowering the pass rate? >> I'm happy to speak to that, Chrissy. By the pass rate, do you mean the accepted

107
00:29:31.279 --> 00:29:46.480
cut score? >> Yeah, for each test. So, I mean, if we just say we're going to say we're going to have that national recommendation and it's 252 for one test >> and then next year we're like, well, why don't we take it to 242 and then 232?

108
00:29:46.480 --> 00:30:06.000
What what keeps us from doing that? What are what are the >> So, the cut scores are set in state board policy. So, that would be an item that >> would be okay >> that would need to be brought before you all. >> Okay, perfect. And then um please one question. Can you

109
00:30:06.000 --> 00:30:24.480
go back to the chart that had the sort of three-year retrospective? Great. And then I think I was reading that nationally about 12% of candidates qualify for bridge. Is that accurate?

110
00:30:24.480 --> 00:30:39.600
>> It could be. It depends on each test. So I'd hate to like make a general statement. I'd have to go back and look at it. But it really depends on the tests. We have some uh some tests that they that has a higher rate of qualification versus other tests that don't. >> Okay.

111
00:30:39.600 --> 00:30:57.039
>> And I just right on average. >> Okay. So we're [clears throat] looking at a little over 10% of educators who qualify for this opportunity. I just wanted to kind of confirm that piece. On average, to your point, acknowledging that some exams have higher, some have lower uh who sort of hit eligibility. I

112
00:30:57.039 --> 00:31:11.120
just wanted to make sure I I had an accurate understanding of that. Yeah. And one of the really positive things that came out of this um that was unexpected is our candidates are reporting to us that knowing that they

113
00:31:11.120 --> 00:31:26.480
have this option going in has really reduced their test anxiety because if you if you talk to candidates, they say they're not as afraid to not pass as they are afraid to nearly not pass.

114
00:31:26.480 --> 00:31:42.399
Right? someone who scores really low and is not even close to the cut score, they they they accept the fact that like, okay, there's a lot. It's that fear of what if I miss it by one point um or two points. And so, uh the the a lot of the

115
00:31:42.399 --> 00:31:58.559
feedback that we were getting from all the states that have um implemented this, and I think we're at 17 states nationally now um that have implemented the the practice bridge option. they're getting more reports of reduced test anxiety and what they're seeing is an

116
00:31:58.559 --> 00:32:20.399
increase in pass rates because of that that reduction in test anxiety. So that was an interesting anecdotal not numbersbased anecdotal feedback from the field and I want to make sure I have I give enough time to uh my colleagues too. So,

117
00:32:20.399 --> 00:32:35.760
I'm just going to go through this other thing within just a couple of minutes um which is a different option for um flexibility and that's our practice steps. This is different than practice bridge. This is that ability to allow

118
00:32:35.760 --> 00:32:52.240
candidates to take the test in smaller chunks. To my uh to the board members who talked about proctoring and test fidelity, this would all have the proctoring and accountability options within it. Um there would be no asynchronous on their own. This would

119
00:32:52.240 --> 00:33:08.799
all be uh proctored. Um but it really just allows candidates uh to take the test in smaller steps um when they're ready to take it. And I'm just going to go to this particular diagram here. So if I'm taking mathematics, I could

120
00:33:08.799 --> 00:33:24.799
either take the full test 8003 and then if I don't pass, I can retake one of the steps. Again, this would be fidelity, this would be proctored, etc. These are the three steps that make up

121
00:33:24.799 --> 00:33:42.480
this test. So, I could take it whole or I could just take each step individually when I think I'm ready for that step. So, we call those two options a progression path. So this is just an

122
00:33:42.480 --> 00:33:58.799
example of a progression path where Jordan first took the earth and space sciences passed it then the life sciences passed it then the physical sciences passed it and then they'll get an overall cut score or Taylor which would be me this is the way I would most

123
00:33:58.799 --> 00:34:14.000
likely use it is I would take the entire test 85 realize for my results that the only test or subtest I didn't pass for step was the physical science ES step and then I can just focus and retake that

124
00:34:14.000 --> 00:34:29.760
test um or that step to get an overall passing. So again, it's just that extra flexibility and they do all work hand in hand. This is an example of the steps which I just showed you, but again you still could have the bridge which is

125
00:34:29.760 --> 00:34:46.000
someone with the near miss pass. Um, this just shows an example of our new score reports that show all of that uh so that you can see it and then just some resources resources. Again, I know I flew by that. I just wanted to make sure I give my colleagues some time,

126
00:34:46.000 --> 00:35:02.480
too. So, any other questions for me? >> Uh, can you hear me? >> Yes. >> This is Ky Cobb. I'm also the state board, one of the state board members, teacher representative. Um, just

127
00:35:02.480 --> 00:35:19.359
curious, do we are do we have something in place that says how many times on the practice bridge a teacher or potential teacher can take these uh modules? Is there a cut off? Because if you just continue to take test after

128
00:35:19.359 --> 00:35:38.560
test after test, eventually you're going to pass it. And I'm not quite sure that's going to uh label you as an effective teacher once you get into the classroom. >> So I'll I'll try to address that I think and and give that back to So Practice Bridge isn't a new test. It's a it's a

129
00:35:38.560 --> 00:35:55.520
professional development module. So are is your question um related to the number of times is there a state policy on the number of times that they can actually take the test regardless of the bridge? Well, you talked about the modules. If there's a there those are their one

130
00:35:55.520 --> 00:36:13.680
standard deviation um and don't pass the test. Am I understanding that they can then pay that $50 to take the test again? >> So the the the bridge is a professional development module. They're not taking

131
00:36:13.680 --> 00:36:31.200
the test again. It's in lie of taking the test. So if they scored within one standard error of measurement of the cut score set by Tennessee, then they have the ability to be retaught the content, go through um uh practice items and then

132
00:36:31.200 --> 00:36:48.240
do a knowledge check until they pass. I think there was a question like yes, they can retake that knowledge uh sorry knowledge check until they get 100% and then they'll get a certificate of completion. So there is no minimum number of times um set in our system

133
00:36:48.240 --> 00:37:04.880
that they could do uh the knowledge check. >> And I think someone asked this question, but I'm not quite sure if I heard the final answer to it. Uh other states, the other 17 states that have bought on to this. Is there a correlation between

134
00:37:04.880 --> 00:37:21.200
evaluation, teacher evaluation, and those that had to go through this kind of assessment? Um and someone did ask that and that was a lot of the states that have implemented this are following that on their own because it has to do with their own state evaluation system. The

135
00:37:21.200 --> 00:37:36.960
states that I have worked with have not seen any significant differences in reference to those candidates have used this and their um evaluation systems. And again every state has their own evaluation system and what is is is

136
00:37:36.960 --> 00:37:54.240
considered as part of that evaluation system. Okay. Thank you. >> You're welcome. >> One last followup. Um, and Michael, you probably can help me on this or Nathan or someone. Is it possible to within state board

137
00:37:54.240 --> 00:38:11.839
rule or policy to limit the number of practice bridge attempts a candidate could take in order to gain lensure uh within a state? That's a good question. I'm happy to jump in on my

138
00:38:11.839 --> 00:38:27.040
thoughts, but I'm sure others want to weigh in as well. Um, I would think the answer to that is yes, though, Lisa, do your systems allow for monitoring that, reporting that, and then it would be up to the department to

139
00:38:27.040 --> 00:38:43.599
to be able to implement that as well. So I think from a policy perspective we can kind of say whatever we want but if it's not implementable then we could run up against some challenges. >> Yeah. At this time we haven't had any

140
00:38:43.599 --> 00:39:05.839
states request that. So I would have to go back and see if there was a way that we could limit that um and and what that would look like. Have you been able or have there been states that have done the what's it

141
00:39:05.839 --> 00:39:20.800
called for the test? If you fail just one portion, they can retake it. What was the name of that? Your >> steps. Sorry. Steps. >> Yes, steps. Have some states adopted steps but not the modules because I'm

142
00:39:20.800 --> 00:39:36.480
that I think that makes more sense. If they just fell a portion, they need to they can retake that portion. Why would they retake portions of the test that they did really well on when they can just retake the the failing section? >> Do you know any s have just done one and

143
00:39:36.480 --> 00:39:52.240
not the other? >> We just implemented it with our new elementary foundations uh assessment which was open in spring. We've only had two states um adopt that right now. One of them does not allow bridge and that is Arkansas because they do a negative

144
00:39:52.240 --> 00:40:08.640
two SEM anyway. So they aren't and then the other is West Virginia and they do allow bridge as well as the steps. So um they have chosen to implement both. Um the other states that are currently working on adoption that I work with um

145
00:40:08.640 --> 00:40:33.359
do allow for both. Um I'm trying to Yeah, all of the ones that are currently adopting elementary steps are going to allow for both. >> Thank you. Well, if you have any other questions that come up, Taylor and and Michael

146
00:40:33.359 --> 00:40:59.200
know how to get a hold of me. I'd be happy to uh provide any additional answers. And um I do intend uh to be at board meetings. So, if you have any questions about anything, please don't hesitate to reach out. Thanks, Lisa. Um, Taylor, you want to

147
00:40:59.200 --> 00:41:15.680
introduce Matt? >> Absolutely. Um, at this time we are going to hear from my wonderful friend Matt at Pearson and he is going to present uh some additional alternative assessment options. So, at this time,

148
00:41:15.680 --> 00:41:32.480
Matt, I will allow you to uh intro yourself a bit better than I have and take it away. >> Thank you very much, Taylor. I appreciate that. Uh this is Matt Devartio. I am with Pearson uh joined actually by one of my colleagues, Jerry Duga, who's on the call as well, but at

149
00:41:32.480 --> 00:41:49.280
one point, Jerry didn't have sound, so he may be watching us just move our mouths. Um I haven't gotten an update from Jerry to find out if he actually can hear us or not, but I'm hoping he can. Jerry and I are both responsible for leading the activities for the program delivery of what's called the National Evaluation Series program in

150
00:41:49.280 --> 00:42:28.000
Tennessee. Um, let me share my screen. [clears throat] Can you all see that on on your screens? See nodding along. That's great for a thumbs up. Thank you, Michael. [clears throat] All right. Um,

151
00:42:28.000 --> 00:42:44.560
a little more about me and Jerry and Pearson. First, uh you all have been working with ETSs for a long time. It's certainly understandable relative to your needs for the practice series. Just wanted to say that um for my part, um I've been with Pearson for 24 years.

152
00:42:44.560 --> 00:43:00.400
Throughout that time, my responsibilities have been wholly in educator lensure testing program development and delivery. Uh Jerry joined us just in the last few years and like I said, he and I are collaborating together to deliver the program in Tennessee. Um with respect to educator lensure

153
00:43:00.400 --> 00:43:16.480
assessments, we deliver educator lure testing programs about 35 states across the country. You can see those on the map here. They range from you know large states like New York and California and Florida and Texas to you know progressive states with respect to their education vision like Massachusetts. And then of course there's lots of states

154
00:43:16.480 --> 00:43:31.760
throughout the southeast not least which includes Tennessee and you can see the others here. Um I wanted to kind of put that frame out there just so that you all know that you know what we're doing and where we're doing it. Um it's just that just brings a little bit of credibility to the fact that we're delivering the national evaluation series to you all in Tennessee. Now

155
00:43:31.760 --> 00:43:47.280
um with respect to the flexibilities following the program launch of the National Evaluation Series program, Terry Terry, excuse me, Taylor introduced us to you all back at the May board meeting. I was there. I had an opportunity to listen in on the questions you all had. Taylor reached

156
00:43:47.280 --> 00:44:03.440
out to me and said, "Hey, you know, we'd really like to have an opportunity for you to talk more with the board members about some of the questions they And of course, we're here to talk about those those two flexibilities that are already available as part of the National Evaluation Series program in some states, but not currently yet being used in Tennessee. And of course, it's up to

157
00:44:03.440 --> 00:44:17.760
you all to decide whether or not I can adopt those. So, excuse me. The there are two um what we call the National Evaluation Series Flex Assessments and then the National Evaluation Series Tesllet based

158
00:44:17.760 --> 00:44:34.560
assessments. I'll give a brief brief explanation of both and then we'll go into a little more detail about both of those. I would ask that if you have questions, please interrupt me because just openly and transparently I'm working with two screens. I've got the PowerPoint in one and my notes in the other and I can't quite see you all at

159
00:44:34.560 --> 00:44:49.920
the moment. So, please please uh do feel free to interrupt me and ask questions. With respect to the flex assessments, this is the flexibility where a candidate who takes what I'm going to call the regular test, meaning the test

160
00:44:49.920 --> 00:45:05.119
that you would already be familiar with in terms of an examination that a candidate shows up and takes at test center or is takes at via online proctoring at home. We'll call that the regular test. The flex assessment is an opportunity for those candidates who narrowly miss the passing score to be

161
00:45:05.119 --> 00:45:21.680
able to submit a flex submission. What they're doing is to provide a deep and rich analysis of a specific national evaluation series test competency that ends up demonstrating their depth of their subject matter knowledge in that respective area. And I'm going to show you one of those so you can actually see

162
00:45:21.680 --> 00:45:38.480
and we'll talk more too about the the being close to the passing score. The series testbased assessments are opportunities for candidates to take versions of the examinations in a test center or at home through online

163
00:45:38.480 --> 00:45:52.720
proctoring whereby they take, you know, bite-sized pieces, smaller modular units, each of which is anchored to a specific domain of knowledge and skills instead of sitting for one large examination. But cumulatively, those modular units add up to what we think of

164
00:45:52.720 --> 00:46:12.240
as one large exam. So, um, why are we doing this? Well, [clears throat] we've had feedback just like what you all have heard that there are workforce pressures and access demands that are really reshaping how states think about assessment. Um, Lisa talked about the

165
00:46:12.240 --> 00:46:27.359
the various states that have adopted some of their programs or in a similar situation where the states that we work with are adopting such situations as well. And really what's happening is we've got stakeholders at all levels and asking for flexible pathways but doing so in a way that reduces burden but

166
00:46:27.359 --> 00:46:42.880
without diluting standards. And of course that's the key question that always comes up as at least the board members and some folks in similar situations like yourselves. How do you get to be able to achieve that situation of reducing burden without diluting standards? So um let's talk about the

167
00:46:42.880 --> 00:46:59.920
flex assessments first. Um, let's start with the basics of of the the who, the what, and the how. [clears throat] Um, what we're talking about here is

168
00:46:59.920 --> 00:47:16.319
individuals who we've kind of come up in a couple conversations before, not the least of which includes the conversation with Lisa about candidates who are close to the passing score. Like ETSs, we're using the standard error measurement as well. in the states that are using flex assessments right now is also within one

169
00:47:16.319 --> 00:47:32.079
standard error measurement. In terms of what they're doing, they're doing an analysis of a topic on the specific assessment though that demonstrates the depth of subject matter knowledge. They are indeed doing this independently and completing one task and submitting it for scoring. What do I mean by

170
00:47:32.079 --> 00:47:49.280
submission? Well, we already have an e portfolio platform where candidates are able to upload artifacts of information to be able to be associated with scoring for portfolio tools. We're applying that same methodology with our e portfolio platform for candidates to be able to submit their flex assessments. So, this

171
00:47:49.280 --> 00:48:04.640
actually already exists and it's already in place now and we're having great success with it. So, let's take a look at uh what one of these actually looks like. I'm going to use foundations of reading, the foundations of reading exam

172
00:48:04.640 --> 00:48:19.599
as an example. I'm choosing this one because the foundations of reading test is used by lots of different states. Tennessee is one of them. And in Tennessee, foundations of reading is actually used or can be used for multiple endorsement areas. So people

173
00:48:19.599 --> 00:48:36.640
who seek to take uh seek endorsements and take national evaluation series exams in areas like early childhood, elementary education interventions, special education, special education vision and more in addition to taking their content assessment in the respective discipline, they also have to take foundations of reading or another

174
00:48:36.640 --> 00:48:51.839
reading instruction competence assessment. So because it touches so many people or can touch so many people, I want to use this one as the exam. So, foundations of reading, that test, the regular exam is organized into these five major domains of knowledge and skills that are in the left side of the

175
00:48:51.839 --> 00:49:06.880
table. [clears throat] Like the first one saying foundations of reading development that those domains are further split into individual competencies that further describe what's being measured. So, for example, in foundations of reading development, there are these four competencies,

176
00:49:06.880 --> 00:49:22.160
competencies one through four, each of which are even further broken up into specific descriptive statements. The document that actually displays all that is on the website right now. It's available for free. Anybody can go there and access it. You don't need to go register. Anybody can find that information.

177
00:49:22.160 --> 00:49:37.520
What's important about this is in the regular exam and there are 100 multiple choice questions that cover the three domains in the top part of the screen. And then there are two constructed response items or open response items. You would think of as essays that measure foundational reading, reading

178
00:49:37.520 --> 00:49:53.520
skills, and reading comprehension. So 100 multiple choice and two constructive response. So when a candidate takes that examination in the open response area in the example on the screen for competency or

179
00:49:53.520 --> 00:50:10.559
objective 10, they have a fair amount of work ahead of them to do relative to the open response or the essay that that they would be responsible for. This information is on the website right now too. So for example, and I'm going to do a little bit of reading here, so bear with me. In this particular competency or objective that they have to prepare

180
00:50:10.559 --> 00:50:26.559
an essay for on the regular exam, their charge is prepare an organized developed analysis on a topic related to the development of foundational reading skills. But they have to be able to touch on the four elements that are on the bottom half of the screen. So they actually have to do the level of effort associated with analyzing, interpreting, and discussing accurately and

181
00:50:26.559 --> 00:50:42.640
appropriately the results of an assessment of foundational reading skills for an individual student. and they build on that to demonstrate the ability to select appropriate examples from a student's reading performance to identify a strength or a need. They demonstrate the ability to select and accurately describe the appropriate effective instructional strategy to help

182
00:50:42.640 --> 00:50:59.599
them uh improve. Then they have to demonstrate the ability to explain the effectiveness of that selected instructional strategy. So that's kind of a big deal. There's a fair amount of work to do at a test center because we want to make sure that we have that fidelity. And I know that that was a question that came up already. When it comes time for the flex, meaning

183
00:50:59.599 --> 00:51:16.400
the individual who got close to the cut scoreer and taking what I'm going to call the regular examination, they're doing this exercise still from home, but there are specific elements to it that are particularly unique and make it what it is. So, what are those things? First,

184
00:51:16.400 --> 00:51:31.680
they provide some background information. [clears throat] So, these instructions that you see on the screen right now are part of the handbook that's available to the candidates. So, first they have to tell us which descriptive statements um from the test objective that they're going to

185
00:51:31.680 --> 00:51:47.359
address in their submission. Meaning on the previous screen, those numbers one through four. And then they have to do some work to actually choose student data. So, indicate the student data or examples you will use in your submission. They get to choose their own authentic student data or examples or

186
00:51:47.359 --> 00:52:02.960
sample data from relevant coursework that they actually took during their academic career to develop their topic and address the selected descriptive statements that are represented in the framework. Then they describe the topic that's being addressed and then they actually have to provide us with the list of sources they're using to prepare

187
00:52:02.960 --> 00:52:17.839
the submission. In other words, it's got to be evidence-based. We want to know what they're using as a resource tool to be able to demonstrate evidence associated with their submission. That's just background information though. The job then thereafter is we

188
00:52:17.839 --> 00:52:34.720
need from you a written analysis of up to three pages that address the specific descriptive statements that you are being asked to write to using the information you provided in your background uh information to us and

189
00:52:34.720 --> 00:52:50.319
submit that three-page document as well as the background information via our e portfolio platform for scoring. >> [clears throat] >> So, the individuals, I'm going to kind of do a little quick recap here. The individuals are taking this are folks who have taken the regular exam failed

190
00:52:50.319 --> 00:53:06.000
but got close to the cut score and have the option then to be able to go and take the flex if they so choose. They could just simply choose to retake the regular examination, but they have the opportunity to submit the flex in either path, meaning the examination, the

191
00:53:06.000 --> 00:53:22.160
regular examination retake or submitting the flex. If they pass either, they've met the requirement associated with in this example passing foundations of reading and then they can, you know, move on from there. If they if they fail the regular examination on a second attempt or they failed the flex, they

192
00:53:22.160 --> 00:53:46.079
have an opportunity to come back and retake. Let me pause here and see if you all have any questions so far before I talk about Teslas. make it so I can see it while I'm near my screen.

193
00:53:46.079 --> 00:54:02.000
>> I have one question. Um, hi, I'm Erica from the state board staff and my question is about um, so in this example that you gave, I presume that this person that the reason that they failed the original exam was because of that specific open response question, they

194
00:54:02.000 --> 00:54:17.280
performed inadequately on it. Um, what if what happened if they performed inadequately on the multiple choice section instead? >> So, that's a Erica, nice to meet you. Great question. So, um, I'm going to just be direct. Your

195
00:54:17.280 --> 00:54:35.440
assumption isn't correct. meaning they could have the the when someone takes the regular exam, [clears throat] their performance on multiple choice and constructive response is pulled together to yield a single score of either passing or not passing. So in other

196
00:54:35.440 --> 00:54:52.559
words, they've got the 85 multiple choice, excuse me, yes, the 85 multiple choice questions here, the two con, excuse me, the 100, I apologize, 100 multiple choice questions here, and the two constructive response items here. [clears throat] those two things, multiple choice and constructive responses, the output of

197
00:54:52.559 --> 00:55:07.760
those are pushed together and they receive a pass or a fail on the collective assessment. [clears throat] So regardless of whether their performance was particularly poor in multiple choice and or the constructive response items, if they fail, but they get close to the cut score, they have

198
00:55:07.760 --> 00:55:23.920
the opportunity to take the flex that they want. The reason why though that we are driving the flex the way that we are is that we want them to particularly demonstrate their knowledge and skills in this area in the flex because it's an integration of their knowledge and

199
00:55:23.920 --> 00:55:39.680
understanding. Meaning the topics that are covered in the multiple choice section are ultimately integrated into this the elements of the constructive response. So in other words, we're not just saying, well, you didn't do so hot in the multiple choice. Just go back and take that particular piece. We'll talk about some of that stuff in Teslas in

200
00:55:39.680 --> 00:55:56.079
just a second because the the integrated approach to and I'll go right back to here. The constructive response elements and therefore the flex is is is deep and broad. It covers all the

201
00:55:56.079 --> 00:56:19.183
elements across the perspective framework, all the elements that are being measured across the entire assessment. >> Okay. Thank you. >> You're welcome. Any other questions for folks? All right, I'm going to keep going then.

202
00:56:19.183 --> 00:56:37.760
[clears throat] Um, let's talk Tesllets. [clears throat] So, Tesllets are mod or a modular design. Historically, generally speaking, when you think about tests in a testing center or at home

203
00:56:37.760 --> 00:56:53.280
online proctor testing, there have been generally with a few different flavors, the top two on the screen test designs. Candidates either are exposed to a single assessment where they take one assessment that measures multiple domains of knowledge and skills which

204
00:56:53.280 --> 00:57:09.920
and has compensatory scoring across the respective domains. meaning a secondary mathematics assessment for example that covers and I'm just making this up right now algebra geometry and statistics right that's an assessment that covers those three domains of knowledge and skills and we're [clears throat]

205
00:57:09.920 --> 00:57:26.480
producing a score another design is subtests or you take those particular pieces and you broke them break them out so separate shorter tests across multiple domains and knowledge and skills requiring each individual subtest to be passed uniquely so in the secondary mathematics assess

206
00:57:26.480 --> 00:57:42.799
example, they take algebra full stop, they take geometry full stop, and they take statistics full stop. And then they have to pass those things uniquely. We've been doing both for a long, long time. Came up with a Tesla model to actually take the best of both worlds.

207
00:57:42.799 --> 00:57:58.079
So, separate shorter tests across multiple domains of knowledge and skills allowing compensatory scoring across the Teslas. It's kind of a lot going on in there. So, I'm actually going to show you examples of what I mean. So start with that single test design.

208
00:57:58.079 --> 00:58:14.400
So just pretend that what you see in gray is a test and it measures algebra, geometry, and statistics. And the passing score is just the black line wherever that number is. Well, [clears throat] I'm going to do show you my crude

209
00:58:14.400 --> 00:58:30.240
drawing. Okay? And what I'm showing you is I'm overlaying performance of three people on what the test looks like. So the person in blue did particularly well on each of

210
00:58:30.240 --> 00:58:45.280
algebra, geometry, and statistics above the passing score. Right? So good for blue. They passed. They move on. They're headed towards getting their endorsement. The person in red performed particularly well in algebra, well enough in statistics, and not quite so hot in

211
00:58:45.280 --> 00:59:02.480
geometry. Person in yellow was sort of in a similar situation. They did, you know, reasonably well in geometry and statistics, but not so much in algebra because it's a single assessment and it produces a single score. In this particular example, because the test is

212
00:59:02.480 --> 00:59:18.079
compensatory, meaning in the red person's example, this the algebra and statistics um performance was particularly good. It helped pull them across the finish line for passing the assessment overall despite the fact fact they were a little lower in geometry. So

213
00:59:18.079 --> 00:59:35.200
this model it's a compensatory design but it's a long test because there's a lot to cover. So someone could be you know in a test center for let's say four hours. It's a long time and they have to be ready for all those domains at the same time. Have to be ready for algebra, geometry and statistics on that day of

214
00:59:35.200 --> 00:59:50.720
that prospective examination. And if they do fail, they're coming back for the whole thing. Even if they blew algebra out of the water and they are fantastic, they're retaking algebra because it's a single assessment. So contrast that with subtests. So

215
00:59:50.720 --> 01:00:08.079
picture the same kind of gray area, but split it into three pieces. So now I've still got passing, but now I have three standalone elements that are subtests in algebra, geometry, and statistics. But so I have to uniquely pass each one. So if I take blue, yellow, and red and I

216
01:00:08.079 --> 01:00:24.480
overlay them again, look what happens. Okay, so same level of performance, if you will. Blue performed particularly well. Passed each subtest uniquely. Good for them. They passed the assessment overall. They can go get their math credential.

217
01:00:24.480 --> 01:00:39.920
In the compensatory model for the single test design, I said that red and yellow passed. Well, look what happens here. Red has passed algebra and statistics uniquely but did not pass geometry and and and Mr. Yellow passed geometry and

218
01:00:39.920 --> 01:00:56.880
statistics but failed algebra in this model. Neither red nor yellow are going to get their endorsement mathematics despite the fact that they passed algebra and you know they passed two out of three because they have to pass everything to move on and get their endorsement for secondary math. So in

219
01:00:56.880 --> 01:01:13.200
the first model all three of them moved on. second model, only two of them did. But why then? Why would we want to do that? Well, there's some advantages here. They can take the subtest when they're ready for each one. So, for example, if they covered algebra and

220
01:01:13.200 --> 01:01:29.599
let's say they were a four-year degree program and they covered algebra when they were junior, but geometry and statistics when they were senior, maybe they go take they want to get the algebra portion taken after their junior year. They bank that and put it away and then they come back and take geometry and statistics after they have geometry and statistics training in their senior year. So there's benefits to being able

221
01:01:29.599 --> 01:01:46.079
to bank performance, but you do have to uniquely pass each one and therefore it's not competitive. So there are pros and cons to both models. So what I want to put in front of you then is I'm going to keep on the left side of the screen the subtest model just as a as a reference point.

222
01:01:46.079 --> 01:02:02.319
And then I want to talk about what we're doing for Teslas, which is on the right half of the screen. So what I'd like you to picture is a situation where you've probably all seen this. Picture a church, okay? And out in front of the church, there's the big plywood sign that they've they've painted in white.

223
01:02:02.319 --> 01:02:18.960
And over that, they've painted a thermometer, okay? And they're trying to raise money for the cracked stained glass window that they've got in the church, right? And you drive by one day and that thermometer painted on that board is empty. You drive by three days later and they've raised some money for the the stained glass window. Well, some

224
01:02:18.960 --> 01:02:34.880
of the thermometer is painted in, right? [clears throat] And then you drive by five days later and more of the thermometer is painted in. It's because they've raised more money and you know in route to their goal. And then you drive by two weeks later and they've got the hooray we did it right and the signs out. Thank you to all our donors. We have enough money. We're going to go

225
01:02:34.880 --> 01:02:49.920
ahead and proceed with our stained glass window repair. Keep that vision in mind. That's what's happening on the right side of the screen with the tesllets. Okay. What I'm saying is individuals can still have the benefit of taking the individual tesllets when they want in

226
01:02:49.920 --> 01:03:06.720
this case algebra geometry statistics separately or take them together if they want. And what they're trying to do is fill that thermometer with their performance on each of the respective Teslas. So they get to take the Teslas when they're ready. They get to bank the performance on each one. They get the

227
01:03:06.720 --> 01:03:24.640
benefit of the those performance stacking and therefore it gets to be compensatory in terms of the performance. So we didn't perform quite so hot in geometry. I can still make up for it by the fact that I have algebra and geometry per or statistics performing a little bit better for example. Um, and what we are doing with

228
01:03:24.640 --> 01:03:42.000
a particularly unique feature of the Tesla model is we're making it so that the candidate can go back and choose which Tesla they want to take. They're not taking and passing and failing each one of them individually. That's subtests. What we're saying is, okay,

229
01:03:42.000 --> 01:03:58.160
Mr. read, you didn't do so great in geometry. But if you think that you can come back on statistics where you perform particularly well and you can do even better and it helps you pull geometry across the finish line, then good for you. We're already doing this.

230
01:03:58.160 --> 01:04:13.760
Your neighbor to the southeast in Georgia is doing this for their entire program. Their entire program right now is made up of Tesllets. So Georgia decided we want to do a complete overhaul on what we're doing for educator certification testing here in the state of Georgia. What can we make

231
01:04:13.760 --> 01:04:30.640
that look like? We proposed a Tesla model. They're doing it across the board. [clears throat] So the idea is Georgia moved to this idea of we like the idea of the compensatory relationship in a single assessment. We like the idea of being able to do subtests with respective banking uh scores and having candidates

232
01:04:30.640 --> 01:04:46.880
come back and take it when they want. and you get to overlay the compensation across the respective performance for each. We have testlets in the national evaluation series now. We're already doing it. It's it's available for elementary education and it's avail and it will be it's actually registration is already open for and the test launches

233
01:04:46.880 --> 01:05:02.240
for foundations of reading in August 31st. Other states are actually headed in that direction. It would take you all to decide if Tesllets for those respective assessments and other ones in the future are something that you would like to do or not. I do want to highlight one thing about

234
01:05:02.240 --> 01:05:17.119
this too though and then I'll certainly open up for questions which is in this particular circumstance that this is not a situation where we're saying the Tesla model is for example standard error of measurement based the flex assessments

235
01:05:17.119 --> 01:05:32.640
that I talked about first is driven by someone getting within standard error measurement this completely separate this is an option if you choose to implement it where you could say you know what Matt we like we like this idea We're going to take, for example, the

236
01:05:32.640 --> 01:05:49.695
regular foundations of reading and put it aside and we're going to replace it with the foundations of reading test the model. It still covers all the content that I that was is represented by the regular exam. It just breaks it into the respective pieces into into corresponding tests.

237
01:05:49.695 --> 01:06:18.720
[clears throat] Okay, I've been watching the screen. I see people looking and then writing and then looking and then writing. Let me see if you all have any questions for me. >> Someone has to have at least one.

238
01:06:18.720 --> 01:06:35.280
>> Okay, Matt. Um this is aa um help me understand with the Tesla model >> when I look at the um I'm looking at the slide you have up with the considerations and I'm looking at blue, yellow, and red.

239
01:06:35.280 --> 01:06:51.039
If I'm a candidate and I'm taking particular subject area test, I could take like one test on this day because I feel like I'm ready for it based on previous classwork.

240
01:06:51.039 --> 01:07:05.440
Then maybe next month I'm enrolled in a different class and or two months. Then I could take another part of the overall test and my scores

241
01:07:05.440 --> 01:07:24.720
bank up towards the passing overall score. >> Everything you said is 100% correct. >> Okay. Thank you. >> And that passing score would be based on what Tennessee's cut score is.

242
01:07:24.720 --> 01:07:39.599
>> Absolutely. Yes. So am I allowed to compare this to the practis? Will you all allow this just for my simple mind? What I'm say what I'm asking is let's just say that a practice score is

243
01:07:39.599 --> 01:07:55.680
200. Let's say and someone goes in and takes that test and they score 200 even though they may have missed questions. Or if someone does the Tesllet model and they still score 200 even though they may have missed some questions,

244
01:07:55.680 --> 01:08:12.640
they've passed both tests. >> Well, [laughter] this it's a great question. >> It's No, that's okay. This is exactly why we're doing this. The scales that we are using are different than the scales that ETSs

245
01:08:12.640 --> 01:08:29.920
is using for for the practice tests. To be clear, I am not here to say one scale is better than the other. I'm just saying that they're different. And let me let me give you a more concrete example. Um, lots of folks are familiar with the SAT, for example.

246
01:08:29.920 --> 01:08:47.520
And you probably all know that for one segment of the SAT, the scale for the score is 200 to 800. It's just a scale. The the National Evaluation Series assessments in Tennessee are on a scale of 100 to 300. the 100 to 300 on the

247
01:08:47.520 --> 01:09:04.560
tests you all are using for the national evaluation series in Tennessee right now are the scale there doesn't mean anything relative to the scale associated with the SAT that's 200 to 800 and I'm choosing that that one the SAT only because it's so different

248
01:09:04.560 --> 01:09:20.480
meaning if we're on a 100 to 300 scale for the national evaluation series and the SAT is on a 200 to 800 and someone said oh my gosh this person got 800 on the SAT but they only got a 300 on the National Evaluation Series. That statement sounds odd because 800 and 300 sound like they're light years apart.

249
01:09:20.480 --> 01:09:36.159
But what are we actually saying? The person that got 800 on the SAT got a perfect score. Person that got a 300 on their national evaluation series exam got a perfect score. So what I'm trying I I moved away from your practice question only because I was choosing the SAT because the scale

250
01:09:36.159 --> 01:09:51.520
is so different. So, all that's to say, you can't you won't be able to look at a 200 on practice and a 200 on the national evaluation series and ask yourself if they're the same thing because they're not. >> Okay.

251
01:09:51.520 --> 01:10:07.360
>> So, Tennessee would need to set the criteria for a Tesla model just hypothetically if if that's what were to be used. >> Yes. But we would bring we would work we

252
01:10:07.360 --> 01:10:23.600
Pearson would bring to the department ideas and recommendations for what that could look like. We wouldn't we wouldn't walk in the door and say hey board what do you want to do and just leave it to you all to have to figure out. Of course we'll you know we'll have recommendations and methodologies to help you make the decision without

253
01:10:23.600 --> 01:10:40.080
steering you in a particular direction. Uh, one other question and I should have asked this sooner but so forgive me but you won't have to go back with your slide deck with the flex uh portfolio assessment platform.

254
01:10:40.080 --> 01:10:56.400
I'm this is I know there are no dumb questions but you're going to think this is a dumb question. I'm sure there are guard rails for AI generated responses that a candidate might submit. So yes,

255
01:10:56.400 --> 01:11:12.560
however, my sh however is the wrong word. Um I'm going to start with it's a great question mostly because everyone is talking about the role of AI particularly at home, you know, generated material. So a couple of

256
01:11:12.560 --> 01:11:28.480
things. First, in some of the states that we're working with, the states have their own AI policies associated with the expectations of what the candidate should be able to use AI for versus not. Even further than that, some states have said, "We state are not making a

257
01:11:28.480 --> 01:11:44.560
recommendation for what the AI policy should be." In part because lots and lots and lots or potentially in some states, all of the colleges and universities that are parent uh future educators have their own AI policies. in in in a state that I'm thinking of and I'm actually not going to name it right

258
01:11:44.560 --> 01:12:01.120
now, they chose not to have a statewide policy as it relates to their assessments because the individual programs have their own AI policies and because they differ, you know, from preparation to preparation program, they didn't want to actually overlay a policy on top of that. So they're they're letting the educator preparation

259
01:12:01.120 --> 01:12:17.040
programs have their own. With respect to the the role of AI in the flex though you wouldn't believe the number of conver you would believe the number of conversations that I take part in that are sort of the tugof-war and what I mean by that is one conversation says

260
01:12:17.040 --> 01:12:32.400
well boy no one should be using AI to help them prepare their response and then the next the next conversation almost in the same breath is but all these people who are going to become teachers need to know the proper use of

261
01:12:32.400 --> 01:12:49.920
AI because they will be using it as it relates to the role of AI in their preparation of lesson plans, implementation in their classrooms, etc. So then because those two things are in the same breath, the question ends up becoming so what do we tell people they can can do and cannot do on a tool that

262
01:12:49.920 --> 01:13:04.960
they're actually creating from home to submit for scoring when the hopefully the next day they get their certificate, the day after that they're performing in the classroom and using AI a whole lot. My wife is actually sitting in another room right now and she's an elementary school teacher and a reading specialist. She's using AI every day, but she's

263
01:13:04.960 --> 01:13:27.360
using it in a way that actually benefits her. I'm giving you a long answer. Um, but it it's it's an ongoing conversation. [clears throat] >> Thank you very much. >> Other questions from folks? >> Matt, this is Jerry. >> Hi there, Jerry. Jerry, I think must

264
01:13:27.360 --> 01:13:43.040
have sound if he's joining us. Hi there, Jerry. >> I just wanted to say I was able to solve my connectivity issues. Or should I say Barnes & Noble was able to solve my connectivity issues. So, so but I did want to go back to the question about

265
01:13:43.040 --> 01:13:57.760
the two different scale scores for a second to make sure that there's a pass understanding. While we use a scout score uh from from 1 to 300, practice uses from 1 to 200, a passing score on both of those

266
01:13:57.760 --> 01:14:14.400
assessments represents an appropriate amount of knowledge that a candidate should have in order to pass the test. So, while a 200 on one scale, you can't say that's a 200 on the other scale, those scales are appropriate in determining what is an appropriate

267
01:14:14.400 --> 01:14:49.120
passing score on each of those distinct exams. So, I just wanted to make sure that there wasn't any any question about that. >> Thank you, Jerry. >> Any other questions from folks? I guess I have one I I think you've

268
01:14:49.120 --> 01:15:04.880
already addressed this but just to be clear there are some areas where we have not adopted NES series assessments am I understanding you correctly we don't necessarily have to have adopted the regular assessment to adopt one of these tesllets or do you have to have

269
01:15:04.880 --> 01:15:21.120
both >> uh oh thank you Michael and I understand your question so I'm going to take your question and break it to pieces realizing it's kind of ironic given the Tesla conversation that's not intentional [laughter] [clears throat] with respect to the

270
01:15:21.120 --> 01:15:36.560
assessments that Tennessee has adopted. You all have adopted assessments that cover 59 different l different different endorsement areas. Right now there are a few endorsement areas for which there are not assessments which I think is the second part of your

271
01:15:36.560 --> 01:15:52.159
>> conversation. And to your last point, no, you do not need to be in a situation where you will have already adopted what I keep calling the regular test. >> Yeah. before adopting the corresponding Tesla version. That said,

272
01:15:52.159 --> 01:16:07.760
the the Tesla assessment that is out there and doing its thing now in states that already want to use it in the national evaluation series is for elementary education. You did adopt the regular elementary education assessment. The next one that is coming out is the

273
01:16:07.760 --> 01:16:24.320
foundations of reading Tesla version which launches August 31, so six weeks from now. Um, and there will be other Tesla examinations on their heels. If any if in any of those situations, meaning the LN Teslas that already exist, the foundation's reading that's

274
01:16:24.320 --> 01:16:40.239
coming out imminently or future Tesla versions, you have the opportunity to adopt any and all of those, regardless of the choices you've already made in terms of which assessments you've already adopted versus not. >> Got it. That's helpful. And I and in this case it seems like

275
01:16:40.239 --> 01:16:56.880
states adopt them assessment by assessment individually as opposed to adopting it as a full program saying we accept any and all tests. Does that is that right? >> Yeah. The answer to your question is yes. I will pause that just for a second

276
01:16:56.880 --> 01:17:13.199
to differentiate about what I said regarding Georgia. When Georgia chose to go the Tesla route, they went a slightly different path than what we're bringing to you. So, I'll start with the second thing first. We're bringing to you all the National Evaluation Series existing

277
01:17:13.199 --> 01:17:28.400
testing program and the assessments that we continue to add to it. Georgia said we want to start from scratch and we're going to build everything from scratch specifically for Georgia. And when we do all of that, we're going to build them all and test them all.

278
01:17:28.400 --> 01:17:49.040
>> Got [clears throat] it. Thank you. >> You're welcome. >> Matt, can you share the cost of a Teslate for a candidate? >> Yep. So, um I'll give you the the the short

279
01:17:49.040 --> 01:18:04.880
version is it's the cost of the regular examination simply divided by the number of tesllets. What I mean by that is I want to pick a nice even number so I can do the math in my head. [clears throat] So let's say that the

280
01:18:04.880 --> 01:18:22.960
test is $120. The you know the regular full examination if it's 120 if if we build the Tesla based version of that examination and it has six Tesllets in it all we're doing is 120 divided by

281
01:18:22.960 --> 01:18:43.600
six. of each Tesla is $20. >> And is there a number of days between a candidate, let's say they have to take one tesllet and they do not meet the criteria to have a passing score, how many days before they could retake

282
01:18:43.600 --> 01:18:59.199
another Tesllet? >> Yeah, two this great question. Two answers to your question. So, first let's just say that we're going to continue use the math example. So, if it's tesllets in

283
01:18:59.199 --> 01:19:14.239
algebra, geometry, and statistics, and they just take the algebra tesllet, regardless of their performance, we're going to ask them to wait 30 days, [clears throat] but they haven't taken the other two Teslas yet. So, they can go and register for the other Tesla now and take them

284
01:19:14.239 --> 01:19:47.440
when they're ready. So the the retake wait period is only based on the fact they took a specific test. You're welcome. Anyone have any other questions for our friends

285
01:19:47.440 --> 01:20:07.120
who have joined us from UTS and Pearson? >> This is Sarah. Thank you all so much. I I had a similar question to Dr. Maxwell on sort of the use of AI for the open-ended. I assume um it's similar like with Pearson and ETSs, but is there

286
01:20:07.120 --> 01:20:22.159
anything different in the practice that you would say about that if that's applicable? Because I think um I'm just still thinking about sort of the possibility of disingenuous responses versus, you know, really allowing people to have this opportunity. That's the thing that's kind of bumping in my head

287
01:20:22.159 --> 01:20:42.239
right now is how you guard against that. I think that was for the practice version, right? Sorry, [laughter] not to call you out directly, but if there's anything else that you would add besides what Matt had shared, I'm just curious to know how you >> No, I I wouldn't I wouldn't add anything

288
01:20:42.239 --> 01:20:57.840
else. The for the for our practice bridge though, it's not a portfoliobased. So, it's not an essay based. It really is that professional development where they're going through online um objectives, reteing the practice, and then they do a knowledge

289
01:20:57.840 --> 01:21:13.920
check at the end. So, um I mean I think we discussed this with the question of it's possible that someone could go out and and Google let's say the answer and then answer it just in similarities to those pieces that they could create not their own work in a portfolio, right? I

290
01:21:13.920 --> 01:21:31.400
think both both pieces have that potential um for that but um without it being proctored and without it being watched that someone's actually doing that. Yeah. So, I wouldn't add anything else. >> Thanks.

291
01:21:37.679 --> 01:21:52.800
>> Well, I think we are just about out of time. Um, I just want to thank our our guests, uh, Lisa, Matt, Jerry, and our friends at the Department of Education. And also just note that I I think it's a really healthy conversation that we're

292
01:21:52.800 --> 01:22:09.760
having. Uh just to be able to continue to talk about what it means to hold a license in Tennessee and consider, you know, the rigor that we expect of prospective educators and and the qualifications to become licensed. Um, I'll also just add, I mean, we

293
01:22:09.760 --> 01:22:26.080
border eight states and so we have a unique situation where about 50% of our licensed teachers come from another state and so in some aspects we're kind of dependent on all the wonderful lensure policies of those eight states.

294
01:22:26.080 --> 01:22:42.400
And so we, you know, while we're we're really kind of trying to put a stake in the ground in in Tennessee, there are certainly end runs folks can make to be able to transfer their credentials or the the program completion uh elsewhere, get a license in another state, and

295
01:22:42.400 --> 01:22:57.520
ultimately transfer that license here. So, it's kind of a unique situation we're in, I think, because we border so many states. But again, I'm just uh grateful for the fact that we're still having conversations about teacher quality. uh that hasn't always been true. We've had a period where we had to

296
01:22:57.520 --> 01:23:15.760
talk a whole lot more about shortages. So, um thanks to everyone. Any final words or thoughts? >> You know, [clears throat] Michael, I just I I think this whole process has been very helpful, especially, you know, with in a time

297
01:23:15.760 --> 01:23:31.440
like this when we have two new board members uh who have just come on the board in the last several months. uh to see the balancing act that we need to do and to give them an opportunity to to listen to some of these options while

298
01:23:31.440 --> 01:23:46.239
also listening to their consciences about about uh what they think the best way forward is. And that that isn't always something that can be done uh in a 10-minute discussion or a 15minute

299
01:23:46.239 --> 01:24:06.880
discussion in a board meeting. Uh so topics like this are of a you know real special use especially as as folks are kind of gearing up and finding their footing on the board. >> I agree Nathan U. Michael these are very

300
01:24:06.880 --> 01:24:22.320
helpful. I've enjoyed each of the member learning sessions that you've conducted or the team has conducted for us. So, I just really appreciate um Lisa and Matt and um Jerry being here and providing us

301
01:24:22.320 --> 01:24:45.120
with the information that helps us to try to make the best informed decision possible. We're always happy to contribute. So, these conversations are as important to us as it is to to you folks. So, we thank you for your time.

302
01:24:47.520 --> 01:25:03.040
All right, on that note, hope everyone enjoys the rest of their week. And uh you know, this isn't the last time we'll talk about this. Obviously, I think the department said November uh would be when we'd see this again. So, there's certainly time between now and then. If you all think of any additional

303
01:25:03.040 --> 01:25:19.199
questions that you didn't get to ask or um want to run by uh the folks who have joined us here, please feel free to send them our way and we'll make sure that you get answers. But I hope everyone has a good rest of the week. >> Bye everyone.

304
01:25:19.199 --> 01:25:22.440
>> Bye folks.