WEBVTT METADATA Video-Count: 1 Video-1: youtube.com/watch?v=uy6yvdaPblg NOTE MEETING SECTIONS: Part 1 (Video ID: uy6yvdaPblg): - 00:00:05: CEC2 President Introduces AI Guidance Conversation and Speakers - 00:03:00: DOE Requests Flow Modification, Questions Over Deck - 00:04:39: Public Comment 1: Parent Concerns Over Educational Impact - 00:07:58: DOE Responds, Acknowledges Concerns, Guidance and Guardrails - 00:10:03: Public Comment 2: Moratorium for Lower Grades, Data Privacy - 00:12:16: DOE Response: Exploring Age Appropriateness, Data Protection - 00:15:17: Public Comment 3: Professional Development and AI Literacy - 00:16:24: DOE Response: System-Wide AI Literacy Training Initiatives - 00:17:45: Public Comment 4: Ambiguity in Guidance and Student Usage - 00:21:35: DOE Response: Addressing Academic Integrity, Family Involvement - 00:25:41: Q&A: AI in Lower Grades and Tool Approval Process - 00:27:44: Q&A: Online Learning Ineffectiveness and Efficacy - 00:30:09: Q&A: Tool Efficacy Framework, Council Member Healey - 00:30:41: Public Comment 5: Transparency, Family Engagement, Research Partners - 00:35:20: DOE Response: Data Transparency, Themes, FAQ Creation - 00:38:37: Q&A: Group Projects and Student Breaches - 00:40:16: Q&A: Tracking Data Usage, Vendor Accountability - 00:42:59: Q&A: Energy Consumption, AI Generated IEP Goals - 00:45:55: Q&A: Schools and AI Platforms Before Guidelines, Value - 00:47:34: Q&A: AI Literacy Training Materials, School Leadership - 00:50:01: Q&A: Reading Regulations, AI Startup and Logo, DOE Clarifies - 00:52:28: Q&A: Resistance to Stop AI, DOE Protecting and Providing Equity Part: 1 1 00:00:05.600 --> 00:00:26.640 Okay. So, uh I guess we can begin. Uh let me send my question and I'll I'll end. Okay. Um well, good evening everybody. Uh my name is Craig Slutzkin. I'm the president of CEC2. Uh thank you so much for joining us tonight on this 2 00:00:26.640 --> 00:00:42.960 important conversation about uh the DOE's proposed guidance on AI. As many of you know, AI is already a part of the world our students are growing up in and it's something families, educators, and students themselves are thinking about with a lot of energy and in some case very strong feelings. Some people see 3 00:00:42.960 --> 00:00:58.719 potential and others have very real concerns. Tonight's meeting is meant to give our community a clear, grounded look at the DOE's proposed approach. It's not a debate. It's not a final vote, but it is an opportunity for families to hear directly from the DOE to ask questions and to share what you 4 00:00:58.719 --> 00:01:13.520 want the system to keep in mind as this guidance continues to shape to take shape. We're very grateful to have uh deputy chancellor uh Pete who will be joining us shortly and Tara Karazza from the office of the her of the first deputy chancellor here with us tonight. 5 00:01:13.520 --> 00:01:30.640 uh she will walk us through the proposed guidance where what it covers and how the DOE is thinking about AI in classrooms and across the systems. I also want to acknowledge and thank our superintendent Kelly Maguire for being here tonight and continuing to support those conversations in CEC2. Uh I also 6 00:01:30.640 --> 00:01:46.079 just want to note that we have is that deputy oh hi deputy chancer Pete is that yes >> chief academic officer >> chief academic officer sorry chief okay sorry about that I I do want to acknowledge that several my uh >> sever of my other council members are on 7 00:01:46.079 --> 00:02:02.880 I see council member Cody Linquist I saw Gavin Healey I saw Tam Reed on earlier I don't know if I'm missing anybody if I am I apologize and I'll I will definitely recognize you in a little bit. Uh here's how the evening is going to run. Tara's going to present uh for a little bit and after that we'll open the 8 00:02:02.880 --> 00:02:19.120 floor for questions and comments from families. Depending on how many people wish to speak, we may need to set a time limit per person so we can hear as many voices as possible. But we'll see how many people uh sign up to speak. If we don't get to your question, uh if you or if you prefer to share your thoughts in 9 00:02:19.120 --> 00:02:34.560 writing, you'll certainly be able to submit comments after the meeting as well. We have about an hour tonight. Our goal tonight is simple. to make sure families have clear information, a space to ask questions, and a meaningful role in shaping how this guidance evolves. So, with that, I'd like to turn over to 10 00:02:34.560 --> 00:03:00.080 Tara and Dr. Pete. And let me Sorry, I have to make Let me make the other uh Yeah, let me make her >> Sorry about that. Uh make co-host. Okay. Okay. Um, thank you first of all for 11 00:03:00.080 --> 00:03:17.280 organizing this and we're really excited to be able to engage. I have a little bit of a sore throat, so just mind that in the sound quality of my voice. Um, if it's all the same, I would like to ask permission to just modify the flow of 12 00:03:17.280 --> 00:03:34.720 the meeting a little bit. Um, we did not bring a deck because Sarah and I wanted to make sure that we gave the most time to answering questions. Too many times we show up, we have these elongated decks and then we consume lots of time 13 00:03:34.720 --> 00:03:50.560 and we don't leave time for for speaking um and for the engagement. So I would love to just jump into the questions and you know engage in a conversation um to get as many questions answered as possible if that is appropriate for um 14 00:03:50.560 --> 00:04:06.159 for this audience. >> Yeah, sure. Absolutely. I apologize. I thought that you that you were going to go through the quick piece, but that that's perfectly fine. >> It did shift. We from doing these a few times, Craig, we know that we never have enough time for all >> No, that we wanted to be honor that. 15 00:04:06.159 --> 00:04:22.240 >> Totally get it. So, uh, we did have I'm going to put it in there a signup sheet, uh, to meeting group. Uh, right now we only have we don't have that many people that have signed up, but I'm sure we'll add we'll have a lot 16 00:04:22.240 --> 00:04:39.440 more. Um, let's see, uh, the people in terms of let's see, Cynthia is not on. Mary is So, the first person that signed up that is here is Mitch. Uh, S Mi Mitch, I'm going to ask you to unmute yourself, Mitch. And it sounds like you 17 00:04:39.440 --> 00:04:56.960 wanted to uh say something. And if you'd like, please sign up. If you have questions, comments, anything. Um, and Mitch, >> can you Okay, Mitch, I think you're ready. You're set. >> Yes. Thank you for the time. Um I was 18 00:04:56.960 --> 00:05:12.400 kind of expecting that there would be more of a uh speaking session before uh public speaking but okay. Uh so evening all I'm a parent of a district 2 student in 8th grade. I'm also a software engineer professionally 19 00:05:12.400 --> 00:05:28.080 and I had a few concerns to raise about the AI um guidance that the board put out. The one of the glaring flaws with it is there doesn't seem to be any sort of evaluation for educational impact in 20 00:05:28.080 --> 00:05:47.039 it. As as a result, I'm confused as to why we are pursuing this when there's no indication that it's actually going to benefit the kids. In fact, increasingly there's evidence that it does the opposite, limiting kids abilities to 21 00:05:47.039 --> 00:06:02.639 think through things on their own and to do the productive work of uh generating writing. The the simple practice of having suggestions provided 22 00:06:02.639 --> 00:06:17.440 uh to rephrase things erodess both self-confidence because it by implication suggests that the writing wasn't good enough and further switches the brain from a productive mode into an analytical mode. 23 00:06:17.440 --> 00:06:34.479 making kids think about whether to accept things rather than think about what they want to say in the first place. There's a lot of concern that I've seen about not wanting kids to be left behind 24 00:06:34.479 --> 00:06:51.520 in the big push for AI. But if these products are as revolutionarily easy to use as the companies are promising, then that's nonsensical. They should be able to there the whole point is to make them easy to use, easy 25 00:06:51.520 --> 00:07:09.120 to figure out. So why are we training kids spending all of this education time prepping them for something that's supposed to be easy to use? Like you can learn how to use an iPad more efficiently, but there's a hard skill 26 00:07:09.120 --> 00:07:23.840 limit on that. And all of the evidence that I've seen on my teams of developers, user experience designers, product managers using AI for their to supplement their work has been very 27 00:07:23.840 --> 00:07:39.680 limited and has provided more work to do than it has saved time by using it. So I encourage the board to take a very skeptical view of this. There's a lot of people who want a lot of money 28 00:07:39.680 --> 00:07:58.080 that's generated by pushing these um AI products to prop up their stock values. And we don't have to buy into it. We can give our kids back the time that they were using to learn with from their teachers instead of being stuck on screens even more than they already are. 29 00:07:58.080 --> 00:08:22.160 Thank you very much. So I I would I would start by saying that your concerns are extremely valid. Um especially because the guidance itself, we acknowledge that there are long-term impacts that we are taking a 30 00:08:22.160 --> 00:08:38.479 very critical look at and we're also equally cautious about impact. One of the reasons why we put out the guidance first is because we wanted to be clear about the guard rails because AI is 31 00:08:38.479 --> 00:08:53.200 already actively being utilized by students and by educators and to not address it. There is an inherent danger there as well. But we also took a pause, our first pause to 32 00:08:53.200 --> 00:09:10.399 hear back from our various stakeholders through our 45day window which closed on May 8th. And now with more than 6,000 respondents, we are now taking a humanfacing 33 00:09:10.399 --> 00:09:27.600 um deep dive into the data so that we could analyze trends. And so that data will be the first foundation that will support us in thinking differently I would say about how we move um forward. 34 00:09:27.600 --> 00:09:47.440 There's learning that will be captured from that um from the feedback. So I just want to say that um as well but acknowledge that the concern is extremely valid. Okay. Um, I don't actually see any other 35 00:09:47.440 --> 00:10:03.920 questions, but I know there are so many so many uh questions that we have. I also want to acknowledge Council Member Gloria Chu who joined us. Thank you, Gloria. Good evening. Um, Cody, do you have any Did you want to ask a question? Council member Lungquist. 36 00:10:03.920 --> 00:10:20.480 >> Yeah, I would love to. Thanks. Um, thank you so much for taking the time to meet with us. Um I actually have gathered some um questions and general concerns from some people at various schools that um I leazison with. Um two the part is 37 00:10:20.480 --> 00:10:36.800 twofold. Um number one a lot of people are um curious as to whether or not you have discussed a complete moratorum um especially for the lower grades prek through 8th grade. Um a lot of the parents that I have spoken to in the 38 00:10:36.800 --> 00:10:54.720 elementary grades um feel that although they might see a need in the upper grades in high school perhaps with digital literacy or preparing kids to use these products in um various ways. Most people cannot see a single valid 39 00:10:54.720 --> 00:11:10.480 reason for using AI in the classrooms or on any of the Chromebooks or iPads for kids who are younger than 8th grade. So, I'm curious as to whether or not that is discussed because it does feel like I know in the guidelines there are um 40 00:11:10.480 --> 00:11:26.000 breakouts for certain um age groups, but it does feel that it's a little bit um it it it feels sometimes that the younger grades get left behind. And then the next question I have is about um this is a personal question I have about 41 00:11:26.000 --> 00:11:41.360 the state audit that came out recently slamming New York schools for their lack of student data privacy oversight. Um, I know this is a different topic, but it's sort of related to the same thing. One of the concerns I'm hearing from a lot of the parents is that they're worried 42 00:11:41.360 --> 00:11:59.760 about student privacy. And if we can't even get our arms around data privacy with the programs that we currently are using, how uh can you ensure and guarantee that our students privacy and data will not be breached with AI? uh 43 00:11:59.760 --> 00:12:16.720 which one of the main things with these AI tools is to take data and privacy from children. So two questions one any discussion of a moratorium complete moratorium for the younger grades and also how can you um say that you're specifically going to protect our data. 44 00:12:16.720 --> 00:12:33.760 So thank you. >> So again another really great question. One of the things that the 45day uh feedback window is doing is it's giving us an opportunity to explore a variety of options for students 45 00:12:33.760 --> 00:12:52.720 connected to um age appropriateness of AI usage um and screen time usage as well. um some of our preliminary information just from our respondents have have identified that as a concern area. So we 46 00:12:52.720 --> 00:13:07.680 are looking at the data and we are exploring a vast a vast majority of possibilities. Um can I say explicitly yes no? I cannot at this time, but we 47 00:13:07.680 --> 00:13:24.000 are exploring um a a menu of potential options, but we are still combing through the data and we want to do a really great job at that so we can capture the trends and essence and 48 00:13:24.000 --> 00:13:40.000 themes that families and educators and students have um provided us as to the direction that we should go next. as it relates to data privacy and like some recent data breaches. I think again that 49 00:13:40.000 --> 00:13:57.839 is something that our data privacy team is working very diligently on ensuring that we can continue to protect data privacy and continue to strengthen our data privacy walls um particularly in 50 00:13:57.839 --> 00:14:16.160 this era of AI usage and technology usage in general. Um, one thing and we see someone raised their hand, but just in response to that, >> um, I feel like one of the things that has really upset people is that yes, we 51 00:14:16.160 --> 00:14:30.240 talk about how we want to strengthen data privacy, but how can you strengthen data privacy while we're using these tools? And also I just want to say that um I have reports from middle school students where chat GBT is on their um 52 00:14:30.240 --> 00:14:47.839 school devices. Um a lot of uh even if there is um you know a policy in certain schools to not have these tools on there, a lot of the teachers don't realize that they're on there and they're still on there. So um it just feels like uh a lot of people feel like 53 00:14:47.839 --> 00:15:03.040 the train is leaving the station and we're standing behind it saying wait. So um yeah, that's the comment I wanted to make. I see other hands are being raised now. So >> I appreciate that comment and it is absolutely noted um in general. Thank 54 00:15:03.040 --> 00:15:17.120 you though. >> Thank you. Uh and and uh let's go to Cheryl Rothman. And if you have qu if you want to make a comment or a question uh just you I think at this point you can use the raise hand feature. That's perfectly fine. Uh Miss Rothman. Oh, and 55 00:15:17.120 --> 00:15:35.120 let me unmute mute you. Go ahead. Uh, hi there. I'm not sure. Can you hear me? Okay. >> Yes, we can hear you fine. >> Super. Um, I'm wondering if you can talk a little bit about like professional development and the way that teachers 56 00:15:35.120 --> 00:15:51.360 and paras and other educators are being um kind of onboarded to the world of AI as well. I'm um the mother of a prek kid. Um and I've noticed that a lot of worksheets and questions of the day and 57 00:15:51.360 --> 00:16:08.320 that sort of thing are using and I think videos that they're watching in class have a lot of AI generated content and I'm not even sure I'm not even sure that the teachers are aware that it's AI generated content. So, I'm sort of wondering where the media literacy training and sort of AI 58 00:16:08.320 --> 00:16:24.800 guidance is on um materials, not just curriculum building, but actual materials that are being disseminated in class um and how teachers are are using their skills to discern uh what's appropriate and accurate and what's not. 59 00:16:24.800 --> 00:16:39.519 >> Sarah, do you want to talk a little bit about um the training that we're launching? So um we we're we're doing AI literacy training currently across across all schools and the entire DOE. 60 00:16:39.519 --> 00:16:56.639 Um prior in the past in the past three years my team digital learning and innovation has been working with um districts and offering AI literacy training also across the system. Um, and so we're really focusing in on how do we 61 00:16:56.639 --> 00:17:13.120 train from a systems level from superintendent. We're starting a superintendent AI learning series this Friday actually um for all superintendents and district offices to engage in. We have two separate sessions. one focused on K to8, another one focused on high school, so we can 62 00:17:13.120 --> 00:17:30.720 differentiate the types of supports that are needed and the AI literacy training differences that um would be needed across those two grade bands. Um we're also running a principal PLC uh that goes through early June. Um, and in 63 00:17:30.720 --> 00:17:45.919 terms of the AI literacy training for educators, that's open to the entire system. That's gone out in P Digest um, and uh, across our channels for educators. And then on June 4th, Chancellor's PD day, we'll be hosting a 64 00:17:45.919 --> 00:18:06.559 number of AI literacy training sessions as well. >> Thank you. Uh, Council Member Tam Reid, uh, you have a question or you want to make a comment? You should be able to unmute yourself. >> I muted in just a little slow. Um, hi. 65 00:18:06.559 --> 00:18:22.480 Thank you so much for coming and yeah, questions and comments. I think what I'm hearing from families that I've spoken to, from my son and and his friends um is a lot of just like the ambiguity around 66 00:18:22.480 --> 00:18:39.679 what we've seen so far in in terms of guidance or potential guidance. It's really confusing and just very vague. Like I I understand the safety concerns, the data privacy. What I'm really not seeing is how AI will either be allowed 67 00:18:39.679 --> 00:18:54.960 to be used by students in what context within the classroom and what contexts at home on individual accounts um for different assessments for group projects. So, there's a a very heavy focus right now on what I'm reading on 68 00:18:54.960 --> 00:19:11.679 teacher usage, but there's not the same guidance at all on on student usage. Um I know that it it has been used a lot for instructional reasons um in certain schools and classrooms um sometimes with 69 00:19:11.679 --> 00:19:28.000 accelerated learners if you want to call it that where they're covering a lot of material um and they'll they'll use AI in in the class and work silently on you know get taught by by AI but if they don't finish what they're doing in the classroom they have to use AI at home to 70 00:19:28.000 --> 00:19:44.240 learn the content and then they're testing on it. Um, so that's a big concern. I would like to see very very very clear guidance on how AI is being used by teachers with students and what's okay and what's not okay and using it during instructional time or in 71 00:19:44.240 --> 00:19:59.919 place of instructional time, not in tandem with it. There's also been a lot of AI tutoring um going on where parents have been told to just use AI. If your student's falling behind and you can't afford a tutor, AI is a great tutor. We 72 00:19:59.919 --> 00:20:15.840 all know the problems with that. So, some clear guidance on what you're as a teacher, what you're telling parents is is safe or a good idea. Um, also, I just had a couple more points. I was wondering about the, you know, I heard Mitch what you were saying loud and 73 00:20:15.840 --> 00:20:30.159 clear. I've been looking at a lot of the studies coming out. You know, I'm a professor. I'm a writing professor, so this is really close to home for me. Like, how are we using AI or not using AI um in the classroom? And from my own experience with my son who's a D2 74 00:20:30.159 --> 00:20:47.200 student um at Clinton, he said that, you know, as someone who's generally lacks confidence, he's a student with disabilities, he's not always sure of his ideas. He has great ideas, but he thinks that AI will always have better ones and that his might not compare to 75 00:20:47.200 --> 00:21:04.559 to AI's. And so it's it's really limiting like what he feels comfortable in doing in terms of creative critical thinking, originality of ideas, just producing and generating ideas when he's told just to use AI to generate those those ideas for him. So I guess the question is what are the parameters 76 00:21:04.559 --> 00:21:20.080 going to look like for student usage? How are you going to know what's actually happening in the schools? And to follow up on that, can students opt out of using AI if they are ex, you know, very opposed on their own, not 77 00:21:20.080 --> 00:21:35.200 their parents, but if the student does not want to use AI, like is there a way for them to not have to engage with it? >> Thank you. No, thank you um Council Member Reid for bringing up 78 00:21:35.200 --> 00:21:49.679 a great rich list of things that I've just written down um to kind of turn key back and building out our FAQs um is going to be a part of that. What I want to just acknowledge is that our 79 00:21:49.679 --> 00:22:07.600 guidance to your point it does not um reflect nor does it address um the nuances around academic integrity. Um the plan for that um was aiming to live within the playbook. But 80 00:22:07.600 --> 00:22:24.480 I do believe that these are additional points that feedback has surfaced similar to what you have surfaced that we need to really be able to provide like actual case study like real use examples so that teachers understand 81 00:22:24.480 --> 00:22:39.039 um what they should and shouldn't do and what students should understand as appropriate use and not appropriate use. How they should utilize it, how they should not and if they don't want to. we should address that as well. I think 82 00:22:39.039 --> 00:22:55.600 that um again that is those are like the AI essentials that we want to be made clear for educators and and students. So I I have no rejections to anything that you've said. I think 83 00:22:55.600 --> 00:23:11.520 that these are additional ideas that we have to add to the list and make sure that this next iteration um it it really clarifies that for educators, for school leaders, for families, and for students. 84 00:23:11.520 --> 00:23:27.840 and and I just wanted to the reason I I included families is because as we're doing these engagements with different districts, we are um really gaining momentum in terms of the things that families want in terms of co-development. And one of the things 85 00:23:27.840 --> 00:23:43.679 that um I'm not I don't even recall which district because it has come up in multiple districts, but one of the ideas that a parent shared was um they said, you know, Dr. Hey, we want like it's great for you to create educator facing 86 00:23:43.679 --> 00:24:00.320 and school leader facing and central facing, but families want a familyfacing AI guide as well that will give families insight as to here's what you do at home. Here are the questions that you can ask. Here are the signs that you 87 00:24:00.320 --> 00:24:17.919 should be looking for. Right? Ask your child. Explain how you got this answer. If they cannot explain it, it is likely that it is supported possibly by AI or some other means. So we want to really support parents in asking really good 88 00:24:17.919 --> 00:24:35.840 and solid clarifying questions that help them join together with educators around helping students navigate and to maintain the ownership of their thinking, their critical thinking, and the ownership of of their outputs. and 89 00:24:35.840 --> 00:24:52.960 ensure that the cognitive load is is limited if not none non-existent. We don't want the offloading um to take place because that does in fact interfere with the learning of of the student. So I really embrace what you've 90 00:24:52.960 --> 00:25:08.720 said and I thank you for bringing um bringing that up council member Reid. >> Thank you so much. >> And Dr. Thanks. >> Um, just to add to that to Normma on how we built the the current guidance as is, it's based on our existing policy and 91 00:25:08.720 --> 00:25:26.640 the focus of tool use is based on the educational educationally purpose-built tool use, not on commercial tools outside of school. So, just want to make sure we norm it was based on our existing um policies, chancellor's rags, state and 92 00:25:26.640 --> 00:25:41.919 federal laws as well. Thank you. We have a couple of questions in the Q&A box, also in the in the uh chat box. So, I'll start off with the Q&A box. Uh and I do want to get to uh Council Member Healey as well. So, one question in the Q&A box is, "Are 93 00:25:41.919 --> 00:26:04.799 students in a using AI in lower grades like K through two?" >> So, Dr. P. Yeah. Um, so there's it depends on, you know, the district that they're in and it depends on it depends on if um there is an Irma approved tool. 94 00:26:04.799 --> 00:26:21.200 Um, but there's not a central there's not a requirement across our across our district that students use AI or schools use AI empowered tools. Um, and so if a tool that is an educationally 95 00:26:21.200 --> 00:26:38.480 purpose-built tool has an AI empowered piece to the product and it's potentially being used in those grades, that's a possibility. Um, and any of the tools that are Irma approved, they can't they are not 96 00:26:38.480 --> 00:26:56.559 allowed to train on student data. um we have to abide by Ed Law 2D which makes those um those things very clear and we are required compliantly to ensure that that any tool that's in our system does not do that. Um, and so I I can't I 97 00:26:56.559 --> 00:27:11.919 can't speak to every tool being used across the system, but there are tools that have AI empowered features that potentially are being used in K to2. But it's something that based on the amount of feedback that we've had based also um on the 98 00:27:11.919 --> 00:27:27.120 research that's coming out uh not not solely on tool use but screen time and just the impact of technology for our younger children. We're actively reflecting on the feedback, reflecting on research, working with research um 99 00:27:27.120 --> 00:27:44.640 partners to inform our next steps on this and what it looks like from the K to2 3 to 5 gradeband um appropriateness for any technology used beyond not solely AI. >> Great. Uh the next is more of a comment 100 00:27:44.640 --> 00:28:02.080 uh but you can you can respond if you wish. With all the data on how ineffective the majority of online learning techbased learning was during the pandemic, why are we going down this road again of experimenting with our kids again? >> Sorry, can you repeat the question or 101 00:28:02.080 --> 00:28:16.880 can I look at it? >> Can you can you get into I don't know if you can see. >> Yes, I see the Q&A. Sorry. Go ahead. >> Okay. So, I'm going to read it again for everybody in case they can't other others can't see it. With all the data on how ineffective the majority of online learning/techbased learning was 102 00:28:16.880 --> 00:28:31.919 during the pandemic, why are we going down this road again of experimenting with our kids again? >> So, I'm going to break that response. First of all, valid valid question. Um the pandemic was 103 00:28:31.919 --> 00:28:48.080 was pushed on us and there was definitely a gap in terms of teacher preparation for teaching in that type of environment and the amount of of online instruction that had to happen and then the as a result of that the amount of 104 00:28:48.080 --> 00:29:03.360 technology that came into schools. And so um in terms of AI and and where we are now comparing comparing the pandemic to right now is not what we see as 105 00:29:03.360 --> 00:29:20.559 apples to apples. With that said we know from the pandemic that teacher training and leader training is critical to uphold our academic rigor rigor and highquality instructional implementation. And so that's the area 106 00:29:20.559 --> 00:29:36.720 based on both the feedback and the past three years of work that our team and different teams across the organization have been doing. Really coaching for teachers and then teacher training is a critical way that we are mitigating having anything like the pandemic 107 00:29:36.720 --> 00:29:52.720 happening again in schools and upholding our high quality of academic rigor through that through that teacher training. And that's what we're committed to and actively doing right now. Thank you. We'll go one more one more one more question in in the chat and 108 00:29:52.720 --> 00:30:09.120 then we'll go to council member Healey. Uh how do we know that the AI tools work if not testing them with our children? Being Irma approved does not mean that they are effective. So it's both basically about the efficacy of the AI tools. 109 00:30:09.120 --> 00:30:24.559 So this is something that we actually >> yeah we're actually we're do not we're actually doing we know that Irma is has been put in place and it's uh it's a baseline for data privacy. The Irma process is focused on data privacy. What 110 00:30:24.559 --> 00:30:41.760 we're actively building is a tool evaluation framework and scorecard to have a higher threshold of how we're evaluating the efficacy of tools in terms of instruction. Thank you. Council member Healey, would you like to ask your question or make a comment? 111 00:30:41.760 --> 00:30:58.399 >> Yes, thank you. Um, thank you for this opportunity. Um, sorry, I've had a lot of questions about about the guidance. The the 45day window window goes very quickly when you're when you're a parent with a job and with a lot of other responsibilities and not someone, you 112 00:30:58.399 --> 00:31:14.640 know, doing this like full-time. Um so I you mentioned about the sort of like family facing aspects of this and um one of the things um before you had mentioned is that you're doing um AI literacy training for for superintendent 113 00:31:14.640 --> 00:31:30.000 and for for principles at you know both the K to8 and high school levels and I think for for our purposes I think for transparency it would be very helpful um if you could commit to to sharing that you know those materials and those from 114 00:31:30.000 --> 00:31:45.600 trainings with with um you know with parents and families I think >> because the the guidance is you know is is one thing it's a little bit um a little bit abstract but but the actual training that our that our superintendents and principles are 115 00:31:45.600 --> 00:32:02.000 getting on this I think would be extremely sort of illuminating to to take a look at um that that is um one thing um I think the other thing is that you know there's in terms of the the tech issues. I mean certainly a lot of 116 00:32:02.000 --> 00:32:19.120 tech came into our um schools during the pandemic and and you know perhaps that was inevitable but um I think I I take issue with some of the sort of things we're hearing about the inevitability of of AI. I think it's quite different. Um and we also see a backlash. I mean there 117 00:32:19.120 --> 00:32:35.919 are people like myself who have a child who was you know in elementary school during the pandemic and you know had really no other choice but to you know you to kind of really really sort of ratchet up his engagement with tech which I think I think ultimately was 118 00:32:35.919 --> 00:32:52.159 um you know we're still sort of grappling with the effects of that um but then on the other hand we have younger parents who are coming in you know post pandemic who who have not who don't have the same relationship to some of tech products and are really trying very hard to give their, you know, 119 00:32:52.159 --> 00:33:09.679 children a sort of a more screen-free uh life. Um that it's it's maybe harder for those of us with with older children who went through the pandemic to to do. Um so I feel like there's, you know, maybe an opportunity here for you and your team to try to sort of, you know, 120 00:33:09.679 --> 00:33:26.720 address that. I think I think screen free schools would actually be incredibly attractive to to a lot of uh families. Um that's one thing and then I think the the last thing I just want to mention is you know I think in general you know parents trust their their 121 00:33:26.720 --> 00:33:44.240 school principles and teachers but we don't necessarily trust the the edtech companies that are pushing a lot of these AI products and I think you mentioned before some of the research partners and I'm a little bit curious about you know who are the research partners who are working with uh New 122 00:33:44.240 --> 00:34:01.120 York City public schools on this um I I mean, it's wondering if they're, you know, teachers college or or or, you know, something like that. I guess I'm a little bit um wary because we've heard a lot about, you know, recruitment of superintendent from, you know, sort of Google fellowships and things like that. 123 00:34:01.120 --> 00:34:17.359 And there's like a huge amount of it's sort of a wild west atmosphere of people really pushing these products on us. And um and you know, when I hear people kind of rushing to sort of sort of get into this market, and it's a market um Yeah. I'm going to ask you just to wrap up 124 00:34:17.359 --> 00:34:33.440 because we have a lot a bunch of questions in the chat, please. Thank you. >> Thank you. Um I'm almost done. >> I know. I know. >> So, I mean, we when we had this kind of kind of wild west atmosphere and these, you know, you know, this rush to sort of make decisions to to move forward. I 125 00:34:33.440 --> 00:34:50.320 mean, what I always tell, you know, I used to be a practicing attorney. What I always tell my clients is when people are rushing you to make a decision, that's when you need to put on the brakes. That's when you need to slow down. And I mean I think there's an opportunity here for us to maybe extend this 45day window. Maybe take it a 126 00:34:50.320 --> 00:35:05.280 little easier. You I think you've seen the backlash at at PEP meetings and you know in other other fora and um you know maybe it's something that you know you should take back to the chancellor's office and you know maybe if not a moratorum you know perhaps you need to 127 00:35:05.280 --> 00:35:20.240 you know give this some more time to sort of sort of percolate. Thank you. Well, I can take back the the research question to our RPSG team and we can we can come back to you on on that for sure 128 00:35:20.240 --> 00:35:35.839 >> and then just to you know assure you go ahead Dr. Pa do you want to jump in? >> Yeah, I just wanted to say I wanted to respond to the 45day window. So the 45day window is the starting point. So I just want to be clear to everyone on this call that we created the guidelines. We engaged in uh lots of 129 00:35:35.839 --> 00:35:53.440 rounds of feedback and iteration to even create the first draft of the guide book. And one of the things that we continue to say is that it's just like phase one. It literally is phase one of like about four phases that we're engaging in. So we've closed out phase 130 00:35:53.440 --> 00:36:09.839 one. Um we are going through all of the the data and the guidance. Our next step is to really make that data um from the the survey results um transparent to all of you and to make clear what are the 131 00:36:09.839 --> 00:36:24.880 themes that we've come across, what are the top concerns that families have elevated and then to start to talk about the best way to address those top concerns. And all the things that you all are sharing are questions that are 132 00:36:24.880 --> 00:36:42.480 being created that will advise our release of FAQs. Um, and we have quite a healthy bank of FAQ questions, more probably more than 100 questions. Like we want to make sure that we're giving answers to all the questions that are coming up in all these different 133 00:36:42.480 --> 00:36:57.040 engagement sessions that we're having. Um, which is why we're capturing these questions um, you know, all along. So the 45day is just phase one. There are multiple phases that are that are still in the works. And one of the things that 134 00:36:57.040 --> 00:37:13.839 we have as you said from PEP meetings, from different CEC engagement meetings, from the chancellor's listening tour and so forth is that parents don't want to be told. Parents don't want to this to just come top down. They want to be co-developers. And the first step to 135 00:37:13.839 --> 00:37:29.920 being a co-developer is giving an open window for the partners essentially to let us know what the top concerns are. What are your thoughts about a variety of different things and to take that information and 136 00:37:29.920 --> 00:37:46.880 then turn key it back like give it back to you to say so here's what everyone said. Here is what more than 6,000 people have said. And now from this we start we enter phase two which is the co-development of a resource, a tool, 137 00:37:46.880 --> 00:38:03.440 guidelines, guard rails, um pauses, a host of things um that that can come out of that. Um but we want to do it right. Um so again, this is phase one. That is all that that was and now we're moving 138 00:38:03.440 --> 00:38:19.520 into phase two where we are looking at the data and then within the upcoming weeks we will be sharing this data very publicly um with everyone so we can start the process of co-developing that is what I do know factually and that is what families have said that they want 139 00:38:19.520 --> 00:38:37.599 they want to be co-developers of our AI essential guides for families school leaders students um and educators and and also where the playbook finally lands when it lands. So I just want to say that as well. 140 00:38:37.599 --> 00:38:54.160 >> Thank you. Uh so couple few we have a bunch of questions in the chat now. What happens with group projects? If one student breaks AI academic integrity unbeknownst to the other members of the group. >> So I'm going to say if it is discovered the outcomes will be classroom by 141 00:38:54.160 --> 00:39:09.359 classroom case by case basis. But I think first of it and and I'm a mother. So as a mother of two children, I'm when I think of projects, I'm thinking of my son who's in 11th grade. I I want to create and suggest that we create an 142 00:39:09.359 --> 00:39:25.839 environment that is not overly punitive in a learning environment. I think we have to have conversations first with our young people to say, "So what were your thinking around this? Was it that you didn't know the information? Was it that you needed additional support? What 143 00:39:25.839 --> 00:39:42.640 was it? I think a lot of times with children, we we rush to to like apply punishment or a consequence without asking them the the most important question, like what was your thinking and why? And really get behind that cuz sometimes I find when children are doing 144 00:39:42.640 --> 00:39:58.640 things that we don't necessarily agree with, sometimes it's because of reasons that we didn't even know exist. Um, and that's me answering this question not so much as the CEO, but as a mother who is also the CEO. So, I I want to say that 145 00:39:58.640 --> 00:40:16.079 um just any rules that are going to be made will ultimately and potentially impact my two children as well. So, I go into it with that lens first. I want someone to ask my child why first and then and then go we go from there, wherever that may be. But I but I want 146 00:40:16.079 --> 00:40:32.160 to give our children an opportunity to know that they're they're loved and they're cared for and it's not like this environment that's going to beat them down and and be overly punitive. That that's my stance on that question. Thank you. Does the DOE have the capacity to 147 00:40:32.160 --> 00:40:48.640 follow up with each single platform to track whether they are storing and using our children's data as intended and agreed upon? Um so Tara, do you want to take that in terms of how we've strengthened uh vendor accountability and consequences of violations? 148 00:40:48.640 --> 00:41:04.640 >> So in 2024 in fall 2024, we um implemented an update on AI policy to our Irma process and our requirements from vendors. We added an addendum to our data processing agreements that requires all vendors to share with us 149 00:41:04.640 --> 00:41:21.359 the actual um data trail and and data mapping of how that is being used and to ensure that we can um make sure they're being compliant with not training not using any student data to train their models. 150 00:41:21.359 --> 00:41:36.000 understanding what types of models that they're using, making sure they're in compliance with our state, federal, and chancellor regulations. And so for a K12 system, we actually have the highest level of rigor in terms 151 00:41:36.000 --> 00:41:53.440 of data privacy and compliance. Um, >> actually in the nation. >> Yes, in in the nation for a K12 system. Um, with that, we're looking to improve it all the time based on all of the feedback and student data and privacy concerns that we're hearing and al and 152 00:41:53.440 --> 00:42:12.079 with that also the um the new and daily evolving models of AI and how rapidly um AI is evolving, how we can ensure that we're constantly leveling up the Irma process um to to keep pace with the 153 00:42:12.079 --> 00:42:28.319 changing nature of technology and also using our size and scale and our role as the largest public school district in the country to ensure that we're um holding holding vendors to a higher level of accountability. Um, we are if 154 00:42:28.319 --> 00:42:43.599 if there's further questions around that too or deeper questions around that, I'd love to get them in writing after this so that we can make sure we filter them back to our data privacy uh our chief of data privacy and then to our IT division DIIT to make sure that we're answering 155 00:42:43.599 --> 00:42:59.200 those questions effectively with the correct people who lead those um who lead that work. Okay. >> Uh shouldn't we require these AI platforms to state how much energy and water they are consuming? I want to say two things about that. 156 00:42:59.200 --> 00:43:14.480 One, we do have a sustainability division within the DOE and we are partnering with them too. Sorry, that's my alarm. Um, excuse me, someone's at the door. I think it's my husband actually. So, I'm safe people. 157 00:43:14.480 --> 00:43:29.200 Um, sorry about that. Um the we have partnered with the sustainability division within the department of education and we are starting to have like critical conversations that really start to look 158 00:43:29.200 --> 00:43:47.200 at the broader impact um of of that. These are new um discussions that we are are having and thinking about ways that we embed that into um how we kind of look at um the environmental impact. So 159 00:43:47.200 --> 00:44:04.160 that is super valid. We are currently not doing that as this is new but we do you know recognize it as a an important topic that we must you know pay attention to. Um, as far as earth, we get we only get one. So, it is important 160 00:44:04.160 --> 00:44:19.839 and we want to make sure that we address that and we we are one other I saw one question in the chat that I I don't want to >> break Robert's rule of order. >> No, go ahead. >> To make sure that we did answer or just 161 00:44:19.839 --> 00:44:35.440 at least draw insight into it. It says, "I've recently read goals um on IEPs and language on um progress reports that were AI generated and I understand this is against policy." So, yes, it is against policy and when those when you 162 00:44:35.440 --> 00:44:49.920 do come across those things, you should immediately contact your principal and bring that to their attention or ask those critical questions and then the principal will go take the next step from there. But one of the things that we are doing because it is it is all new 163 00:44:49.920 --> 00:45:06.079 to everyone is that we have created um this is an internal training on our we learn system uh for all DOE staff that does a deep dive around the AI guidance and it talks about the guardrails the the the grows the glows and and all the 164 00:45:06.079 --> 00:45:21.280 things that they should know and be aware of. So as we are doubling down as they say on awareness it is a part of professional development and training. So that is something that we are doing um and thousands of people have taken it 165 00:45:21.280 --> 00:45:38.960 but um we're still pushing all educators and school leaders to engage in this this training um so that they can educate themselves more about what are what things live in the green, what things live in the red, what things live in the middle um so that we can get 166 00:45:38.960 --> 00:45:55.440 better together. So if that is a a unique case, um please make sure that you just notify the principal so that we can address it um right away and make sure that we educate that particular teacher um around around that if if that is the the case. So I just I I thought 167 00:45:55.440 --> 00:46:11.280 that that was really important to to surface and thank you um council member Reed for bringing that up. >> Uh thank you. So a few other questions. uh schools have already started to use AI platforms even before the guidelines uh are approved. I wonder about the 168 00:46:11.280 --> 00:46:26.160 value of these guidelines when the schools are not even waiting for the final document. >> So two things um one one of the reasons this statement is exactly why we needed to produce the guidance because in the 169 00:46:26.160 --> 00:46:43.520 absence of guidance people were doing what they wanted because the AI train has definitely left the station. So people are just doing and not knowing what the guardrails are. So we needed to take a first step which is phase one which is to create the guidance and the guardrails. Once you have awareness and 170 00:46:43.520 --> 00:47:00.240 you know better you start to do better. The second step really was to create the training deck so we can continue to educate and um to to go to go with it. Like saying nothing does does not help us in this case. So that's why the guidance is there. So, um I can't see 171 00:47:00.240 --> 00:47:17.359 like who put that comment in, but it's a very um it's like app propo. It's a very great comment and I wanted to shed light on it that you're 100% correct. Some people are utilizing AI um even with without the guidance which is why we needed to put it in because there were 172 00:47:17.359 --> 00:47:34.319 no clear guidelines for educators. now they're more around the guidelines and they know about the different processes that they may not have been aware of um you know in the first place. So we just want to kind of like you know draw attention to that. 173 00:47:34.319 --> 00:47:51.680 uh there is a uh request for that the AI literacy training materials to be shared with super materials for the superintendent and principles to be shared and I'll add to that would something like that be shared with the the various SLTs of the schools. So, um, 174 00:47:51.680 --> 00:48:08.880 superintendents have been provided with resources, um, to engage in a variety of engagement sessions as it relates to their unique districts and their leadership within their districts. And we really leave that up to our superintendents to start 175 00:48:08.880 --> 00:48:24.640 to engage the different leadership groups within their district with the resources that have been provided um, to them. So superintendents have been provided resources and we believe that superintendents know their districts best so that they're able to engage in 176 00:48:24.640 --> 00:48:38.800 the leadership. But I will also say Tara if you want to talk a little bit about the training that we are doing with in partnership with FACE around um school leadership and and parent coordinators and things like that. 177 00:48:38.800 --> 00:48:54.880 So, um, we've met with all of the district level, I think, I'm trying to think of the acronym, the district family leadership coordinators and done an overview and training for them and face also has provided um supports for 178 00:48:54.880 --> 00:49:13.040 them out to out to the field. Um, we've done the same thing with parent coordinators. um tomorrow we're or sorry not tomorrow's not Friday we're doing another engagement with parent coordinators um with our partners at DIIT and so again like Dr. Pete said, 179 00:49:13.040 --> 00:49:28.880 um, the channels we're using the channels that we have that are existing, uh, to ensure that we're not creating more work for folks across the system that we're hooking into the regular engagement channels, but we also will take back the request and I hear you 180 00:49:28.880 --> 00:49:44.880 Craig as well on the SLT part of it and and that's been part of the feedback we've gotten as well. How do we engage CC's and SLTs, which is why why we're here and, you know, open to doing these feedback sessions to make sure we're >> we're really um engaging in a different 181 00:49:44.880 --> 00:50:01.200 way and engaging in the way that that we're hearing um that that you all want a different type of engagement. Dr. has said at many of these um we can say we're engaging but if you don't feel engaged and you don't feel like we're 182 00:50:01.200 --> 00:50:18.559 taking action based on the engagements and the feedback we're getting we have to do something different and so that's what we're really committed to. Where can we read the state and federal regula regulations on using AI in schools? >> Currently there's there's 183 00:50:18.559 --> 00:50:34.079 >> none. >> Yeah. >> Yeah. There's >> I was like that's a wait a minute eas that you >> No no THAT WAS A TRICK QUESTION. I was like what that's a trick question. So currently there there isn't like that 184 00:50:34.079 --> 00:50:52.880 what what does govern us um is with is found within our guidance we identify the governing rules and regulations chancellor's rags and and a host of those things in terms of the guidance um Tyra I don't know if you could just put the link to the AI um guidance in the 185 00:50:52.880 --> 00:51:08.400 chat for people >> did already for me >> did okay look at you quick start but I will grab it again uh >> and I think that So that's where you would know what the current regs are um with it. 186 00:51:08.400 --> 00:51:24.720 >> So there's a comment in here. It's not a question, but but I think you might want to react or be interested to see what you your comment. Your thoughts are. I saw the the website for an AI startup that says it helps teachers grade. They say it was trusted by teachers in many school systems and included the NYC DOE 187 00:51:24.720 --> 00:51:40.800 logo. >> Well, we can't control who includes our logo. Yeah. >> So, let me say that. Um, I'm trying to look at the the picture. >> I'm trying to download it, too. If you could Yeah, if you could I don't know what the tool is. 188 00:51:40.800 --> 00:51:56.880 >> Um, >> I'm not I don't I don't know what this tool is. Um, and I will send this over to our DIT um platform so that they can um kind of check it out. This is a first 189 00:51:56.880 --> 00:52:13.440 for me. So, it's not it's not probably a DOE or sorry, NYCPS. >> No, I I haven't this is not familiar to me at all. >> Um, but I but I think we should send it over because when when we find these types of things, we should put it out there and cut and paste a a web a logo 190 00:52:13.440 --> 00:52:28.800 from anywhere. So, >> yeah, absolutely. I >> I don't know about this personally. So, >> okay. A comment of why is there so much resistance to stop this process? I still don't understand what the DOE is trying to accomplish. Could could you say more on stop which 191 00:52:28.800 --> 00:52:43.040 stop the process? >> I'm assuming stop AI. I'm assume it's the implementation of AI or incorporation of AI. I I I don't want to I don't want to I don't want to paraphrase but but I'm guessing it's the whole bringing AI into the system. 192 00:52:43.040 --> 00:52:59.119 >> AI is already in the system. We are not bringing AI into the system. Um AI is in the system. So, I just want to be clear. >> No, >> her daughter. >> That's my my uh sevenyear-old. 193 00:52:59.119 --> 00:53:18.079 Um AI, we're not bringing it in. It's it's here. We are responding to what we need to do to create equity, safety, um all all the things in order to really protect students, but also be ensure 194 00:53:18.079 --> 00:53:33.359 that they that they are future ready. But but it's here. So even if we closed our eyes and spent around three times and did nothing, AI would still be here, teachers would still be using it, students would still be using it. um at 195 00:53:33.359 --> 00:53:49.680 home they would still be using it. Um so our only alternative right and the AI is a I don't want to call it out its name but it's a little invasive. It's here whether we want it to be or not. It's on our cell phones. It's it's in our 196 00:53:49.680 --> 00:54:06.160 algorithms. It's in our Alexa tell me the weather. It's in every all the tools and different devices we use whether there's screen time televisions off. It's in our cars our navigation systems. It's everywhere. So, closing our our eyes and saying, "No, no, no, stop, 197 00:54:06.160 --> 00:54:21.680 stop, stop, doesn't actually get us to what we want. Stop, stop, stop." We what we need are guardrails that really do the protection, that really ensure that cognitive offloading, our kids, our students are not victims of cognitive 198 00:54:21.680 --> 00:54:36.400 offloading to make sure that they are not victims to the bias, make sure that tools and vendors are held accountable and more. So that's what we are attempting to do and that's what we we need to do. Um but we're not bringing AI 199 00:54:36.400 --> 00:54:52.160 into schools. It's here now. We have to figure out how we protect. That is the real answer of it. And I know that that may not be an answer that is welcomed or or given a warm hug by all. But it is here whether we want to acknowledge it 200 00:54:52.160 --> 00:55:08.559 or not. Um, and now we must do what every parent wants to do, which is to protect. Um, and that that's that's the tiger mom in me. Like when something is upon me, how do I how do I stop it? How do I, you 201 00:55:08.559 --> 00:55:24.800 know, ensure it in in every in every way? Um, and removing someone said remove AI and protect. As long as a student has a phone that a parent have given them, we don't remove AI. AI is now in their hands by every phone device 202 00:55:24.800 --> 00:55:41.599 that they get. AI is there. So, how do we as parents, how do we as DOE and partners with parents create guard rails that help us to address what is here in our camps in the right way. Period. The 203 00:55:41.599 --> 00:55:58.400 end. >> Well, we're coming up at 7:30. Uh so we our time is our time tonight is is is over. Uh, but I put in the chat and I'll I'll put it in again. Um, if you want to email us at the CEC, uh, I'm going to 204 00:55:58.400 --> 00:56:14.240 put it in again. Uh, cecschools.nyc.gov your comments, further questions. Uh, we will we're happy to, um, route it to Dr. Pate, doc, uh, Miss Carrosa, and the superintendent Maguire. I'm sure he I know he is very concerned and eager also 205 00:56:14.240 --> 00:56:33.760 to hear uh, of all of your comments and concerns and questions. Um, so I'm going to put this in again. CEC2schools.nyc.gov. I have no idea. Okay. I have no idea how I got boldfaced. So maybe AI. I don't know. there if there are if there are any questions not to cut you off and you 206 00:56:33.760 --> 00:56:50.480 know to shift the protocol if there are any like questions that come to you if I would love if you could um either share them with the superintendent myself Tara so that we could add it to the FAQ bank because that's what we've been doing regularly D22 sent us 50 questions and 207 00:56:50.480 --> 00:57:06.799 they're now all a part of the bank so there is no limit we want to get all the questions um you know answered so if there are questions that you have please share them out um with your CC president, with your superintendent, he will get them to us directly and we will add them to our bank and get answers to 208 00:57:06.799 --> 00:57:22.880 them. >> Yes. And and and we recognize that the 45day comment period concluded on f last Friday. That doesn't mean that Tara Dr. Pate is is no longer going to be anything. It's a it's a continuous process. Mr. McGuire, Superintendent 209 00:57:22.880 --> 00:57:38.240 Magcguire as well, it's a continuous process. The DOE does want to hear uh your questions and comments. Uh, so I do want to thank Dr. Pate, Tara, thank you so much for this for this session this evening. Thank you for your time. >> No, thank you. >> I want to thank I want to thank uh 210 00:57:38.240 --> 00:57:55.200 Superintendent Magcguire for for uh joining us this evening uh and all of my my uh CC2 uh council members. And of course, >> you want to remind everyone, Craig, when our next meeting is in case anybody has more questions they want to bring to us as a council? >> Yeah. So, our next meeting because of 211 00:57:55.200 --> 00:58:11.760 because is is uh Wednesday the 26th at 6:30. Uh if you go to our website www.ced2.net, uh meeting information is there. You can join virtually, you can join in person. It's usually on Wednesdays, but that Wednesday is a uh is an off day for the 212 00:58:11.760 --> 00:58:27.839 New York City school system. So, we moved it to uh Tuesday the 26th. Uh I do want to thank all of the uh families, the parents. Uh I think there were some students on here as well for to participating tonight. Uh we know it's uh dinner time for many of us or or uh 213 00:58:27.839 --> 00:58:44.640 you know putting our kids to bed. Um but I do want to thank everybody for their time and perspectives. Uh always good to hear uh all perspectives one way or the other because it's our power is our voice or our voices. So thank you again and uh have a good night everybody. 214 00:58:44.640 --> 00:58:47.440 >> Thank you all.