WEBVTT

METADATA
Video-Count: 1
Video-1: https://stpaul.granicus.com/player/clip/5931?view_id=37&redirect=true

Part: 1

1
00:00:01.701 --> 00:03:35.450
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

2
00:03:48.600 --> 00:07:09.468
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? >> Saint Paul City Council out of committee to copies. CN Coleman Johnson committee member door. Committee member last year here, committee member McVeigh. Committee member Donnelly here. Chairmaker here. welcome everyone to our committee meeting summer. >> And we have a packed agenda today. We have 2 items. One is the Avenue semi final report from our last committee pending are thoughts and to potential revisions. And then we have we're moving straight

3
00:07:13.541 --> 00:08:27.279
into topic review for our next topic. So we just never never take a moment pause here at audit committee. So I will welcome to Brett to take us away. And like you were just saying, I think we're going to try to get. 30 minutes. 30 minutes. We so we can do. Fantastic. Look, thanks, everybody, and appreciate your patience with us. As it relates to our timeline, so I have a couple of intentions today. I'm going to do this sort of walk through the presentation that you had attached in the agenda. >> I'm happy to take questions as part of the presentation have take broader questions. And then I'm just to continue sorry. Continue that feedback. Process written feedback that you want to offer through. The 2 us will continue to take that knee has invited us today, too. Have some specific conversations with her is Greg and with cold and with some of the attorney, team so that we can get some additional feedback as it relates to the reports that we're still striving for a June 30. Final version. But I think if we need a fish that because we're still receiving some feedback to you all, if you're like, you know what, I can't like the 19th, but I can make the, you know, insert date. That's between the 19th in the 30th

4
00:08:29.282 --> 00:09:40.552
will continue to take that feedback and do that final pass. To make sure what you get is the final edited version of of what we've got available. And as you know, since we're in long-term relationship with each other, we are here and around. So happy to continue to do additional work that you So. And maybe I'll just open by saying a giant. Thank you to as always and a giant. Thank who spent a lot of time with us walking through the got a system, spent a lot of time with us back and forth as it relates to folks that we might survey did a lot of back and forth of there as it relates to just city rules about getting surveys inside the walls of the city system. so I think just want to say that. Incredible. Thank you. You are fantastic team for us to work with. So. >> So I'm going get started. But I will say, you know, pause me along the way as we as we making our way through this. All right. So a little bit about that kind of put just a quick reminder. Those research questions, the method. So the kinds of things that we did to amass all of the pieces and parts for this. We'll talk a little bit about that. Gov Q a platform itself. Some of the things that we know from taking a look at that platform and taking a look at that data. We're going

5
00:09:41.880 --> 00:10:48.860
to talk a little bit aout those conversations with the staff conversations with here, jurisdictions and also sort of other folks leaving Minnesota City's office of the state auditor. >> The data practices office. We'll talk a little bit about the data we have from the surveys that we did of department leads and of Coral users. kind of the key findings and recommendations that we have and then just invite discussion. And I think close by saying as always, we look to the audit committee to and pull off draft recommendations. Del do that stuff you saw we had a pretty light executive summary. >> With a set of kind of a major recommendations and kind of the back of the report. But again, look to you all to say, move this from a justice. We'd like to see this here. So relieved to have looking for your. If you're directions that you're really setting the mark for what you expect, the the broader city team is going to do as sort of a response to this. you all know this. taking a look at that public data request process. We really had 3 kind of big questions. >> We have lots of sub specific questions, but kind of the 3 overarching guiding questions. Has it operating overall process with the

6
00:10:50.557 --> 00:12:00.959
experience of the public with that process and what are the kinds of opportunities that exist to strengthen that process overall? so we did lots of things. We touched loss of lots of pieces and parts. if you remember, we had a slightly larger scope. We've sort took a look at sort of managing the scope so that we could make sure that we're really sort of considering things that would hope. Informant advise the city as it relates to next steps here. So first thing we did was took a look at as many documents as Nia Greg possibly find and sent to We took a look at policies, training manuals, the internal workflows and some model guidance from the state's data practices office. did a literature review as we typically do. That's kind of looking at peer reviewed literature. It's Great literature. Kind of what's the best practice information we have out there. We did have conversations with folks from Hennepin, Minneapolis, Duluth, State Auditor's office, the League of Minnesota Cities and the data practices office. We also have information from you, the websites or the online publicly available sites from all of the jurisdictions that you asked us to take a look at. But those are the folks that we actually virtual interviews with. We did survey

7
00:12:04.698 --> 00:13:11.905
of the department leads and portal users and then >> we had a bunch of your Gov Q a data that me and Greg pulled for us and we took a look at that at a really high level. >> If you'll recall early on, we debated how deep we were going to go inside that data. And so really trying to keep that sort of data snapshot at a high level. Really? Just take a look at turnaround time. What is that distribution look like? Volume those kinds of high-level questions that you are primarily interested in. And while that's a lot of data, 75,000 plus requests over this time period. So all right, let's talk a little bit about just kind of the current process and the key data. So just the platform itself. >> Q A is the city's kind of platform for receiving tracking managing all of those requests cost across all the departments. Right? So it is what it's the backbone that the city staff are using its key capabilities. It's got that public request portal. So folks can sort of submit 4 requests. It's got the ability to be able to rout requests across multiple departments. It's got what are called parent child request so I can attach a big request. A lot of baby requests sort of keep those all together.

8
00:13:16.579 --> 00:14:30.184
I can manage documents. I can think about reduction. It's got some dashboards, some dashboards that came kind of out of the box and then Sun reported. That in particular, some of your department's really had kind of built out in particular for some of their needs. The city uses what we would call a decentralized model. So common backbone, the kind of common in taken tracking system. But really each department is doing the things that they need to do to be able to respond to requests in the ways that if it's the best for their particular department. And so you'll see a little bit in the survey, what that distribution of behavior looks like across those departments. One thing you'll see in the report, we did do a comparison just of capabilities. You'll see a grid in with next to request and just FOIA, always things kind of funny name just for you. a I'm the kind of the other the other kinds tracking systems sort of management systems that are out there for possible consideration. So we do make a recommendation that if at some point it seems warranted, the city might consider examining alternative or possible systems to use. These are at least a couple of the systems that are the best in class on the market. So we

9
00:14:32.540 --> 00:15:48.462
just did a comparison of those sort of basic capabilities and we're close. But I think just a caution across all of these all, you know, they really are about record of tracking and management. They are not. I'm going to go into your system and pull stuff out and retrieve it. That is still a profoundly manual process. That is still something where staff are searching through emails and folders and documents department databases. And that would happen regardless of the system choice that you made and for all kinds of reasons, especially from a security manager point that probably makes sense. But I do think it's considering that it is inevitably going to a set of hours labor and expectation for city staff to have to navigate as they were managing through what these requests look like. There is no automated fix that is going go in and find all this kind of stuff and pull it back. It just you know, so really isn't isn't available, at least through the mechanisms that are out there for public data requests tracking and management. >> So a little bit about volume. So, you know, we pull back all of those srt of 75,000 plus, we look to 22 through. 25 25 data was not the full year of data. And, you know, it often takes a

10
00:15:50.399 --> 00:17:06.700
while to wrap up the previous year's request. So we're going to see a little bit of a sort of lag experience as it relates to that 2025 data. >> So I think a few key notes growth year-over-year. Yes, you actually are seeing requests come through. So I think the stress that the staff are feeling is a real stress. So that volume of requests has to stop. >> You've got an average closed time that actually has improved over time from kind of a 2 day average to about a 12 day average from 22 to 24. But you have got a really wide range for what it actually takes for folks to actually be able to accomplish that close. So you've got a median of 4 days. Lots of stuff is pretty quick. And then you've got some requests that are lots and lots and lots and lots of days, right? So you really do have a volume of complicated requests that are taking the staff a pretty significant period of time to make their way through. So I would say that the data are distributed, not in a normal fashion. If we were to think about kind of you know, bell curve distribution. >> Got about half a percent of sort of open requests. >> Meaning you're closing really, really, really well. So there's not a big volume of those I suspect you're hearing a lot about those opens. But

11
00:17:11.800 --> 00:18:24.885
in the volume of you're not seeing that giant open just yet. Yeah, the question is, questions I can jump in head? That is a shocking increase over 3 years. Yes. Are. We is that comparable to the increase the other cities are seeing do we have any idea of what's driving that? >> So we don't have specific volume information from the other jurisdictions. But when we did have those conversations, you'll see a little bit here and a little bit of the report. Everybody is reporting that pretty significant uptick overtime, right? Like just increasing appetite for access to >> public available data every place and Audrey was the one doing those jurisdiction calls. Every place was like, yeah, we're watching the volume. Just jump, jump jump. So it is startling. Think when we finally cracked open, look at the data, looked at the data we were. We were surprised, but it was confirmed by the jurisdictions that we talked to help. Everybody is seeing that that increase. >> Can't ask poll question. Yeah, which is I was sitting with a 2022 as random start date for looking at this. Do we have And maybe there's candidate. Can we can. was They because digits dring

12
00:18:39.914 --> 00:19:56.989
COVID face our lives. As you recall, was a working remotely. So there was a lot of interest about connecting with other jurisdiction. >> And that portal at a lot of interest. So we thought it was it's good. Starting point. Interesting. thank you. We go back. >> Yeah. And you were fully fully online and you a right few were done on ramp period, right? You are fully stable inside a to to Yeah. Yes. So you've got sort of a subset of departments with kind of higher open request rates. And I think it's because simply because of the complexity of what's coming through. >> But again, we're looking at high level count and volume information we intentionally we're not looking at the detail inside with those requests were and that was really an agreement that we made as it related to our scope. Initially. >> As you can expect, police department is the lion's share. >> The of the requests safety and inspections next site administrator that there's sort of which department and then which type of requests are coming through. So there's sort of a couple of ways that we think about those variables. just want make sure reading this right to the police from is 95% of all

13
00:20:01.610 --> 00:21:17.870
requests. And the next highest is 2.2%. >> Yeah. Yeah. So it's because of lions share. year. >> Lion's share everyone. Okay. Yeah. Yeah. You have seen in your human rights. You're a trio area. Some requests that have sort of grown year every year. >> So that's a bit of a Yeah, I just kind of the like, like I alluded to that previous like you've got some complexity and some of the non police requests that are coming through, which likely result in some of that longer close time. So I think kind of the the take away of the population perspective improvements are going to have especially improvements related to like police response are going to have the broadest impact because they are again, the lion, lion share. But again, that citywide consistency and you're going to see a little bit of that threaded through the recommendations. And we heard that and some of the department feedback. It's gonna matter across all of the department's right sort of got one Big Department. But given that you're seeing this volume increase, you know, kind of year-over-year, I suspect, you know, you're going to start to see that the 8 across all of your departments, especially with those departments that have got those complex

14
00:21:17.938 --> 00:22:30.475
requests. >> So let's talk about talking to folks. So just from a staff in effective scuffed you as a tool effective for tracking routing communicating with requesters kind of keeping that documentation. It's got some advantages. As I said before, it doesn't go do the job for you. It doesn't that which we've the records, it doesn't. Search email shared drives all the kinds of things that really is the lion share of the labor. But the staff are pouring into and then repositories that are kind of outside the platform. >> So repository that allows you to communicate with the requester. Does a few minutes. We'll have some flexibility and there are departments that have done a little bit of work flow work, particularly police have done some work for work. But that does mean that you've got some departments that have done some of that and some departments that haven't done some of that. Some departments that certainly some departments that they have other ways. He just got some inconsistent workflows. >> Across those departments so and then I think the age old, it's got a lot of cool stuff to do. >> But if you don't use it regularly or have the time to dig into it, you're just not gonna be able to unleash all

15
00:22:31.879 --> 00:23:54.760
of those capabilities. So sort underutilized power of the tool yeah, that that exists. Let's see kind of challenges across the system. again, manual. >> Staff have to do a lot of interpretation. Routing splitting of requests and a lot of stuff that comes in centrally is broken apart into different pieces and rerouted. And that's lot of different ways. Those email requests are really resource intensive. So right. I want all the emails related to right. That is a lot of volume of information that's got to be sort of pulled back and processed by staff in order to be able to be a plated. And I think there was a sense of what we've watched all this volume increase and we have not had a whole lot of increase in staff to be able to manage process through this through this volume. So I think that, you know, data practices act. I think folks feel like still is highly interpret abull. It is not a 0, 1, black and a sort of set of expectations rules. And so I think just that that is something around which folks are so the feeling and then I think there's this sense that like folks who are putting in request don't appreciate the amount of time that it takes for city staff actually to be able to respond

16
00:23:56.960 --> 00:25:10.186
to those and what we want folks to have high expectations of of the city, a turnaround time, the volume of hours that it takes to put to do that response and to do that close as pretty substantial. And so that that's where the move over here. >> That sort to disproportionate sort of matching reality I think is is sort of a challenge for folks. >> insights from the pier jurisdictions. So I think again, back to this question, consistent themes. The question growing in volume and complexity everywhere, especially those e-mail intensive Those gve me all of the e-mails and serve the rest of the sentence here. reinforcing technology helps with that tracking. But most of the work is done outside of the system. Somebody said 90% of the work in their particular area. So an jurisdictions are primarily using Gov. Q A other. There are jurisdictions that are Excel spreadsheets to track. Yeah. Clear written procedures to find staff roles, regular training. >> From peer jurisdictions, kind of the things that are really good differentiators about being able to kind of open up the process, make sure you're navigating sort of effective and efficient programs. That proactive

17
00:25:14.393 --> 00:26:33.736
disclosure that we've talked about So stuff that could be released public proactively being released public so that folks are repeatedly asking for it. It's something that Kim sort of head off at the pass a little bit of this workload and just improve that public access more broadly. And I think just broadly say post challenges are not unique. So this isn't a particularly exceptional set of experiences. This is a set of experiences that that peer jurisdictions are happening. And kind of thinking about that structure intake. How are folks putting in the request and heard a magic managing workflows and department expectations. Probably 2 of the things that would help sort of unable just some additional efficiencies. So that's something that we heard from from the pier jurisdictions. All right. A little bit of sort of survey feedbacks against 13 department leads. Give us some feedback. About two-thirds of folks are, you know, kind of users for at least 2 years. Most folks. Art. >> Somewhere in that kind of like logging and weekly, they may be logging in less Most of the folks that were responding or processing requests themselves. And then they are also assigning request other folks. Right? So it's a little

18
00:26:34.473 --> 00:27:47.760
bit of a both that's happening. If they have direct rate, the functionality of Q A about have said somewhat functional about a quarter, very functional about a quarter. you know, not functional. So to have a distribution, the things that folks found difficult understanding what's really needed in order to be able to fulfill a request, kind of unpacking that and having that figure that out. thinking about deadlines and closing requests. So sort of just being accountable to that constantly. That constant list TikTok you that folks are trying to manage and then kind of running the reports to try to figure out should be looking at reports that are to be able to manage this. Right. Look, I think there's sort of a variable used across departments of kind of that reporting capability. would help? Guidance on best practices and we're close. What should I be doing? Request intake form. So again, your public portal doing some work kind camp amping up are elevating that being that much clear for folks to put information and so that it's easier when it shows up in the system for city staff to respond to automation. Some of those kind of features saying if some of that could be enabled where possible and then just kind of training

19
00:27:48.813 --> 00:29:01.550
that was definitely this like I got trained. And I could probably use aother train. I'm kind of kind of said a few back. And I think just kind of a cross department communication and figuring out who's responsible when request he's got multiple parts. That spans multiple departments is just complicated and everybody was like that's just hard right We're crossing departments. It's just going to be it's going to be noisy. It's going to be tricky. And that's going to be hard part because we're gonna go back and forth and back and back and forth. The I feel like I'm talking about like a day in the life show. First. Again. So and we talked about this last time. We surveyed a whole lot of portal users. We did not hear from the tunnel portal users. But what we did here is here. >> So a lot of thinks it was easy to do which department to submit a request most folks found getting into that portal. Pretty easy to do. Lots of folks that it was accessible in the preferred language. I will comment that that's probably biased set of humans who are responding to the survey. But, you if we have the data that we have. >> About three-quarters said filling out that request was the at about a quarter said, you know what, it's somewhat difficult or it's pretty

20
00:29:02.353 --> 00:30:10.860
difficult. Improvements. And again, we offered to folks the option to select particular improvement. Said this was not coming up with my own idea. This was rally in partnership with Greg and the outlets being really thoughtful about what kinds of improvement improvements we're going to offer for folks to select so folks really want to know about teir request status next steps, right? So some clear information about that. Instructions and examples of what to include. So what I put in a request, what should I be putting in? I'd love a little bit more insight about that department rolls. So there's a little bit of kind of picking department, but like understanding what it is that the department's to will say that's a little bit of if you'll recall the last project that we did inside out city speak versus folks who are coming to that portal and may not be as familiar with departments and all of the things that are really kind of insider lnguage courses, everybody across the city language and then just kind of like simplify what that request form looks like and some links to publicly available data. You can hear that theme over and over. If I don't have to ask you for it would be great if I could just have access to it his home. >> Sorry from the question.

21
00:30:12.557 --> 00:31:30.767
I >> So surprised given that 95%, their quest coming are for the police department stress that 29% of our request or Extrapolating little bit. But don't necessarily feel clear about which department dishes submitting to what To me like it was likely the obvious, if your screen to the police department or not, is it 95 do all suspect understand that we don't have a ton of respondents here. That 95% people are like very clear at the police department and that 5% that are like is this dsi is guy is a public works or do you think people are like is this dsi or the police department that I should be submitted this request too? I suspect I mean, I suspect a couple things are going on, right? I suspect it really is that like 5% are like I am really not sure where this is sort of going to go. I also suspect and we sort of had a sense for this like you've got some repeated requesters and you have some repeated with questions that are not about police there about other things. And so I suspect we also heard from those folks in this right since we had such a low response rate on folks who wanted to respond responded to this. Right. And so I suspect that >> Not a uniformly distributed side of humans who are who are

22
00:31:33.772 --> 00:32:55.185
responding here. So a little bit of a boost. and that response. So, yeah. All right. So some recommendations. >> All First one, which I think again, we heard a big better use of that go up, come a tool to there's some step inside of it. kind of a hammy like actually enabling some of its functionality. Leveraging those features through training, sure practices crossed a partner of you doing that stuff. So making the best use of the current platform. First. >> And you'll know at the before exploring the possibility o other platforms. Right? So it's kind of like make sure you really maximizing use of what you've got hands on. Coming back around that structured intake Really unpacking with that online portal looks like, you know, are there some things that you can revive public request more specific information? Would you say that back and forth clarification, Burton, rerouting. thinking about kind of that that front and that input peace, citywide procedures. Again, you're sort of let every department make a set of decisions for themselves about what makes sense as it relates to that management. And so really considering the possibility of say, you know what, across the board, we're going to have

23
00:32:57.322 --> 00:34:04.721
some consistency. S we're going to roll the same way. We're gonna document the same way. We're going to communicate same way and we've got those lots of pieces and parts either parent and child are both tied up case. We're going to behave in the same fashion with manage those because right now it's really dependent upon who's kind of the primary after for the first start. And then that is what dictates how the rest of that request to stop Staff training and support regular training, some refresher sessions, some peer learning what you do have lots of folks that are touching the system. Each of whom have some expertise and the thing they've dug into some really letting those folks actually get exposed to each other and appear fashion just a bill that consistency to build that confidence. So again, we did hear from folks who are like, yeah, I got a train to the city the 1st half hour and then been on my own sense OK, so. >> So just a chance to be able to come back around to really create that lie, almost community of practice for folks who are using which may be helpful. Taking a look at that reporting data pretty regularly. So and I don't have a sense for how frequently you're taking a look that. Just that look,

24
00:34:06.191 --> 00:35:26.202
where are we at where the bottlenecks, where we moving things who's doing well and can we learn from who seems to be stuck in having longer close times and where do we need to potentially lean and offer some additional support or some additional improvement opportunities? Back to the stuff folks are requesting frequently make that easier to find ed. And I think that's both publicly making it easier to find as well as internally for the Internal Department staff. So within departments, stuff that we're going to ask for a lot across departments, stuff that need ask for a line if it's possible to make those easier to access inside the walls or upon. You know, quite literally the city's website. So that staff are just not having to respond to some of the same tings. And then I would say periodically, take a look at whether or not technology is meeting the city's so if you're volume and requests the same way, they have grown over the next year, too. Then it may be, you know, at some point, a place where you're like, alright, we want to consider an alternative to go Q A from a system vantage point. Let's see public casing information. So in addition to the portal itself, just sme of that like again, and I think me and Greg already knew this when they were starting think through what kind of

25
00:35:27.272 --> 00:36:39.409
questions we were thinking about asking users. Think about amping up that FAQ, thinking about some more guidance materials and thinking about just kind of status communications. Just so folks know what is happening, right? Because I think a lot of the feedback you're getting is just don't know what's happening. So just some expectations about what that communication patterns look like. And then there's enough further down the you know, kind of the can you allow so routing based upon able that inside the system? Can you think about mre proactive data that's really strikes home more and more and more of that disclosure that happens without having to help folks put in requests. >> And then I know again, you're thinking about to practice this stuff, right? So what does it mean to have that? So expectation and what load are they going pull off of those department heads? What is that balance going Frank? So that is the story that we have so far. You have a big report to make your way through with lots of data you're also going to get technical appendix, which were assembling, which is essentially everything that peer jurisdictions gave us that they said it was OK for us to share. And then we're just going to pull a lot of

26
00:36:42.481 --> 00:37:59.280
stuff from the state just to make that available inside that technical appendix. And you can see the documents that are going to be in there in the back of this existing support so that we really want pay attention to the so the primary data collection in the existing report itself. and again, happy to take feedback through Friday. Having takes you back after Friday and we are going to have some sit-downs with some of the team. City attorney staff just to get some additional feedback about things that we should be as it relates to the report. >> Well, a lot of food for thought. Thank you, Director Brett. I am curious to see we want to make sure we have enough time for next conversation. But I want open it up to my colleagues in the committee just to see what questions or thoughts people have as Eunice Johnson Dyson is common. >> Well, thank Well, thank you for kind of getting this to us so we can look at it. And I think from a procedural point, I just have question around, how far in the weeds who we, too, are able to get since 95% of the work happens outside of the. Q system itself like how far and we think it's were you able to get with just understanding what happens after the requests come peace? >> Yeah, yeah. So what you're

27
00:38:02.234 --> 00:39:17.107
reading in the report is what we were able to get into We'd manage point. will say kind of sort of the cracking open. Those individual requests and seeing what they look like from a level down from a detailed vantage point is not something that we did and it was a move that we did on purpose, right? We really were kind just trying to look at high-level volume types trends, that kind So, but I think the you know, we interviewed Greg and then we you know, sort of the department's survey so you can get a sense for, you know, I'm telling you most of the work that we're doing is outside of that outside of that system. So what you're seeing is about the deaths that we've got to OK without frankly, of packing every single one of those 75,000 requests at a level down which we really intentionally setting a boundary around at least right now. This may be something that in the future you say, you know what, in 2 years we'd like to look at this again. Maybe go a little deeper. But you're in a particular. Scenario right now where we put that. Okay. Thank you so much. great question. Thank you. also offer the thanks. It's obvious that this is something that's taking up a lot of the time and resources. So having just more insight into the process and

28
00:39:19.244 --> 00:40:30.447
what's working in this super helpful your last answer might have and the answer to this question as well. But in this case, further Graham and Ation to make frequently requested information easier to find and looking seeing that. >> You know, next step for the city might be figuring out what that information is. This strictly rquested figuring out if we could and how he would make it public. >> You have anything else to share that or anything else that came up in urals research has to. Anything that will bring up consistent near that. Other jurisdictions mentioned they have proactively made public in order to try friends of the that's a Let me talk to about whether or not we heard that in interviews. I think we heard it at high sort of like general as much as you can make public made public. >> That's what they like. Best practice literature says lucky to make public public. But we'll take a look at those interviews again to see if there's some specificity about like, hey, here's what we made public. Yes, so what we can take a look at that. And then I will say that potentially is again, where we'd pull open those requests probably a qualitative analysis of those requests. We might be able to show something like again. We've put some edges around this, but that is a future

29
00:40:33.585 --> 00:42:06.750
possibility. If you were to say, hey, let's take a look at a random sample so that we can get a sense for repeated requests, maybe generate a possible qualitative list of most frequently. What of that could be public. that could not be public. Thank you, Mr. >> I'm just curious in terms of the 75,000. You weren't able determine if they were discreet. 75,000 his home could have been repeat. Requests could you tell if they were? Individual? >> And they were I mean, it's 75,000 individual mean they could have that they could have been repeat Somebody could have yet coming and we didn't. You have enough information to be able to say we're going duplicate based upon anything, right? Because we didn't know what we didn't pull I who requested, right? We're really trying to sort of >> only have a sense for this sort of straight out right? but at least from the staff vantage point logged into the system. >> It shows up as if it is 75,000. Plus. Individual may be not independent but individual requests come in through and could you tell if they peak times? >> We have 10 there. Yeah, I don't think we pulled 10 stamp, but that's a great. Okay. Time stamp. Yeah. Yeah.

30
00:42:11.616 --> 00:43:30.974
Yeah. >> And so getting at time of year, nearly a year. are what was happening? In the world? Might be. And so what they were requesting particularly since is related things like to the police. But curious if case up and Evan Yeah. Yeah. Yeah. We really just going give flag on year right inside that. Oh, yeah. And again, trying to be. >> I think careful about what we were. What we were allowed to take a look and make available publicly. But that's also a great, perhaps additional like. >> There's some time stamp questions, some qualitative. You know, sort of repeated public. And I sort of ask kinds of questions that we can maybe do a next level down analysis. Yeah. Thank you. >> I need to wrap year knowing that there may be other questions that really encourage my colleagues on the committee to reach out to Dr and between now and Friday. Send your feedback particular things that you would love to see presented when this comes to the council. Like additional questions you want to see dug into and doctor. But I also think that would be a good idea going forward to have now that we have such a strong partnership with our administration to have you come into a presentation with maybe the the council members

31
00:43:34.379 --> 00:45:18.348
and the mayor and administration after presented to the council. So and maybe any relevant members of her staff sums think about that now want Thank you so much. going repress a body of work. I have a ton of questions not asking right now because and taking some time. But thank Thank you. Thank you. Thank you. With that, we're going and we look forward to having you back to council. Other. So we'll see what everything for today you put your microphone closer to him. I think looking at July yet. So we cordon 8 cells. July was the date. We have a mission discuss, possibly. >> All right. We're going to have it to our next topic, which is new the Edge u N a or over to Kenya. Geysers to us too. Us discuss and rank the score. S I think and what it means that you're muted. Okay. Well, thank you, And Chairmaker as news 4 of us. Matt, I guess you can call our risk Assessment subcommittee to go over the list of proposed there were 5 of them. And I think the the key takeaway >> they're all very good. >> Audit topics. They all scored very highly. On the scoring matrix that we used and so that affirms we have good topics. Any of them would be a good topic for an adit so just leading off with that

32
00:45:22.621 --> 00:47:11.128
and then getting into the details. So first off, I think. What we propose is combining 2 of them, which are the business licensing and building code operation. lot of topics and the reason for combining these is because they're both within dsi. And my understanding is they make use of the system Dsi using a new a system right now. So it might make sense to combine those. And then so then the other topics for college savings Accounts, District Council reform and street maintenance and potholes. So we went through our We're exploring Matrix, which you have copies in front of you. so we have rankings of low medium and high for 6 different. Questions. we had 2 of the projects tie. Well, actually, we had 2 times. So the top 2 scoring topics. We're the dsi business licensing and building code operation, which had mention first. But with the caveat that because of the new dsi system, we recommend maybe that being the last topic did in 2027. So I think what we're looking at is maybe the remainder of 26 and then also looking at calendar year 27 for scheduling. These are topics. And so we kind of look that may be there being 3 slots for timing of putting those in and so that being the case potentially because of this new system, though, the

33
00:47:18.670 --> 00:48:59.235
learning curve getting that all in place that that would take maybe the 3rd slot would be a recommendation. And that scored the highest. tying with that project was the College savings Account project and that one, I think mainly scored highly because of the backing of the mayor's office and the interest doing that. Audit to look at that program, kind of a performance evaluation. Is this program running as intended and so that would be a recommendation. For the next one. So those 2 are tied. And then the final 2 were also tied. But with only one point difference we nearly had a four-way tie. But then the final 2, then we're District Council reform and street maintenance and potholes so in thinking about if we have 3 slots, it might be difficult to. Decide which one of these 4 projects now to dropping guess it would be a choice between district council reform and street maintenance and potholes, at least as it comes from the recommendation from the risk assessments subcommittee. So any questions about what we did, we we met it was last Yeah. So community visors. McVeigh Dilworth and then myself. All right. Well, thank you for the report and thank you for the Probably the most important. Part of our job is to figure out what we actually

34
00:49:03.241 --> 00:50:27.610
want to look at. And this is a huge part of it. So thank you for doing this. Just one clarifying question for Open it up to others. So not seeing that shooting on the business licensing But I'm assuming that all initiated off Questions from a couple things. Our compass. And again, our charter goal is to choose 3 topics today to recommend to the council, but also wnt as usual, make sure that we're doing due diligence with the relevant department directors and staff in each of those topics prior, bring them to the council. It's great to able to speak with the mayor, but we also want to make sure we understand from the departments that, you know, there will be buy into these topics. they want to get down to 3. So I'm looking to folks for questions and comments. Mrk. Thank you. I will start off just by saying how I am that we have these topics because I think that residents really such a wonderful job. >> This audit cycle and I'm really proud present Saint Paul because >> how many people actually came to district council meetings share their ideas and their concerns with us. I think that's really valuable, especially in this day and age There's a lot of Eroding confidence in our public institutions. I think what we do here on this committee is

35
00:50:31.820 --> 00:51:55.698
really important terms of demonstrated that we're all the government accountability and proving public service delivery for the folks who live here in Saint Paul 300,000. so. I have a couple comments but regarding District Council perform again. This is from the district councils themselves from teir staff and also from officers and their boards. And so it's important to keep that in mind that they're going to have some motivation behind one, trying to get But still. They really would like us We would highly encourage us to consider looking they're organization or structure their finances, how they operate to your core services because a lot of them are struggling financially right now. Wake the conversations with some of the executive directors are district councils. They having some difficulties, especially with indirect cost. With pet, still. And I think, you know, I'm talking about Madam Chair. And there's also few It's not just one of them, though. It's actually multiple Bres is concerned. And I will say that with District Council reform. I think given how many that district councils brought this up in terms of the board members and their staff. I think that don't do this within this audit cycle. It's going to. Create some problems

36
00:51:57.702 --> 00:53:23.185
in terms of our relationship as a committee with the district councils going forward in terms of our ability to work with them to bring the public can. And back for future topics simply because they may say like brought this idea, we really wanted to go wth this idea you brought to the full attention, the committee and I. They may just be like we want to really go through this process in the future because they are are eyes and ears on the ground here in Saint Paul. And so that's something keep in mind. I'm not seeing this really needs to be maybe this year 2027, but it's definitely something we need to look at some point whether it's in the next 3 years, if we don't get to do with this year next year want to make committee aware that particular. The other thing is that college savings accounts? I'm sorry, I'm sorry my points and getting cold. So bear with me here getting emotional about the work of the attic. Oh, yeah. the best of us. I'm direct. With the college savings accounts. I know it's a really big of. This is something that is mayor, she is a proponent of this because I know she had a hand in creating this program years ago. I will say this. There is a lot of information out there already about how this. How the program is working.

37
00:53:27.992 --> 00:54:45.201
There's could be more information. >> The problem is we don't have Depending on what comes out from the office of Financial Empowerment, there's different metrics that are using based off of how you frame the question. In terms of like participation and then also lake how many? Because of the fact we don't even automatically roll kids when they're born here in Saint Paul into the program creates difficulties, which the usual because most unit programs are on a max Rowland. But I'm getting out and getting on my wheel house here. >> So that being said, part of me is wondering if that topic. I think there's merit for us to do an audit that I do think that I also wonder if this is almost something that could be evaluated during the budget process as well program. That could be something that had between the mayor. And her administration and the council again, I'm just playing these ideas out there in the sky thinking aloud this point trying to determine what makes the most sense for us as a community be doing and seeing that conversation at this point. So I'm Dan Rambling, chair. Thank you, Mr. Ek they never rambling. Other thoughts, Mr. Fallon, great work by the team. Sincerely like we have. >> For Austin topics. I love

38
00:54:46.804 --> 00:56:01.349
the River X, taking all the information and get it to this point has a real achievement. Thank you so wherever we land, it's a good outcome. Okay. So when I worked in a factory, we do believe that that's and we would sort of use criteria to decide what we would to improve. And one of the he points to be like if we did the lien event on this part of this process, could we extrapolate those findings to different parts and processes so that? What's gined is not just in this one thing. It's over here and that's kind of where I'm at with college town where it's like my counsel to the city is to say if you're just going to look college town and that's the oly lens you're looking at this. It's probably not a ton of value. But if you're going to say we really need to look at the programs across the city, how we fund them, how we evaluate performance. Grant management and we're going to take the findings from college bound. And use it as a basis for improvement across the city. For all our programs, the Knights, you talk about and it's really that's a question whether the city. How your approaching this, what you're thinking about programs. And so that would be my counsel to say, like. Think about if there's a genuine interest to learn about program, effectiveness and integrity

39
00:56:03.353 --> 00:57:26.300
and weather, there's a bandwidth to to take these findings and put it across the city or not. And I think that's some of the deciding factor. Whether you see value in that topic or not. But again, good job here. Tese are all great topics and I think you find value. Going for a in from Thanks. Fallon. >> Thank you. I really appreciate just kind of listening to everyone's feedback. I think and also to folks at took extra time to go through the scoring an actually Laker view the topics and give your thoughts on your roof. Exploring, I think that that's also really important because we have to, you know, kind of see where folks land. I know during the last meeting we talked a little bit over on our own scoring something speak from my vantage point I I think for some of this, I like to think about like the next steps. I'm like what we do with the information that given to us. I'm unclear what the first for the cost savings accounts, peas, I think I'm a little unclear like. Reviewing whether or not work. It's doing what it's intended and then trying to understand where we would go from there. We found that it was or even if we thought it was and and I just so I'm a little like. I'm kind of like what the outcome create and also what the outcomes being. That would be

40
00:57:28.838 --> 00:58:50.510
something that I would really. Have a I think we should if we were to evaluate the college savings account, I think we should have a clear understanding of what that means the findings especially because there is an opportunity, a thing being that believe that it sounds like it was, you know, the mayor's request. I think that would be what I would want to know is like. To what not but kind of 2 wooden. Mike, what exactly are we aiming to look at and what we're looking for? Because when I look at the description, it's like what is what does equity when it comes to like the effectiveness the equity of the program that all of the others like they're very like look at organization, the core functions of the district councils and look at the former infant formulas. For the building code operations. There's also kind of like that. specific specificity that I know where they who they be talking to. Are they going with the program evaluation of just like the economy, efficiency, effectiveness and equity of both college town in college. From like that to me requires a little bit more of a follow-up. So I respect that it had higher scoring and I think to the committee's reviews. But I think I think that's decision that we go with, that's going to have a look needs to have a little bit more questions asked

41
00:58:52.321 --> 01:00:13.667
because I think the description as written. Its objective. Not research base. And that's a little concerning for me. And then also the flip side, I'm just like what the findings are just like. What does that You know, so that is that the program it's tied to multiple funding streams, is, you know. It my eyes. It carries actually a little bit more risk on the operational side on the financial side on compliance and the reputational sigh like to me like those risks matter of what the findings would be. If we're not confident in the finding themselves. So that would be the question that I would ask for probably for the administration. Also for that for for folks that are looking through this topic because just. My brain goes into have multiple external funding sources calling into this as well. And I just would want to make sure that we were saying after success for says, making the Korean a public facing documentation to ask questions and things that nature. That would be the only thing of concern for me of that topic. If we move forward. If that makes sense Thanks for The school then and then I'm going to move us towards. We have to drop a lot. quick proposal for to withdraw us. Appreciate the subcommittee that put this together. It's really helpful. I can to see that al of our

42
01:00:15.738 --> 01:01:27.674
topics are strong and we think could sort meet the basic requirements that we have set out. >> I gt that to my mind. The Dsi one is a no-brainer. As is the sermons potholes for reasons that I think are educated. I last meeting, but those are actions of the city. There are things that we need to very well. There are things that we spend a ton of money on. We hear about them from residents every single day. I think we'd be remiss to not move forward with those who So to my mnd, it sort of leaves the between college savings accounts and District Council reform. It was a I would be a vote for the store formats. The 3rd topic, I think both as a mark of appreciation like MRSA was stating, you know, how much work the district cancels Putin to getting us feedback on this. Sure. Some of the Councilmember Johnson Rice about the college savings accounts. The other one that I just got a lot is the plot of trust or district That's reflected to some extent in our city finance funding of system. But not Lee and Hayley reflection we don't give them very much money little after to be the voice of the community and that they have a great influence in our city and so understanding how they work, how they currently working, how we can support them, how we can take

43
01:01:28.811 --> 01:02:39.826
advantage of quite significant role that they play is to the benefit of everybody in the city. So does a B through that. I would be supportive of moving forward with. Great. Thanks for coming. Well, I think I'm Open it up to kind assuming that's not a formal motion that sort of a proposal. >> To see what others think. It nears some of what we've heard around the table and I'm curious how others would respond to the idea of >> having our 3 B. Dropping cosas accounts essentially and having the other 3 victims that we move forward with. What's your opinion on something curious? I actually. Would rank. I would agree, but because I would rank the first for amateur a little bit differently for college bound. I think it's funny. We started we were interested that topic in part because we were just the mayor was really interested. I think as a result, you want us or sweets fighting off the mayor to be 0 interested. But in some ways because the mayor has expressed interest in evaluating this program. And That has, I think also convey that to her own staff because there is so much data available, I think the likelihood that I think 2 is actually more accurate reading for this one topic has not been studied last 5 years, but

44
01:02:40.895 --> 01:04:06.979
has plans to address the topic. I think it's more likely than not that that will be looked at and I think some of these oher topics, maybe not so much as I had not thought about the responsiveness to the district councils wanting to be studied. I think I've always thought about this topic is something that would not maybe be welcome by them. But as our community engagement chair just a testing, too. And also maybe saw member of the committee at Times the subcommittee. I think that is really that sits well with me. So. I I would support the proposal. just chair in anchor. I think that's a very important point. risk or 18? >> If if it if there are plans in place already to address the cllege town you know, that would change the results of or scoring on that, which we were not aware of the time, but yeah, what I'm hearing around the tables, you know, there's concern about the scope of that engagement. We'd need to be very careful about what that is. I love the suggestion from adviser Donnelly about, you know, could it be looking citywide programs as a whole like best practices? What's working? What doesn't what, you know, Reagan and do similar programs in the future. What lessons learned can be applied to those programs that could make

45
01:04:09.717 --> 01:05:25.570
sense but yeah, the the scoping would be key and if we would move forward with that 2 council mmbers, Johnson's comments, saying what questions we want answered those and be the draft, a haunted objectives yet. What questions are we wanting to answer with the engagement? And that would be keep. He cites a ? >> here. You would agree. and by the way, totally would echo that. The committee, the subcommittee that I'm not sure that the city for sure has plans to address the topic. So it is a little bit of a maybe 2.5 on that one. But I think just given what I've what I've heard the mayor say, feeling like it might be more likely than not. So no were hidden from the The subcommittee by any I also just want to note that the 3 that we're talking about advancing. >> Are all. Pretty hefty. >> And so we're going to have to have some difficult conversations as a group about scoping, for example, District Council reform. Revisit the financing organization core functions of Saint Paul's 17 district councils. So I think have >> to say that we shouldn't advanced the is I think as a group we could make an argument that it's really worth doing all of these in

46
01:05:27.729 --> 01:06:44.671
fall and it just might take more time. And we're talking about phases of each of these. Rather than just tying to take a small bite off of each one. But we're going to talk about that. I doubt that we are going to be, you know, 3 or 4 months study of all of a street maintenance or all of business licensing and building could all of our district councils. So 6 months age? easily see. And obviously we Dr Brad, about that. But I just that is heavy out as we consider take us through 2020 through 2028. Actually potentially. That would mean fewer traffic selection meetings for us. That's good. But still. >> I just want to and I been listening to the conversation and think I think I want to clarify when we had the conversation about college town and that's not mean that I'm trying to say that we should be doing college. Now. And these the description is high level. But the conversation that we had was a little more detail. And it's unfortunate that it didn't. And Im not in but it's unfortunate it didn't make its way to this document to explain kind of what we were thinking because when as I recall, we were talking about like the economy, it's whether it is it sustainable. What is it? You know, so there's just there's different top things

47
01:06:46.741 --> 01:08:16.647
that we're looking at with equity and even. Mr. Fay said. It's not something that if you're born in Saint Paul, you automatically get that's the equity. Because if you have to apply. Inequity. On many levels because there's lot of people packed fact, parents that want to apply. So and it's not because they don't want to they mean that. Gets passed them against, you know, there's different reasons why people don't apply. We have all sorts of scholarships out in the world that people don't apply students on a plan for. So if it's if this is really was what they wanted to do, then they would get it automatically as you were born in Saint Paul Hospital, whatever you get and so then and effectively and efficiently, I think lose. There's a little more depth to the conversation that we had just in terms of what we were thinking. But I also as Mr Downey was talking the thing, you know. I don't know if it's a full audit that needs to happen for this particular that program. I don't know if I'm splitting hairs, but a review of some sort that helps to understand where it is. And if that's something that we can. Talk to the mayor about and then doing that. I think that would be great because I do think there needs to be something that answers

48
01:08:19.386 --> 01:09:26.251
some of these questions that the 2 gentlemen are bringing out at and that we had we had that discussion that if we just say we're not going audit or not have it on the on it moving forward and then I get answer. That's just so first of all So really appreciate that point. I think it's there's there is a middle ground between where auditing and we don't think it needs and consideration whatsoever. >> I wonder if maybe there is a communication that comes from this body, for example, to the mayor, especially given that this topic came up, you in our discussion with her to say. I think that would be actually really good closing. The feedback loop of this is we ended up not to side and out of this topic for these reasons. That don't mean we don't think this program needs And I really appreciate the points Mr. Donnelly made solicitor echoed about and those are likely things that could be. Considered applied to future programs. Current programs deciding whether or not to start or continue existing programs. Not just this one. So. If the committee, if we do decide to not move ahead with that audit, I certainly support some sort of communication to that effect and maybe even and I want to recognize that Doctor Owens is financial current maybe even a

49
01:09:31.858 --> 01:10:49.667
conversation with team as they are. Getting into the role in looking at the program about what they're finding, what their plans are that something that this body can certainly invite and engage in? And we've done that in the past in vain department directors to come talk to us about programs. Separate from auditing as for quick question, Madam Chair, is it possible that we could and work in concert since we have the director now? could be actually work in partnership E perhaps CD review. And they come back to us as a committee after Tate the administration to stir. internal evaluation report back to us and then we can. If say the council needs to make some legislative changes or whatever to the program, I don't know if it's set up and city code or not. And I imagine it must be. Correct me I'm wrong, I don't changes to 30's, be ay reforms to the program after having that kind of dialogue with the administration, if they do their own internal review and then they come back after evaluation to the committee. And what have you? Because I still worth brings valid point. And we did have conversation about, like, for instance, like the fact how the program some of count. I the counter structured person's life, how governor

50
01:10:53.139 --> 01:12:12.817
the section 5 9, all that, too. So was a part of our conversation. I can't call freshman was that we had we had a robust conversation in But 20 minutes, I think now. we question yes, just a clarifying question. I think just like and no the timing for we're picking 3 topics because we feel like we have the timing just completed the topics that we have for life. You we just talk about yet. How many like Howard picking the topics, how many topics and then also just like. Is that because we think that that some them will be able get done in the course of in the course of the 2027. Is that why we're picking 3 at this time? So great question. We We are. We typically complete 2 out the year. >> And rather than spending a lot of tme choosing just one topic when we often have one very close runner-up will be decided, as we're always looking to continuously improve and become more efficient, even as not a committee was that we choose 2 topics at the same time. So we have one selection process that yields to results. One takes about half a year. The other one takes about half a year. We are inevitably running a little behind schedule read something at the start of 2026. Right now, these are becoming sort of the end of 26 beginning at 27

51
01:12:14.221 --> 01:13:37.902
audits. The idea here is to send 3 recommendations to the council at the council, choose to. Which would be the next 2. And as I was saying earlier, maybe not even the next 2 for the next year, but to that might take even longer than that. But that's thing. Okay. I I think with that then my follow-up comments would be. I have a lot of questions left around the college savings accounts and I would hate for it not to be able to be considered. At a later point in time like when it comes to a follow-up conversation, like I have questions, I'm not sure. Typically feel like. Coming up, have a conversation about But I think like. I have a couple questions about just really where you know, to be clear and also. With this particular piece, I think it's a question for me is more at this is around program evaluation of the actual program itself or if it's around auditing what has been done. Like and what the intent of correcting or like taking the information that's given to do something with it. I think that they are distinct of things. And that's actually part of my hesitation around this topic. And I'm supportive of what is being considered. I just I do think that those 3 topics together, like if I had to choose between the pot holes in it,

52
01:13:40.907 --> 01:14:54.249
the U.S. peace, the building culprit piece because those are 2 substance. Because savings accounts for me would make the most sense because I don't think you would take as long as the district outperforms going to be. So I just I'm just those 3 together. And knowing that we're only doing 2, I'll just say there are 3 heavy ones that probably will take a long time like going through a system that's been in place for decades. It's going to so I just realistically don't know if they'll be able I would really valued Dr Brits and put on just what she thinks will be like realistic, tangible pieces. By the time this gets to council because that I mean, I am curious to know how long I think it's going to take to go through 17 systems. Anticipated taking quite a bit of just offer that and be done. But and those will be my follow-up for the administration and for the director of like what would be the goal of this study because they see the description. Hear the conversations that matters to me about whether point you know what? The results with cuts that which may be tabled today. But so I have a crazy suggestion based what you just said, chair, especially because we get to make our own rules in the audit committee interest we could also rather than deciding between these

53
01:14:55.530 --> 01:16:06.455
today we could. >> Do the follow-ups with the department directors that I mentioned before we make a final decision and that could include talking to Dr Britt about each of these topics and getting her estimate of what we're really talking about and that in terms of the scope and timeline and we could then come back in July 2 with all that information and decide based on all of that. What which which are free. Is that something that would help people and making I don't want to rush U.S. unnecessarily and seeing some heads nodding and I'm seeing some. I think process. I mean, as long as we're going to do that Constitution anyway, I think it does make sense to kind of have that by so okay. And also just say that to the extent that the subcommittee talk about things that really weren't reflected here. That you want to share their in an email afterwards or just other considerations about specifically how savings accounts, for example, feel free to do that. Like this does not to be the only record of the conversation of what's happened. It's yeah, Channing, correct. I think we're kind swirling around a new idea, too, that I didn't want to get lost. The idea of maybe willing departments doing a self-assessment that maybe

54
01:16:08.726 --> 01:17:32.941
that could be an option. You know, we could develop the questions kind of like what I was saying before the the audit objectives, questions this committee would like answered about a program and department is willing and, you know, has the capacity they can do the self assessment and report back and that could be deliver ball and then this committee, you reviews decides is that Would this be right for an audit in the future? But maybe that's you know, you said we're innovative committee that can be, you know, something new that we look at as a possibility? maybe sort of an audit light of a self-assessment and then that frees us up to do something else. But still. That story kind of where the tap into topics okay, I like that. Thank you for bringing that back. Holmes. We did not want it jail. Actually. Madam chair with federal departments, what's on sub programs within federal agencies at the Government Accountability Office that give it the atually adopt as a practice here at the audit committee for our city. >> To speak actually have more thorough understanding of things going on departments and regular basis. that provides more information as well. >> So I'm hearing I want emphasize, this is great that we're going you know, all that. We're going to talk to departments and talk to Bret

55
01:17:36.881 --> 01:18:02.971
about these topics. We're also going to raise the possibility of a self-assessment sort of in the guise of where you are you planning on doing this anyway or would you like our support and helping some questions and doing this on your own without outside consultants doing it then we will bring that back in July and we will have a real decision before us at that point. Thanks, everyone. best conversations I went over. I think it's worth it. We are adjourned.