Evaluating Safety Programming

 Prior sections outline the myriad ways in which the Black and Brown residents engaged in this project think about safety and thriving. The findings provide a framework to consider the interconnectedness of safety and thriving and the ways in which it can be constructed. The indicator domains described in Measuring Safety pertain to the material resources and experiences that residents raised as essential to community safety. Taken together, they present a more expansive understanding of community safety that incorporates socioeconomic factors, neighborhood conditions, and various data from public systems. 

In this section, we discuss the remaining domain of Community Power, which was identified by residents as core to the creation and maintenance of safety and thriving. We think that this domain is best explored in the context of the Mayor’s Action Plan for Neighborhood Safety (MAP), which works to increase neighborhood safety and well-being through the mechanisms of community power. While there are certainly many other ways to consider the measurement of Community Power, we believe that doing so through the evaluation of MAP will provide the most expedient means of using available programmatic data to assess the relative power of particular communities working on neighborhood safety in NYC.

Below we lay out what we know about MAP’s goals, what we heard from residents and staff about their experiences of MAP, and what we recommend as a framework for the evaluation of MAP that incorporates the expanded understanding of safety and thriving presented in this report.

What is MAP? 

MAP is a community-based approach to increasing neighborhood safety at fifteen NYCHA developments across New York City by creating opportunities for residents to identify key issues underlying crime and participate in the decision-making to address these priorities. MAP’s vision is to co-create safe and thriving neighborhoods in places impacted by historic disinvestment where residents determine how government best serves their needs and priorities to address root causes of crime and proactively build safety.

The core strategy of MAP is to enlist residents to identify and address the factors underlying safety in their communities. They do this primarily through NeighborhoodStat (Nstat), an extensive participatory problem-solving process, where development-specific resident teams engage their communities in the articulation of priorities and the implementation of solutions, in coordination with City agencies and community-based partners. Solutions get turned into projects that are funded by MAP and can differ across developments, ranging from physical space revitalization to public events and ideas for new programs. For issues that cannot be addressed through local action alone, MAP coordinates a city-wide policy-making process that engages City agency leadership to fill resource gaps and co-develop policies that can address neighborhood conditions. Additionally, MAP coordinates access to existing government resources including more than a dozen programs that connect residents to education, mentorship, and employment opportunities, and builds relationships between residents and the spaces they live in. MAP is committed to empowering residents by giving them resources to organize and access to influence government systems. Critical elements include community organizing staffing in government and in community, a vast network of partnerships with community leaders, community organizations, and City agencies, as well as concrete funding, services, and additional resources to back up the government pledges and carry out the work.

 
Image from iOS (10).png
 

What did we learn about MAP?

This project engaged the staff who design and deliver MAP programming as well as residents who participate in it. In our conversations, we asked staff and residents about their experiences with MAP and how MAP impacts community safety. We sought to understand the elements of MAP that are meaningful to residents and staff, and what might be amplified to increase the efficacy of MAP programming. 

Click the plus to expand

+ MAP builds community and reduces stigma

At its core, MAP is oriented around building resident relationships to create safer communities. Through public events, NStat meetings, space revitalization projects, and programming, MAP creates opportunities for residents to get to know one another and build relationships. Community building does the work of destigmatizing relationships between residents by establishing familiarity and trust. Space revitalization projects are a form of community building that gives residents an opportunity to improve derelict public spaces that are “hot spots” for crime. Residents report that space design projects can eliminate the stigma surrounding parts of the neighborhood by creating spaces that invite recreation and play, and build connections. 

 

+ MAP builds platforms for communities to organize

The participatory processes run by resident teams are perceived as incredibly important to changing people’s relationships to spaces and one another
. They not only do the work of building familiarity and trust, but also provide pathways to space stewardship and civic engagement. MAP engagement coordinators (MECs) and resident team members are often individuals who have a history of community organizing and a desire to make their communities better places. MAP provides them with a platform to amplify the work they’re committed to and enroll others in the definition and execution of it. 

 

+ MAP builds hope around change with government

MAP processes are resident-centered, which counters traditional notions of how the government operates and can contribute to a sense of hope around change. Both staff and residents value the emphasis on resident priority setting, co-design work, and advocacy, and feel that MAP follows through on their commitment to amplifying resident voice. They cite the importance of MAP as an instigator of positive change that can provide pathways for residents to get involved in community initiatives that have the potential to make a difference. By bringing government agencies to the table, MAP provides a more direct and human face to government and exposes residents to the possibility of driving change in government systems. Residents talked about how impactful it is to see a project from inception to completion, even if it is as small as a mural, it builds capacity and faith in the agency of residents. 

 

+ MAP is social work at the neighborhood level

MECs and Borough Coordinators function like neighborhood social workers
, actively identifying resources and helping residents get their needs met
. Formally, this happens through NStat, which facilitates deep outreach into communities to identify and prioritize needs and connect residents to resources. Much of this happens more informally beyond the bounds of their actual responsibilities. It is common for staff to get calls or texts from residents at any time of day with a request for assistance. This includes things like accessing a public benefit, filling out a job application, and finding a daycare. Once a need is identified, staff work hard to problem solve, and coordinate resources and connections with other agencies. For residents, MAP brings government services directly to them and is often letting them know about myriad existing resources that they could get connected to. Additionally, the way that MAP does outreach is high-touch, which was said to be important to reach residents who might not normally seek services and those who are resistant to believing in the value of a service. 

 

+ MAP is a tailored approach

MAP’s commitment to community-led decision-making is evident in how it is implemented across sites. While there are certain similarities between sites, like NStat and access to programming, the implementation of MAP can look quite different from place to place. In part, this is due to the iterative nature of MAP, which evolves to meet resident priorities both within and across sites. Varied implementation is also a consequence of listening to community needs and meeting them with the array of community organizations, services, and programming that make sense for that place. Staff see this as a strength of MAP, despite the administrative challenges in delivering tailored approaches by site. 

 

+ Partner accountability is not given

Staff mentioned the significant challenge of holding other agencies accountable to the priorities and needs surfaced through MAP
. There were many instances relayed where partner agencies fail to follow up or follow through after receiving input from residents. When this happens, Borough Coordinators and MAP leaders do what they can to call and advocate for residents, but there are limits to what staff can achieve from this position. Staff are desirous of tools and mechanisms to encourage deeper accountability with government partners. There is also an appetite for a form of accountability that doesn’t require MAP staff to produce it, instead enabling residents to push agencies directly. 

 

+ Priorities are left on the table

Additionally, sometimes residents' priorities might not be possible to address within the construct of NStat or MAP, particularly if they require significant policy advocacy at the local or state level. There can be tension at times between the desire to be community-driven, and the reality that staff are facilitating and administering a program, which structures and constrains what residents can take on. Staff and participants struggle with the limits to what can be accomplished with the current budgetary and scale limitations of the program, which can often feel like surfacing numerous priorities and being able to act on only a few. Staff and residents expressed a desire to increase resident access to decision-making over government funding beyond MAP and to expand the capacity of MAP to respond to resident priorities.

 

 

What do these findings mean for the evaluation of MAP?

The key takeaway from this project is that safety is far more than incidents of crime and violence. NIS proposed a handful of domains and indicators for measurement that would better represent the safety of a neighborhood. While this may be a new paradigm for many to associate things like economic security and access to public services as safety, our findings align well with the existing approach of MAP. MAP works with an expansive understanding of community safety that actively solicits residents to define their safety priorities across five areas: Economic Stability; Health and Well-being; Physical Space; Safety and Justice; and Youth Development. These categories are closely aligned with the indicator domains developed by NIS:

NIS Community Indicators MAP Priority Areas
Economic Security Economic Stability
Economic Readiness Youth Development
Public Services Health & Well-being
Physical Security Safety & Justice
Built Environment Physical Space
Local Economy ~Physical Space/Economic stability

Given that both NIS and MAP developed these priority areas from participatory community-based processes, it is no surprise that they are similar. While NIS’s indicator domains are meant to measure safety in Black and Brown communities, MAP priority areas are used to organize activities that aim to increase community safety in these communities. How then, might the activities of MAP interact with the measurements of safety proposed by NIS?

Building from a theory of change

For the purposes of evaluation, it is important to articulate how activities or actions intend to create a particular impact. We find it helpful to start with a simple theory of change that outlines the fundamental cause-and-effect relationships of an initiative or system. 

“If we [actions], in order to [goals], we expect [impact].”

Based on documentation shared by the MAP team and our conversations with MAP participants, NIS developed the following statement: 

If we create a governmental response to neighborhood safety that amplifies resident voice and power, in order to direct investments to address resident safety priorities and provide accountability, we expect to create safe communities where residents can thrive. 

From our perspective, this statement provides a scaffold that articulates the core premise of the MAP approach that is unlikely to change: a focus on resident power. This articulation of MAP’s theory of change reflects the very clear message that we heard from residents, MAP staff, and MAP leadership: Black and Brown communities are not safe because they don’t have power.

 

Centering community power in the evaluation of MAP

This framework for directing evaluative work looks at the types of activities that MAP engages residents in to build towards community power and the continuum along which community power can be built over time, with a focus on addressing safety priorities.

Community power is the ability of communities to develop, sustain, and grow an organized base of people who act together through democratic structures to set agendas, shift public narratives, influence who makes decisions, and cultivate relationships of mutual accountability with decision-makers.

Specifically, MAP is designed to engage residents in relationship-strengthening activities (Base Building), skill-building opportunities that increase resident capacity to identify and address problems (Community Capacity), and more formally organize around their priorities and develop solutions (Resident Voice). It also advocates on behalf of residents to make connections to services and increase the responsiveness of agencies to resident concerns (Government Accountability).

In addition, MAP staff and residents expressed a desire to lean into power-building, in order to increase agency responsiveness to resident concerns and develop resident capacity to direct government priorities and investments, within the MAP program and beyond (Exercised Power).

Community power enables MAP to do what it does best, respond to the priorities of residents with a mix of custom programming, services, and activities that make sense for a particular community.

Evaluating the impact of tailored programming cross-site is a very complex endeavor that puts more emphasis on the metrics of a particular program or service. Year-to-year the resources and programming delivered by MAP evolve, but the core activity of building community power does not. Community power may lend itself better to cross-site evaluation, particularly if programmatic variance continues to be the norm. 

Because MAP aims to build the capacity of residents to increase their power in government processes, any evaluation of MAP should be participatory and center resident voice. Community power is not a simple concept and will necessarily look slightly different across MAP sites. This should be leaned into, as residents spoke passionately about the importance of power to the creation of safety. In addition, NIS believes that certain phenomena explored in previous evaluation work, such as social cohesion and collective efficacy, are better positioned within the context of community power, as components of collective action. Within each category below, we recommend places where residents’ perceptions could be engaged to monitor the progress of MAP and direct programmatic improvements.

 

Measuring Community Power

Resident goal: An organized and engaged community that acts together through democratic structures to set agendas, shift public narratives, and cultivate relationships of mutual accountability with governmental decision-makers.

Click on the Indicator categories below to see how Indicators are measured:

+ Base Building

Base building describes activities meant to engage and expand participation and strengthen relationships. These indicators measure participation across MAP activities that engage residents in participatory processes and events.

Indicator Indicator Measurement
Resident footprint Ratio of residents engaged in MAP activities / the number of residents eligible to participate
Attendance # of unique participants in MAP activities
Return engagement Rate of returning participants in MAP activities
 

+ Community Capacity

The cultivation and use of knowledge and skills that can contribute to resident capacity to identify and address safety problems. These indicators focus on MAP skill-building activities and resident engagement over time. 

Indicator Indicator Measurement
Capacity building # of participants in workshops and skill-building opportunities
Policy working group attendance # of residents who participated in policy working groups
Resident facilitation # of resident-led public meetings
Resident stakeholder team attendance Rate of attendance in stakeholder meetings for resident stakeholder members
Resident stakeholder team tenure Avg tenure of resident stakeholder team members
Resident + staff perceptions of capacity Measurement of capacity over time. A survey is recommended. See here for potential approaches.
Resident stakeholder team attendance Do residents feel like they have the skills and knowledge to participate?
 

+ Resident Voice

The ability for residents to formally articulate safety issues and plans for consideration. These indicators track MAP’s process for engaging residents to articulate issues and plans. 

Indicator Indicator Measurement
Action plan creation # of action plans
Issues identification # issues identified through participatory processes
Idea generation # of ideas contributed in participatory processes
Investment decisions # of decisions made about the investment of public funds
Resident perception of process Do residents see their needs and solutions reflected in the action plans
 

+ Government Accountability

Government responsiveness to individual needs through service connections, as well as to the community-level issues raised by residents. These indicators reflect the activities that MAP undertakes to make the government more accountable, as well as the responsiveness of government agencies to the issues that residents raise through MAP.

Indicator Indicator measurement
Agency responsiveness to service connections % of complaints addressed (by agency)
Progress on resident-identified priorities % service connections facilitated by MAP that result in enrollment in benefit/assistance (by agency or program)

% service connections that result in referral to a program that’s a better fit (by agency or program)
Progress on resident-identified priorities Have the appropriate government agencies taken action in response to the priorities and issues identified (in action plans and other forums)?
Action plans implemented # of dollars available for resident investment (through grants and contracts)
Investment in resident projects % of desired policy/need areas where MAP has agency connections
MAP relationship building Percentage of households with a computer of any kind (desktops, laptops, tablets, or smartphones)
MAP staff perception of agency relationships Does MAP have the agency relationships necessary to respond to resident needs?
Computer access Were resident needs addressed through service enrollment?

What needs are not being addressed?
 

+ Exercised Power

The ability for residents to see their recommendations implemented, and to direct government investments and programmatic and policy decisions. These indicators track the longer-term impact of MAP’s work in supporting the development of community power.

Indicator Indicator measurement
Resident perception of power within MAP Do residents have the leadership opportunities they want within MAP programs?
Resident perception of participatory governance impact What are the impacts of participatory governance on their community?
Resident perception of program influence What action steps has the government taken to change programs or create new programs based on their priorities and guidance? (Regular tracking.)

Have programmatic changes been made or new programs created based on their priorities and guidance? (Annual tracking.)

Programmatic priorities from the conversations we had are laid out in Economic Readiness, Public Services, and Built Environment indicator categories.
Resident perception of policy-making power What action steps has the government taken to change policy or create new policies based on their priorities and guidance? (Regular tracking.)

Have policy changes been made or new policies created based on their priorities and guidance? (Annual tracking.)

Shorter-term policy priorities from the conversations we had are laid out in Physical Security and Local Economy indicator categories.

Longer-term policy priorities are laid out in the Economic Security indicator category.

Previous Section

Measuring Safety