How do nonprofits make sure AI tools don't exploit vulnerable populations data

A Genuine, Actively Managed Ethical Challenge

Responsible nonprofits work to prevent AI tools from exploiting vulnerable populations’ data through several key practices: clear data governance policies, genuine efforts toward informed consent despite real power imbalances, limits on third-party data sharing, and evaluating tools against the humanitarian principle of doing no harm.

Establishing Clear Data Governance Policies

A foundational safeguard involves establishing clear policies specifying exactly what data is collected from program participants, how that data can and cannot be used, how long it’s retained, and who within (and potentially outside) the organization has access to it — providing a documented framework that helps prevent data from being used in ways beyond its original, stated purpose.

Genuinely Grappling With the Challenge of Informed Consent

Obtaining truly informed, voluntary consent from vulnerable populations is a genuinely difficult ethical challenge, since individuals dependent on humanitarian aid or nonprofit services may feel implicit pressure to consent to data collection they don’t fully understand, out of concern that declining could affect their access to needed assistance — responsible organizations work to address this by making consent processes as clear and genuinely voluntary as possible, and by ensuring declining consent doesn’t actually affect access to core services where feasible.

Limiting Third-Party Data Sharing

Responsible nonprofits generally limit sharing collected data with third parties — including AI tool vendors themselves — without explicit, well-justified reasons directly connected to the program’s purpose, reducing the risk that vulnerable populations’ data ends up being used in ways beyond what program participants understood or agreed to when their data was originally collected.

Evaluating Tools Against the Humanitarian Principle of Doing No Harm

Beyond simple legal compliance, many humanitarian and nonprofit organizations specifically evaluate whether a given AI tool’s data practices align with the broader humanitarian principle of doing no harm, carefully considering whether a specific data collection or AI tool use could expose vulnerable individuals to risks like data breaches, unwanted surveillance, or unintended harmful use, weighing these risks seriously against the tool’s intended benefits before adoption.

Why This Requires Ongoing Attention, Not a One-Time Policy Decision

Given how quickly AI tools and their underlying data practices can evolve, responsible data governance in this space generally requires ongoing attention and periodic reassessment, rather than being treated as a policy decision made once and then left unexamined as tools, vendors, and organizational practices continue to change over time.

Bottom Line

Responsible nonprofits work to prevent AI tools from exploiting vulnerable populations’ data through clear data governance policies, genuine efforts toward meaningful informed consent despite real power imbalances, limits on third-party data sharing, and evaluating tools against the humanitarian principle of doing no harm — an ongoing ethical responsibility requiring ongoing attention rather than a one-time policy decision.

Go deeper

• What Are Legal Aid Organizations Doing With AI Chatbots?