Department of Education

Authors’ note: For this report, the authors use the definition of artificial intelligence (AI) from the 2020 National Defense Authorization Act, which established the National Artificial Intelligence Initiative.¹ This definition was also used by the 2023 “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.”² Similarly, this report makes repeated reference to “Appendix I: Purposes for Which AI is Presumed to be Safety-Impacting and Rights-Impacting” of the 2024 OMB M-24-10 memo, “Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence.”³

The U.S. Department of Education has several existing statutory authorities to consider in addressing the challenges and opportunities of artificial intelligence (AI). Governing for Impact (GFI) and the Center for American Progress have extensively researched these existing authorities in consultation with numerous subject matter experts. However, the goal is to provoke a generative discussion about the following proposals, rather than outline a definitive executive action agenda. Each potential recommendation will require further vetting before agencies act. Even if additional AI legislation is needed, this menu of potential recommendations to address AI demonstrates that there are more options for agencies to explore beyond their current work and that agencies should immediately utilize existing authorities to address AI.

The Department of Education has already started to consider the risks and opportunities of AI, resulting in its 2023 “Artificial Intelligence and the Future of Teaching and Learning” report⁴ and its various taskings in the 2023 “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.”⁵ Artificial intelligence may be able to provide new kinds of education opportunities,⁶ but it also comes with risks, including those related to student surveillance and discipline, biased plagiarism detection, student loan misinformation, and discrimination in admissions.

The Department of Education’s authorities to address AI include titles VI and IX of the Civil Rights Act, the Americans with Disabilities Act (ADA), and the Higher Education Act (HEA).⁷ These statutes can be used to address impermissible discrimination using AI technology and provide various requirements for contractors servicing student loans.

Equal access to education is paramount to social and economic advancement,⁸ and educators increasingly rely on AI and related technologies to advance access to education.⁹ However, the use of AI in education may also cause unintended harm.¹⁰

There are several areas of potential concern in using AI in education, including:

• Student surveillance and discipline: School software and online systems track various school interactions, performance, and other information.¹¹ In many cases, AI tools, which may train on datasets with racial bias,¹² are then deployed to “predict” future misconduct, track academic performance, and develop personalized learning programs.¹³ The May 2023 Department of Education report, “Artificial Intelligence and the Future of Teaching and Learning,” notes: “Examples of discrimination from algorithmic bias are on the public’s mind, such as … an exam monitoring system that may unfairly identify some groups of students for disciplinary action.”¹⁴ The finalized Office of Management and Budget (OMB) AI guidance issued for federal agencies in March 2024—the OMB M-24-10 memorandum on “Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence”—lists “monitoring students online or in virtual-reality,” “recommending disciplinary interventions,” and “facilitating surveillance (whether online or in-person)” as uses of AI by federal agencies that should be presumed to be rights-impacting.¹⁵ The Center for Democracy and Technology (CDT) found that 88 percent of teachers report their schools use AI-powered software to monitor student activity online, and two-thirds of those teachers report these data are being used to discipline students.¹⁶ The same study found that 40 percent of schools monitor students’ personal devices.¹⁷

  Alarmingly, this surveillance is even more harmful to LGBTQ+, Black, and brown students, who are already overrepresented in incidents of school discipline and law enforcement involvement.¹⁸ Additionally, Black and Latinx students are more likely to use school-issued devices than their peers and, therefore, are more likely to be subjected to increased monitoring and related negative consequences, such as discipline.¹⁹

• Plagiarism detection: Many schools use online platforms or software that purport to detect plagiarism in work submitted by students.²⁰ However, these programs—equipped with AI-detecting capabilities—have been found to falsely conclude that students have used AI tools or otherwise plagiarized their work.²¹ Even when these tools work as intended, they fail to consider the many reasons why students may use assistance when submitting written work, especially as an accommodation for students with disabilities or those for whom English is a second language.²² The OMB M-24-10 AI memo lists “detecting student cheating or plagiarism” as an AI use by federal agencies that should be presumed to be rights-impacting.²³
• Federal student loan misinformation: As the Consumer Financial Protection Bureau (CFPB) has reported, banking institutions increasingly rely on AI-generated chatbots to provide consumers with important information about their products and services.²⁴ Given the changing nature of federal student loan repayment and the on-ramp period for resuming repayment,²⁵ loan servicers’ usage of chatbots, especially any chatbots that utilize generative AI, could expose consumers to misleading information about their federal student loans due to their unreliability from hallucinations.²⁶ Hallucinations are described by the National Institute of Standards and Technology (NIST) as “the production of confidently stated but erroneous or false content.”²⁷
• Undergraduate admissions and scholarship awards: Half of the higher education institutions currently use AI technology to review incoming applications, including initial screenings of transcripts and recommendation letters.²⁸ The OMB M-24-10 AI memo lists “influencing admissions processes,” “determining access to educational resources or programs,” and “determining eligibility for student aid” as uses of AI by federal agencies that should be presumed to be rights-impacting.²⁹ Advocates have highlighted that the unrestrained use of AI in school admissions may replicate and favor past prejudices, as AI programs may take into account the demographics and characteristics of past students and give those same groups preferential treatment, often to the detriment of students whose communities have historically been pushed out of higher education.³⁰ This is especially troublesome given the U.S. Supreme Court’s decisions in Students for Fair Admissions v. Harvard and Students for Fair Admissions v. University of North Carolina, which prohibited the use of race as a factor in college admissions outside of narrow circumstances.³¹

The Department of Education has a unique opportunity to regulate and place safeguards around the responsible use of AI in the classroom. The recent 2023 executive order on AI directs the department to develop policies that will ensure “safe, responsible, and nondiscriminatory” uses of AI in the education context, including by providing for human review of AI determinations where possible and developing an “AI toolkit” for education leaders.³² In May 2023, the department released its “Artificial Intelligence and the Future of Teaching and Learning” report.³³

Notably, the OMB M-24-10 AI memo for agencies identifies several rights-impacting AI use cases in education, including: “detecting student cheating or plagiarism; influencing admissions processes; monitoring students online or in virtual-reality; projecting student progress or outcomes; recommending disciplinary interventions; determining access to educational resources or programs; determining eligibility for student aid or Federal education; or facilitating surveillance (whether online or in-person).”³⁴ It also warns that AI-related “[decisions] blocking, removing, hiding, or limiting the reach of protected speech” may infringe on protected rights.³⁵

Additional ongoing and existing agency initiatives are also likely to protect students and consumers from the potential harms of misaligned or inaccurate AI.³⁶ For example, the Department of Education’s existing guidance relating to the Family Educational Rights and Privacy Act applies if protected records or information are collected, analyzed, or distributed using automated technologies.³⁷ Moreover, a proposed rule to restrict misinformation in the advertising market for secondary degrees should also apply when institutions deploy automated algorithms to target those deceptive ads.³⁸

The Department of Education has already proposed several protections for students that target AI harms, including the many recommendations in the “Artificial Intelligence and the Future of Teaching and Learning” report.³⁹ It has also taken on extensive litigation to enforce its existing protections.⁴⁰ However, the department can take additional actions under its existing statutory authority to address the unique threats to students posed by AI technologies.

Title VI protects individuals from discrimination based on race, color, or national origin in programs or activities that receive federal financial assistance. Title VI states:

No person in the United States shall, on the ground of race, color, or national origin, be excluded from participation in, be denied the benefits of, or be subjected to discrimination under any program or activity receiving Federal financial assistance.⁴¹

Title VI applies to all the operations of a school district or college that receives federal funding, including its admissions, financial aid, recruiting, academic programs, student discipline, grading, class assignment, physical education, athletics, and housing.⁴² Past guidance has clarified that Title VI also covers national origin-minority groups, including when students cannot speak or understand English because of that identity.⁴³

Title IX protects individuals from discrimination based on sex in education programs or activities that receive federal financial assistance. Title IX states:

No person in the United States shall, on the basis of sex, be excluded from participation in, be denied the benefits of, or be subjected to discrimination under any education program or activity receiving Federal financial assistance.⁴⁴

Title IX includes protections from sex discrimination, including: “sexual harassment; the failure to provide equal athletic opportunity; sex-based discrimination in a school’s science, technology, engineering, and math (STEM) courses and programs; and discrimination based on pregnancy.”⁴⁵ An interpretative notice has explicitly recognized sexual orientation and gender identity as a protected class under Title IX.⁴⁶

Section 504 prohibits the discrimination of individuals with disabilities in programs and activities that receive federal financial assistance from the Department of Education. Section 504 provides:

No otherwise qualified individual with a disability in the United States … shall, solely by reason of her or his disability, be excluded from the participation in, be denied the benefits of, or be subjected to discrimination under any program or activity receiving Federal financial assistance.⁴⁷

The Americans with Disabilities Act, as amended, expanded the definition of covered disabilities under Section 504, emphasizing its wide applicability.⁴⁸

The Individuals with Disabilities Education Act (IDEA) provides federal funds to educational agencies to assist students with disabilities.⁴⁹ Each participating school must identify and evaluate students with disabilities and develop individualized educational programs (IEPs), including a statement of the services the school is providing the student.⁵⁰ The IDEA grants the department the authority to promulgate regulations and guidance that govern participating schools.⁵¹

Under Section 504 of the ADA and the IDEA authority, the department has issued several guidance documents, including “dear colleague” letters and technical assistance documents, and has implemented regulations.⁵²

Under 20 U.S.C. § 1087(f), the Department of Education is authorized to enter into contracts with qualified institutions for loan servicers as long as they meet certain criteria, which the secretary can set.⁵³ The department has promulgated several servicing-related regulations under its rulemaking authority under Title IV of the HEA⁵⁴ and has published several guidance documents.⁵⁵

Education is at the front lines of the AI revolution, and the Department of Education is already engaged in examining the challenges and opportunities of AI. Its existing statutory authority across various education levels provides a variety of options to address the potential challenges and opportunities of AI. GFI and CAP hope this chapter will offer thoughtful options to educators, administrators, and, ultimately, students as AI becomes commonplace in the education system.