Copyright Baltimore Sun
Leaders in Baltimore City and Baltimore County are proposing a review of artificial intelligence security systems under contract after officers recently pulled their weapons on a teenage student whose bag of Doritos was mistaken for a firearm. In the last two years, school systems in both jurisdictions have signed multimillion-dollar deals for gun detection technology with separate companies: $2.6 million to Omnilert from the county and $5.5 million to Evolv from the city. But after last week’s incident at Kenwood High School in Essex, and purported lawsuits and failures involving Evolv, Baltimore City Councilman Mark Conway said he is “keen to establish better safeguards” for public school students. According to a news release Monday, he will put forward a resolution to convene oversight hearings on the procurement and efficacy of AI surveillance machines. “As we continue to see false positives and false negatives for these AI systems, it is clear we need a more rigorous process of vetting before we subordinate public safety decisions to algorithms,” Conway, chair of the City Council’s Public Safety Committee, said in a statement. In February 2024, the Baltimore City Board of School Commissioners unanimously approved the AI tool for its 26 high schools. The tech contract with Evolv was included in a nine-item funding vote without public input. Conway’s spokesperson Luca Amayo said in an interview Monday that because last year’s contract “went through like a breeze,” there is a need to scrutinize and evaluate AI scanners as a tool. The “level of rigor that should be pursued” with technology that affects the children’s welfare “has not been pursued,” he said. “It’s more about asking questions than indicting the school system,” Amayo said of any potential oversight. With their statement, Conway’s office included a finding by the Federal Trade Commission last year that Evolv made false and deceptive claims about its products’ capabilities when marketing — a practice that purportedly led to a class action lawsuit by its investors. While the FTC’s injunction in federal court did not challenge the efficacy of Evolv’s technology, a settlement required the company to give its 2022 and 2023 education customers the chance to opt out of their contracts. In a statement sent to The Baltimore Sun on Monday, a spokesperson with Evolv pointed out that the FTC did not challenge “the core efficacy of Evolv’s products” or the AI tech behind Evolv’s security system. “Evolv Express® has a proven ability to consistently detect a wide variety of threats, with countless examples of successful threat deterrence, including recent incidents where students were prevented from bringing firearms into schools,” Alexandra Ozerkis, chief marketing officer for Evolv, said. “We stand behind our technology and are proud to partner with over 1,300 schools around the country to help make schools and communities safer. Last year, an Evolv weapons scanner had 118 false positives, detected 12 knives and detected no firearms in 3,000 searches during a pilot program in New York subway stations. The system saw a failure rate of 4.29%. In April, two men were able to bring firearms into Melbourne Cricket Ground in Australia despite Evolv security scanners being used at the time. According to Conway’s office, the City Council is generally limited in its ability to interact with contracts made through the school system. However, Amayo said there is room to regulate when a deal crosses into topics of public safety, and even more room when dictating whether a business can operate in city limits. Andre Riley, executive director of communications for Baltimore City Public Schools, said the school system utilizes human intervention with Evolv to prevent incidents where students could be hurt due to the AI security system. “[False positives] could happen with a regular metal detector, however that’s why we have that human element,” he said. “It’s part of a layered approach that includes human verification to ensure that we are acting appropriately in the best interest of our students.” Last week, Baltimore County Police officers responded to a tip from Kenwood High School administrators based on an Omnilert notification. Officers drew weapons on 16-year-old Taki Allen and his football teammates as they sat outside of the school after practice. In the body cam footage, Baltimore County Police officers are seen detaining Allen and his bag before realizing the mistaken gun was a blue Doritos bag. This is not the first time an Omnilert security system has had a critical error. An Omnilert system failed to detect a weapon used in a shooting at Antioch High School in Nashville this past January. Baltimore County Police supported the responding officers’ actions in a statement, declaring that after the BCPS Safety Team had already canceled the alert, Kenwood High School Principal Kate Smith contacted a school resource officer. “The officers responded appropriately and proportionally based on the information provided at the time,” according to the statement from Baltimore County Police. BCPS Superintendent Myriam Rogers defended the Omnilert error last week, declaring that the system “did what it was supposed to do” despite a student being falsely detained. “It’s truly unfortunate — we never want any student, whether it’s during school hours or not, to be in a situation that is frightening,” Rogers said during a news conference last Thursday. “But it is important for us to remind everyone that the purpose of the system is to keep all of our schools safe, all of our students safe.” Principal Smith said the situation was “upsetting” for Allen and the other students involved, promising to provide counseling for every student involved. Izzy Patoka, Baltimore County District 2 councilman, said he empathizes with Allen’s family after the incident at Kenwood High, calling for more discretion on how, and when, AI is used in school settings. “This is an example of how the office of the Inspector General can help BCPS to improve some of its systems,” Patoka said. “Clearly there is a failure here. I think to a degree the police were caught in the middle of this failure. When a police officer hears that there is a weapon they have to be careful. In my opinion this is a complete failure of the AI system.” Have a news tip? Contact Luke Parker at lparker@baltsun.com, 410-725-6214, on X as @lparkernews or on Signal as @parkerluke.34. Contact Chevall Pryce at cpryce@baltsun.com.
