NYC’s government chatbot is lying about city laws and regulations

When you observe generative AI information in any respect, you are most likely accustomed to LLM chatbots’ tendency to “confabulate” incorrect information whereas presenting that info as authoritatively true. That tendency appears poised to trigger some critical issues now {that a} chatbot run by the New York Metropolis authorities is making up incorrect solutions to some necessary questions of native legislation and municipal coverage.

NYC’s “MyCity” ChatBot launched as a “pilot” program last October. The announcement touted the ChatBot as a means for enterprise house owners to “save … money and time by immediately offering them with actionable and trusted info from greater than 2,000 NYC Enterprise webpages and articles on subjects reminiscent of compliance with codes and rules, accessible enterprise incentives, and greatest practices to keep away from violations and fines.”

However a new report from The Markup and native nonprofit information web site The Metropolis discovered the MyCity chatbot giving dangerously improper details about some fairly fundamental metropolis insurance policies. To quote only one instance, the bot stated that NYC buildings “will not be required to just accept Part 8 vouchers,” when an NYC government info page says clearly that Section 8 housing subsidies are one among many lawful sources of revenue that landlords are required to just accept with out discrimination. The Markup additionally acquired incorrect info in response to chatbot queries relating to employee pay and work hour rules, in addition to industry-specific info like funeral residence pricing.

Additional testing from BlueSky user Kathryn Tewson reveals the MyCity chatbot giving some dangerously improper solutions relating to the therapy of office whistleblowers, in addition to some hilariously bad answers regarding the need to pay rent.

That is going to maintain occurring

The outcome is not too shocking if you happen to dig into the token-based predictive models that energy these sorts of chatbots. MyCity’s Microsoft Azure-powered chatbot makes use of a fancy means of statistical associations throughout hundreds of thousands of tokens to basically guess on the most definitely subsequent phrase in any given sequence, with none actual understanding of the underlying info being conveyed.

That may trigger issues when a single factual reply to a query won’t be mirrored exactly within the coaching information. In truth, The Markup stated that no less than one among its exams resulted within the appropriate reply on the identical question about accepting Part 8 housing vouchers (whilst “ten separate Markup staffers” bought the wrong reply when repeating the identical query).

The MyCity Chatbot—which is prominently labeled as a “Beta” product—tells customers who hassle to learn the warnings that it “might sometimes produce incorrect, dangerous or biased content material” and that customers ought to “not depend on its responses as an alternative choice to skilled recommendation.” However the web page additionally states entrance and middle that it’s “educated to offer you official NYC Enterprise info” and is being bought as a means “to assist enterprise house owners navigate authorities.”

Andrew Rigie, government director of the NYC Hospitality Alliance, informed The Markup that he had encountered inaccuracies from the bot himself and had acquired studies of the identical from no less than one native enterprise proprietor. However NYC Workplace of Know-how and Innovation Spokesperson Leslie Brown informed The Markup that the bot “has already supplied hundreds of individuals with well timed, correct solutions” and that “we are going to proceed to deal with upgrading this software in order that we are able to higher help small companies throughout town.”

The Markup’s report highlights the hazard of governments and firms rolling out chatbots to the general public earlier than their accuracy and reliability have been absolutely vetted. Final month, a court docket forced Air Canada to honor a fraudulent refund policy invented by a chatbot accessible on its web site. A recent Washington Post report discovered that chatbots built-in into main tax preparation software program gives “random, deceptive, or inaccurate … solutions” to many tax queries. And a few artful immediate engineers have reportedly been able to trick car dealership chatbots into accepting a “legally binding provide – no take backsies” for a $1 automobile.

These sorts of points are already main some corporations away from extra generalized LLM-powered chatbots and towards more specifically trained Retrieval-Augmented Generation models, which have been tuned solely on a small set of related info. That type of focus may develop into that rather more necessary if the FTC is successful in its efforts to make chatbots liable for “false, deceptive, or disparaging” info.