Mamdani to kill the NYC AI chatbot caught telling businesses to break the law(themarkup.org)
119 pointsby jyunwai3 hours ago9 comments
  • hashberry34 minutes ago
    > The Office of Technology and Innovation spent nearly $600,000 to build out the foundations of the MyCity chatbot, which will be used for future chatbot offerings on MyCity. [0]

    This was experimental tech... while I admire cities attempting to implement AI, it seems they did not spend enough tax dollars on it!

    [0] https://abc7ny.com/post/ai-artificial-intelligence-eric-adam...

  • andsoitis2 hours ago
    Why did NYC release it in the first place? Did they not QA it?

    Or was it perhaps one of those cases where they found issues, but the only way to really know for sure that the deleterious impact is significant enough by pushing it to prod?

    • cheald9 minutes ago
      Remember that many people are heavily are happy-path biased. They see a good result once and say "that's it, ship it!"

      I'm sure they QA'd it, but QA was probably "does this give me good results" (almost certainly 'yes' with an LLM), not "does this consistently not give me bad results".

      • themafia6 minutes ago
        > almost certainly 'yes' with an LLM

        LLMs can handle search because search is intentionally garbage now and because they can absorb that into their training set.

        Asking highly specific questions about NYC governance, which can change daily, is almost certainly 'not' going to give you good results with an LLM. The technology is not well suited to this particular problem.

        Meanwhile if an LLM actually did give you good results it's an indication that the city is so bad at publishing information that citizens cannot rightfully discover it on their own. This is a fundamental problem and should be solved instead of layering a $600k barely working "chat bot" on top the mess.

    • drillsteps5an hour ago
      >Why did NYC release it in the first place? Did they not QA it? How do you QA black box non-deterministic system? I'm not being facetious, seriously asking.
      • pegasus17 minutes ago
        The same way you test any system - you find a sampling of test subjects, have them interact with the system and then evaluate those interactions. No system is guaranteed to never fail, it's all about degree of effectiveness and resilience.

        The thing is (and maybe this is what parent meant by non-determinism, in which case I agree it's a problem), in this brave new technological use-case, the space of possible interactions dwarfs anything machines have dealt with before. And it seems inevitable that the space of possible misunderstandings which can arise during these interactions will balloon similarly. Simply because of the radically different nature of our AI interlocutor, compared to what (actually, who) we're used to interacting with in this world of representation and human life situations.

        • themafia5 minutes ago
          > radically different nature of our AI interlocutor

          It's the training data that matters. Your "AI interlocutor" is nothing more than a lossy compression algorithm.

        • 6 minutes ago
          undefined
      • 32 minutes ago
        undefined
    • thedanboban hour ago
      > Why did NYC release it in the first place? Did they not QA it?

      Considering Louis Rossmann's videos on his adventures with NYC bureaucracy (e.g. [0]), the QAers might not have known the laws any better than the chat bot.

      [0] https://www.youtube.com/watch?v=yi8_9WGk3Ok

      • direwolf2023 minutes ago
        Considering the previous mayor's relationship with the law, it could be on purpose.
    • elgeniean hour ago
      QA efforts can whack-a-mole some issues, but the mismatch of problem and solution is inherent in any situation in which a generator of plausible-sounding text gets pointed at an area where correctness matters.
    • fragmedean hour ago
      Why do you think OpenAI let a red team loose on GPT-5 for six months before releasing it to the public?
      • bluGill27 minutes ago
        For the image. There is no way a red team can find all the issues in 6 months. They can find some of the biggest, but even getting all the issues fixed in 6 months seems unlikely.
    • erxaman hour ago
      > Why did NYC release it in the first place?

      Perhaps a big fat check was involved.

  • sylens3 hours ago
    > The bot, built using Microsoft’s cloud computing platform

    When is the last time there was positive news involving Microsoft? This bot could've easily been on AWS or GCP but I find it hilarious that here they are, getting dragged yet again

  • kittikittian hour ago
    Being in and around the NYC area, while also knowing plenty of small businesses, I'm glad Mamdani killed this bot. Telling bosses to steal tips from their employees is run-of-the-mill corruption and common over here. The vibe for businesses is that everyone has to be exploiting someone else or have a schtick. If you were to talk about morals, you would be ridiculed. Most lawyers wouldn't even prosecute small businesses for this. It's probably why the agent was put into production, the level of business ethics in NYC is cartoonishly evil.
    • patrickmay17 minutes ago
      In the case of stealing tips, that's wage theft and the New York State Department of Labor has zero sense of humor about that. They will definitely investigate all claims on that topic. It might be too little and too late for the individual affected, but the business will pay.
  • 1970-01-018 minutes ago
    He is turning out to be a benevolent, law-abiding mayor that just happens to be communist.
    • direwolf207 minutes ago
      What's that supposed to mean?
      • 1970-01-017 minutes ago
        The previous mayors were none of these things
  • cmiles8an hour ago
    We’ll likely see a lot of these AI pet projects get axed in the coming year or two… especially things rushed out in the early phases of the AI bubble when folks were desperate to appear to be using AI.
    • chasd0031 minutes ago
      yeah i hope the problems stay to somewhat humorous themes like convincing a car sales bot to sell you a car for $1 and not more serious issues like convincing a bot to metaphorically launch the ICBMs.
      • toomuchtodo25 minutes ago
        "The WOPR did a better job avoiding thermonuclear war than most humans would" is my hot take.
  • toomuchtodo2 hours ago
    > A spokesperson for the mayor, Dora Pekec, confirmed in a text message that the new administration plans to take down the chatbot. She said a member of the Mamdani transition team had seen reporting on the bot from The Markup and THE CITY and presented it to the mayor as a possible place to save funds.

    Journalism works.

    • atq211937 minutes ago
      It does. And it works best if you elect politicians who are willing to listen.
  • monero-xmran hour ago
    [flagged]
    • geoffegan hour ago
      To ride NYC's free busses, you must have a two minute conversation with a chat bot. (/s)
  • terespuwashan hour ago
    What else to expect from Eric Adams.