1202 pointsby galadran7 days ago61 comments
  • chrisweekly7 days ago
    Right, so it's not just git, it's a bunch of system-provided utilities; macOS trusts bad certs, and these utils that trust the OS are affected. I use homebrew for my git and curl but haven't yet checked to confirm whether that matters here. Kind of outrageous that Apple's guidance is to edit the cert chain manually, and to expect that subsequent OS updates may revert your mitigation, reintroducing the vulnerability (as happened at least once so far).
    • stingraycharles7 days ago
      To be honest, git deferring to the OS is the right thing to do. It's macOS that's the problem, not git.

      "Ten days later, macOS and all applications that rely on it still trust the compromised eMudhra certificates" would have been a better title.

      • emn137 days ago
        Yeah, definitely: having lots of separately distributed and likely rarely and unreliably updated trust roots would not inspire a lot of confidence.
        • gchamonlive7 days ago
          In turn the centralized OS authority needs proper maintenance otherwise it's just as bad if not worse, because of the larger blast radius
          • brookst7 days ago
            I don't know about worse. Imagine a world where there is no OS authority, each app does their own. Do you think all important apps would be updated in 10 days, including auto-updates?

            Apple fucked this up, but I really don't think the situation is "as bad if not worse" than if MacOS just didn't do this.

            • mikepurvis7 days ago
              Isn't that the world that Docker gives us, basically?
              • Shish2k7 days ago
                Yes - difficulty in shipping security updates to a hundred different container images, each of which is vulnerable in their own slightly-different way because they all decided to ship an OS alongside their app - is one of the top criticisms of the docker way of doing things
                • stephbook6 days ago
                  Just run it in kubernetes and mount the certificates with trust-manager
                  • paulddraper6 days ago
                    > just

                    > kubernetes

                    • gchamonlive5 days ago
                      If it were 5 years ago I'd have the same reaction, but today it's fairly simple to setup a single node deployment with minikube or k3s, and LLM makes it very easy to maintain deployments.
                • musicale6 days ago
                  > docker

                  If only we could somehow share libraries and configuration files across apps rather than duplicating them and having to update them for every app individually.

                  We could call it a "shared library" system and an "operating" system. Someone should really invent this.

                  I mean, even docker could maybe use a shared base image or something right? ;-)

                  • mixedCase6 days ago
                    Now make it immutable and layered, like Docker images, and you might be onto something... Ah well, sounds too good to be true. We might have to nix the idea.
                    • mikepurvis6 days ago
                      Should it also predate Docker by a decade? That would be crazy.
                • xp847 days ago
                  What kind of psychopath ships 100 distinct container images? I mean, maybe if you’re talking about all of Netflix or something, but… I suspect even they have base images that they use as a starting point.

                  There’s very little actually running in a docker container. ca-certs may actually be an exceptional case though, since it’s not code that’s running but a kind of data.

                  It’s not like if you use an Ubuntu image for a node.js app that you get PHP too and have the PHP vulnerability present in your web app.

                  • stackskipton7 days ago
                    You still have to rebuild each Application Container with new CACert and deploy them. Depending on teams are working, that can be disruptive.
                    • anamexis7 days ago
                      I would say that if rebuilding your docker containers is a disruptive process, that is a process issue.
                      • stackskipton6 days ago
                        I would agree but it doesn't change the fact that many companies do not have CD and have processes that must be followed for rollouts.

                        It's same problem we have when libraries get massive security vulnerabilities. Updating them is not always as easy as many would like.

                        • mikepurvis6 days ago
                          I think the number one thing that bites a lot of container workflows is when you have a separate "base image" build that depends on external, mutable package sources using tools that don't support locking, such as apt.

                          This gets worse for each additional such layer, for example if you start from ubuntu:whatever and have a system deps layer, then an application deps layer, then static assets, then your app's code. Unless you think about it upfront when designing the pipeline, it can be challenging to trace all that state backwards and figure out how to safely re-build thing-from-x-weeks ago but with a single unsafe package bumped forward.

                          Every shop with competent devops people present should be set up for this, but it usually doesn't come for free.

                    • GoblinSlayer6 days ago
                      AFAIK a container build just puts files into a base image, this creates a sort of a commit chained to the base image commit. This means to update the base image you only need to rebase that commit on top of the new base image branch automatically.
                      • notpushkin6 days ago
                        I haven’t seen tooling for rebasing OCI images yet. Considering you’d still have to do this and roll it out to the server, you can just re-run the CI, which would build the image again, but also test that it works with the new base image.
                        • mikepurvis4 days ago
                          You actually can sort of do this just inside a modern docker file, with COPY --link

                          That version of copy overlays the copied content without any knowledge of its underlay, so if the underlay changes, the hash of the layer created by COPY remains identical.

                          Obviously there are lots of situations this doesn't work for, but say you have a pool of Ubuntu packages (the system layer) and then you separately copy a CMake installspace or built uv project from a build stage, that project layer can be identical in hash if it's copied onto an updated system layer.

                          In practice, it's hard to get this to work because it's very easy in docker to accidentally trigger a rebuild for some random reason, but all the pieces are there.

            • lakeviewterrace7 days ago
              > Imagine a world where there is no OS authority, each app does their own.

              That's the world we live in now thanks to Google and Mozilla.

              • emn136 days ago
                Surely there's a nuanced difference between "every app distributes this independently" and "a very small few apps that bear the brunt of hostile exposure support independent mechanisms".

                It's also worth remembering that both of these specific organizations were instrumental in major, positive security shifts that eventually hit OS's too, including shorter-lived certs, limiting the point of CAs, adoption of TLS 1.3, and likely other things. If anything, they deserve _more_ trust than the OS distributors on this specific front, perhaps also because they don't share the OS distributor's incentive to prioritize stability for paying customers, which can be at odds at times with harsh, prompt security measures.

            • musicale6 days ago
              > Apple fucked this up, but I really don't think the situation is "as bad if not worse" than if MacOS just didn't do this.

              https://support.apple.com/en-us/149555

            • gchamonlive7 days ago
              It makes no sense to imagine this hypothetical scenario and compare it to reality, you have to imagine a similar world with a OS authority, but it only gets updated every 6 months. Which is worse in the worse case? I'd say the one with the lower blast radius.

              But this abstract exercise is only to highlight the importance of Apple getting its shit together, because such a scenario exists with Windows and the culture of downloading exes and msis and it's honestly way worse than having a central but responsible authority.

          • sfoley7 days ago
            Remember this is the same company that demands they're the only ones who can possibly decide what software their users get to install.
        • sebazzz6 days ago
          So many Java trust stores never seeing this update.
      • whizzter6 days ago
        Yes, but the irony here is that Apple used to (or perhaps still has) forbidden using other HTTP(S) clients in iOS apps (or perhaps Appstore in general) than the built-in out of security concerns like this would address by using 1 central storage instead of multiple ones were one won't get timely updates.
      • ToucanLoucan7 days ago
        > To be honest, git deferring to the OS is the right thing to do.

        Tangentially related: I fucking hate so much that browsers now all have their own DNS resolvers and caches, it's just Another Fucking Thing to debug when you're working especially on locally hosted stuff. This is not a browser's job, it's a task that should fall to the OS.

        • bombcar7 days ago
          It is super annoying especially since it's not even "browser has its own DNS and cache but uses your name servers" - more and more "browser bypasses all your shit both on the OS and the network and uses DNS over HTTPS to do God knows what".
          • xp847 days ago
            Wait, did I miss this? Can you share details on what browsers started doing this?
            • ToucanLoucan7 days ago
              Can confirm, Firefox did this forever ago. Become my ritual during a rebuild to turn DNS over HTTPS off because it fucks everything up.

              Edit: Found it, February 2020: https://blog.mozilla.org/en/firefox/firefox-continues-push-t...

              • pelican07 days ago
                > Become my ritual during a rebuild to turn DNS over HTTPS off because it fucks everything up.

                Out of curiosity, what are some of those things it messed up for you? And what kinds of builds is it affecting?

                • ToucanLoucan6 days ago
                  Like I said, internal DNS. I have my own DNS server with a small set of records for my own internal sites, mostly automation tools alongside the websites I've built and maintain for small businesses near me. The latter are the easiest to explain, for example, if I built joescoffee.com I wouldn't work in production obviously, I'd work on a copy hosted on my local server at joescoffee.myhouse.com, make my changes and then push that code live. Because Firefox and Chrome (and likely others) roll their own DNS server though, I have to configure that separately, because while my machines all get their DNS from the DHCP leases I've already configured which point all DNS queries to my internal DNS server, DNS over HTTPS bypasses that and contacts public DNS servers directly.

                  And like, I don't need that. My DNS server is properly maintained and queries those exact same servers itself when you request something like google.com or apple.com. Putting a separate friggin DNS query AND cache in a browser is just more trying to protect people from themselves IMO, and introduces this weird issue where you can't hit an internal site in Firefox, even though you're running `nslookup` on your freaking computer, and it's giving you the proper IP address. And paradoxically putting the IP address in makes the browser find it when the domain you can see correctly resolving in terminal isn't resolving in the browser is fucking obnoxious.

                  It also would render a pihole useless because Firefox despite the PC's DNS not resolving ad provider's names, WILL resolve them, ensuring an attack vector for malware remains wide open even if you've tried to close it.

                  And while I'm bitching about this: STOP DEFAULTING EVERY CONNECTION TO HTTPS. Any properly configured website is going to do this anyway, the browser also doing it, again, just annoys the piss out of anyone trying to host stuff locally because why the fuck would I get certificates for servers that aren't even exposed to the damn Internet!?

                  • Dylan168076 days ago
                    > And while I'm bitching about this: STOP DEFAULTING EVERY CONNECTION TO HTTPS. Any properly configured website is going to do this anyway, the browser also doing it, again, just annoys the piss out of anyone trying to host stuff locally because why the fuck would I get certificates for servers that aren't even exposed to the damn Internet!?

                    Attackers won't do it. The browser itself has to put barriers in front of http for public domains.

                    But yeah it shouldn't get so fussy for local IPs or local DNS.

                  • emn136 days ago
                    While you say many reasonable things and describe a perfectly reasonable setup and desire - surely it's clear that your situation is _extremely_ unusual? Browsers are absolutely _also_ reasonable in prioritizing secure defaults for setups that are not similarly controlled. After all, precisely the thing that makes your situation managageble and reasonable - that being you the knowledgeable admin - is also the thing that allows you better than the average user to change defaults.

                    Both from a sheer numbers perspective and also from the perspective of qualitative differences between these audiences browsers are doing the right thing - prioritizing security in low-to-no-trust environments assuming low-to-no-competence administration.

                  • solyin6 days ago
                    QFT! My http://nas.lan isn't getting a cert, nor public DNS. I know what I'm doing. Firefox, stop fucking things up, it's hard enough to use you instead of Chrome, don't make it harder by copying everything they do!
                  • pelican06 days ago
                    Fair enough, carry on.
                  • bombcar6 days ago
                    My browsers shit themselves if I try to go to http://191.168.0.1 and meanwhile https://1.1.1.1 is a thing.
            • 6 days ago
              undefined
            • bombcar7 days ago
              "In May 2020, Chrome switched to DNS over HTTPS by default."

              Welcome to hell. https://en.wikipedia.org/wiki/DNS_over_HTTPS

              (My current chrome reports no at https://one.one.one.one/help/ so maybe I changed it or maybe the default's different now)

        • senderista7 days ago
          Indeed, it makes it harder to block addictive sites like this one in HOSTS :)
        • pcthrowaway6 days ago
          Doesn't libcurl do this also?
      • bombcar7 days ago
        You can have it "both ways" if you want, but it quickly becomes very muddy - defer to the OS, but override the deferral in some cases.

        Certificate management is fifty kinds of all messed up - and in THEORY everything is designed perfectly and properly, in practice it's all bandaids on bandaids that nobody cares about as long as "things mostly work".

        • vikramkr7 days ago
          This doesn't have the vibes of something that's designed perfectly even in theory. This more has the vibes of "this was the least worst option"
          • bombcar7 days ago
            If you look at the FULL design, it's actually pretty good - things like https://en.wikipedia.org/wiki/Certificate_revocation_list CLRs have actual thought behind it - but I know of no system that actually IMPLEMENTS everything, for various practical reasons.

            So from a spherical cow perspective, it's well-designed; from a "this has to work in the real world with real things and people" it's ... workable by violation of the design all the time.

            • Perseids7 days ago
              Trust relationships are fundamental even in the most abstract view of the Web PKI and without the band-aids of Certificate Transparency etc. that trust relationship is: Some random commercial entities that are completely independent of the DNS, make fully trusted statements about the validity of DNS record ownership. It's bonkers to the core. If someone would design it from scratch, of course the domain registrars would be the ones issuing the certificates via their established relationship to the customers, and they would only be able to issue them for the domains they are responsible for. And no: DNSSec does not solve the same problem and is a tragedy all of its own.
              • bombcar6 days ago
                In a roundabout way that’s effectively where we are now since everybody and everybody uses lets encrypt certificates which are just a fancy way of saying I controlled the DNS at some point
                • thayne6 days ago
                  Even before Let's Encrypt was a thing, that is how most DV (Domain Validated) certs were issued. It was just a more manual, more expensive process to do it. GP's point is since the system depends completely on trusing the DNS chain anyway, why not have the registrar [sic. it should actually probably be the registry] act as the CA as well and cut out the middleman, and the need to verify the domain from multiple paths to prevent MitM.
            • thayne6 days ago
              CRLs were a great (ok, maybe just ok) design in the early days when the internet was small and the number of CAs and revoked certificates was small. But it was never going to be scalable. And it has the really big problem of needing to distribute these lists of revocations. OCSP has a different set of tradeoffs.
            • vikramkr7 days ago
              I would argue for anything security - if there is a difference between how it works in theory vs in practice - then the theory is flawed. Physics gets to have theory/practice distinctions, but engineering doesn't because the bridge can't fall down.
              • thayne6 days ago
                In Physics, if a theory doesn't describe reality, the theory is incomplete. Of course all theories are ultimately incomplete, the goal is to find theories that match reality better than the ones before.
              • bombcar7 days ago
                Exactly, and it's a very real problem with the IETF-way of building things, and why "this is what Chrome does, copy it" becomes the real standard.
      • angulardragon036 days ago
        Wouldn’t be true either. Stuff that uses the OS certificate store (that you can view in the keychain) won’t trust the compromised certs.
      • throwaway20377 days ago
        Correct. I Googled to ask about location of certs on disk: /etc/ssl/cert.pem

        Is it possible for users to remove the eMudhra certificates themselves? I have no experience with MacOS, so I don't know if certain actions are prohibited. To be clear: I am not suggesting that normies do this! Rather, I am asking if it is possible for highly skilled users.

      • stkdump7 days ago
        I guess we are going to learn how important secure communication is based on the stories (or lack of stories) about how it is being exploited in the wild.
      • tomkarho6 days ago
        I could swear I saw that title here earlier today almost verbatim maybe even referencing the same link.
      • Barbing7 days ago
        Exceeds character limit, but admins can modify if there’s a better shorter title.
    • woodruffw7 days ago
      Homebrew ships its own CA bundle so its own curl, etc. are not affected. But the system versions of those tools will continue to be affected until Apple patches their own bundle.

      Edit: Git is still affected, we're fixing that now.

      (Source: I am one of Homebrew’s maintainers.)

      • astrostl7 days ago
        I'm using the latest Homebrew git (2.56.0 confirmed with `which git` and `git --version`), and it remained vulnerable until I manually patched the SYSTEM per https://support.apple.com/en-us/149555
      • throwaway20377 days ago

            > Homebrew ships its own CA bundle
        
        No trolling by this question: Can you explain your decision? I can see some pros and cons of this decision. Was it simply removing dependency upon Apple if there was a disagreement on which CA certs to trust? (You can act without permission from Apple to add/remove CA certs?) Also: How does Homebrew decide which CA certs to include/exclude? Do you get a CA cert pack from Oracle/Mozilla/Google/Microsoft/Amazon/Apple?

        Outside of an enterprise environment, I never thought about having multiple "golden sources" of CA certs on a single host. (I have no experience working outside of classic enterprise environments, so please do not read my surprise as immediate dismissal of this idea!) For those unaware, in enterprise environments, it is very common to use a common CA cert pack from Oracle/Mozilla/Google/Microsoft/Amazon/Apple, but add 2-5 CA certs from your own company that act as trust roots.

        I did some Googling on this matter. It looks like the "Big Four" of CA cert packs are: Mozilla, Microsoft, Apple, and Google. (I guess there is one or two in China also, be leave that alone for the moment.) What happens if they are in disagreement? This is a complex problem! If anyone is involved in this process, I would like to learn more.

        • woodruffw7 days ago
          > Can you explain your decision?

          Not easily, because it's closer to a convergence of smaller engineering decisions into an emergent choice to ship a CA bundle.

          But the short version: lots of tools ship their own CA bundle. This is not ideal for distributors like Homebrew, because people expect one CA root program for the entire distribution. As a result, Homebrew tries (and sometimes fails) to apply a consistent root program across its tools. Apple's own roots are not inherently best suited for all Homebrew users, so we end up shipping a composite of their roots and the Mozila root program.

          > What happens if they are in disagreement?

          They are in disagreement, all the time! Root program operators are totally free to manage their trust roots as they please. The only forcing function is a desire to honor users' connections; differences in root programs end up getting papered over somewhat by cross-signing between different CAs.

          • throwaway20375 days ago
            Thanks for the helpful reply!

                > lots of tools ship their own CA bundle.
            
            Can you give an example for Homebrew? And did the package maintainer explain why?

            With the exception of Java, on a modern enterprise Linux box, all packages use the same CA cert pack.

      • lukasgelbmann7 days ago
        Homebrew’s git (unlike homebrew’s curl) seems to default to using the system’s CA bundle, not homebrew’s. Is that a bug?
        • woodruffw7 days ago
          Edit: Sorry, I forgot that I had patched my local bundle. This might indeed be an oversight, I'm looking at it now.

          Edit 2: This is being fixed now[1]. Everything below is stale.

          I couldn't reproduce this:

            % which git
              /opt/homebrew/bin/git
            % git ls-remote https://git-emudhra.we-have-a-problem.com/check.git
              fatal: unable to access 'https://git-emudhra.we-have-a-problem.com/check.git/': SSL certificate problem: unable to get local issuer certificate
          
          Make sure your `git` binary is Homebrew's, not the system/XCode CLT one. If you see `/usr/bin/git`, you're not using the Homebrewed Git.

          [1]: https://github.com/Homebrew/homebrew-core/pull/321878

          • bombcar7 days ago

                $ git ls-remote https://git-emudhra.we-have-a-problem.com/check.git && which git
                2975b9930644b321e18ecae09e29c15b955e6103 HEAD
                2975b9930644b321e18ecae09e29c15b955e6103 refs/heads/main
                /opt/homebrew/bin/git
            
            We watch; we wait (I guess)
          • smmeis6 days ago

                $ brew list git --versions
                git 2.56.0_2
                $ which git
                /opt/homebrew/bin/git
                $ which curl
                /opt/homebrew/opt/curl/bin/curl
                $ git ls-remote https://git-emudhra.we-have-a-problem.com/check.git
                2975b9930644b321e18ecae09e29c15b955e6103 HEAD
                2975b9930644b321e18ecae09e29c15b955e6103 refs/heads/main
            
            It seems that the curl formula still needs patching?
          • lukasgelbmann7 days ago
            > “This is being fixed now”

            Wow, that was fast!

      • forkerenok7 days ago
        > (Source: I am one of Homebrew’s maintainers.)

        I would've preferred a doc or a code repo link as a source. Otherwise, this just reads like a brag. A decent one, to be fair :)

        • woodruffw7 days ago
          I’m currently on mobile but you can look at the source (and Git history) for the ca-certificates formula for the patch.

          (Sorry if you thought it was a brag, it’s just supposed to be context so that everybody is on the same page about the reason for a comment.)

    • Fatnino7 days ago
      Why are we ok with Apple or Microsoft or Mozilla deciding on a wide selection is CAs for us to trust?

      I understand why they originally let each CA in, but it's definitely not one size fits all users.

      A user in the US does not need to trust a Turkish CA the same way a user in Turkey does.

      There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.

      • jeroenhd7 days ago
        > There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.

        There is. On macOS go to Keychain Access > System Roots and disable the kind of trust you want to disable. There's probably a scriptable method for that.

        On Windows, open the management console to "Certificates". Then go through the list and move/remove/distrust any certificates you dislike. There are also powershell tools you can use.

        On Linux, the situation is a bit annoying. You can remove all CA certificates manually and make the directory read-only to break any attempt from your distro to interfere with the list. You need to check your distro documentation to see if they use /etc/ca-certificates or /etc/ssl or something like /etc/pki for reference. Some distros offer a partial GUI as well.

        Programs that deem themselves smarter than the OS (true in the case of macOS and various Linux distros), like browsers, the JVM or NodeJS applications, can have their own cert stores.

        You will need to acknowledge half a dozen certificate prompts for every website you visit after this, though. Make sure to compare the fingerprints of the CA root certificate with the real ones you might find elsewhere, because it's easy to generate your own CA and copy all certificate information signed by your self-generated key.

      • blmarket7 days ago
        There are three futures, each being dystopia than current:

        - Only ISRG(Let's encrypt) being the sole CA and benevolent dictator of the internet. (then forced to be dissolved due to antitrust?)

        - Each country controls their own CA and you need to purchase 200+ certs in order to make your website generally accessible. (then country XXX will charge you $200/year)

        - Big techs owning OS/Browsers control their own CA and cross license each other (then Apple still being lazy handling incidents) (then they form a cartel and block others, like current Tier 1 ISP, email senders, credit card networks)

        edit: put enter between numbered items.

      • mort967 days ago
        Someone has to decide on a set of default CAs to trust... It's natural that it's your OS vendor, since the trust store is an OS thing.
      • pizzafeelsright7 days ago
        Have you checked with the newsgroups and gopher web on why this became a thing?
      • xp847 days ago
        Honestly that last paragraph I totally agree. I did edit to “never trust” these clowns’ root CAs in Keychain Access on my Mac, and this is the first I’m hearing that there was any possibility of that being flipped back. Will have to look into that.

        If I can speculate though on why these bundles come from the browser or OS and aren’t fully in the user’s responsibility. You’d have to start from scratch for it to not have a bunch of stuff the user had ‘no need for’ to be present, and the user would need to see a cert window semi-frequently that gives the possibility to start trusting a CA, and the bottom 99.99% of users in sophistication has no idea whether they should say yes or no. I mean, without being signed by a root CA you already trust, even the name on the new CA could be completely made up. “Do you want to trust the ‘Apple Root Certificate Authority’?”

        • pishpash7 days ago
          Yet they ask you what keyboard type to use, what Wifi network and password to use. Is it too much to ask, do you want to trust "key authorities" from outside your country? Users used to see computers as inherently technical devices and expect to get asked technical questions.

          But even if not, you get asked whether to turn on Siri, whether to send telemetry, whether to sign away your rights, etc., all of which are consequential but prompts are massaged in a way where it's nice and easy for the user to answer a certain way without having a clue. Makes me think user comprehension isn't even a problem.

          Where I suspect the problem lies is that it will not fly to make it seem like US-based CA's are more trustworthy just because the majority of sites roll up to them. Then you'll need to do some non-country-specific human tiering, and who's the judge of that?

          • gruez7 days ago
            >do you want to trust "key authorities" from outside your country?

            So what happens to americans visiting french sites?

            • pishpash7 days ago
              "It seems like you're visiting a site registered in XYZ country. Do you want to start trusting XYZ country key authorities?

              [Do Not Trust], [Trust Once], [Trust Forever]"

              • gruez7 days ago
                Then users just proceeds to hit trust for everything, just like people blindly accept ssh keys.

                The country comparison doesn't do much either, because there isn't a good guarantee that they'll correspond. Some guy posted a repo to distrust every CA except the top 30, and that has a bunch of weird examples, like a bunch of russian sites being signed by a Greek CA. Not to mention some of the biggest sites use CAs that don't correspond with their country of origin. GlobalSign is Belgian for instance. Guess how many of their customers aren't Belgian?

                • pishpash7 days ago
                  Isn't the whole point of this to decide what "shady" CA's to trust? If a Russian site bothered to get signed in Greece, some people will trust the site, and some people will distrust the Greek CA's. That's how trust works.

                  You can add other options too, "Trust for 30 days" or something. I wish SSH known hosts has an auto-expunge.

      • bombcar7 days ago
        Because nobody is going to be arsed to figure this shit out, even the PGP fanatics don't bother.

        So someone has to do it, and a pop-up on first boot asking you which "certificate providers" you want to trust would be as pointless as the "which browser do you want to use" ones were.

      • paulddraper6 days ago
        ?

        The idea of the internet is that you can reach it across the world, globally.

      • Ajedi326 days ago
        If you want access to the whole internet and not just part of it, it has to be "one size fits all users". (At least with how the internet works currently.)
    • 7 days ago
      undefined
    • 6 days ago
      undefined
    • undersuit6 days ago
      I noticed a machine of my was using a Chinese mirror for Debian apt. Switching to the US apt source brought in updates to cacerts and libbrotli. Not trying to insinuate anything, my mirror was just outdated, and yours (plural) could be too.
    • kridsdale17 days ago
      I asked codex “find out if this Mac is still trusting that bad cert and remediate if so”. I got a few commands out of it to run myself with sudo, and now I’m good.

      Thanks for being on the ball, Apple.

      • jeroenhd7 days ago
        Rather than prompting the mechanical parrot, you can also refer to Apple's own website: https://support.apple.com/en-us/149555

        I don't know why they haven't rolled out an update if they've already wrote a whole article about patching the problem yourself.

    • 7 days ago
      undefined
    • 6 days ago
      undefined
    • probe_user7 days ago
      Second probe: confirming the fixed comment path resolves a parent comment correctly.
    • probe_user7 days ago
      [flagged]
    • andrewmutz7 days ago
      Apple gets a free pass on this website. If Google or Microsoft did the same thing, the torches and pitchforks would be out.

      But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.

      • stouset7 days ago
        Shallow or misguided criticisms of Apple—of which there are countless—get shredded on this website. This is not a shallow or misguided criticism of Apple. It is clearly and obviously unacceptable.

        Better luck next time.

      • robby_w_g7 days ago
        Apple makes incredible hardware and could easily be the best development machine on the market. But I ultimately prefer Linux because I own the maintenance story and it's simple + fast. System packages and xcode are horrible experiences, and there's no reason for this to be the case in 2026.
        • nullwarp6 days ago
          As a non mac user who's had to setup xcode and an apple dev account for the first time ever to compile and publish an app store app update for my company - the entire thing soured me on ever even attempting mobile app development.

          It really surprised me what mobile devs put up with, just an utterly horrible hostile experience.

      • zoobab7 days ago
        It has the davantage that you can finally find out what those apps are sending back home, including your private data.
      • zahlman7 days ago
        > Apple gets a free pass on this website. … But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.

        I have no idea how you've come to this conclusion, and clearly nothing of the sort is actually going on here.

      • moi23887 days ago
        I’m somewhat of an Apple fanboy, and I find this absolutely unacceptable.
      • blackcatsec7 days ago
        We clearly just don't know what complex level security Apple has put into their OS. Clearly their OS isn't vulnerable or else they would patch it! (/s)
  • jrochkind17 days ago
    That MacOS hasn't removed from OS-level /etc/ssl seems like a big deal, affecting more than just OS-supplied git. Is /etc/certs not used by most graphical Mac software, I guess? The chaotic state of OS-level cert management, on Linux as well, is pretty dismaying.

    If Apple is still trying to brand themselves as security-forward, you'd think removing the revoked CA certs from /etc in a system update would be a pretty easy thing to do?

    • hyper_reality7 days ago
      Following the eMudhra incident I made a tool to distrust all except the 30 most essential CAs on my Linux and Windows systems (https://github.com/hyperreality/distrust). I was shocked at how messy the situation is on Linux, with browsers, CLI tools, programming languages all having their own stores and distros building the bundles differently (update-ca-certificates in Debian vs update-ca-trust in Arch or Fedora). Or like Firefox not using the certs in /usr/share/ca-certificates/mozilla/ but its own per-profile NSS store instead. There has to be a better way.
      • jeroenhd7 days ago
        Browsers were the original TLS trust store vendors. Operating systems started shipping trust stores because their OS came with a browser, not because they were the natural root of trust. Browsers also have tools to update revocation near-live, while operating systems lag behind by hours, or days in this case.

        For your tool, remember to nuke Windows' automated certificate authority updates (separate from normal Windows updates) and possibly the automated intermediary cache. Microsoft publishes a list of trusted CAs/intermediaries periodically and they get automatically applied to Windows so websites don't need to send the full cert chain during TLS negotiation, and it has a tendency to cache trusted intermediaries as well.

        Mozilla's trust store is one of the first to get updated when a revocation happens, as well as Google's CRLsets. Downloading full CRLs repeatedly would be a waste of time and bandwidth, that's why there's a specific distribution method for CA updates. I would extend your tool with a daemon that checks for revocation automatically, so you don't accidentally eMudhra yourself while browsers and the OS get updates next time something like this happens.

        • 6 days ago
          undefined
      • acdha7 days ago
        I was also surprised to find Java’s key tool returning successfully for the delete request without actually doing anything: list, delete, and list again would still show the CA. Debian had this right during the 90s, all they had to do was follow the system convention.

        This feels like all of the C code getting attacked now where everyone spent the last three decades building up cruft because fixing it would be a chore, and now the separate bills are coming due all at the same time.

      • Aperocky7 days ago
        The better way then comes with the drawback where one vendor has all the power.

        Then someone would ask "there has to be a better way" again, when the reason for the new inconvenience gets forgotten, and it oscillates.

    • jeroenhd7 days ago
      Apple's official API had revoked the certificate within hours. Only the Unix compatibility layer they sort-of maintain has this problem. This is hardly the first or only outdated Unix compat tool they ship with their OS.

      Not great for unaware developers, but most macOS users are fine. Those who are aware can follow the steps in https://support.apple.com/en-us/149555 until Apple bothers to roll out an update.

    • cloudfudge6 days ago
      I used to manage OS level trust root management for all internal machines at a hyperscaler. Hundreds of thousands of machines. Even when you have an edict from a common VP or CEO, and most people are even using the same linux distributions, it's hard to get people to adhere to or even understand that there is a standard. Throw containers, embedded devices, windows and mac in and it's just a total nightmare unless you exert strict control on how every single machine is set up. Which, of course, is impossible because cowboys gonna cowboy.

      Fundamentally, people don't want to think about the intricacies of how we decide whether or not to trust a particular cert. They want it to just work without thinking about it. This includes that if they have a custom private root they need to trust, they just drop it various places until it starts working and call it done. And sometimes they do weird things like adding intermediates to their trust store because they were trying to call a service that has their cert chain configured incorrectly, and that "fixes it". Again, it's a nightmare.

      The fact that linux, a system notoriously cobbled together by 873 different factions on the internet, is even more of a mess is not surprising in the least.

    • kylecordes7 days ago
      The thing that makes this especially odd is that Apple apparently has a fast-track way of shipping security patches for the OSs that don't require the user to go through the lengthier OS upgrade process. But for some reason, they haven't used it here.
      • mixdup7 days ago
        That fast-track method was used, but it only applies to "graphical" software which really means, software that uses Apple's macOS specific APIs and frameworks, not the lower level UNIX stuff like /etc/ssl

        I suspect the fast-track background security updates can only update the databases and revocation lists that live in the proprietary APIs, not any arbitrary file on the system

        • genxy7 days ago
          I see other people making this statement, but it sounds like a really really weird excuse.
          • acdha7 days ago
            It’s not an excuse but an explanation: there are separate stacks owned by different teams and they haven’t been forced to collaborate. If you use an internal CA you’ve already had to deal with the easier additive version of this problem because that fails in an obvious, less harmful manner than quietly accepting something which needed to fail.
            • genxy7 days ago
              It doesn't explain why Apple who 1) sells itself on security and 2) ease of use doesn't 3) update what it installed and protect its users. Why are they not doing those things?
              • mixdup6 days ago
                I wasn't trying to excuse anything, just explain that they haven't done nothing, but also why it's still an issue. I agree with you they've dropped the ball here. If I were a betting person I'd guess this is just something that slipped through the cracks and there will be changes whether that's to the release process for pushing OS updates, to the fast-track security update process, or to the OS so that everything is working off the same trust store
          • mdavidn7 days ago
            Agreed. Why does macOS ship with two copies of the trust store? Shouldn't the process to update the primary store automatically derive the other?
        • jonny_eh7 days ago
          Or they just missed a spot?
      • saagarjha7 days ago
        Yes because that thing is completely useless and provides basically no value
      • genxy7 days ago
        Apple has one job here and it isn't to make UIs that look like melted plexiglass. I am fucking pissed.
    • 6thbit7 days ago
      Yeah, why may that be hard? That’s the question

      I don’t think it’s gross incompetence, since they documented the steps to remove. They may have some unexpected dependency on the file? Looks mysterious from the outside.

      • lukasgelbmann7 days ago
        I think the main problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Certificate revokations are needed to handle this correctly.

        Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, and maybe this is what Apple should have done (it’s the workaround they recommend!). But it means that some small percentage of certificates that are fine won’t be accepted.

      • giancarlostoro7 days ago
        People would have to use it in some drastic catastrophic way for them to force it out in an update, which they might push it out in the next macOS update, which don't come nearly as frequently as Windows bloatware updates do.
        • pc867 days ago
          This seems important enough to just send out a patch update that removes the cert and does nothing else. But apparently at least a couple people at Apple disagree; or, something I'm not familiar with makes that type of update impossible.
          • xp847 days ago
            You have to admit that “just a quick patch” isn’t automatically low-risk. You could cause a large blast radius if you rush any patch out without proper testing.
            • pc867 days ago
              I agree 100%, but I keep going back to the blast radius of a compromised CA and my gut instinct is still "get rid of it and push a hotfix" but I completely acknowledge it's a lot more complicated to ship a MacOS update than just branch off of master and push a new commit, and my experience is almost entirely the latter.
          • giancarlostoro7 days ago
            I mean, I agree with you, I'm just basing my thoughts of what I've seen managers at different companies prioritize as "important" even when engineers cry that a security issue should be of higher priority...
        • dudeinhawaii7 days ago
          I would put Apple updates in that bloatware category given I was forced to upgrade my phone after iOS and updates eventually took up 50% of storage.
        • behringer7 days ago
          amusing you criticize windows updates here where the macos update schedule is woefully inadequate.
          • drfloyd517 days ago
            One is too delayed. The other is frequent but full of crap.
            • giancarlostoro7 days ago
              The issue with Windows is they eventually force it on you, no matter what.
              • alt2277 days ago
                What, MacOS doesnt?
    • LoganDark7 days ago
      > Is /etc/certs not used by most graphical Mac software, I guess?

      Correct, those usually use Apple's networking APIs which all go through their security stack, which already picked up the OCSP revocation within about 2 hours in most cases.

    • insane_dreamer7 days ago
      this only affects terminal apps, which the vast majority of MacOS users don't run; the fix for graphical Mac software is already patched

      my guess is their thinking is that if you're running those apps on Mac, you're a developer or know what you're doing, and can edit /etc/ssl/cert.pem until they push an update

      they also have to consider that such users may have custom changes to their cert.pem file, and they probably don't want to rush an update that would potentially fu-bar those changes

      • cloudfudge6 days ago
        If you're editing /etc/ssl/cert.pem yourself, you're setting yourself up for failure. Use the proper mechanism to add your own trust roots (on macos it's "security add-trusted-cert").

        Whatever reason Apple had for not updating /etc/ssl/cert.pem yet, I am certain it's not out of consideration for users that might have put custom certs in there.

      • LtWorf7 days ago
        Certainly you're aware that every certificate sits in a separate file and an update can just remove the interested files?
        • insane_dreamer7 days ago
          I would be aware of that if it were true, but (surprise!) it's not

          not convinced? take a look at your /etc/ssl/cert.pem

          • LtWorf6 days ago
            dpkg --listfiles ca-certificates

            Well looks like they should take the hint from a better vendor such as Debian then :)

        • xp847 days ago
          It doesn’t look like that, it looks like it’s a cert.pem file.
  • truelson7 days ago
    Here's what apple (themselves) recommends to patch: https://support.apple.com/en-us/149555

    Pretty much every developer on mac should probably run their patch.

    • gamer1917 days ago
      You have to re-run that every time you do a software update btw

      Per https://emudhra.we-have-a-problem.com/apple.html

      > macOS 27.2, released on 26th October, did not fix the issue and undid the workaround that users had applied by hand, leaving them vulnerable again.

      • Arbortheus6 days ago
        Apple’s handling of this has been a disgrace. It took 4 days before my iPhone and MacBook Pro got the CRL, and now I hear they have done this. Dreadful.

        Apple, for the company that prizes itself on believing privacy is a human right, why are you failing your customers so badly on PKI management?

    • Recurecur7 days ago
      That’s not a ‘patch’, it’s several command lines you need to copy and execute, some optionally.

      It’s completely insane to me that Apple didn’t script this with interactive prompts as needed, and non-interactive options for automatic installs.

      • unfitted25456 days ago
        This is such a Linux way to resolve the issue lol
        • dewey6 days ago
          * Unix way
          • idkeickeixkeidj5 days ago
            No, no. They’re right. It’s a very “Linux” thing to do. Applies to Unix as well? Yes. But it is most definitely what one comes to expect of Linux, not Mac.
            • dewey5 days ago
              I get it, I was just being pedantic ;)
    • herf7 days ago
      First time I've seen AI-authored instructions from Apple. I guess it is hard for them to just post a new .pem because it could be MITM'd.
      • gregoriol7 days ago
        Why would it be more secure to post 4 manual commandline instructions than a single file?
      • gamer1917 days ago
        Safari (and other browsers) is safe. The issue is terminal tools such as curl and git
    • truelson7 days ago
      I reposted this as it wasn't clear from previous comments with this URL that there's a patch that's worth running in there from Apple themselves
      • alt2277 days ago
        Why arent they just pushing this patch out on their security update channel?
  • qaz_plm7 days ago
    • davidcollantes7 days ago
      Indeed!

        On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.
      • ChrisRR7 days ago
        Why are they not just pushing out the update asap? Unless they're relying on the compromised certs, then it feels it should be incredibly trivial to push
        • mike_hearn7 days ago
          Because to Apple once you open a terminal you leave the supported world behind. This is a standard thing for them. Apple is not really a UNIX vendor and certainly not a Linux vendor: they sell macOS which is primarily a consumer OS and has its own APIs for everything.

          Tools common on Linux like curl and git come with the OS basically for developer convenience. Apple doesn't write those programs, they don't use Apple's APIs or platform features, and thus in some sense you aren't really meant to use them in Apple's eyes. Ports of UNIX CLI tools exist in a kind of semi-supported netherworld on macOS. They run but they aren't fully integrated or supported parts of the OS.

          • mikeocool7 days ago
            This case not withstanding, Apple regularly includes patches to the built in CLI tools in security updates, so it's a little disingenuous to suggest those tools are not really supported.
            • xp847 days ago
              I read GP as a descriptive comment not a justification. They are showing with this issue that they don’t truly support the cli tools in a way we ought to rely on. Whether they occasionally provide some support doesn’t mean they’re supported. Only full and competent support counts, everything else counts as “it’s best-effort at best”
          • idkeickeixkeidj5 days ago
            > and thus in some sense you aren't really meant to use them in Apple's eyes.

            That doesn’t seem to be the case at all. I’d wager that in Apple’s eyes you’re very much meant to use these tools, they just expect you to know what you are doing if you go down this route.

        • ratorx7 days ago
          The issue might be Root CA vs intermediates. The root CA that eMudhra rolls upto may be used for other intermediates which were not compromised or revoked. Removing the root CA will mean that these valid certificates no longer work.

          This is technically a problem in the CLI - it needs to support revocations properly, not rely on root CA verification only. I’m not sure though whether there’s a standardised mechanism (that Apple supports) for CLIs.

          > These steps affect every tool that uses /etc/ssl/cert.pem. Afterward, tools that use only this file might not be able to verify some websites whose certificates chain to these root CA certificates through intermediate CAs that weren't affected.

          • matharmin7 days ago
            In pracitce, those root CAs have already been removed in major browsers. If your server uses them, you would likely have needed to migrate to another root already.
            • 7 days ago
              undefined
          • brewmarche7 days ago
            Apple’s own security framework is supported by curl with OpenSSL or GnuTLS (build with ./configure --with-apple-sectrust). That blocks the intermediate certificates via the same mechanism Safari uses for example.

            Not sure why macOS doesn’t ship such a build of curl.

            • Orphis7 days ago
              It's relatively new I think.
          • zarzavat7 days ago
            PKI works on trust. When a CA does things that violate that trust, their root gets removed. That's how it's supposed to work.
          • ChrisRR7 days ago
            I'm no expert in the matter, but weren't only the intermediate CAs compromised, not the root?
            • ratorx7 days ago
              Yep, but the certs.pem file only contains the root and does not know anything about intermediates.

              So the only way to fix apps that don’t handle revocation of intermediates (eg the ones that only look at certs.pem) is to revoke the root.

              • brewmarche7 days ago
                I think it’s possible to add intermediate certificates with distrust marker attributes to bundles with OpenSSL. But other libraries consuming the bundle would not understand it, so the problem remains.
          • xp847 days ago
            > Removing the root CA will mean that these valid certificates no longer work.

            Great, that’s fine. They can reissue.

        • teki_one7 days ago
          I think since 10.15 the built-in command line tools in MacOS are there mostly for backwards compatibility. This was clearly communicated about scripting languages, but there was a feeling back then that this was about the whole lot (all cli tools).
          • abujazar7 days ago
            Git comes with Xcode, it doesn't come with macOS by default.
        • MomsAVoxell7 days ago
          Theory: Apple are moving away from user-controllable development tools, towards an AI-hosted application development framework which removes the divide between user and developer, and thus the need for specialized developer tools.

          The writing is on the wall for the command line tools.

          (We can already see this tooling devaluation happening all over the place. Apple have their variant of this response to the AI tooling-paradigm tsunami…)

          • JoBrad7 days ago
            Apple seems like an odd company to lump into this group. Any evidence for this? I removed the cert using a CLI command on my Mac.
            • MomsAVoxell7 days ago
              They have been treating the command line tools like the bastard child of their software portfolio, and then there’s that whole drive to remove the Unix’ness of MacOS .. Swift/SwiftUI is a drive away from lower tooling layers, and .. there is no doubt they already have a developer agent of their own, internally, pushing their abilities to make projections of hardware requirements and specifications for the next 2 - 5 years …

              I’m not saying they’re going to deprecate XCode today, but I do believe they will “agent’ize” Developer Tooling drastically in the next year or so. Agents are becoming the most important user of command line tools.

              Because they have to, in order to catch up with NVidia, their only real competition. If you want to understand why I think that, its because I have been developing and deploying cross-platform Linux-centric (but MacOS and Windows targeting) applications, mostly for the ARM platform but also with Intel (SIMD on both) .. for the past few months, and I haven’t touched an IDE in a month. On DGX Spark. Which turns out is a truly, utterly, mind-blowingly great platform to do cross platform high performance application development…

              • pasc18787 days ago
                Apple has pretty well always got users to do things via the GUI – look at the Lisa and Classic MacOS.

                CLI is just a thing that NeXT had and even that made it easy to use GUI tools.

                Xcode is a GUI tool – it happens to expose some cli tools but they could be covered up – however some developers would complain a lot so I don't see that happeneing.

          • HFguy7 days ago
            Command line tools seem to mesh well with AI. Why would they move away?
            • MomsAVoxell7 days ago
              Precisely because the agent is becoming the user of the command line tools now, these are all being abstracted behind an interface which blurs the developer/user requirement…

              The original question is, why aren’t Apple fixing the CA problems for the command line tools (whereas they have done it for .app’s such as Safari), and the reasoning is, because the user will just have to fix it themselves, because Apple don’t care about the command line users - they see that these command line tools are, indeed, becoming the front-line for agentic development, so .. presumably, they have an .app bundle which does away with this problem, to be shipped on the horizon ...

          • zf000027 days ago
            The command line tooling lets you/AI manage a lot of your app store interactions though. It's much easier to use that way then trying to deal with the Steamworks UI and doing things manually.
            • MomsAVoxell6 days ago
              My theory is that Apple don’t care about the stock OS cli tools because they are moving towards making XCode more of an agent harness ..

              Anyway, its just a theory.

          • ilvez7 days ago
            That would be a killer move!
      • binaryturtle7 days ago
        Interestingly on a much older OS (Yosemite) cURL, etc. links to Security.framework which uses the system's trust store instead a separate .pem file in /etc/ssl (I can't find any .pem inside /etc/ssl)

        I wonder why they stopped doing that and now just seem to do "standard" builds and end up with this (at least) 2 separate trust store situation.

      • bestouff7 days ago
        Some depending on the executable, sometimes it goes through /etc/ssl/cert.pem and sometimes through the Apple framework. This OS is a mess.
    • samgranieri7 days ago
      I think this is basically related to apple's SIP system protection: they haven't figured out a way to update this quickly and would need to roll this out in like an OS update, which can take a while.

      They might need to make a way to do this outside of os updates

    • ChrisClark7 days ago
      Are they just saying "do it yourself"? Can't they just send out an update?
      • iAMkenough7 days ago
        They did send out an update this week…that undid this mitigation for users that already removed the certs.
      • nailer7 days ago
        I imagine their response would be: "we need to parse a new file format, that may have been edited by users, reliably. That takes time."
        • xp847 days ago
          Why would they need to? Can’t they ship an update that runs an executable that uses the already-existing tooling to update the pem? They’ve posted the cli instructions, so, ship that as a shell script and run it. It should be idempotent, whether the user has already “untrusted” it or not.
        • seam_carver6 days ago
          Isn't the file overwritten every update?
          • nailer6 days ago
            I hope not, that would be data loss.
            • roryirvine6 days ago
              The most recent MacOS update does indeed revert any manual changes you've made, and re-adds the offending cert: https://mstdn.social/@jschauma/117513596272504904

              Their attitude seems to be that you don't own anything outside of your homedir, and touching system-managed files is done at your own risk.

  • faxmeyourcode7 days ago
    While there is no proof that anyone knew about these weak keys for the year and a half that they were trusted by virtually everybody, and there's no evidence of MITM attacks happening in the wild because of it, you can imagine the payoff that a three letter agency would have if, for even 10 days, there was an opportunity to harvest credentials from important people in high places.

    I have no evidence or proof, but I am convinced this wrap/unwrap key generation debacle was intentional and the weak keys were discovered and kept secret by the agencies or governments and used to exploit their adversaries.

    You don't need to perform a traditional MITM attack _if you own the airport_ or you _own the free public wifi_. The traffic can be harvested in clear text and we have to come to terms with the _remote possibility_ that this was going on for over a year and a half. If not that long, then certainly for the past 10 days.

    • 6thbit6 days ago
      This would've been true a while back prior to major browsers checking for certificate transparency log entries as part of the handshake.

      So, I do agree this could be problematic, on specific cases where people not using a major browser were accessing resources some is interested on, on a network they have control of. Makes the surface quite small.

      The threat here is more so for 'backend' services than regular browser users.

    • ryeights7 days ago
      This is misleading. In order to read traffic you would need to swap out the end-entity TLS certs being used. Phones and browsers (at a minimum) enforce certificate transparency, so connections would fail.
      • faxmeyourcode6 days ago
        But the point of the OP is that git, curl, pip, python, etc don't use certificate transparency. That's why it's so egregious that apple still trusts these certs by default in these non-browser edge cases.
    • ycombinatrix7 days ago
      Buddy, 3 letter agencies don't need these weak keys, they straight up own root CAs.
      • faxmeyourcode6 days ago
        My point is less NSA and more NSO. The UAE was denied a root CA[0]. DarkMatter applied to be its own root CA until it was denied[1] over concerns raised about potential spying/espionage[2].

        There are people from these CAs who were denied in the past explicitly for concerns about spying who now work at eMudhra[3,4].

            0: https://bugzilla.mozilla.org/show_bug.cgi?id=1474556
            1: https://bugzilla.mozilla.org/show_bug.cgi?id=1427262
            2: https://groups.google.com/g/mozilla.dev.security.policy/c/nnLVNfqgz7g/m/TseYqDzaDAAJ
            3: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c43
            4: https://emudhra.com/en/blog/scott-rea-joins-emudhra-as-senior-vice-president-global-strategic-advisor
        
        Again, I have no proof. There may never be any proof. But I think it's time we think about who might have found out about this weak key when it was first issued a year and a half ago and trusted by the whole world and what they would have done with that information.
  • kryptiskt7 days ago
    I'm not being sarcastic when I say that Apple should copy Windows Update, I have no idea how they have got this far without a method to add small patches outside monolithic OS updates.
    • tedd4u7 days ago
      They have it. It’s called Xprotect. I’m surprised it hasn’t been used to remove the bad certs.

      https://fullmetalmac.com/cybersecurity/macos-security/xprote...

      • saagarjha7 days ago
        XProtect removes malware, not certificates
        • tedd4u3 days ago
          My bet is the XProtect update mechanism could be used to update the certs.pem file.
        • tonymet6 days ago
          what are the capabilities? presumably it could a apply a diff, patch or overwrite a file. in this case it's not even in the protected partition.
    • tonymet7 days ago
      Microsoft even has multiple express channels. For example Windows Defender definitions are practically real-time
      • acdha7 days ago
        Just like Apple has. The problem here is whoever made the decision not to use that capability:

        https://support.apple.com/en-us/102657

        • tonymet6 days ago
          not quite. this is more similar to Windows Quality Updates. Defender updates are real time (up to multiple times a day, and delivered p2p)
    • reaperducer7 days ago
      I have no idea how they have got this far without a method to add small patches outside monolithic OS updates.

      You're either not an Apple user, or never noticed Settings → General → Software Update → Automatic Updates

      • xp847 days ago
        I think they meant that Apple’s updates seem to be the opposite of incremental, and they’re not wrong. If anything, Apple has moved in the exact opposite direction in the past decade, moving towards using enormous cryptographically-signed volumes (“sealed” is the term). This makes Apple really happy because they can make sure the users aren’t allowed to modify any of the “system“ files themselves for any reason, but also means that they can’t incrementally update things as easily as they did before. They ship multi-gigabyte “patches” every time.
        • acdha7 days ago
          Being able to trust the OS is a huge advantage, though, and they built the rapid response mechanism to handle those updates:

          https://support.apple.com/en-us/102657

          The big question is why they’re not using it for things like this, where a tiny layered file system update would benefit almost all of their users immediately.

        • tonymet6 days ago
          but the volume with these certs isn't protected. you can edit the files with vim. i'm curious what the holdup is.
  • commandlinefan7 days ago
    It seems like we've all just... given up on certificate revocation? CRLs and OCSP were designed for exactly this sort of thing, but they both seen to have been abandoned as unworkable with no replacement.
    • acdha7 days ago
      It’s a hard problem: if you query on use, you not only introduce delay but also leak information about what servers people are connecting to. Dealing with that starts looking a lot more complicated (bloom filters per CA of recently-revoked cert IDs?) needing to be implemented into multiple libraries and the failure mode is an outage unless you fail open.
    • boston_clone7 days ago
      Same here - I genuinely thought this would have been easily, and almost immediately, remediated vis-a-vis CRLs. I'm not in the vuln management space anymore, but is there some sort of caching system used which would affect software differently than a typical browser?

      Editing to add a link to this Apple support article which is unsurprisingly detailed - https://support.apple.com/en-us/149555

      • zerocrates7 days ago
        As far as I'm aware the typical command-line stuff and most things just using SSL libs or language standard libraries don't do revocation at all by default.

        Similar reason (though with the opposite effect) as how a cert can be fine for browsers but fail for curl when they don't send an intermediate: the "vanilla"/library/command-line stuff is just checking for a path to a trusted cert and not doing the CRL or AIA stuff a browser does.

        • boston_clone7 days ago
          Yes, exactly what the Apple article highlights! Appreciate the extra context.

          CLI-based activity references etc/ssl/cert.pem. While they say this will be updated in the future (likely the next dot release), admins can also proactively remove the eMudhra entries.

    • ycombinatrix7 days ago
      [dead]
  • gringofyx7 days ago
    A flat cert.pem file (or similar) can't say "distrust this intermediate".

    macOS already has the verifier needed (Security.framework/trustd path that protected Safari). It would be good if MacOS routed CLIs through it.

    • josephcsible7 days ago
      > A flat cert.pem file (or similar) can't say "distrust this intermediate".

      Why does that matter in this case? Can't this be fully mitigated with next to no collateral damage just by removing the trusted roots that signed them?

      • Ajedi327 days ago
        The root certs may have signed other intermediaries that are still trusted.
        • zenexer7 days ago
          They aren’t trusted, though. Didn’t Chrome ultimately choose to drop the roots, not just the intermediates, from their root program?
          • lukasgelbmann7 days ago
            Chrome dropped one of the eMudhra roots. The problem is that the weak eMudhra intermediates were cross-signed by Sectigo. ~~Browsers (at least Chrome) didn’t drop the Sectigo root, instead they store a revokation for the cross-signature.~~ (edit: this is wrong, see josephcsible’s answer below)

            Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, but it means that some small percentage of certificates that are fine won’t be accepted.

            • josephcsible7 days ago
              Wasn't the cross signing root AAA Certificate Services, which modern browsers had already dropped trust of before this incident?
              • lukasgelbmann7 days ago
                Ah, you’re right. Though unless I’m mistaken again, Safari (but not Chrome and Firefox) still trusts AAA Certificate Services and handles this issue with revokations.
            • xp847 days ago
              This seems like extremely acceptable collateral damage to me. Sectigo should already be working with their customers to reissue immediately since they are the ones who signed the dumb intermediates. Trust only matters if there’s accountability. If root CAs want to keep their trust, it’s on them to not sign things that are bad. That’s the responsibility that comes with the license to print money that being a root CA gives you.

              If I’m an OS vendor I would have been putting them on notice days ago that their root was being dropped and they have X days to reissue certs and deliver a new root cert or otherwise they can just accept that they’re not qualified to be a root anymore.

              • direwolf206 days ago
                How should Sectigo have detected the badness?
                • atanasi6 days ago
                  Verifying the documentation of the key ceremony, what scripts they used, the attestation from the HSM…
        • LtWorf7 days ago
          I trust nothing that came from emsign
  • paaloeye7 days ago
    > Anyone can intercept any HTTPS connection these tools make.

    yeah, it's not ideal but the scale of the threat is way overblown tbh. You have to compromise a lot to be able to make a MITM here.

    Let's not treat it like PHP2Shell.

    • dspillett7 days ago
      Don't underplay it either though: with a compromised but unrevoked signing cert all it takes is a successful phish, or some problem to allow DNS poisoning (or hack a wireless AP, or …) to direct you to a fake site or (potentially worse as it might be less immediately noticeable and could quietly collect credentials/data for a while) send your automated tools to a malicious API.

      It may not be directly exploitable on its own, which is the impression some reports give, but it is a serious hole to leave open especially with the current rate of old security issues being found, as it could be _incredibly_ useful to a malicious actor as part of an exploit chain.

      • jfyi7 days ago
        Yeah, it increases the attack surface of mitm attacks to your network from the local os level.
      • commandersaki7 days ago
        Doesn't affect app bundles though, only *nix commands. This'd be detected pretty quickly, at least by the significant portion of patched devices.
        • dannyw7 days ago
          Unless you're clever enough to only poison user agents known to be macOS and CLI. You'd probably get by.

          xz-tools was only caught because it had a bug causing slow startups and someone was micro-benchmarking Postgres and decided to dive in further.

          • genxy7 days ago
            One person, at that particular time. The person and the task were also a couple standard deviations away from the norm, the world was incredibly lucky.
          • throwawaysoxjje7 days ago
            The cert is provided before the user-agent string is
    • rekoil7 days ago
      It's pretty easy to compromise things like McDonalds or Starbucks WiFi hotspots though, just setup a Pineapple with the right SSIDs and a 5G modem, funnel your target domains through your mitmproxy via the DHCP DNS server you're providing, and that's basically it.

      I agree it's not PHP2Shell, but its certainly not being "way overblown" here.

      • 17186274407 days ago
        Why would these public network would allow guest hosts to communicate with each other?
        • HiJon897 days ago
          I think they're saying you create your own malicious wifi network and people will unintentionally connect to it. This network specifies your malicious DNS server via DHCP, and everything looks secure to the client. I think the main issue with this in practice is that all the major browsers have been updated to not trust these certs.
          • rekoil7 days ago
            So you target some host like ssh.dev.azure.com, find all the hooks in the proper repo the client is trying to access, make a new commit that puts your persistence payload in one of them that you expect will be executed soon, this script can additionally "correct" the git state when it finally runs, and moves its persistence to some more appropriate location.

            Users will be none-the-wiser because they're not accessing these hosts through a web browser, and you net a few more compromised hosts in your inventory.

            • commandersaki7 days ago
              Yeah also got to be lucky that the users are using https and not ssh. You're very limited in what you can mitm, you'll be trivially detected by patched devices (though maybe not acted on because nobody is the wiser), but then you'll need to serve objects with common/conflicting hashes -- fixing up that state synchronisation (what you call "correct") to then do nefarious things is not trivial at all. Extremely difficult to pull off successfully for git anyways or to do it in stealth.
              • rekoil7 days ago
                Ahhh, true, SSH isn't the same certs, my bad... well, both Github and Azure DevOps default to HTTPS-based cloning anyway...
        • ctippett7 days ago
          They wouldn't. A device like the Pineapple broadcasts its own copycat wireless signal and tricks unsuspecting clients to connect to it thinking it's the real one.

          Edit: Holy moly we all jumped on this comment at the same time didn't we.

          • jfyi7 days ago
            > Holy moly we all jumped on this comment at the same time didn't we.

            It's good to see push back against this narrative that is trying to develop that this isn't any kind of big deal.

            This has been every machine on the internet compromised for at least 12 hours and probably some that are not going to ever get patched.

            • rekoil7 days ago
              > probably some that are not going to ever get patched

              This is a HUGE deal for embedded devices, we can now mitm HTTPS traffic on lots of embedded devices running Linux produced between 2018 and now.

        • jon-wood7 days ago
          They don't need to, a Pineapple (or anything else that can run a wireless access point, you could do this with a laptop) just needs to advertise the expected SSID and do so with a stronger signal than the official ones for people's devices to connect to it instead of the actual network.
        • jrmg7 days ago
          It’s extremely common to see other Macs pop up in the ‘network’ panel in Finder in the independent coffee shops around me. I imagine they’re just signing up for a business plan and using the routers the ISP gives them with a custom SSID and password.
        • SXX7 days ago
          You dont interact with anyone. You just set your own hotspot and people will connect to it.
    • breakwaterlabs7 days ago
      The near-universal use of TLS and the absurd ceremony we go through to ship root CAs everywhere is all premised on the idea that MITMs are not terribly difficult.

      The entire reason one can rest easy as they connect to their coffee shop's WPA2-PSK network, is the security provided by TLS.

      And if you don't think an ISP or agency would abuse this if it could be done undetectably then you haven't been paying attention these last 20 years (hence why DoH is now a thing).

      • yonatan80707 days ago
        Even a simple rouge AP is enough to become a MITM for this.
        • itintheory7 days ago
          Why is the color of the access point relevant?
          • yonatan80706 days ago
            TIL rouge means red in French

            Either way, I mean rouge in the sense of an AP that uses the same SSID and password as the AP of a coffee shop or office to have devices automatically connect to it without their users noticing.

            • ButlerianJihad6 days ago
              In English, rouge is a type of makeup.

              “Rogue” is a completely different word. Rogue access point.

              • yonatan80706 days ago
                My apologies. Thanks for the correction
          • zdp76 days ago
            Da Red goez fasta. Hence the router beats the non red router to the punch.
    • mschuster917 days ago
      > You have to compromise a lot to be able to make a MITM here.

      It's trivially easy to mount an ARP spoofing attack. I did that at school two decades ago and on-the-fly rewrote <img src=...> to hardcore porn images for funsies.

      mitmproxy [1], a popular toolkit for making that easy, has the option to generate "eMudhra" certificates on the fly now.

      It is trivially easy to compromise things, all you need is physical access to the network.

      [1] https://news.ycombinator.com/item?id=50189232

      • sersi7 days ago
        I don't know why you're downvoted, any public wifi access point in a shop could be compromised and users would never notice.
      • treesknees7 days ago
        Client isolation is very common on wireless and wired LANs these days in enterprise and guest network environments. You also have tools like dynamic arp inspection to block poisoning attempts via the network switch.

        If you get past all of that, you still need a victim who is actively using a cli tool on macOS using https.

        It’s not impossible, but the conditions are much more specific than those needed to inject porn into a webpage on a poorly configured school network in 2006.

        • 9dev7 days ago
          …and yet it doesn't help at all if a laptop sees an AP with strong signal in its vicinity and connects to that instead of the meticulously configured AP that sits behind the counter with a slightly weaker signal.
          • treesknees7 days ago
            You’re not actually making the point you think you are. This comes back to the original comment stating you need to compromise a lot to MITM here.

            So you need to somehow trick macOS into remotely connecting to the wrong bssid, and then arp poison a user who happens to be using a macOS cli tool that’s using https.

            Sounds pretty targeted and unlikely to happen.

            • 9dev7 days ago
              Go to a random coffee shop that allows people to work at the tables, and you’ll find a few developers with MacBooks there pretty soon. They will be connected to the local hotspot, which either has no password at all, or WPA2 with the password written on a billboard behind the counter. Spin up an evil twin, then wait for the connections to roll in.

              I don’t think that’s either targeted or unlikely?

              • yonatan80706 days ago
                Exactly, I do this all the time for doing out-of-office demos with our company's robots.

                I just set up my phone's hotspot to be the same SSID and password as the network the robots already know from the office, and I'll have the robots, a couple phones, and a laptop connected in 10 seconds.

      • raverbashing7 days ago
        > all you need is physical access to the network

        Yeah that's not so much "trivially easy"

        • breakwaterlabs7 days ago
          Do you have a cell phone? Do you use Wifi? Do you use an ISP who, in the last few decades, has performed any form of interception or traffic-tampering?

          I hazard that the answer to all three of those is "yes".

      • paaloeye7 days ago
        > all you need is physical access to the network.

        yeah, it's trivial to punch somebody in the face in the morning, all you need to do is to be by their bad in the morning when they are waking up.

        • mschuster917 days ago
          Public wifi hotspots tend to give you that opportunity. Large commercial hotspots have inter-client communication blocked, but your average coffee shop style network? You're lucky if it even is an isolated VLAN from the shop's cash register lol.
        • cryptonym7 days ago
          You don't have physical access to the network, doesn't mean other actor aren't here. Working for providers, governments, or being able to impersonate them. This is a serious issue, it shouldn't be downplayed.

          If you think it's not that important, it becomes a critic about TLS and basically saying that ciphers for web traffic are useless.

        • bawolff7 days ago
          Have you never used wifi at a coffee shop?

          Physical access to networks isn't that hard to come by.

          • paaloeye7 days ago
            I have, but good luck predicting where I'm going to have my next cuppa. Outside of petty crimes mentioned, I don't see how it could be industrially weaponised to make actual damage.

            Regardless, real bad actors are busy compromising your next `npm install` anyway. Much better reward.

            • rolosa7 days ago
              Why would I care about you specifically, and not the dozens of other people working at the coffee shop? Does the world revolve around you?
            • bawolff7 days ago
              If they are specificly targeting you, then they probably dont need to use the coffee shop. The coffee shop is for low skill attackers targeting people at random.

              If someone has it in for you and hired a high skill attacker, its not a stretch to imagine they compromised your home network.

            • voidUpdate7 days ago
              Compromising a specific person is difficult. Compromising any old person is trivial
            • 7 days ago
              undefined
            • dahart7 days ago
              > good luck predicting where I’m going to have my next cuppa.

              Why are you assuming there isn’t someone else trying to hack you at the next cafe? Actual damage can happen one person at a time, and can occasionally lead to larger scale compromises. Randomizing your location isn’t going to reduce your risk. Now that everyone who’s paying attention knows they can MITM your Mac, you shouldn’t trust any network.

          • ForHackernews7 days ago
            Great, you have an inconvenient way to possibly-maybe see some random's gmail inbox at Starbucks.

            You'd be better off just mugging them in the parking lot.

            • itintheory7 days ago
              > gmail inbox

              Except it won't impact gmail unless you're accessing it through curl. The browsers already distrusted the compromised intermediates via CRL within hours of disclosure, and removed the affected root via update.

              • 7 days ago
                undefined
            • jfyi7 days ago
              Sure practicality is always an issue. It's an issue that attackers will go to great lengths to resolve too.

              What if it's the Starbucks at 1730 Pennsylvania Ave NW, Washington, DC 20006 for an extended period of time? Compounded with other security issues like password reuse...

              We can always draw a line and say, "that's not practical". The assessment an attacker is likely to do is going to be "is this practical inside my budget for the value I hope to acquire?" or maybe even just "waow, lulz".

            • alt2277 days ago
              No, you have a pretty easy way of intercepting someones git credentials and repo access. Arguably more dangerous than a gmail inbox.
    • chpatrick7 days ago
      You can take your laptop to a busy place, make a "free wifi" hotspot that runs everything through mitmproxy and you can steal everything.
      • itintheory7 days ago
        I think "everything" is an overstatement. Browsers included the compromised intermediates in their CRLs within hours of disclosure.

        So this isn't like Firesheep [0]. Yes, there would be vulnerable traffic, like the git client this thread is discusing, but it's not "everything"

        [0] https://en.wikipedia.org/wiki/Firesheep

        • alt2277 days ago
          If you see a developer using git on a Mac laptop in a coffeeshop, you can intercept that and gain their repo and probably password hashes pretty easily.
    • chjj7 days ago
      > You have to compromise a lot to be able to make a MITM here.

      This is just an argument against TLS in general. Use plain http if you believe this.

      • caaqil7 days ago
        Appealing to complexity is always fun. Just throwing "a lot" in there for good measure and you won't have to think about practical security implications.
    • tensegrist7 days ago
      php2shell was overblown perhaps but this is being blown quite correctly
    • abujazar7 days ago
      The TLS trust chain is the last defense against MITM attacks. The prevalence of MITM attacks is in fact the very reason TLS exists. I'd say it's a lot worse than PHP2Shell because it doesn't only affect poorly maintained servers, but potentially ALL credentials and secrets a user sends over a compromised network with compromised CAs on their device.
    • commandersaki7 days ago
      The window of opportunity to MITM has passed. If you attempted an MITM from an easy vantage such as telco, ISP, or public wifi you'd be trivially detected as a significant portion of devices are patched, including on these unpatched macs as only the *nix commands are affected and not app bundles.
      • acdha7 days ago
        You might be detected but if, for example, you were smart enough not to MITM the same client twice after an error, I suspect a ton of people would ignore it and just assume the airport/coffee shop WiFi was flaky.

        I’d also bet more things use libcurl behind the scenes than we appreciate, and that at least a few of them have things like silent updates which retry after a failure without notifying a user. I’ve seen a lot of agents which just try to periodically check in because they’re designed not to be noisy on lousy networks.

    • catlifeonmars7 days ago
      This is way worse than PHP2Shell in scope of impact and its effects will stick around for much longer. Well, as long as the validity period of the compromised certs anyway
    • educar7 days ago
      If you hijack the DNS, the laptop can be pwned quite easily. Think of all the coffee shops with unprotected routers out there.
      • yonatan80706 days ago
        Exactly, I've been to public places where the Wi-Fi password is on the wall and the router's admin password is "admin".
    • jon1287 days ago
      [flagged]
  • kd9137 days ago
    They just released MacOS 27.2 a few days ago? How and why did they not do a patch at least then?
  • thought_alarm7 days ago
    Delete /etc/ssl/cert.pem and replace it with the one from MacPorts.

        # /usr/bin/curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
        Your client successfully connected to the site. That's bad!
        
        # sudo mv /etc/ssl/cert.pem /tmp/
        
        # sudo ln -s /opt/local/libexec/openssl3/etc/openssl/cert.pem /etc/ssl/cert.pem
        
        # /usr/bin/curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
        curl: (60) SSL certificate problem: unable to get local issuer certificate
        More details here: https://curl.se/docs/sslcerts.html
    • paaloeye7 days ago
      What? Who's still using MacPorts?
      • phone_book7 days ago
        Anyone running Intel macbook because Homebrew dropped support for them because Apple dropped support for them. You can still use Homebrew but it means rebuilding from source. I downloaded uv recently and it took about 8 hours to fully install because it had to build llvm, uv, some other stuff all from source.
      • xmodem7 days ago
        Heaps of us. I'd been using it for years before Homebrew even existed and I never felt that any of the problems that Homebrew purported to solve were real.
      • xp847 days ago
        I know, it always makes my jaw drop when I realize that it’s still a significant number of people, since I remember it being a thing I used circa 2005. I don’t think git was even a thing back then.
      • dear-leader6 days ago
        Intel Mac users. I recently installed MacPorts for the first time after Homebrew dropped support.
  • paxy7 days ago
    This should have been a macOS security update on day 0. Dragging their feet over it is sadly typical when it comes to Apple software nowadays.
    • alt2277 days ago
      Why do people still think of Apple as a leader in security? This kind of thing happens all the time with them these days.
      • paxy7 days ago
        Their branding, mainly.
  • ltbarcly37 days ago
    Microsoft had a small golden age with Azure and Windows 10, and now they are going right back into a Vista-era-like rot. Github is down an awful lot. The reliability is worse than the product I work on for work by like 2 orders of magnitude and we have no ops team. It's a joke.

    Windows 11 is awful, lots of the fixes and modernization started in windows 10 has totally stalled, leaving users with 5 different places to change a setting instead of the one old place or one new place. Look how many different UI's there are to adjust sound card / sound settings or display settings. It's a complete mess. Their attempts to jump into the AI game have been a complete joke, despite already having integrated Cortana years ago. Putting Cortana on an llm is such an obvious thing to do, and they already did so much work to make that fully integrated into windows and force it on everyone even though it was trash, and now it's still trash and they don't seem to be able to adapt at all.

    Now they can't respond to a complete insecurity in one of their major products in weeks? Why do they even validate certs? Why doesn't revocation work in their app? I don't think they can really be trusted at this point.

    • Xiol7 days ago
      Did you misread the title?
      • ltbarcly36 days ago
        They changed the title, it said github client before I believe.
    • tonymet7 days ago
      Microsoft were the good guys in this case.
  • 6thbit7 days ago
    There must be a long list of things that aren’t fully patched for this yet, on laptops, on servers. Think of packages that ship their own trust anchors, isn’t there a rust crate that compiles with some root certs embedded?

    On one hand there’s unmaintained packages that won’t update dependencies. On the other hand, fixed dependencies did not necessarily backport to all possible versions in the wild, making update as a fix unviable for those.

  • lrvick6 days ago
    People use Apple products because they want to delegate all authority over their operating system to a corporation that knows best. If you trust Apple, you should clearly also trust eMudhra.

    If you are finding yourself not trusting eMudhra anyway, then you might want to choose an OS that lets any well vetted and capable security researcher in the world contribute to it.

  • mFixman7 days ago
    Question for people who know about this: why do we have hundreds of root certificate authorities, when a vulnerability on a single one can break encryption?

    Wouldn't it make more sense to have 5 or so that can be more closely monitored, so something like the eMudhra certificates would probably be found before the certificates are deployed?

    • trebligdivad7 days ago
      And which countries shall run those 5?
      • samatman7 days ago
        The permanent members of the Security Council, obviously.
      • mFixman7 days ago
        The US, which already centralises the internet by running .com and developed most of the software using these certificates?

        It's not like they cannot introduce a bad certificate right now. We would be just be taking the ability from all other countries from doing the same.

  • cromka7 days ago
    Things like this is why I moved back to Linux after 10 years on macOS.
    • petcat7 days ago
      Not sure what Linux offers that's any better. Didn't Debian ship trivially weak SSH keys for over 2 years because they were trying to silence Valgrind warnings and the random maintainer had no idea what they were doing and there was no oversight or peer review whatsoever?
      • usef-7 days ago
        That's one issue from one distro 18 years ago. Have there been recent things? This thread is about a today's MacOS, and an issue known publicly for more than a week.
      • 17186274407 days ago
        > the random maintainer had no idea what they were doing

        Upstream was involved in it as well. Sure it was a communication problem, but it's not like the maintainer just decided to this unilaterally. Also this WAS undefined behaviour, so from a language perspective it was already broken, even if it worked accidentally in practice.

        • direwolf206 days ago
          The Debian maintained patched by "fixing" two very similar looking "bugs" on different but identical lines of code:

          One that mixed uninitialised data to the entropy pool.

          And another one that mixed initialised data to the entropy pool with the exact same function call. Which was, in fact, almost all of the entropy.

      • bawolff7 days ago
        In 2008.

        I don't think its wise to make current security decisions based on screw ups from almost 2 decades ago.

        • petcat7 days ago
          Well that "screw up" was so egregious that it should take at least a decade to regain any trust.

          I don't really care one way or another. I use Linux and I use macOS. But I'm not naive to think the security is any better in one or the other.

          • cromka7 days ago
            > Well that "screw up" was so egregious that it should take at least a decade to regain any trust.

            Do you use macOS? Because if you did, then how did you recover from them allowing anyone to straight-up log in as root without password? And then mess up the fix for that bug? That wasn't 2008, that was 8 years ago.

          • alt2277 days ago
            > it should take at least a decade to regain any trust

            Then as it has been 2 decades you should be all good right?

          • fg1377 days ago
            Security practice was generally very, very bad two decades ago, by today's standards, across the industry.

            I don't see the point of being so fixated on that.

            • petcat7 days ago
              I was responding to the OP saying "stuff like this is why I'm moving back to Linux". It's nonsense.
              • alt2277 days ago
                No, pulling up a bug from 18 years ago to argue against the fact that linux is secure is nonsense.
              • cromka6 days ago
                And yet you never responded to my "login to macOS as root without password" argument.

                That's convenient!

          • ajjahs7 days ago
            [dead]
        • simondotau7 days ago
          Somehow I doubt many people will be saying that about eMudhra a decade from now.
        • baobabKoodaa7 days ago
          Two decades
        • efficax7 days ago
          i think you mean almost 20 years ago old man
          • bawolff7 days ago
            I edited my comment
      • kobalsky7 days ago
        You are going so back that gmail was served over HTTP (not HTTPS) in that era.

        If that's the worst you have against Linux then it looks like it has done very well.

        • cromka7 days ago
          Not really, no. HTTPS was used, except for log-in only. But cookies traveled unencrypted, so you had access to the account anyhow.

          And not just GMail, this was virtually every service out there. From a perspective, it's laughable how bad security was back then.

      • mattstir7 days ago
        Besides singling out a single bug from a single distro two decades ago, Linux offers maintainers who actually care to push security fixes within a few hours of globe-spanning vulnerabilities being disclosed? Not sure how that's confusing or controversial.
    • boxed7 days ago
      It's funny. When Apple goes in and changes peoples systems deeply without consent then it's Apple being evil. When they don't, it's Apple being evil. Heads I win, tails you lose.
      • 1970-01-017 days ago
        Let's pretend there's a night and day difference between pushing some MP3 onto all iPhones and distrusting a chain of certs that hold the value of '4'. At minimum Apple need to revoke everywhere these joke-level certs live and as fast as possible, and on all devices which also means out of support. Breaking is acceptable here as CA trust is fully compromised.
      • mattstir7 days ago
        Yes, there would be crowds of angry people, furious that Apple... released a critical security patch? Framing removing compromised certs as "changing peoples systems deeply" seems intentionally in bad faith.
      • 17186274407 days ago
        It's funny, when I do steal something I get called bad, and when I do nothing and turn the blind eye to others stealing, I get called bad too.

        Morality doesn't solely depend on action or inaction, it also matter what you act or don't act to.

      • simondotau7 days ago
        I've seen this referred to as the "All Roads Lead to Rome" fallacy.
      • cromka7 days ago
        Eh? Changes systems without consent? Is that the ONLY way to update a fatal flaw in system? Normal update channels don't work?

        Besides, how does this relate to my Linux point, where no one installs anything without you knowing it?

        • boxed7 days ago
          Because in the linux situation, you wouldn't get such an update automatically anyway so it'd be worse?
          • cromka6 days ago
            How would it be worse? Linux distros patched things almost immediately and you'd get a notification, typically after a daily check.

            How is that worse than NOT having any sort of fix from Apple?

      • kqp7 days ago
        I don’t think you understand what consent means.
  • devy7 days ago
    Like many suggested, a proper fix might be harder than simply revoking a bunch of certificate like the blog authored assumed without breaking downstream features and functions in a large ecosystem and security conscious platform like macOS/iOS.

    However, there is certainly a chance that Apple team didn't think the impact was that bad given all the other protections in place.

    When Steve Jobs was alive, his Apple company email was leaked and published and Apple customers emailed him for features and fixes all the time and he personally responded once a while. Tim Cook did the same a couple times too. Perhaps the blog author should write in to the new CEO John Ternus or SVP of Software Engineering Craig Federighi too to see if they can fix that.

    https://www.apple.com/leadership/

    • iAMkenough6 days ago
      John Ternus’ team ended that tradition and doesn’t accept emails from his customers. Just ask Siri AI.
  • saagarjha6 days ago
    I would not be surprised if what happened here is that Apple had hardware to release this week and needed to get 27.2 out the door with support for them, and it was "too late" to merge in the certificate store changes. Of course, I think most people here would argue that this is completely unacceptable and I would more or less agree with them, but unfortunately Apple has never been capable of validating a release quickly or pushing small patches. They've tried and, well, most of those efforts have not worked at all. I wonder how they will deal with a new full chain targeting their OSes every week or whatever AI is going to do.
  • thefilmore7 days ago
    Apple can fix this by building curl with SecTrust [1].

    [1] https://eissing.org/icing/posts/curl-apple-sectrust/

  • kqp7 days ago
    Can we get a title edit? OP’s title implies this is a Git bug, but the bug applies to half of macOS, including curl, openssl, python, perl, and ruby, and to only the macOS version of Git. The original title is less misleading, but personally I’d have titled it something like “macOS still trusts the compromised eMudhra certificates in git, curl, openssl, python3, perl, ruby, more”.
  • altcognito7 days ago
    I'm not that familiar with the MacOSx system, but do they have a "minor" update path that I'm not aware of? It seems that everything is an 8gb update. Is that why this update isn't getting put out?
    • pram7 days ago
      Yeah they do, it's called "Background Security Improvements" but they rarely use it lol
      • bon0r7 days ago
        Now would be the time :)
  • davidcollantes7 days ago
    Running (uname -a) Darwin arrakis.local 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:07 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T8132 arm64, I get the following for both:

        curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)    
        More details here: https://curl.se/docs/sslcerts.html
    
        curl failed to verify the legitimacy of the server and therefore could not
        establish a secure connection to it. To learn more about this situation and
        how to fix it, please visit the webpage mentioned above.
    • lukasgelbmann7 days ago
      You could be running a different version of curl, e.g. homebrew. Try `which curl` and `/usr/bin/curl <url>`.
    • jpitz7 days ago
      Uname is not enough here to identify what has or has not been done to your system to deal with the compromised certificates.
      • davidcollantes7 days ago
        No, no, I meant it to show that the machine is running the current/latest macOS, and the results I am getting from curl on the two URLs.
        • microtonal7 days ago
          Maybe you have another curl in your path?

              $ uname -a
              Darwin MacBook 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:21 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T6030 arm64 arm Darwin
              $ which curl
              /usr/bin/curl
              $ curl https://dv-g3b-emudhra.we-have-a-problem.com/
              Your client successfully connected to the site. That's bad!
          • davidcollantes7 days ago
            Aha! Touche! Indeed, I am using brew's curl ("/opt/homebrew/opt/curl/bin/curl"). That's why; the one at "/usr/bin" connects just fine (facepalm).
  • j457 days ago
    Appreciate the breakdown.

    It's totally wild that there isn't an interactive script/check already built to test, detect, and run the solution on all fronts.

  • __MatrixMan__7 days ago
    If you're not already in the habit of contacting your git remotes over ssh, its probably one worth building. This is gonna keep happening.
    • mort967 days ago
      I use SSH for things which are private or which I may want to push to, but I use HTTPS for everything I just need to download. SSH typically requires that you create an account with whatever the git host is and upload your public key.
    • kbolino7 days ago
      SSH in typical use is not vulnerable to a compromised root certificate, instead it's "just" vulnerable to you having no idea whether you can trust that host key or not, so you just blindly say yes and hope for the best. At least if the key changes, you'll know something was wrong; maybe it was this connection that got compromised, maybe it was the first connection that got compromised, maybe the legitimate host just rotated their keys, who knows. SSH security against MITM is mostly based on lack of motivated attackers and wishful thinking.
      • __MatrixMan__7 days ago
        Its no more wishful than expecting a root of trust to stay trustworthy or expecting a trusted name to consistently resolve to the same party.

        And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.

        • kbolino7 days ago
          Yes, it is about the same, that's the point. If you are worried about this or future compromises of TLS/PKI, using SSH naively is not any better.
          • __MatrixMan__7 days ago
            Maybe it's the same probability of being a problem, but it would be a problem introduced by different kinds of threat.

            If you're worried that you'll be targeted as an individual by somebody who cannot coerce a CA but can tamper with the channel you're using to get the public key, use https.

            If you're worried that you'll be targeted along with many others by somebody who can coerce a CA, use ssh and ensure that some folks are comparing key fingerprints such that at least one is close to the source.

    • paaloeye7 days ago
      yeah, it's a good habit to have. Unfortunately, GH defaults to the TLS transport.
  • 01CPMD31745 days ago
    I‘m grateful I found this page. Ran the command in the article a s discovered my machine was a sitting duck via Git.

    Seems like via AI everyone is discovering their systems are compromised. Can’t blame Apple for making the discovery. 10 days and no patch is another story.

    Hope they get it together and fix this soon.

  • hmartin7 days ago
    I worked with GPT to create this script based on the Apple doc, also with helper options for ensuring homebrew's python and java are up to date:

    https://gist.github.com/hbmartin/d485b07691d9dc2a66937f50719...

  • bluechair7 days ago
    To get mine to finally update, I had to leave my laptop plugged in for a couple of hours. There’s some trustd process that runs every so often; I couldn’t figure out a way to manually trigger the sync.
  • tonymet7 days ago
    Why is the cert.pem not generated from the keychain storage? Does someone want to vibe-code a utility to do that? It seems like the better approach (one master read-write storage with replication)
    • xp847 days ago
      I might find that a good tool, too, but it’s worth pointing out that even doing that upon any change to the keychain trust settings won’t fix the full problem here. The bad “4” certs are also signed by another root, Sectigo, so when CRLs are not being used (a case that seems common for the cli tools), then the certs downstream of the “4” certs have an alternate path to a trusted root. Unless you also nuke the Sectigo cert. which I would personally, but I have a different risk tolerance for breaking random sites than I guess Apple expects its whole customer base to have in the aggregate.
      • tonymet6 days ago
        good point -- since the revocation is not recorded in the /etc/ contents.
  • kleash7 days ago
    Ran security find-certificate -a -c eMudhra /System/Library/Keychains/SystemRootCertificates.keychain on my work laptop out of curiosity and yep, still there :|
    • direwolf206 days ago
      The emudhra roots were not compromised
  • waffletower6 days ago
    Actually, command line tools like git and curl could choose to support Certificate Transparency as browsers do, and the need for piecemeal manual updates could be reduced.
  • direwolf206 days ago
    Lots of software does. It's been great for MITMing your own devices to find out what they're doing. It doesn't defeat cert pinning though.
  • biglyburrito7 days ago
    • 9dev7 days ago
      That page is an insult to their customers. So what, if I want to be protected from a security issue you acknowledge, I'll need to make modifications to my system configuration?
    • 6thbit7 days ago
      It’s interesting they call out as “optional” removing sectigo roots.

      If you’re not removing them you better be sure anything that runs on the system has full revocation support including intermediates. Which is.. kind of an impossible thing to assert as true. Which is why most trust stores just removed that as well last week.

      • lukasgelbmann7 days ago
        Yes, it’s insane. Step 2 does next to nothing for security without step 3.
  • 7 days ago
    undefined
  • teknologist4 days ago
    Intelligence agencies still love this one weird trick...
  • dennisjoseph7 days ago
    looks like people are buying the stocks of the company(EMUDHRA) after the hit they took last week > https://www.google.com/finance/quote/EMUDHRA:NSE?window=1M
  • aaomidi7 days ago
    This file is used by a bunch of CLIs. Apple is fundamentally dropping the ball here from a security perspective.
  • hackerbrother7 days ago
    It seems like these trusts should all be delegated to the OS unless there's a really specific need.
  • franktankbank7 days ago
    What would cause this delay from apple? Is there something behind the scenes that can explain it?
  • amelius7 days ago
    Don't run important stuff on your consumer devices. Apple is a consumer brand.
  • bethekidyouwant7 days ago

       which git
       /opt/homebrew/bin/git
    skill issue tbh.
  • 7 days ago
    undefined
  • meta-level7 days ago
    /e/OS from 2026.10.21 also seems to be still affected
  • WesolyKubeczek7 days ago
    Doesn’t macOS have a way to generate cert.pem from the keychain?
  • BadApple20266 days ago
    > https://support.apple.com/en-us/149555

    I applied those commands.

    Then, last night, I applied the macOS 15.8.2 from Apple.

    Voila, the eMudhra certificates were back. :-(

    So I re-applied those commands once more.

    Come on Apple, you can do better than that!

  • bcvvvy7 days ago
    I have been wondering recently if Apple are bothering to maintain the Unix side of the OS. Seems like they occasionally remove or deprecate stuff but don’t put any serious effort into maintenance.

    This is just another one of those smells.

    • xp847 days ago
      It was arguably always halfhearted at best. Look at how behind they were on many things for years. They update things, I assume, when necessary to make the build toolchain work, and possibly to hold onto their POSIX compliance status which some us government buying contracts demand. The fact they changed the default shell to zsh instead of that decade(s)-old version of bash they used to have (avoiding a GPL license change, I think?) was a miracle.
  • 7 days ago
    undefined
  • zoobab7 days ago
    The only thing that works is fines.
  • InsidiousWombat7 days ago
    That's disingenuous title

    It should be "Apple still ships compromised CA in /etc/". Git is just a victim like every other app using system CA

    • okr7 days ago
      Is using the system CA not the way to go?

      (on linux you always have the update-ca-certificates command that downloads the root certificates from a trusted™ source. but sure, it will not help, if each command starts to bring their own ca package :)

      • Zambyte7 days ago
        Exactly, so the title wrongly frames git as trusting certs that it shouldn't, when it's correctly trusting the system certs, and MacOS is wrongly trusting certs.
      • mschuster917 days ago
        The problem is macOS has two system CA stores, keychain and /etc/ssl/cert.pem.

        That is unequivocally bad, similar to Java which ships its own keystore.jks and many applications shipping its own copy on top of that. True nightmare fodder.

        • krackers6 days ago
          The thing is Apple used to do this correctly. Back when they still shipped OpenSSL (albeit some ancient version), they had actually patched OpenSSL to do cert validation against the native keychain and only ever had Keychain as the source of truth. You can see this in https://github.com/apple-oss-distributions/OpenSSL098/blob/O...

          And curl was patched to use the native TLS implementation (SecureTransport) which obviously used Keychain as well.

          For some reason when they switched to LibreSSL everywhere they decided to ship their own copy? I don't know what they were thinking.

        • WesolyKubeczek7 days ago
          Also certifi in every Python venv, keystores in every single container, nested keystores within that, it’s certificate matryoshkas all the way down!
          • mschuster917 days ago
            > keystores in every single container, nested keystores within that

            At least for that, in $OLDJOB I went and created a new keystore.jks myself to account for an internal CA, so my successors can just update that file, restart the containers and that's it...

        • kbolino7 days ago
          It's also an indictment of Apple's relationship to open source software. Never mind contributing back to upstream, they haven't even made the change in their own fork to support their own keychain.
      • RetroTechie7 days ago
        > Is using the system CA not the way to go?

        Yes, and that makes not dealing with this issue worse on Apple's part. If MacOS' setup somehow doesn't allow untangling a (still deemed trustworthy?!?) root certificate issued by eMudhra from the broken intermediate certificates, then revoke that whole chain including the root & accept collateral damage.

        Failing that just provides an incentive for apps to do their own chain-of-trust implementations. Which like you said is not the way to go.

    • broken-kebab7 days ago
      So git on MacOS does trust compromised certs by default? What's wrong with the title then?
      • Liquid_Fire7 days ago
        It's not factually wrong, but it implies that this is something specific to Git, and possibly might even be interpreted that it is Git's fault.

        In reality it's macOS shipping the outdated certificates, and it affects various other command-line tools, like Curl.

        • broken-kebab7 days ago
          Fair. I didn't read it as something specific to git cause I know how it works with MacOS
        • boxed7 days ago
          Not macOS, but XCode dev tools. Still, it is Apple that ships them for sure.
          • SahAssar7 days ago
            Does /etc/ssl/cert.pem not exist if you don't install XCode?
      • publlus_enigma7 days ago
        Because it isn't got specifically, it is anything that uses the installation of OpenSSL provided by Apple as part of macOS (at least is my understanding).
      • smithofsmiths7 days ago
        The problem is that the title implies that it's somehow the fault of the package. That's it, that's the problem; the title is disingenuous in its claim.

        "Ten days later, Apple Image Viewer on macOS still trusts the compromised eMudhra certificates" doesn't sound as catchy, but is equally correct because the apps use system CA.

        • xp847 days ago
          No, unless it’s were a cli image viewer. The gui parts of the OS, and what’s written by Apple, use keychain.

          Macs have two completely independent root CA stores. The cli and all the opensource parts use /etc/.

          Curl project did actually release support to compile it with default keychain support a year ago, but of course Apple never updates their cli tools, so expect to see that in macOS 37.

          Note: I’m not defending Apple here.

    • JoBrad7 days ago
      The lede is certainly buried, but calling out an affected development toll on a tech site isn’t a bad call.

      Pretty disappointed that Apple hasn’t pushed out an emergency patch to address this, though.

    • sscaryterry7 days ago
      Agreed.
    • anothermoron7 days ago
      [dead]
  • yourmahbub6 days ago
    its very good
  • jasonm236 days ago
    Well, you know, Apple invest so much into development

    : - /

  • commandersaki7 days ago
    Not really a big deal, the window to do an MITM has passed.
    • computerfriend7 days ago
      No, it is open for as long as these certificates are trusted.
      • commandersaki6 days ago
        When a significant portion of devices are patched (including gui apps on mac), pulling off an MITM without detection is pretty much impossible. That would be usual culprits of telco, ISP, or public wifi. That's what I mean by the window has closed. Anyone attempting is taking significant risk of being detected; at least in my neck of the woods would be illegal to do as well.
  • fHr6 days ago
    small indie company lmfao
  • arbor-group7 days ago
    [flagged]
  • 40PT7 days ago
    [flagged]
  • rbaRBA7 days ago
    [flagged]
  • ajjahs7 days ago
    [dead]
  • jjjegly6 days ago
    [dead]
  • 6 days ago
    undefined
  • jon1287 days ago
    [flagged]
  • paaloeye7 days ago
    Albeit helpful, the site is kinda cringe. It leaves a bad taste in my mouth knowing that the author is affiliated with Mozilla.

    Of course they would throw shades at Apple. That being said the situation isn't ideal and I'm sure Apple would have shipped an update if it were easy and warranted.

    • lucideer7 days ago
      Affiliations aside I think I'm happy to see shade thrown at anyone who hasn't bothered to even issue a patch 10 days after what has been one of the biggest news weeks in security in memory.

      Fwiw, as critical as I am of Mozilla in recent years, they (along with the Chrome team) have been pretty solid on this whole debacle over the last week. Apple has been far from the only target of their collective shade-throwing.

    • therealmarv7 days ago
      Your apps and browser don't trust them. Why should your CLI. It's easy and warranted for Apple and look how much cash and engineers they have.

      Apple should publish a hotfix now. Work on a better solution in the long-term. They just don't care about security for developers is my take (it's like a niche group for them?)

      Mozilla or not does not matter here.

    • MaintenanceMode7 days ago
      A concern for me on this site is that they give the shell commands to edit the /etc/ssl file on their site, instead of just making the Apple link (which they also list) as authoritative on the matter. I don't think users should get in the habit of cut/pasting sudo enable shell one-liners from "random" websites, especially and specifically when the vendor has the very same commands in their published fixed for the issue. In short, this site should have just said "go get the fix from apple".