E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?
https://www.wsj.com/politics/national-security/china-cyberat...
The truth is things do occasionally happen and we should be prepared, even if most of the time they don't
Keep in mind who the President is.
If so, successful test.
These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.
> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.
https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...
I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.
Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.
Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/
(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)
Like all the doom and gloom after the Twitter layoffs predicting the site would implode and go permanently offline "within a month" which...never happened.
It's also ironic in the sense it implies the indignant people were so bad at their jobs they designed and built a system so fragile it would collapse without constant intervention from thousands of individuals.
You do realize it's possible for an organization to be overstaffed?
I realize your point, but its fair to say maintaining a nationwide physical wireless infrastructure may not be the same as hosting tweets, particularly when outages strike.
That's correct.
In the case of Twitter, it was disclosed that many of their systems were running out of date EOL software, to the point of being a security liability, which raises the question: if the systems weren't being maintained, wtf were all those people doing? Taste-testing the free food and cappuccinos?
You do need too many people to work with that. Cutting them is asking for pain.
This is more likely a management problem rather than a staffing problem. Lower level management knows about these kind of things but often they are not incentivized to make them a priority due to a culture focused on growth and “winning”.
When Twitter did, its CEO may have slept at the office for weeks to make sure problems were resolved.
On the other hand, the Verizon CEO may be shopping for a new boat
Many think Twitter has imploded, though it's online.
> You do realize it's possible for an organization to be overstaffed?
It's possible to be understaffed or appropriately staffed. Anything is possible!
I can't even begin to imagine what those 100k people actually do. For starters, none of the telcos actually develop their own equipment - they buy pre-made from vendors like Ericsson. Often that includes ongoing maintenance too. The only "engineering" is building the back-office and customer-facing UIs, and even that is often outsourced (as a rule of thumb, if something can be outsourced, telcos will do it: https://berthub.eu/articles/posts/5g-elephant-in-the-room/).
Customer service might be part of that number (assuming that too isn't outsourced), but even then 100k feels extreme.
10k is ok although leaning on the more bloated side. But 100k?
To who? What evidence is there either way?
Is it an often-repeated story online? That sort of information is both well known and unreliable - it's well-known misinformation.
Big Tech companies operate much more complex systems (for starters, they actually build greenfield stuff instead of buying ready-made equipment from a vendor and plugging it in) and have way less headcount.
You're building new cell towers, managing countless failed backhual links (thanks to fiber's natural enemy, the backhoe), working with whatever obscure bugs your MVNOs have managed to uncover, certifying new cell phone designs, and still working on upgrading everything to 5G while simultaneously planning for 6G (keeping in mind that the 5G network architecture looks radically different than the LTE architecture). Much of that work is necessarily physically distributed across the entire country.
Not to mention dealing with end-user sales and support, which unfortunately often needs physical stores.
I'm not going to say whether 100k is too many, but there's a lot more involved here than just maintenance and monitoring - especially if you want your network capacity to keep up with growing demand.
The problem is that this is a culture problem and once a company is ossified it is really hard to enact such change from the inside even if you wanted to because everyone enjoys the status quo (and who doesn't wouldn't be there to begin with).
Another example: have you seen the UK & EU banking scene and the boom of fintech and "neobanks" around 2017 like Revolut, Monzo, Starling, N26, etc? They managed to build from scratch on relatively shoestring budgets their own implementation of a consumer bank, something that their legacy competitors still can't replicate despite having way more budget and resources.
Unfortunately, the telco world is an oligopoly and they don't like new entrants (banking in the UK was actually a much more level playing field in comparison), so we can never actually see an experiment that proves or disproves my theory.
Yes, this explains most of our jobs.
The main downside is that you have different numbers for each eSIM, but that doesn't really affect me because I use Google Voice for SMS.
https://www.google.com/appsstatus/dashboard/incidents/THtvkD...
But yeah I probably should have clarified that Google Voice has been a pretty terrible UX and quality overall for years. I really need to just bite the bullet and port my number out.
I was previously on Google Fi, and MVNO roaming is bloody fantastic - I always had reception.
As I've said elsewhere, never underestimate Verizon's incompetence. A couple years ago they shut down vtext service which also happened to host their UAprof's for most Android phones. Without a working link to these descriptors, those devices would download any MMS (pre-RCS picture messages) in compatibility mode at the lowest resolution.
I'm in a similar boat to you in wanting a LoRa mesh. I tried out MeshCore on the LilyGo T-Deck+ hoping it would be a device I could hand out to family members, but I found the hardware and software disappointing.[0] But I'm weirdly tempted to try the LilyGo Pager.
[0] https://mtlynch.io/first-impressions-of-meshcore/#testing-th...
Ironically I was planning to port my parents from Verizon to US Mobile to save them some money since they aren't financing any devices (there's a sale from them that ends today) and I've just done that on the first line and I now have service on that line with Verizon, where the remaining Verizon postpaid is still dead.
"Verizon engineering teams are continuing to address today's service interruptions. Our teams remain fully deployed and are focused on the issue. We understand the impact this has on your day and remain committed to resolving this as quickly as possible."
As someone responsible my whole career for uptime and network response, I really feel for the engineers, at the same time hoping my service comes back up soon. SOS
It came up for around 5-10 minutes at 15:00 EST, but is currently still down.
Something like a routing configuration, BGP failure, or underlying network misconfiguration would cause seemingly bizarre results with some phones working and some not with no obvious correlation. Compare Access Point Names under Mobile Settings on android, and whatever the equivalent is on iPhones, and check things like whether 5G allowed and data roaming is enabled.
If it's a cyber attack of some sort, then there's all sorts of different attack vectors that would cause these outcomes.
I assume state on state cyber attacks are commonplace but get minimized to avoid public fear.. perhaps this will be the first notable one.
Made worse by the fact Estonia is a more networked society than, for example, the US.
https://downdetector.com/status/t-mobile/ ~ 1,600
https://downdetector.com/status/att/ ~ 1,500
https://downdetector.com/status/verizon/ ~ peaked at ~169k, dropped to 67k
https://downdetector.com/ shows verizon, tmobile, and att. BUT if you look at the magnitude of the outages for tmobile and att vs verizon it's fractions of a percent. Likely those people with tmobile and att reporting when they have trouble communicating with verizon customers.
(Note, you have to click on the providers to see absolute magnitude -- the graphs are scaled to show relative outage over time within a given provider; order of 150k verizon vs 1.5k others)
I noticed that a little while ago too. Very bad UI. Tempted to post in r/dataisugly.
Looking at the replies some of the people who are more worried about the outage seem to be the OF models
Not judging or trying to make a point. Just find it interesting the ways things are interconnected
Do you use your car’s built in navigation function — that you paid for — or do you plug your phone in and use its free Google Maps or Apple Maps to navigate?
Edit to say: my Verizon FioS and cell service are both working fine, no noticeable interruption at any point today.
Second edit to say: never mind, downdetector's home page normalizes report spikes so 1k and 100k both look identical.
I did manage to roam onto an international network on the boarder near me in New York/Canada, so some bits of the core seem functional for authentication.
When I roam internationally I appear to be on Telus's 3G network (no LTE) for data and voice is falling even further back it looks like.
It's a citizen-run mesh network that allows text communication. It's not perfect and not 100% reliable (signal and hops limit delivery guarantees) but it's better than 0 ways to communicate in this kind of event!
BART Defends Cutting Off Cell Service In Subway
https://www.npr.org/2011/08/16/139664637/bart-defends-cuttin...
And will happen again.