I actually think it’s the opposite. I think people will move back to their own hardware with OSS deployment like dokku. If everything is more bespoke the attack surface is smaller as is the pay off. Hacking railway got access to lots of api keys, hacking some random vps gets less payoff.
How is that ever going to work out? The speed of discovery for LLMs is so fast, these places are getting breached more often than not due to a human errors. Its really hard for machines, let alone humans, to guard against a lot of LLMs going against them 24x7.
I have a bridge to sell you if you think everyone on the Azure team is sub-moron. Infra security is a hard problem to solve, GCP and AWS don't quite have a perfect track record either, perhaps AWS stands out as a better option but it's hard to argue about it.
I don't think the blame lies with the intelligence of the Azure team. I would argue that there exists structural flaws in the business incentives which deprioritize 'users' from 'customers', along with the stark lack of consequences for security failures.
That's a problem across all of tech, I know a startup that recently had a security incident, lost over 40K USD and none of the heads have rolled they haven't completely revamped their security posture or anything like that.
While there are incentive problems across all levels, The political quagmires that make it impossible to move at any reasonable speed, and that lead to counterproductive ideas getting large budgets are in my experience very common at the highest sizes. It's true that on the other edge of the spectrum, small startups make poor choices, but many of said startups will fail due to said choices. So by the time you have a viable company making good money, the really badly aligned companies are gone.
So when I look throughout my long employment career for best security and organizational practices, all the highlights are still relatively, relatively undersized companies, with headcounts in the 3-4 figures. It's as they grow, and start to be multi-headed behemoths, that they decay again, as what is good for the responsable mid-level exec, for the business unit, and for the company as a whole start to have nothing to do with each other. That's when you can smell the decay in the hallways. A few large companies manage to stave off this rot a little more than others, but it's inevitable. Then nothing important can be done in a timely fashion.
It didn't help that azure (especially in the beginning) was a cobbled-together mess of ancient Microsoft tech stacks that were never intended to talk to each other.
Yeah, if Exchange goes down, you get a day off. If the little boutique email provider you chose goes offline, you get a day in calls with everyone yelling at you.
Azure (and AWS) offer Gov clouds, and probably on prem air-gapped deployment of their full stack of Fabric for that type of NEED security customers who can pay for it.
It is trade-off between risk and access/productivity with a cost dimension always.
Depending on your threat model you could go to typewriters /paper as some Russian department did famously few years back too.
With a lot of PaaS type vendors you are effectively exposed to both the vendor themselves and the public cloud they run their stack on top of.
Broadly I think you're much more likely to be the victim of a platform-level breach involving an immature startup PaaS than a hyperscaler cloud, however I think for small teams the alternate risk you take on is that AWS/GCP/Azure are full of landmines and extremely easy to misconfigure in insecure ways if you don't know what you're doing. Especially if you have AI telling you confidently wrong things.
When I was at Microsoft we took security cripplingly seriously to the point we could not deliver features and our competitors could because the security process was convoluted and had multiple reviews and approval steps.
Having crippling bureaucracy does not correlate with good security outcomes, except maybe negatively. However, having a history of unending egregious security failures, like Microsoft, does correlate with poor security outcomes. For that matter, having a present of unending egregious security failures also correlates with poor security outcomes.
Having crippling bureaucracy does have better correlation with security outcomes than the “move fast, break things” philosophy that most of the software industry lives by.
I’m not even blaming Zuck for that, he didn’t really invent it, though he certainly popularized it — he just made an astute observation of the dynamics in software and turned it into a company motto.
Neither is correlated with good security outcomes against the prevailing threat landscape of profit-motivated, commercial, professional attackers with tiny budgets of a few million dollars. The standard is not "better", it is "adequate".
Correlation is not an absolute measure here, it’s relative. As is the entire concept of security.
As is what constitutes the standard of “adequate.” Which BTW is the standard in ALL engineering (not just security or software) because we live in a world where economic pressures apply and costs must be managed. Trivial example: bridges only use an “adequate” amount of steel and concrete because they are not free. The word “adequate” fundamentally involves the concept of tolerances. There’s a reason we consider over-engineering a bad thing.
So which development approach would you say is better correlated with good security outcomes?
I do not see how that is relevant. Are you saying that trivial cross-tenant compromises are not egregious security failures demonstrating their systems are unfit for purpose because other companys make products that have even more egregious security failures that are even more unfit for purpose?
My bridge that collapses is adequate and acceptable because other people make even worse bridges that collapse even easier? No, the bridge still collapses, everybody gets a F.
Practically every software security firm in the country was a vendor to Microsoft in the mid-late oughts and I know the Microsoft you're speaking about, but the vibe I've got lately is that things are different there now.
They get access to the real frontier models that the plebes don't have access to. From that perspective, the probability is higher that they'll find and fix issues before the infra startups who haven't kissed the right rings.
Yes and no. They (or anybody else) are not guaranteed to be secure, of course; see all the compromises regularly found in cloud providers; Azure especially has had so many cross-tenant vulnerabilities (there’s one on the front page right now!) that I’m amazed the concept of public multi-tenant clouds still exists.
But they are also very highly incentivized to be more secure. They have millions of customers (hence a correspondingly higher expectation of trust) and billions in revenue from the trillions in economic activity happening on those platforms (hence a correspondingly larger target on their backs.) This means exponentially higher liabilities. And that probably matters more than any other factor.
Adding on to it, even though they may be using AI as well. This could very well be a placebo and deniability factor but the fact remains that they have many many qualified employees and the chances of bugs passing through that are much less compared to other systems. Also it's that the systems are very severely pen-tested and have years of bugs removed out and the incentive of billions of dollars at stake that you mention that they have for the trust premium.
They will still make mistakes but they have the resources to make them less.
I doubt that such intense level of thorough testing can occur in a smaller company.
I still love small VPS providers and use them myself but it is worth knowing why enterprise is willing to pay a premium over those to these hyperscalers (it is a trust premium)
IMO, There is no "middle-ground" to the situation.
I wanted to write a blog post about it as I was thinking the same but I guess a comment will help me get my thoughts across just as much.
Either we are gonna see a convergence to hyperscalers as you mention, or we would go back to completely indie companies which compete on perf/price.
There could still be some value to "nicer to use" certainly but with AI, we are gonna see almost every product as being "good-enough" if I am able to convey my sentiment.
An anecdote is: Proton's an middle-ground company. I am seeing the Verge article which states that their CEO has gone a little crazy over the use of AI and the wording makes me lose trust within it. Proton's whole point is security and yet it's having this, It has some stocks in non profit and is a swiss company. It shouldn't have been this way and would've been one of my least anticipated companies but I guess they want growth at all costs and the growth has been slightly lesser than expected and now they are burning the good-will in many ways.
I have to wonder that even if proton might have good engineers, then within this pressure by their CEO/CTO. I cannot trust the engineering talent because of these issues because they are very much forced by their company to use AI.
Proton is also really expensive.
Now just today, also, with a company so big as binance getting hacked and losing 6.5 Billion$. My trust within security just because its a large company is less.
On the other hand, I know a guy called Vikas on lowendtalk who operates his own mail platform called mailbig. I know he vibe-coded the landing page and might've vibe-coded something but it is really cheap. I also know francisco whom I have talked to who runs namecrane. (I think mailbig uses namecrane in many ways)
Perhaps E-mail can be a sensitive matter because you have to place trust in the provider even more because the protocol is not really that much secure but suppose it's an matrix instance (where it is genuinely encrypted) then that shifts it even further. I do trust NameCrane and NameBig from what I can gather.
I'd much rather support them and their cheaper deals than some other middle-corp. Sure some security factor can come in but also, the difference feels comparatively less and the balance has been made
The market will slightly squeeze as Hyperscalers will introduce the solutions of the middle-ground and the new players will everyday enter into the field with more competitive pricing just as well.
So in short, Enterprise will move towards the hyperscalers. Generic Consumers will move towards the cheapest option. The moat of the middle ground where UI/UX was the differentiator is being turned down. There would be some mix and match and there will be cost-savings for Enterprise to move from hyperscalers but it would depend on a lot of factors.
This will be the downfall of the torrent protocol. If you think governments (US and China) are going to tolerate people downloading copyrighted games, movies and music without paying the rightsholder, I have a datacenter to sell you.
SOTA open models need tens of thousands of dollars worth of hardware to run.
I get your analogy, and it feels right, but the compute requirements to run these models are extreme. That analogy falls apart when it necessitates having a corporate 3rd party run the models for you. They will not tolerate breaking laws.
If China says no more, then the best the masses are gonna get on their high-end setups are the 30B class models. Which are decent, but not "hack my bank", errr, "protect our bank" decent.
Piracy flourished because you only need an operating laptop to participate. LLMs you need a server rack of GPUs to participate.
I have news for you. A lot of enterprises are moving a lot of workloads back on prem and are expanding their datacenters and on prem GPU compute. Also replacing a lot of SaaS with internal tooling.
I ran some stats on producthunt a few months ago, there are 22 times more daily SaaS launches now than in 2020, the AI impact on the industry is hard to fathom. The blog I wrote (AI-assisted) if you're curious: https://www.mahl.me/blog/product-hunt-is-swamped-now/
How these Paas (Railway before) are being breached? Poor choices or something else? Scary Nonetheless. I thought AI advancement will make the self hosting painful. If PaaS are no good, then what can be done?
Modern PaaS is tough because there's a general expectation that users (and agents) are able to see and read back environment variables in their management UI or CLI - which means that you can't segregate credential values from the public web environment. See e.g. the visibility icons on https://northflank.com/docs/v1/application/secure/inject-sec...
And sure, you can encrypt at rest, and perhaps even enforce that only re-encrypted versions are ever seen by the management/API web server. But fundamentally this display requirement means that if your management/API web server is compromised, users' secrets can be obtained by a process of impersonation.
Stuff made during the move fast and break things era now doesn't hold up very well in the AI era. Or to say another way: startups are very vulnerable now - this new environment is unforgiving towards cutting corners and/or having inexperienced devs
Why would ai advancement make self hosting painful? Apart from increased cost of hardware (which you implicitly would pay for in PaaS as well anyways).
If anything, AI makes running linux boxes much easier than 4 years ago; it's now faster and the learning cliff is gone to get the correct CLI commands and digging into system logs.
When you build something small, you can easily see all the doors and properly guard them.
As you build something bigger and bigger, quickly, with several people building, guarding all the doors becomes exceptionally difficult without a kind of aggressive discipline which is contrary to the "go fast" culture.
Cybersecurity in practice until now, for most companies, has been an insurance problem, not a quality problem.
Such companies are quickly discovering that insurance is a post-hoc partial correction and not a proper solution.
This situation will not improve anytime soon for most companies.
The solution does not require any new magical tools.
I feel like these breaches started to accelerate after Anthropic wrote a full post about how easy it is to abliterate glm 5.3 to make a frontier-level cyber model.
Now, to be clear, abliteration was no state secret, but that blog post was pure accelerationism.
Duh, the idea that those capable of actually utilizing these models for these purposes weren't aware that they could use these models for these purposes up until Anthropic released a blog post about it is silly. Anybody who is even a bit tech literate has been well aware of this inevitability for basically as long as open-weight models have been a thing.
The timing of all of this is much easier to pin to the creation and proliferation of capable of open weight models. Anthropic calling this out is not the cause for it; it's just an obvious correlation.
That was just 30 days ago. This has been happening for months. Part of the "Black October" narrative is just vulnerability research; Calif didn't abliterate 5.3 to find PHP2Shell.
Yeah IIRC calif research on php2shell preceded all of that but I still stand by the thought that Anthropic has been the most accelerationist AI company despite its lip service to the notion of safety.
The idea that attackers were waiting for Anthropic to tip them off on this is kind of risible. It's been a huge topic amongst practitioners for the better part of a year, and practitioners are usually a step or two behind dedicated criminal or state-sponsored attackers.
Right, that's kind of my point. Also: it's a good blog post (the Anthropic one), but it observes abliterated 5.3 models were already public. It's not like people who break into computers didn't know that already. They're not waiting for Anthropic to tell them about open weight models; they already live in the world of open weight models.
You are running a machine with tons of vendor provided agents/scripts to do god knows what on a cloud provider which itself runs tons of agents/scripts to manage that infrastructure and then you expect to be secure.
And no, before the labs there was a degree of fear for everyone to hack into some one else system. With the labs changing the narrative, anyone can just blame the agents for the breach, whoever did it would only get fame and no blame
Run agents and everything they build. Secure execution from the first sandbox
Getting hacked while in the business of selling a secure sandbox isn't really a good look. It seems like can't even trust anything these days. The other day OpenCode got hacked and leaked everyone's secrets inside .env to attackers and now this. Sigh.
Unfortunately I'm not longer using PaaS tools. Instead I use Azure for work and vps now for all my personal stuff. With AI it's easier than ever to lock a server down tight and mitigate this kind of exposure.
I'm actually pretty pissed at this now. I have a throwaway appstore app that uses northflank for the backend, now I have to go in and rotate revenuecat secrets, apple secrets, ai secrets. For an app with basically 2 users.
Can I even delete the app on apple's app store or whats the process like what a pain in the ass. I'm pretty done tbh.
There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises
For Entreprise, the shift will be towards hyperscalers as they want trust and less blame (Nobody got fired for picking AWS)
For consumers, we care about pricing and stability and control and we will move back to our own hardware or cheap VPS providers and self-host things.
The middle ground would be where the slight pain could be at in my opinion.
[0]: https://news.ycombinator.com/item?id=50307410
Azure would beg to differ.
https://news.ycombinator.com/item?id=50301854
So when I look throughout my long employment career for best security and organizational practices, all the highlights are still relatively, relatively undersized companies, with headcounts in the 3-4 figures. It's as they grow, and start to be multi-headed behemoths, that they decay again, as what is good for the responsable mid-level exec, for the business unit, and for the company as a whole start to have nothing to do with each other. That's when you can smell the decay in the hallways. A few large companies manage to stave off this rot a little more than others, but it's inevitable. Then nothing important can be done in a timely fashion.
So, you can always say: Look, it’s not us (especially not my org!). It f’ing Microsoft/AWS! Everyone got hacked.
You and your career are gonna be just fine.
But it you actually NEED security, it's clearly not an option.
It is trade-off between risk and access/productivity with a cost dimension always.
Depending on your threat model you could go to typewriters /paper as some Russian department did famously few years back too.
Broadly I think you're much more likely to be the victim of a platform-level breach involving an immature startup PaaS than a hyperscaler cloud, however I think for small teams the alternate risk you take on is that AWS/GCP/Azure are full of landmines and extremely easy to misconfigure in insecure ways if you don't know what you're doing. Especially if you have AI telling you confidently wrong things.
Let us look on the front page and see if there are any, oh look here is one: https://www.binarysecurity.no/posts/2026/10/one-root-case
I’m not even blaming Zuck for that, he didn’t really invent it, though he certainly popularized it — he just made an astute observation of the dynamics in software and turned it into a company motto.
As is what constitutes the standard of “adequate.” Which BTW is the standard in ALL engineering (not just security or software) because we live in a world where economic pressures apply and costs must be managed. Trivial example: bridges only use an “adequate” amount of steel and concrete because they are not free. The word “adequate” fundamentally involves the concept of tolerances. There’s a reason we consider over-engineering a bad thing.
So which development approach would you say is better correlated with good security outcomes?
My bridge that collapses is adequate and acceptable because other people make even worse bridges that collapse even easier? No, the bridge still collapses, everybody gets a F.
But they are also very highly incentivized to be more secure. They have millions of customers (hence a correspondingly higher expectation of trust) and billions in revenue from the trillions in economic activity happening on those platforms (hence a correspondingly larger target on their backs.) This means exponentially higher liabilities. And that probably matters more than any other factor.
They will still make mistakes but they have the resources to make them less.
I doubt that such intense level of thorough testing can occur in a smaller company.
I still love small VPS providers and use them myself but it is worth knowing why enterprise is willing to pay a premium over those to these hyperscalers (it is a trust premium)
thats the crux; are you working for the paycheck, or for the truth?
could be both, but pretty much all "i was right" stories start with "everyone else said i was wrong"
I wanted to write a blog post about it as I was thinking the same but I guess a comment will help me get my thoughts across just as much.
Either we are gonna see a convergence to hyperscalers as you mention, or we would go back to completely indie companies which compete on perf/price.
There could still be some value to "nicer to use" certainly but with AI, we are gonna see almost every product as being "good-enough" if I am able to convey my sentiment.
An anecdote is: Proton's an middle-ground company. I am seeing the Verge article which states that their CEO has gone a little crazy over the use of AI and the wording makes me lose trust within it. Proton's whole point is security and yet it's having this, It has some stocks in non profit and is a swiss company. It shouldn't have been this way and would've been one of my least anticipated companies but I guess they want growth at all costs and the growth has been slightly lesser than expected and now they are burning the good-will in many ways.
I have to wonder that even if proton might have good engineers, then within this pressure by their CEO/CTO. I cannot trust the engineering talent because of these issues because they are very much forced by their company to use AI.
Proton is also really expensive.
Now just today, also, with a company so big as binance getting hacked and losing 6.5 Billion$. My trust within security just because its a large company is less.
On the other hand, I know a guy called Vikas on lowendtalk who operates his own mail platform called mailbig. I know he vibe-coded the landing page and might've vibe-coded something but it is really cheap. I also know francisco whom I have talked to who runs namecrane. (I think mailbig uses namecrane in many ways)
Perhaps E-mail can be a sensitive matter because you have to place trust in the provider even more because the protocol is not really that much secure but suppose it's an matrix instance (where it is genuinely encrypted) then that shifts it even further. I do trust NameCrane and NameBig from what I can gather.
I'd much rather support them and their cheaper deals than some other middle-corp. Sure some security factor can come in but also, the difference feels comparatively less and the balance has been made
The market will slightly squeeze as Hyperscalers will introduce the solutions of the middle-ground and the new players will everyday enter into the field with more competitive pricing just as well.
So in short, Enterprise will move towards the hyperscalers. Generic Consumers will move towards the cheapest option. The moat of the middle ground where UI/UX was the differentiator is being turned down. There would be some mix and match and there will be cost-savings for Enterprise to move from hyperscalers but it would depend on a lot of factors.
If you think governments (US and China) are going to tolerate people using models for anti-social behavior, I have a datacenter to sell you.
SOTA open models need tens of thousands of dollars worth of hardware to run.
I get your analogy, and it feels right, but the compute requirements to run these models are extreme. That analogy falls apart when it necessitates having a corporate 3rd party run the models for you. They will not tolerate breaking laws.
If China says no more, then the best the masses are gonna get on their high-end setups are the 30B class models. Which are decent, but not "hack my bank", errr, "protect our bank" decent.
Piracy flourished because you only need an operating laptop to participate. LLMs you need a server rack of GPUs to participate.
We can lower our standards instead
Edit: https://www.sentry.help/en/articles/17824780-was-sentry-cust...
Seems it's not used in production and doesn't contain customer data.
If I go on wayback a few days, I can see Sentry: https://web.archive.org/web/20261026032756/https://northflan...
And sure, you can encrypt at rest, and perhaps even enforce that only re-encrypted versions are ever seen by the management/API web server. But fundamentally this display requirement means that if your management/API web server is compromised, users' secrets can be obtained by a process of impersonation.
If anything, AI makes running linux boxes much easier than 4 years ago; it's now faster and the learning cliff is gone to get the correct CLI commands and digging into system logs.
As you build something bigger and bigger, quickly, with several people building, guarding all the doors becomes exceptionally difficult without a kind of aggressive discipline which is contrary to the "go fast" culture.
Cybersecurity in practice until now, for most companies, has been an insurance problem, not a quality problem.
Such companies are quickly discovering that insurance is a post-hoc partial correction and not a proper solution.
This situation will not improve anytime soon for most companies.
The solution does not require any new magical tools.
Now, to be clear, abliteration was no state secret, but that blog post was pure accelerationism.
The timing of all of this is much easier to pin to the creation and proliferation of capable of open weight models. Anthropic calling this out is not the cause for it; it's just an obvious correlation.
What’s changing now is that models are getting more powerful.
And no, before the labs there was a degree of fear for everyone to hack into some one else system. With the labs changing the narrative, anyone can just blame the agents for the breach, whoever did it would only get fame and no blame
Unfortunately I'm not longer using PaaS tools. Instead I use Azure for work and vps now for all my personal stuff. With AI it's easier than ever to lock a server down tight and mitigate this kind of exposure.
I'm actually pretty pissed at this now. I have a throwaway appstore app that uses northflank for the backend, now I have to go in and rotate revenuecat secrets, apple secrets, ai secrets. For an app with basically 2 users.
Can I even delete the app on apple's app store or whats the process like what a pain in the ass. I'm pretty done tbh.