This is a good thing. Repaying technical debt is always a great thing.

On our own 1.5M loc system, we've found and fixed thousands of bugs, security holes and circular dependencies in the last year. Code quality has increased, test coverage increased. We've also recently set our compiler to treat warnings as errors, and to be as strict as possible. It is absolutely amazing to see the gaps being filled, it would've been an impossible task for human devs and IDE's would've made it very uncomfortable. We've also ironed out all the bugs and failure modes in our infrastructure automation / monitoring. Our next project is to eliminate all 3rd party dependencies, and after that to rebuild our frontends completely using zero frameworks, not touching npm ever again after that.

So when I read this about postgres, I was very happy that they found and pursued this side quest. They can make the codebase bullet proof.

Avoiding 3rd party libraries seems like a false sense of security. You’ll stop seeing CVEs on the scanner reports, but the bugs are still there, just in your own code that is less well audited than public libraries.

I understand, but I would argue its better to have an in-house 1K lines of code that only does what we need and nothing else, it well tested and under our control, than to depend on some 50K lines of code (of which we might only use 10% of functionality and the others just getting in the way) that random's on the internet can modify, sometimes pulling in transient dependencies that we would really not have in a server environment, telemetry and other gremlins. It goes without saying that opensource packages will become even more juicy targets for malware/hackers to get into your systems like we've seen in the past month (and it will only become worse - you will be forced to pay technical debt or die).

The only "third-party" packages that we want to ideally allow/support would be provided from LTS versions of our runtime (in this case, packages provided by the dotnet team and audited by microsoft). So we will always try to build around what the framework provides. Second choice is our own code, third choice is open-source packages, fouth choice is a separate application/appliance (lets say ZenDesk for ticketing). Over time, we will reduce this footprint to only have dotnet team code and our own.

Remember most CVE's can be hidden on servers that are not publicly routable and don't have a real exploitation paths. The bits that are public (lets say a linux edge server with a web server on it, some firewalls, a couple of open ports) - these should be hardened to the max and not allow full access to all the services running behind it. We don't even give our internal services public IP addresses to make near impossible to route traffic to them.

Our git repo's are also not on public infrastructure (serious, think carefully about hosting on public services and if you really need it), never mind making the repo public.

We also have a dedicated secrets/certs manager, so no credentials are in codebases or pipelines. Very strict in this regard.

> Remember most CVE's can be hidden on servers that are not publicly routable and don't have a real exploitation paths.

The thinking of "it's inside so we're safe(r) from attacks" is a false sense of security, as (e.g.) Maersk learned with NotPetya:

* https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/

* https://archive.is/Amlij

From the year 2000:

   NAT (particularly NAPT) actually has the potential to lower overall
   security because it creates the illusion of a security barrier, but
   does so without the managed intent of a firewall.  Appropriate
   security mechanisms are implemented in the end host, without reliance
   on assumptions about routing hacks, firewall filters, or missing NAT
   translations, which may change over time to enable a service to a
   neighboring host.  In general, defined security barriers assume that
   any threats are external, leading to practices that make internal
   breaches much easier.

* https://datatracker.ietf.org/doc/html/rfc2993#section-9

> it's inside so we're safe(r) from attacks

Kind of works until the TV, the fridge, a user's BYOD or your printer has infiltrated your LAN.

So it kind of doesn't.

> on some 50K lines of code

also some 3rd party libraries are overdesigned/overcomplicated because everybody wants the library to do everything for everybody.

This overcomplication can easily introduce security problems that is not present in the 1K solution.

Thinking about Log4J here. The vulnerability was with an obscure functionality that I’ve never seen used. Sometimes simpler is safer.

Of course, sometimes simpler is “I stored the passwords unhashed”, so you have to find the middle ground.

Log4J is a really good example. You have to rely on the taste of the maintainers to not just include endless additional features.

jackson-databind too

> overdesigned/overcomplicated because everybody wants the library to do everything for everybody.

Also a source of significant dependency churn even excluding security bugs.

Whereas with my own code I don't have to worry about scheduling work that's only arisen because somebody outside my organisation has decided to make a breaking change in a dependency. Paying that tax got old over a decade ago, and my patience with it now is close to zero.

No need, just use any of the other libraries

The purpose of the above comment is to highlight the flaw in the original premise. Surely we all agree that we would rather not be using a vibecoded cryptography library. But why is cryptography special? Why stop there? I would also rather not be using a vibecoded regex library, or a vibecoded codec parser, or a vibecoded web framework, etc.

The point being, if you weren't committed to maintaining your own implementation of a library before the advent of LLMs, having access to LLMs does not fundamentally change the calculus. The ability to generate code more quickly does not automatically grant an organization the capacity to maintain more code.

I would rather let someone else pay the price of maintaining 50k lines of code than to pay the technical debt of 1k loc myself.

It really depends on what the product is, what the 3rd party library is used for, and what the alternative is. Unfortunately, some people are too biased towards using 3rd party libraries and don't understand the tradeoffs.

For example, a few years ago I removed a rather large 3rd party mathematics library because we were only using it for a single mathematical function that was very easy to replicate.

Remember, 3rd party libraries need to:

1: Be periodically audited to make sure you are in compliance with the license terms, and preferences of your company.

(1.5: This becomes important during an acquisition as someone will always come around doing due diligence.)

2: Be periodically updated (in case of security bugs)

2.5: Be updated on short notice when a security issue is revealed.

In the case of the mathematics library, replacing it with roughly 20 lines of code and 100 lines of tests is a significantly better tradeoff than constantly reviewing it every time one of the above things happens.

Now, at what point is it "worth it" to use the 3rd party library? In our case we'd need to make much heavier use of the library, probably at least 20 different mathematical functions from it.

I’d be happy to bet with you if I rolled my own log4j I would have never implemented jndi as a formatting argument.

Sure I shouldn’t roll my own crypto but I can write a logger…

Logging is a very complicated security area. You often get sensitive data in the logs and it gets sent across the network to monitoring servers.

Also any part of the program can send data to the logger. This means the logger can receive malicious payloads from anywhere. Maybe your logger handles ascii control characters badly?

I dunno I pretty much always wrote my own logger. Not even a shared library or anything just copypaste into next project or often don't even bother with that, just knock something out on the spot. Even before AI.

The only even slightly tricky part is like "don't mess unicode up", rest is straightforward. Sure it can be complicated with logging sinks, configurability, when to flush, monitoring and so on but in many cases you just don't need all that. You just build it whenever you do need it.

And probably more importantly, AI can write a very simple logger with no extra features than you absolutely need in 5 minutes, if that.

I can’t be bothered to check the exact assembly to see if timing attacks are possible.

But crypto really is just a small thing. I’m not going to design a nuclear reactor but I’ll definitely do stuff that depends on the electricity from one. Similarly one can still do their own authentication; its not like even Apple forgot to rate limit…

It has been proofed again and again that you cannot trust npm/cargo not even AUR, implement you own minimalist functionality without dependencies and just for you own needs seems to be a much better solution.

You’re probably right about some third party libraries, but not about all or even most of them. Package repositories are full of half baked packages that are the result of a single person scratching their own itch once upon a time and thereafter forgetting about it. The more harmful part happens when those get imported into other third party libraries that give the image of being better supported. Supply chain attacks are the new null pointer.

For database programs, I would disagree, this should never become something like the nodejs ecosystem.

I personally only rely on 3rd party libraries that are complete. I will never use a library that gets consistant feature updates. I do make an exception for openssl rusttls and whatnot since they have to update by design.

on the other hand you are not exposed to supply chain attacks

Well, the AI model is your supply chain in this case.

most model providers have a direct financial incentive to make sure that the code that is produced is not backdoored

I mean that is true for anyone that sells software. That doesn't mean it's not a hard problem.

If you reduce the amount of code/code paths you reduce the surface area that vulnerabilities can appear on. A code base with 10 million lines can have a lot more bugs than one with 100k.

Two edged sword. That means the bots also have a library of exploits to try for the components used in your project.

3rd party libraries are attack vectors and if like many many people you’re only using one or two functions then it’s much better to get rid of them.

Agreed. It’s the modern security through obscurity.

Security through obscurity protects you from low effort vulnerability scanners.

It makes you more vulnerable to a sophisticated targeted attack. A competent hacker or AI will quickly find vulnerabilities in your custom libraries.

I would like to agree too but there's a genuine new debate here, with LLMs/agents, the storm of CVEs, supply chain attacks, AI assisted social engineering, etc, some custom code doing the minimal required task might be a better security posture than picking a popular library.

I don't know were it will leave us in the long term and I don't think I'll like the answer.

Public software is going to be attacked by many more dollars of AI time than internal stuff. In a year the library code will definitely be more secure against the next X dollars spent on cracking it.

> ...some custom code doing the minimal required task might be a better security posture than picking a popular library.

Only where the AI solution is verifiable by the dev doing the promoting and review. Cryptography is probably better served by libraries vetted by experts.

Cryptography is certainly a valid exception. But for how long? Two years? Three? At some point even that will fall.

If a human cannot verify it then why should it get the highest possible level of trust?

That would be like investing in things one doesn't understand for themselves.

> A competent hacker or AI will quickly find vulnerabilities in your custom libraries.

So why wouldn’t a competent AI find those vulnerabilities during development? I don’t think the proposal is to have human devs hand write the custom libraries and yolo into prod.

Swords and shields. The advantage of the defender is that they have access to the code (usually not available to attackers). The advantage of the attacker is that they have skill and nowadays the most up-to-date models and SOPs for finding attack vectors.

Also the defender has to be right every time, the attacker just has to be right once.

(Defence in depth does mitigate this somewhat. The attacker has to be right two or three times)

> So why wouldn’t a competent AI find those vulnerabilities during development?

It probably would. Except it’s internal, and not a reportable CVE, so the fix will be deferred.

in theory, a company running custom software should be able to invest many more hours into running those same AI with "white-box" pen-testing (assuming hyperscalers actually start allowing this). An attacker would have to be willing to outspend the company (or get very lucky). If an attacker is resourced well enough to put that investment into owning a single company, you probably don't stand much chance reguardless.

Also, the attacker would have to run black box against the live environment which is much more difficult and likely to be noticed than offline white box.

I feel like security by obfuscation might actually be valuable in the age of llm attacks

Not having access to the source is certainly a disadvantage for LLMs, though I saw an article the other day about a model running a disassembler on the binary, from that reconstructing the roughly equivalent C, and finally analyzing that. So, even if the binary is available there are paths.

Yes but that's not entirely obfuscation, see ie what google does to their proprietary endpoints through protobuff. Something like that but on steroids could skip the majority of attackers using LLM scanners.

I think avoiding all 3rd party libraries is excessive, but engineering is about tradeoffs. In the extreme case, do you pull in a LeftPad dependency or write it yourself?

It also depends on whether “avoid” means a strong preference against or a rigid doctrine to never use no matter what.

One problem I’ve seen with third-party libraries, especially in the JS ecosystem, is that to fix a CVE, you often need to jump a major version or three, and that entails quite a bit of work to adapt to breaking changes. Porting a fix back on your own and keeping your own fork is the same as supporting a first-party library, except it’s not really first party, and you’re not familiar with its innards.

but public libraries can also be audited because they are public

Similar to distroless containers & statically linking all dependencies

> Avoiding 3rd party libraries seems like a false sense of security

To be fair, there is more nuance to your statement.

A lot of it depends on your programming language.

For example with Go, with its extensive and excellent stdlib, you would be foolish to rely on large swathes of 3rd party libraries, because so much can and should be done using the Go stdlib.

On the opposite end of the spectrum, you have Rust with an effectively non-existent stdlib.

And the problem you see frequently with Rust is projects importing half the internet, with a Cargo.toml the length of your arm with hundreds of dependencies. Of course the project has never reviewed its dependencies source code, and the project probably barely keeps its dependencies up to date. Its a supply-chain attack waiting to happen, not if but when.

So with languages like Rust, you have to take an approach of "minimal reasonable dependencies", make an active effort to keep your dependency list short and in-control.[1][2]

> less well audited than public libraries

I also struggle with this particular argument because the reality in open source is "well, its open-source, it has many eyes" is in fact mostly a dream. Since we all know 99% of the time its a small group of maintainers (assuming there is are even multiple maintainers[3]) and not many (if any) external reviewers.

[1] https://tweedegolf.nl/en/blog/119/sudo-rs-depencencies-when-less-is-better/ [2] https://tweedegolf.nl/en/blog/104/dealing-with-dependencies-in-rust/ [3] https://xkcd.com/2347/

Same approach here.

And since we now have coding assistants, maintaining in-house costs near nothing now. Nobody else can break us, only ourselves.

Our next project is to eliminate all 3rd party dependencies...

Just remember the golden rule of crypto is to never fool yourself into thinking you can write your own crypto libraries.

Now this is how companies should be using AI. Not as an excuse to fire half the devs. Less bugs, less failures, better experience for everyone.

We haven't fired a single person yet due to AI. We just save everyone time, make input more accurate, make our accountants better, manage our technicians and jobcards better and so on. We also made our system bullet proof to the point that we only had one weekend failure in the last 24 months, and it wasn't our fault (a third party dependency did a large deployment on a friday night). Our system is now deeply "boring" and its awesome.

Oof, we are 1/2 the size we used to be so we can do more with less. We're doing more but with half the people there are still so many gaps that I'm quitting myself out of frustration.

Perhaps the ultimate use case would be to use AI to scan, debug and find issues etc, but never to generate new code. Still have humans writing and understanding everything, keeping things as clean and concise as possible.

They will learn certain lessons the hard way for sure. As much as I want to spin up 100 agents in a loop on our codebase and let them loose, I also know its not feasible as they can easily go down the wrong rabbit holes or over-engineer the wrong solution. They are amazing but cannot be left 100% autonomous just yet.

Literally half of the work we did the last 12 months was this. Refactor and fix, performance improvements, security hole fillings etc. We have one or two devs doing visible feature work, the rest of us is making the system rock solid.

The only companies who will really need to fire are the big tech companies who hired just because, for empire building, or to keep talent out of competitor's hands. Nearly everyone else is actually positioned to leverage AI to now compete against the bigger players. I'm already seeing small companies have a big surge.

This is how I use it. I don't run any companies (nor am I a part of one right now) but were I to run one I would employ AI as the ultimate bug/vuln hunter. Just spin up maybe 20 agents and say "fuzz this thing to death".

> rebuild our frontends completely using zero frameworks, not touching npm ever again after that

Genuine question, because I don’t know: Wouldn’t it suffice to vendor your current packages and “only” stick to maintenance from there on out, instead of rewriting it all?

This should still achieve the goal of not needing to touch npm, but presumably with less effort/tokens, right? Or is it just that you would expect the result of that process to be less clear and maintainable than a full rewrite?

Actually, I have started with it already but is paused, but the approach is to rebuild it from scratch in parallel to our current legacy system and move our staff over to the new completed screens in batches (we have close to 1000 views in our system, 150 background jobs, with about 15 main categories (think organisational sub-divisions)). So as long as we go slow and steady (even with AI) it will work out fine, I expect it will be done in less than 6 months. One month if we rush but we don't want to rush it, as we are moving some things over from MSSQL to Postgres, so slow and steady is the way.

> from MSSQL to Postgres

How's that going? Any lessons learnt you'd be willing to share?

If you only use MSSQL as a datastore, it's not too bad. If you used all the other features (MSAS/SSIS/etc...), it is much harder. PG also isn't a panacea - you're trading one set of issues for another. The biggest upside is that it's much less expensive :)

Differences: Money/decimals, rounding, case-sensitivity, equality comparisons differences, db/table locks mechanisms, memory/worker management, table/column naming conventions, null fields/columns effects on queries.

Mmm what else. I prefer postgres so I'm very excited about this. So many foot-guns will no longer haunt us like on MSSQL and less db/table locks to try solve (pessimistic vs optimistic locks). Oh and we are going to save a gross amount of money per year, as we will also be able to switch off a couple of Windows Server VM's in the process. These puppies are costing us a ton (I think about half of our infra bill are licenses...).

Just create a branch, spin up a test db and try to port a small feature. Its very easy in most cases. Debian + dotnet + postgres = dream team.

MSSQL query optimizer is one of the best available. If you have truly insane queries, views needing joins on views and CTEs, etc, you might be surprised at how well SQL Server was handling them versus PostgreSQL.

Because they are large codebases with their own technical debt. You're not doing a full rewrite, but just make what you need for you own codebase.

If the parts you need from a larger third party library are small and simple, that makes sense. But I don't think the general idea of reusing battle tested code across many projects is obsolete because of AI.

Ah, I think I see now. Essentially the dependency module is unnecessarily large and (for your own use case) ships with extraneous features/“bloat”.

I think the approach should probably move towards semi-tailored systems.

Maybe you don't need all of Postgres - so provide 2,3,4,5,n versions. As long as every version is covered by (possibly shared) tests, then you can release what works, ignore the rest (bugs and all).

Exactly. And over time a project can drift from its original purpose, so you end up building around a bunch of stuff you aren't using anyway, but you pay cost/risk for having it present.

Like the warnings as errors, and extensive test suites, there's also type safety that helps stir the LLM in the right direction.

You can go with "eliminate all 3rd party dependencies [...] frontends completely using zero frameworks" and be stuck with JS. Or use a typed language (TS, ReasonML/ReScript or even Elm) and an API client generator (available for all those languages) and get type safe API calling from a more strongly typed language.

As much as I prefer not to use NPM and even more prefer (even my LLMs) barely touch JS. A bit of safety is cheap these days, as you rightfully pointed out.

> found and fixed thousands of bugs, security holes and circular dependencies in the last year. Code quality has increased, test coverage increased

Does management appreciate it? Do customer see any value in it?

Good thing is now human does not have to justify anymore. If AI says bugs management seems far more amenable to support fixing. At least in my experience managers take AI reports or vendor tool generated reports more seriously then any feedback from engineer.

If it enables faster feature releases and better performance, probably?

AI can help companies provide more features or cut costs.

> If it enables

This extremely non-obvious and non-transparent to managers and completely incomprehensible for customers.

AI makes it possible to fix bugs and solves issue one wouldn't do before. But just because it's easier now doesn't mean it's automatically worth it. There are tradeoffs (like code complexity or breaking stuff), or it really just wasn't that important to fix if it was in the backlog forever. I've seen devs go down this rabbit hole, and perhaps overdo it. It feels productive, but at some point it might stop to provide value, and other things should've been prioritized. It's just easier to say to the AI "fix bugs" instead of solving real issues. At some point boasting about running 10x agents in parallel to fix bugs becomes a sign that there's a lack of value provided to boast about.

While that is true, there's also the other side of the coin: most bugs are hit, but never reported. Users are pretty unlikely to report bugs and when they do they are very vague about it. That's why client observability is such a boon for teams because they find what bugs users are actually hitting. If the agents are directed using that, then perfect

Didn’t you guys have to ship new products? How a business can afford so many backlog tickets being implemented? I understand fixing bugs can also mean improving the product but still - (almost) the whole world has this balance between new features and acceptable-ish number of bugs

We are multiple developers, each having 10+ main claude agents running different sessions/terminal tabs. I have 10 copies of our codebase on my laptop, each with their own purpose and claude session and history. The other devs have different approaches but we all get similar output. And then we also built an agent framework that runs within our system that has access to our code (at most 60min behind master branch), our infra and our databases, so it can diagnose issues with excellent precision and prove it to us where to find code/data issues.

What are your token budgets like?

Are you able to share which domain your company works in?

Utilities - Electricity/Water/Gas. 98% of our systems are internal facing or third party integrations. Our surface for end-consumers is tiny.

Honestly this is the kind of stuff that needs hardened now and new features skipped for a while.

Years ago I had to work on a server that had some sort of water utility software on it and it was a rather jank piece of software.

> eliminate all 3rd party dependencies

The third party dependencies are also using AI to improve their code.

Good thing if you can afford the vulnerability detection run.

I could imagine scenarios where only well off developers and companies could afford that.

Then if you don't have the money, you are out of luck and risk having vulnerabilities in your code.

And it takes just one adversary to pay the bot & the bot will happily find an exploit for them in your code.

This worry this could lead to a serious power imbalance & furthe damage the free software community...

Short term there will be pain (hacks/malware) but at the same time, it will force the survivors to build bullet proof systems.

Worse case, we enter the Dark Forest era for technology. Read up on this concept and apply it to the tech industry. I suspect open-source might die and everything becomes private, no more public networks, no more open access, as to assure survival. In a sense we are halfway there already as it seems the internet is starting to close down, people not posting their art and music as much anymore, not posting their knowledge for free anymore, as it might be used against them by AI (the other party in the dark forest..). Thus private authentic experiences will become more valuable.

I don't think it'll be that bad:

1. The cost of running AI is dropping down to "free" - there's open weight models, open source tools, and guides on running your own if the long list of suppliers isn't good enough for your use cases.

2. Vulnerabilities have always been a risk; AI is not a panacea. The pre-AI best practices still apply. Furthermore, a huge percentage of vulnerabilities are in 3rd party tools and systems, which are systematically being scanned and fixed and/or exploited, then fixed.

You mention "the free software community" in your last sentence though - this implies the code in question is open source, so it's able to receive plenty of improvements from anyone with access to these tools.

what are the nuts and bolts of how you're doing this? which tools have you found helpful? any advice for people who'd also like to do this?

> But I’m still not convinced AI assisted development is at that point yet;

But the gp said it is, and gave examples.

Can you tell us how did you achieve that kind of work, briefly? I struggle to figure out how such massive bug fixing campaigns or even large scale software engineering are done with ai not taking weird decisions...

I personally started using AI with a very rudimentary "memory" system, and for first weeks corrected it ad nauseam, until, in the small domain I was responsible for, it started doing work end to end fully autonomously. Takes a lot of time and effort to bootstrap, but once steered well enough, it just clicks and less and less is needed to check manually

Yeah a memory system per domain, or per module or even per file works amazing, we have it too. Took a ton of time to improve the code and list down all the tribal knowledge, but now that it's done, it works super well.

Myself specifically, I have a /code folder without about 10 copies of our codebase, each running their own claude sessions & git branch. Then, I have 10 terminal branches. I spend a ton of time planning out & specing the problem with claude, to be as specific as possible and be as constrained as possible so it cannot go on side-quests. I use Opus 5.5 on High effort. That's just on my laptop with what I can mentally cope with day to day. Then we have some other automatic ai goodies running within our custom agent framework that runs within our platform that also helps. Then for apps, I basically plug in multiple phones into my laptop with usb (and emulators if I really need to), and claude can fully drive the ui and testing on the phones via adb as an example. Even on linux/fedora/kde it can drive UI apps a and take screenshots and fix things properly etc. It can ingest all logs & debug chains, so it absolutely flies through its task.

Before we started this journey (about a year ago), we spent a ton of time cleaning up our codebase, splitting large files (10K+ lines) into files that are no more than 2K lines, things like that. Then we let claude scan our important files and summarize them, add comments for future claude sessions. So we've let it built up tribal knowledge per file and the reasons for things to exist. We tell it all about our operating environment, the country and legal framework etc, so every little piece helps it stay honest and on track.

So to start, do something simple like: Scan the files in this folder for database code and check for queries that loads the entire result set into memory and/or queries that loads all columns. Suggest which queries we can filter better or reduce columns being transferred. Once done, also check real db row counts, compare them with the queries in these files and check if they might benefit from an index or partitioning/sharding. Also check for queries that loads data but their results are not used. Don't make changes yet, just do a deep investigation and report back. Feel free to spin up muliple agents to batch the scans.

edit: it goes without saying you need a language with strict typing and a good compiler (we use dotnet). Doing this on python/javascript would be a nightmare.

> I spend a ton of time planning out & specing the problem with claude, to be as specific as possible and be as constrained as possible so it cannot go on side-quests

I still have yet to see this be efficient. The time I spent doing this is way less efficient than simply giving the LLM some high level direction and putting that time towards reading the code it generates and correcting small piece by piece (occasionally doing things manually, but that’s only for more off the rails stuff).

It’s about the same amount of time as up front planning, except I know the code much better as well as end up with less mistakes compared to having it implement a front loaded plan.

Knowing the codebase makes everything else 10x more efficient too - triaging bug reports, planning other work, knowing what work needs done (this is a big one), reviewing PRs, etc.

How much do you spend per day on Claude?

> IDE's would've made it very uncomfortable

Would you mind elaborating?

When you change a single setting in the IDE to make something more strict and you have 1M+ lines of code, you will get 30K items to fix, via red/yellow/blue squiggles in the IDE. Most of those can be simple refactorings, but the performance & UI of the IDE makes it incredibly unwieldy and it will sometimes fight you back, sometime crash internal processes which leave refactorings half baked and so. Now imagine you change 50 such settings, to make the compiler force you to write better code. Changing 200K+ lines of code yourself is a nightmare (and to prove it still does the same thing afterwards). Instead, claude systematically approaches whole classes of errors and convert them to cleaner variants (either using our custom skills or it writing new scripts to do it). I consider myself an advanced user of IDE's, know how to do bulk refactoring and renames and so on (esp with JetBrains tools), and I would say it would be a near impossible task to do as a human. Then there are other developers - so you flip a setting and now 5 other people's IDE's are forcing them to work a specific way which will make you unpopular very quickly and revert the changes. So, since most of us no longer use IDE's as our main tool to build/diagnose, we can more easily flip compiler settings without destroying someone else's week. Hope that makes sense.

Oh it also helps to split refactoring work into three buckets: business logic changes, performance changes (same output as before but faster) and cosmetic changes (rename files/folders/classes, breaking large files up, add logging/comments, domain layering etc). We tend to focus on performance and cosmetic changes for bulk refactors and leave logic changes to more dedicated sessions as to avoid changing what the system do (our domain is fairly complex and we cannot afford breakages). We also managed to port ALL of our code to dotnet 10 lts and have clean boundaries between projects (we have a mono-repo with about 50 projects in it), so we can wholesale upgrade dependencies without a sweat.

> it also helps to split refactoring work into three buckets: business logic changes, performance changes (same output as before but faster) and cosmetic changes (rename files/folders/classes, breaking large files up, add logging/comments, domain layering etc).

Do you have a specific skill to guide an agent to scope the work of an agent so it stays in one particular bucket? Would you mind sharing it?

State of the art agents don't usually need such fixed guidance anymore. I've been using Opus 5.5 with Xcode and simply prompting things like "make sure all of our test Suites use the .dependencies modifier, but only if they use dependencies internally" or "make sure none of our test methods are @MainActor, just access the necessary state using standard concurrency constructs". For larger changes, providing the examples you want is often enough to get it most of the way, or even all the way, there. I've had simples prompts for things like "move all standalone SwiftUI view that seem reusable to our Design framework, and create a list of those which can't be moved without effort, listing by complexity".

Overall, there's much less need for prompt skills anymore. They're still very useful for things like best practices, or things you need to make as reproducible as possible, but big skill and planning sessions aren't nearly as necessary anymore.

IIUC, your point is that changing IDE settings will affect other developers. But can't you enable an inspection locally, apply all or some of the quickfixes and revert the setting before committing? In the longterm, you want to enable the inspection anyway because the point is to prevent developers or agents from comitting new code that violates it (agents can access JetBrains IDEs' MCP Server to get inspection results).

Of course, crashes and bugs can happen when applying quickfixes, which isn't nice. Ideally, they should be reported and fixed. You can also ask an agent to fix the occurences instead of using the IDE quickfixes if that's blocking you.

I guess what I'm trying to say is that IDEs shouldn't make that process harder, they can actually improve agents' results. For example, you don't need to write custom skills to prevent agents from writing some class of bugs if the IDE can detect it and agents have access to the results.

> Changing 200K+ lines of code yourself is a nightmare (and to prove it still does the same thing afterwards).

If an agent does it, you still have no proof, but okay.

Full disclosure, I work at JetBrains, but not in the IDE department.

The problem is, even if I only enforce some rules locally, another developer has a different set of rules. So when we git commit, we ping-pong changes and fix each others stuff over and over again, bloating commits. We used editor config files + analyzer configs in the past (that gets checked into version control too), but there were still too much friction.

Where now, the compiler flags & strict analyzer settings forces claude to do the right things (we explicitly tell claude not try and bypass or lower the barrier). I think it only works because we don't spend 95% of our time in the IDE anymore like a couple of years back, so we as developers no longer feel that friction, as claude tries to satisfy the compiler instead. Its a massive mental load that is just gone, we haven't had any conversations in meeting about this kinda stuff anymore (remember in the past we would have flameouts in meetings or code reviews about linter settings...).

If you're going to roll your libs, be sure to set up agents to scan for CVE analogues of equivalent known vulnerabilities in your own code. It takes quite a bit of work to set this pipeline up, as well as manually selecting high-impact CVEs (relying on the public scores is worthless, because there are some vulns that are catastrophic with a score of 6 vs something meaningless thats a 10).

It's also crucial to scan for "analogues" because otherwise many LLMs autoreport as "not vulnerable" since "this applies to Postgres/whatever and this repo isn't using it"

If you get hit by cyber guardrails you should use a local model to prune any CVE specific language from the reports.

To give examples, every single LLM (and people too tbh) will write netcode that's vulnerable to these CVEs: CVE-2026-27135, CVE-2024-45311, CVE-2021-32567, CVE-2019-9511; without explicitly mentioning the CVE text no LLM finds these either.

> Repaying technical debt is always a great thing.

No. Debt is a tool. Sometimes it’s good to pay it quickly, but sometimes it’s better to pay it as slowly as you can.

No. Technical debt is just an imperfect metaphor.

You never wanna intentionally repay tech debt as slowly as possible. You might be able to tolerate/ignore it for longer depending on the circumstance, but you always wanna get rid of it sooner rather than later.

All else being equal, you're always better off without debt. AI can make repaying the debt very cheap (compared to doing it manually).

Based on what?

False dichotomy. A library that is not a 100% perfect match for a problem you have to solve, will require glue code. Any bug in the glue code is in your own code but only exists because of the library.

I’m not going to eliminate every single third party dependency in my system, but I’m on my way to eliminating 90%, maybe more.

For me it’s less about bugs and more about supply chain security risk. And with AI and only building what I need from a dep, exactly how I need it, the cost of doing this is very low.

I worry that the move towards eliminating library usage will lead to further code fragmentation.

It's a nice idea, and in theory sounds great, but you may be trading 1 library with 1000 bugs for 1 million implementations each with one special bug...

It could also be a bad "thing".

AI-slop reports are filling up the queue and taking away time from volunteers. So contributing to delay in releases/features.

The JS frameworks exist for a reason, recreating them seems a little excessive. Why not skip JS and go to a Rust wasm framework or Phoenix LiveView if the goal is to get away from JS/node/npm?

Our legacy system runs mvc view with bootstrap 3 and some jquery. It's not a SPA, so no framework to replace. But we still have to mess around with npm sometimes.

Depending on the framework you are in, there might be the equivalent of LiveView/Hotwire that works with MVC.

> This year is going to be different. PostgreSQL 19 most likely won’t be released until November… […] So, what happened? The short answer is that LLMs happened. […] There’s a huge amount of code, and that code contains a huge number of bugs. […] What’s different now is that it’s become vastly easier to find those bugs.

This seems…good?

That it might disrupt the project's "traditional" release cadence a bit doesn't seem terribly bad in light of the alternative, which is shipping another major update with bugs which have apparently haunted the codebase for some time.

At work we have put a full halt on feature work because the number of security related tasks has become staggering and urgent.

We will hopefully get over this wave and be able to move forward with much more secure software, but it's crazy right now.

We’ve also been blowing a lot of retainer hours just patching sites instead of building new features. It sucks. Ultimately going to cost clients more.

It's not like no work is happening though. Your clients are getting a more secure website. Maybe the bugs that are being patched would have taken longer to be exploited in the pre-LLM world but the bugs were still there.

Most clients will not agree to pay extra for a more secure web site though. Unfortunately. Unless they’ve been bitten in the past, which most businesses still haven’t.

As long as you put the alternatives clearly on the table, you could let the customers choose, and sign off on taking the risk. You can give a recommendation, but ultimately it's their choice if they bear the consequences resulting from the choice.

Hard disagree. Framing it this way makes it seem like all possibilities are equally valid options. Yet you’ve somehow winnowed the field of infinity to the several options presented. You did that through successive rounds of professional judgement that eliminated options you did not consider fit to propose.

Taking options off of the client’s table is the core responsibility here.

Eh, letting the customer footgun never comes off well. They will always fundamentally blame your advice or lack of advice.

Yeah you’re right but for most businesses increased security doesn’t lead to more sales.

I’m not saying that’s right, I’m saying that’s how a lot of businesses think.

Businesses that are paying attention to the winds though have already figured out that the internet is a hostile place: both their insurance companies and their governments are making noise about security.

Probably not, if you factor in the cost of security exploits. A single compromise can potentially sink the business.

> Probably not, if you factor in the cost of security exploits. A single compromise can potentially sink the business.

Agreed however getting humans to understand probability is tricky.

I had this discussion with my son recently about considering the severity of outcome multiplied by the probability of that outcome been the way to look at some things.

Same here. I got access to Mythos two months ago, it confronted me with decades of nasty bugs I wrote. This has been stressing me out a lot, caused sleep deprivation (the bugs follow me in my sleep, yes, really). All other progress is halted, I've been squashing bugs day and night. I see a light at the end of the tunnel.

Bottom line: it's a good thing. My software will be better after that. Even if it's currently not good for my ego (and my health). But I'm glad for all the tools that help me improve quality.

And sure, I can just ask LLMs to fix the bugs, which works well half of the time, and produces bad slop in the other half. This is where human expertise and judgement is still necessary to avoid letting our code bases drift into LLM spaghetti hell.

While it is certainly good to do estimation oneself and not just blindly follow whatever the LLM outputs, keep in mind the recent PHP-related events ... Just because at some point one is not able to make a PoC, it doesn't mean, that no one else ever will be.

> Even if it's currently not good for my ego

If you don't mind answering, how come? Is it specifically that you dislike how abusable your code was, or something else?

That part was not 100% serious. Only 90% or so.

To err is human, I know that as much as anybody. Before the advent of LLMs, there were several (human) audits of my code for vulnerabilities, but they found little, which I was proud of. And now, Mythos had a quick look and found hundreds and hundreds of bugs (many of which were critical and even more were really emberassing).

Of course, I knew there were bugs that just weren't found because the auditors weren't looking closely enough, didn't have enough time/budget etc., but this unspecific feeling is different from having all the bugs accumulated over a quarter century slapped in your face at once.

Code auditors are a crapshoot. I worked around financial auditors and they were much more consistent but thats likely because standards are more well defined.

If you get a good auditor they can find a ton, but most are box checkers there to make the audit company as much money as possible.

Quite possible. Ours were good, they just didn't benefit from LLM superpowers back then.

After my Mythos experience, I was wondering if there is really still a market for human code audits. I guess yes, but the work will be different, just like coding (and marketing and so on...) is different now. You still need to have an expert to wield the new powerful tools.

The upper management and security team came knocking and told us all of the CVEs from the scanner need to be addressed right now. And the only way to deploy without having to wait for QA approval is to not merge feature changes.

> The upper management and security team came knocking

i would say the recent slew of vuln reports/breaches making the news rounds are doing work!

This is the silver lining i suppose!

Interesting that this could result in net negative software productivity world wide in terms of features right now.

LLMs finding unpatched vulnerabilities faster than they can be reasonably patched, bringing all other software development to a halt.

That's only if you consider security to be unproductive!

If you consider it to be productive work, it's a massive productivity boon happening right now. Of course, product managers might disagree...

Security work is not productive.

Imagine that the whole city has to install iron doors and window grilles, because wooden doors and glass windows are not safe enough anymore.

That's a lot of work, but is it productive in the end? Probably not.

With AI it's the same. World was fine with lax security approach before. Today it is not fine. Tomorrow we might need to prove every code line before shipping, will that be more productive? Hell, no.

> Imagine that the whole city has to install iron doors and window grilles, because wooden doors and glass windows are not safe enough anymore.

> That's a lot of work, but is it productive in the end? Probably not.

Your imaginary scenario defines the solution to be pointless. If that same city was subjected to weekly riots smashing windows & doors, and people regularly died, stopping "productive" work to install things that keep people alive and productive? That's productive.

There is a sense in which productive is being used here that is slightly subtle. Security doesn't really add anything to net output. It is, in other words, a cost. That doesn't mean it isn't valuable.

One way to think about it is that production allows you to pay for stuff. Not necessarily directly but by way of trading. It is the output you use to buy things like security.

Consider an apple farm. What does the apple farm produce to pay its bills? Apples. They could also dedicate resources to pears, &c, and that would add to their production -- to the goods they produce, the goods with which they (indirectly) purchase tractors, fence posts, &c. They could also dedicate resources to security -- more fences, &c -- and this may even be prudent. Maybe thieves are getting taller. This doesn't really add to production, though; at best, it maintains production at the status quo ante (assuming the fence is effective).

True in the sense of physical security, but I think it's not true here.

Security bugs are just bugs, as Linus likes to say. And the bugs are often also crash or correctness bugs (use-after-free for example usually causes segfaults). For the most part this actually is a productivity boon in the sense of getting higher quality goods out the other side.

To use your apple farm example, it would be like an apple farm where, if the apple rots, it's sometimes just gross but also sometimes it explodes and kills you. Would it be productive to reduce the rot rate of the bomb-apples? I think so!

It's a terminology issue. Economists use the term productivity in a certain way and they even use the term "broken window fallacy" to describe this debate.

https://en.wikipedia.org/wiki/Parable_of_the_broken_window

Basically, if you accept this line of reasoning, then someone going around breaking windows and someone else going around repairing them can be seen as good because it creates jobs. But that would be nonsensical and create poverty. That's why economists focus on wealth creation not job creation.

Fixing security bugs is only needed because people exploit them. It creates work but it doesn't create wealth.

Fallacy applies only to incidental breaking of glass like child smashing window paying football.

Protecting yourself from bad actors also doesn't create wealth but it protects wealth you already have.

You know cybercrime and APT groups are not bored kids in their parents basement, there are groups that are running automated tooling to find vulnerable companies and if possible take all their wealth.

Just look at Binance of course, but if you are small company there are also people targeting those especially with AI assistance nowadays.

>Fallacy applies only to incidental breaking of glass like child smashing window paying football.

It does not. Scroll down in the given article, and you will find war as another example.

As GP said, it's a terminology issue. War spending, including defense spending, is not productive, because it does not produce wealth. It does not mean that protecting yourself from enemy countries or criminal is useless, or not worth it.

Nobody claims investing in security is a bad idea. It's just not productive, in the economic sense.

> to illustrate why destruction, and the money spent to recover from destruction, is not actually a net benefit to society

it is distinct from question whether it is productive to defend from evil people

evil people + protecting from them is obviously net negative!

but it is not changing that of evil people are present then reasonable level of protection from them is needed, and setting it up is productive

Just like securing a system is a trade-off, attacking one is too. Risk vs reward and time vs reward. Your personal blog doesn't need to be fort Knox, just patch it properly and you should be fine.

The software is better in the end but not all of it will be productive.

> If that same city was subjected to weekly riots smashing windows & doors, and people regularly died..

The point it, this was not happening before. LLMs made cheap cyber attack possible. But it didn't make the implementation of fixes correspondingly cheap. So now everyone needs to spend extra money/effort to install grills to cover where LLMs fall short.

For postgres, this absolutely makes sense though as long as people are willing to shell out the profits from LLM use to support postgres development. That is one way this could workout for everyone..

And now imagine the introduction of iron has led to crowbars that make smashing doors a lot easier. Now it is quite unreasonable to account the reduction in smashing doors to iron, when the previous improvement was also due to iron.

The physical world comparison is interesting. In the real world we have a small amount of security, but most things could be broken in to. Instead of fortifying everything, we just imprison the people breaking stuff.

Obviously this doesn't work on the internet since most hacking comes from overseas out of our control. Which makes me wonder if there is a possible future where we just block packets coming from countries that don't lock up the hackers.

> Obviously this doesn't work on the internet

Not obvious to me. I am generally a fan of Cyberspace manifesto and the free Internet, I think it's great people can reach outside their ingroup and talk to strangers and ostensible enemies.

But, that doesn't mean we should give up physical security, the guarantee that the connecting party comes from a certain physical location (state, company). I think allowing connections from anywhere by default is fundamentally flawed security architecture, for certain things.

I don’t think North Korean hackers are attacking things from their North Korean IPs. They just buy a residential proxy. How would you prevent that?

I think the physical world comparisons should keep in mind the kind of data a lot of these systems (eg postgres) handle. It's more like securing a bank vault than locking a home's door.

People usually don't install steel doors because most burglary is prevented in other ways. Insecure software will be exploited this way or another, if the vulnerabilities are real and the threat model is valid.

> Insecure software will be exploited this way or another, if the vulnerabilities are real and the threat model is valid.

The algorithm is: if the threat model is valid, there are vulnerabilities and you either have some software everybody knows about or a gang of hackers specifically targeting you, you should patch.

The part of the industry that profits from security theater will want you to patch no matter what, even if your business goes down in flames due to downtimes and new bugs introduced by patched components. They won't remind you to assess the threat model first.

Physical threat models and digital ones are completely different.

Digital threat models are very topologically flat. One exploit made in one place can be instantly copied and used anywhere.

Also once hacked you can be used as a base of operations to attack others. There is no neutrality on the internet.

There has to be a responsibility model for that; somebody who can say “we don’t do X even if the auditors say we should, because Y”. But I see where you are coming from, I’ve seen security theater to cost $$$$$$$ at the org I’m part of.

By this line of thought building nice and usable UIs is not productive too, because users can learn cli tools (they did, DOS) and it costs less to develop them. And all the work in TLS/HTTPS has been wasted because HTTP is enough. And ssh vs telnet.

It becomes productive only once your business stops for days because somebody brought it to a halt by breaking into the servers and deleting them.

In the rest of the world making sure your product works like it is supposed to is considered table stakes. In software we somehow consider checking the parts of it we can think of 'good enough' and the rest unproductive