Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
> Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I suspect a substantial part of this is an avalanche of useless testing.
If you even review PRs still: when was the last time you didn't just skip over tests? And if you ever looked at tests in an LLM-heavy PR, how many of those tests tested something useful, and not just built-ins and trivial behaviours?
There's at least some awareness in the industry of how LLMs generate a lot of boilerplate in business logic. It feels like we're much less aware of how much of it is in tests.
My biggest takeaway (as a one man band 20 leagues below this spending time optimizing Gitlab free plan and self hosted runners): they did not worry about any of this until 100M ARR and more 1B valuation[0].
With the widespread increase in speed/commit cadence/content added to repos, I wonder if there's an angle here for companies with a lot of CI/CD needs to simply start self-hosting their CI/CD machines on premises.
In my experience, self hosting your CI/CD runners had the biggest impact in cost savings throughout, while also allowing for more powerful machines, which directly means quicker CI/CD runs
Self-hosted is the way. Small potatoes in comparison, but I also made the change this year as my CI usage went from <3k mins/$0/mo -> $100+/mo.
Migrated actions to a spare M2 MacBook Air at home, and when offline, uses hosted Actions as a backup. Bonus: has simulators/emulators for ease of running Maestro tests for mobile QA.
>Because the third-party runners sit outside GitHub’s network, they rely on a direct IP link to reach GitHub. The provider traced the hangs to intermittent degradation on that link.
This is interesting. If I read that correctly this means the CI host had a direct peering connection with Github or Azure network.
Why were they not able solve the issue?
the blog mentions that they use a local cache to "reduce the time" but the problem is not solved right?
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Agentic Coding has been a huge strain on CI,
I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
Our CI has thousands of tests and is done in 2 minutes. Everytime it grows above 3 I add more parallelization and keep it ultra fast. Been working great for our team
I think to solve this problem properly, we have to stop treating "build" and "test" as separate buckets of work, to be designed and scaled separately. It's all CI. As soon as you try seriously scaling out tests, you will run into build bottlenecks. To truly scale CI you need a scheduler that understands your build, test environment, and all the glue in between, well enough to schedule it in a way that actually speeds things up. That is very difficult and not something that even the best build tools can do - yes, even Bazel. Bazel can run tests but it's not nearly as good at it than at building.
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
We are working on something here, pushgate.dev. It forces the agent to run tests; we are also working on adding support for static analysis of bundles using Jev. Onboarding is a bit of a mess right now, but send me a message cole@testifysec.com if you are interested.
The next bottleneck is customers. There better be a lot of them, they better have loads of money, and they better be willing to install updates for every fucking release that goes out whether they want it or not because how else are we going to pay for all this?
It's strange that they did not mention reworking the tests themselves. In my experience AI-generated tests are absolute garbage - unless you specifically prompt to reason about coverage to avoid duplication and to merge some.
Another approach is to redesign CI pipelines. Most checks should be done within the agent loop for quicker feedback (using hooks, skills). Linting, unit tests etc.
CI should handle only stuff which cannot be run locally due to resource or setup reasons.
Also CI should only receive pre-validated change candidates with passed checks inside the agent loop.
I did that by self-hosting Github Runners, using Podman to build images (it doesn't squirt your build context through a socket every time!), and using Docker images as content-addressable cache.
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
> Despite our test suites almost quadrupling since the start of the year
I think the SDLC needs to be re-evaluated, tests, specifically unit tests, are just cosmetics. "wow its so easy to get high coverage now!" as opposed to why we're doing it in the first place
I don't find agents to be using tests any differently than a junior or mid level developer. I don't find humans to be using tests any differently than a junior or mid level developer either.
Basically, the tests are never guiding the features, they aren't highlighting regressions, the test is simply modified for the updated application. Unit tests especially.
I have found end to end tests to be useful, and also unblocked. They can tell when the actual user experience has changed, and go back to the feature's implementation to stop altering the user experience in unexpected ways.
additionally, I find this across every industry that was worried about AI. basically, whatever best practice was neglected due to lack of investment into your org is now being done without additional investment into your org. that's a good thing. now we need to look at the purpose of the best practice, and if that purpose itself was already solved in the process.
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
AI coding has made CI a bottleneck, so we reworked ours to keep up
(linear.app)245 points by julian_digital 14 hours ago | 269 comments
Comments
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
If you even review PRs still: when was the last time you didn't just skip over tests? And if you ever looked at tests in an LLM-heavy PR, how many of those tests tested something useful, and not just built-ins and trivial behaviours?
There's at least some awareness in the industry of how LLMs generate a lot of boilerplate in business logic. It feels like we're much less aware of how much of it is in tests.
[0] https://linear.app/now/sharing-growth-with-the-people-buildi...
In my experience, self hosting your CI/CD runners had the biggest impact in cost savings throughout, while also allowing for more powerful machines, which directly means quicker CI/CD runs
Migrated actions to a spare M2 MacBook Air at home, and when offline, uses hosted Actions as a backup. Bonus: has simulators/emulators for ease of running Maestro tests for mobile QA.
This is interesting. If I read that correctly this means the CI host had a direct peering connection with Github or Azure network.
Why were they not able solve the issue?
the blog mentions that they use a local cache to "reduce the time" but the problem is not solved right?
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Anyways great blog post from linear team a lot to learn from it
Did the tests produce four times as much value, though?
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
CI should handle only stuff which cannot be run locally due to resource or setup reasons.
Also CI should only receive pre-validated change candidates with passed checks inside the agent loop.
And this includes _full_ linting and unit tests.
I did that by self-hosting Github Runners, using Podman to build images (it doesn't squirt your build context through a socket every time!), and using Docker images as content-addressable cache.
I will do a write-up about it this week...
I think the SDLC needs to be re-evaluated, tests, specifically unit tests, are just cosmetics. "wow its so easy to get high coverage now!" as opposed to why we're doing it in the first place
I don't find agents to be using tests any differently than a junior or mid level developer. I don't find humans to be using tests any differently than a junior or mid level developer either.
Basically, the tests are never guiding the features, they aren't highlighting regressions, the test is simply modified for the updated application. Unit tests especially.
I have found end to end tests to be useful, and also unblocked. They can tell when the actual user experience has changed, and go back to the feature's implementation to stop altering the user experience in unexpected ways.
additionally, I find this across every industry that was worried about AI. basically, whatever best practice was neglected due to lack of investment into your org is now being done without additional investment into your org. that's a good thing. now we need to look at the purpose of the best practice, and if that purpose itself was already solved in the process.
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.