This is great. I particularly liked this observation:
> Shipping is a social construct within a company. Concretely, that means that a project is shipped when the important people at your company believe it is shipped.
A lot of the ideas in here resonated with me a lot. This is the kind of article I wish I'd read before I spent a few years leading an engineering team at a medium-sized tech company!
This post is about corporate politics, not "shipping projects." To wit:
If you ship something users hate and makes no money, but
your leadership team is happy, you still shipped. You can
feel any way you like about that, but it’s true. If you
don’t like it, you should probably go work for companies
that really care how happy their users are.
When the stated goal is to make leadership happy and not solving customer problems such that customers are "happy", that is pretty much the definition of politics. As subsequently identified therein:
Engineers who think shipping means delivering a spec or
deploying code will repeatedly engineer their way into
failed ships.
I would question the ethics of engineers whom employ a strategy of preferring politics over delivering solutions.
Oh boy. I've been around these trenches long enough to have seen the dark side of 'shipping' strategies.
Rarely in large enterprise environments are projects greenfields. The new system is often there to replace some existing Line of Business system.
Inthese cases the pervers but common shipping strategy is:
1. freeze all maintainance and evolution of the existing system. Seems reasonable until you realize that these enterprise projects are multi year calendar endeavours and the business will not stop changing during that time. The idea is not realy to halt all development, just to inctease the friction of getting anything done on the now declared 'legacy' system above an administrative pain threshold.
2. Commandeer all the IT staff to serve on the 'new' system's project team and overwhelm them with tasks. Make sure they remember any time spend on the 'legacy' (keep hammering in that term) codebase is basically waisting time assuring you'll end up at the bottom of the corporate dinosaur tar pit at best.
3. Force they 'new' system into production long before it is ready. Use all the leverage you build from point 1 to promise 'going forward'. When essential gaps to support the business are pointed out en mass week1, reverse the tables and intigate a very heavy administrative process demanding proof that the lacking functionality is actually required with a full ROI breakdown coutering your handwoven 3 minute overinflated guestimate for implementation. Again, you know essentials are missing, you are just trying to gain time as the longer you can stay deployed the higher the chance you will.
This process is very destructive to not just the business, but also moral of most involved. Most projects taking this scorched earth shipping strategy fail regardless. This is no small contribution to the cynicism oftem found in veteran battle hardened corporate IT shops.
This mirrors my views. In sports, winning fixes all problems; in software, shipping solves all problems. You need wins, and you need to ship. It’s like in Glengarry Glen Ross: instead of “ABC—Always Be Closing,” it’s “ABS—Always Be Shipping.”
You’ll never ship a perfect product; no such thing exists. But if you ship early (and incomplete), users will be delighted—and often change their minds about what they want. That’s great, because you didn’t spend forever on their initial (flawed) requirements.
This is a reasonably good if somewhat depressing blog post about how to do well in the performance cycle at a big company. Now part of that is shipping product, but that’s pretty clearly of secondary concern for the author.
It sounds to me like this person is good at getting paid no matter how it goes for the shareholders or the rest of the team.
*In my experience, projects almost always ship because one single person makes them ship. To be clear, that person doesn’t write all the code or do all the work, and I’m not saying the project could ship without an entire team behind it. But it’s really important that one person on the project has an end-to-end understanding of the whole thing: how it hangs together technically, and what product or business purpose it serves.*
Yes that is pretty much what I see as well, I see how things go to nothing with wrong people who think they can make bunch of requirements and dump them on dev/qa - then be surprised nothing works - and also how much stuff I try to hand over ends up not happening despite people claiming that they are very motivated to pick up a project. If they are motivated and I see nothing happens in 3 weeks I just take it over again and don't even talk to the guy anymore.
I am also quite often seen as an asshole because I don't ask "everyone" for permission and mostly care about my actual boss.
This is called project management, and in my experience seems to be a practice/discipline that's often overlooked or ignored at startups and scaling tech companies.
Generally in my book projects should only be assigned to teams, and managing that project to completion should be part of that team's way of working.
Doing this in an agile manner generally means smaller projects, one at a time.
Excellent article. I'm going to pull out one point out of many that made me think:
> Project competence. You want to aim for something like a NASA mission control vibe
Having read a little about early Apollo-era (and earlier) mission control this was a fun comparison. The astronauts and flight controllers had no idea what they were doing! There was no textbook – they were performing a human first. They were generally young and very cocky people hired out of university, but were successful anyway, for some reasons, including:
1. They were further culled through harsh adversarial simulation training. This made sure only those that could cut it came out the other end.
2. They approached each new mission phase with a trial mission, each time extending the envelope in a controlled manner. (The cowboy moon touchdown on Apollo 11? It's often described as the first moon landing, but it really wasn't. It was just a test to see if they could reach the surface and get back up. The real first precision landing came on Apollo 12. Oh, but that was also just a rehearsal for the first scientific mission on Apollo 14, etc.)
3. They were fluid in how they organised themselves, and allowed mission needs to dictate which positions existed, rather than trying to impose a hierarchy onto a mission.
Going back to shipping, we can squint and look at this as:
1. Fake it, either until you make it or get tossed out.
2. Prefer to ship many smaller things than one big thing. This builds confidence in your abilities, gets you experience shipping things, and allows you more feedback to adjust to what the real project is.
3. Build coalitions across department boundaries, and draw on their support to ship.
Analogies are like scissors: more fun when you run with them.
> fact, it’s paradoxically often better for you if there is some kind of problem that forces a delay, for the same reason that the heroic on-call engineer who hotfixes an incident gets more credit than the careful engineer who prevents one.
I could really relate to this. Having been part of incidents, I fucking hate them and try to anticipate them with various safe deployment strategies. To leaders though it apparently seems like Im not doing anything because “nothing ever goes wrong” lol.
It’s about pleasing upper management. Those are not the same things. The article even says it explicitly - if your users hate it and the market laughs, but executives like it, you’ve shipped. If you’ve given users the best software ever but your executive management doesn’t know, or even actively hates it, you haven’t shipped.
Think a lot of comments here (in my humble opinion) have taken the wrong interpretation of this article. Don't think this is a project-manager role, but is, as stated, typically a "tech lead or a DRI role." This means this person is still technical - would like to think this is how I approach my projects at a company and it is super effective.
Specifically then, as a technical person, this means you don't ignore the requirements and just think about the code. Instead, you need to communicate with both your higher-ups and your end-users, in order to make sure the code you're writing matches their needs - in my opinion, requirements is the part that most developers are weakest in, but if they strengthen would make the biggest difference. Get it as close as possible to being right on the first time (it's very rare that this happens but focusing on doing really good requirements increases your chances that you don't create the wrong system).
a project is shipped when the important people at your company believe it is shipped
Definitely. This can work very much to your favour.
There have been times when I have happily shipped a product containing issues and bugs that would be catastrophic for any customer using it, because I know that no customer actually ever will. The product is shipped and released, everyone is happy.
The catastrophic issues won't ever come up because that shipped product will never be used by anyone.
Big people are happy because the product is shipped.
Team are happy because big people have stopped causing so much friction in their desire to ship.
When the measurement becomes "have we slapped the label shipped on it", then that's what you get. A label.
When I'm shipping on a deadline, one of my favorite things to say is, "there is no future". When someone says they want to do X because they will do Y in the future, I say, "There is no future, there is only shipping. Give me what I need to ship and then we can talk about the future."
Pigs can fly. I know this because if it's got to ship and it's a pig, then sufficient thrust will be brought to bear to achieve flight.
Shipping is some liminal space between sheer will, recklessness, and tautology. It ships because it has to ship, because to not ship is to fall, and failure is not an option.
this mirrors my experience, but it doesn't give much actionable advice and probably only rings true to those who have experienced the same motions. "You only know your project shipped if your leadership acknowledges it". Yes ok great, how do you get their attention. I think this piece could be more easy to understand and influential if the author included a specific IRL project as a case study instead of having a bit of memoir about the vibes of their experience.
I see so much backlash against Project Managers and they're just Outlook Calendars with legs, but this post is basically a love letter to good Project Managers/Release Train Engineers/Delivery Leads, whatever you want to call it.
> Sometimes it’s just an influential VP or CEO’s pet project, and you need to align with their vision.
My biggest career mistake was a project that I had no way to know was the CEO's pet project.
Can't say where (non-disparagement agreement), but oh boy was that a lot of messy fallout — QA awarded me a prize for least bugs on launch, yet at the same time line manger also immediately went from giving me bonuses to putting me on a PIP for having too many bugs in launch.
Learning how to cancel, or quietly bury projects, can be just as important.
You're quite likely to be handed projects that make no sense. Bad ideas are created every day. Even if you ship them, you may be associated with their failure.
They ship because they do project management right:
1. Focus on stakeholder management
2. Know and understand what you build end-to-end
3. Because of 2. you can quickly assess the impact of various options to resolve a problem and communicating this back to stakeholders, generating trust
I like the example of 3. In the blog post.
So many people run projects like they just coast around. Never a moment to pause, reflect and proactively thinking about potential risks and possible mitigations.
Oh my, we never saw this road block coming, how could we have known? Well..
Articles like this are off to me… you’re going to need to quantify and qualify your claims of things otherwise I don’t know how much credence to throw ya.
“I ship at big tech” is a broad statement… as a reader, I need elaboration or I tend to question the rest of the article. Is this just a guy bloviating or does he really know his stuff? And bold claims need to be back up.
Regarding the “realism” proposed here, that leadership defines what shipping means: there’s a good faith and bad faith interpretation.
The bad faith interpretation is that this is corrupt and misaligned.
The good faith interpretation is that each leader has context and motivations that are important.
The right interpretation depends on the exact circumstance.
Getting leaders aligned on the right definition of “shipped” is a separate problem to the one described here, and it’s not the project lead’s direct responsibility. It’s the responsibility of the leadership team, the CEO, and the board.
But the realism proposed in the post is useful for project leads. Meeting leadership’s expectation is not necessarily corrupt, and if it is corrupt, it can be improved by directly discussing and negotiating what it means to ship a project.
I effectively agree with the "destination", so to speak, of this post, but disagree with the path taken to get there.
In general when I ship projects in the context of the org I with for, I _do_ meet the requirements of higher-ups/stakeholders, but not because that's my goal -- but because we're aligned on those requirements. Shipping to make management happy feels kind of incestuous, you should be shipping because you and management both agree on what the end goals of the project are.
This might seem like a subtle distinction, but it can create issues. If you're using your management as your metre-stick without truly understanding and believing in what they want, you'll likely over-optimize for the project's goals that they've stated, but miss the target of actually making the project successful.
This approach is fine if you're primary goal is to get paid, get promoted, please your managers, etc. But if your goal is to create great things, I think you need set your metre-stick based on the project not on the management. From the outside these approaches look very similar -- which is why I largely agree with the "destination" and a lot of the tips in this post, otherwise.
This hasn’t been my experience at Big Tech (in ~20 years). There’s plenty of projects I’ve shipped that didn’t have upper management support but after the user feedback or metrics were positive they got counted as a win. There’s also many smaller projects that you can ship without anyone paying much attention but they can still be valuable either to some subset of users or in aggregate. For example, +1% on some metric isn’t going to get you a call out from the VP but do that every month and now you’re talking about big aggregate gains.
As any worthwhile piece of advice, this post is not saying "Do good things, don't do bad things" but instead "Prioritize one good thing over another".
Hence lots of comments. I wish there were more submissions like this.
If you work at a company like this where leadership does not understand the engineering, the value, or the consequences of their reports but only want to talk about it at review-time, you are not in a healthy environment with an actual leadership.
And you will feel the toxicity created by this kind of poor leadership by way of constant blame games, misalignments, and unnecessary stress over garbage.
> Can we ship right now? Feature flags are the best way I’ve seen to do this, but staging environments also work, and so on.
Feature flags are a super weapon if you are trying to keep the path to production clear, especially if you have a diverse deployment or customer base. They don't have to be a constant binary value either. I've set up "flags" that were SQL queries which produced the actual boolean fact based upon customer-specific data and requirements.
These things do tend to accumulate, but cleaning them up is easy as long as you have a way to report on the current configuration across all deployments. If two or more instances have different settings but their users seem happy with this arrangement, then you go the other way and promote the flag into some kind of official configuration abstraction.
Customer and environment specific branches quickly become a death spiral if you are trying to grow a B2B/SaaS product. I'd much rather fight a spreadsheet of flags & config than rebase branches and resolve conflicts that have silently piled up for the last several months.
The idea of “shipping” as being an emergent state of mind is a great one. A project grows the same way an embryo grows into a full animal: it does so in millions of tiny parts that overlap in discrete ways to produce a continuous spectrum of development. I think some good additional milestones to measure that kind of development might be as follows:
* Your first bug report that you file against yourself which you fix yourself. Your project is stable enough to be usable with a known bug, but you go back and fix that bug when you have time to dedicate to it. Contrast with nothing working until everything works.
* Your first time seeing someone find a flaw that lies so far out of your own mental model of the project that it causes a very jarring feeling, hopefully followed by excitement that the project is now taking on a life of its own outside you / your team as the sole developers and users. An example of this might be someone filing a bug that image rendering is missing from a git visualisation tool where you’d never even considered using it for projects with version controlled image assets.
I feel like there’s an analogy here to a baby’s first detectable heartbeat and a child’s first original that-never-occurred-to-me thought. There must be a lot more in between too, with the social acceptance of a project becoming shipped as the child entering the adult world as its own person.
> no matter the project goal, your leadership team (the people in your reporting chain who care about the project) will always have basically zero technical context about the project compared to you.
This is not universally the case, and I've been much happier when I find teams and orgs where this is not true.
Good leaders understand the details. Find them. Be one.
Yeah, I think TFA gets a lot right about doing software engineering in large orgs.
We call this role something like "lead developer" and it's as much about people, relationships, logistics, and documentation as it is about technical systems — but like TFA says, it's also your job to have the best holistic understanding of the technical system. A lot of more junior engineers are likely to work on a large project, and they will inevitably need guidance to satisfy the constraints of other parts of the system. You are responsible for knowing those constraints and, as much as possible, preventing mismatches when different components come together.
Meanwhile, the non technical staff (whether it's a manager or someone from the sales team, documentation team, customer support team, etc) are going to need someone who can explain the technical system to them in plain language. "Can it do this? How fast? Why can't it do this?" You tend to become their main interlocutor.
It's kind of your job to anticipate problems before they happen, to look into the future so you see what's coming and solve it in advance, and then to show up all the time when something needs help around launch, which it always does. It's very undefined but there's always something that needs doing.
This role doesn't sit well with some engineers who think their job is basically about coding and staying within their lane — those engineers don't get to be lead dev too much, because they don't do well with the organizational side of things. (It depends, of course, whether there is a hands-on project manager who can pick up a lot of that slack, but where I work, we don't always have that.)
(Context: I have worked in a large tech company a little more than 3 years and shipped some things. And before that I shipped a lot more things in smaller places – I like to think that shipping in small shops was a good prerequisite for shipping in large ones.)
This reflects my personal experience as "CTO" of a software factory. I put CTO between quotes because my role definitely shifted away from technology itself more to a delivery role. I have to delegate responsibilities to the right people, act fast when a team member is pulling its weight, allocate resources to solve problems a team can't, and even jump in and work along the team if shit hits the fan. Delivery is the hardest thing; something that works for me is trying to have a plan and not only consider features but a checklist to reach Prod. What I mean is adding all those tiny things that add up to the planning (who's in charge of the DNS? who's setting up the Twilio account? etc etc etc)
Great article. It's not "shipping" as in making some feature a reality, but more contextualized within big tech and perhaps larger sized companies. Understandably, some people might call this "unethical", but it's kind of a "game" that you play (or don't) being in a large, hierarchical org.
> But it’s really important that one person on the project has an end-to-end understanding of the whole thing: how it hangs together technically, and what product or business purpose it serves.
In the before times, we called this an architect. We're too agile for that now, apparently.
A constant focus on keeping the architecture,infrastructure and codebase
quick/ease to ship is to "Keep It Simple Stupid".
Be critical about adding new services or complexity.
Ask yourself can this be solved in a way that does not require
changes to the shipping process.
If no, evaluate possible alternate approaches.
A fanatical dedication to monitoring the production system is beneficial.
The faster you can catch that the last deploy screwed something up,
the less stress and down time there will be.
The shipping process should also include the ability a rollback the latest
deployment. Again less stress and possible downtime there will be.
Achieving these goals is the best way to win the trust and influence
on the people up the chain.
Every project has a role of Release Manager. Some teams are small enough that this role is filled by a team lead, project manager, architect, etc. some teams rotate this role to spread the pain and knowledge. Large teams have a defined employed role with title of Release Manager that could even include a team (think shipping large DoD contractual projects)
There’s whole books on this subject. I think this blogpost does a great job of summarizing the goals for large tech company shipping — well written — nicely done
This aligns with what I've seen and the issues I've had shipping high leverage things at an enterprise. Its way more communication, selling and documenting more that it is a technical issue. Nice post.
This is why I'm a big fan of MVPs and quick iterations after ward.
Launching a significantly complex product is HARD, even if it is an MVP.
Nobody understands how hard, even if they've done it before. It works in staging, must be ready! Then users start trying to signup and actually use the product and the wheels come off.
I've usually been the person that understands it all and launch week is always hell. Or depressingly boring because you have no users.
It's all about the last 10% - it's what makes a project of any size see the light of day. Some thrive on that last 10% (like I do). It's the work of grinding through the final loose ends and the finishes touches, fixing the issues in the live environment vs development, etc.
It's hard and boring, but it's also rewarding.
And then there are most people - who just check out when it's sort of done. I don't like those people.
I think the whole idea of shipping, at least within big tech, should follow slow rollouts starting with 0%. For example: Deploy an empty service in production with 0% rollout. Increment the rollout with each version deployed, including fixes of previous releases.
This may not be great for user experience but great for getting things up and running. Although, upper management will not be happy to see slow rollouts.
The way I've used to describe this to people is, most people's mental model is:
<doing a poor job> --- <doing a average job> --- <doing a good job>
Whereas the more accurate mental model is:
<doing nothing> --------------------------------------- <doing a thing> -- <doing the best thing>
The former mental model is intellectually satisfying because you get to infinitely yak shave over every little tradeoff and doing nothing isn't even on your mental radar.
The latter is deeply humblingly mundane and doesn't get to center us as masters of the universe which is why we all initially resist it. Even for the people who become great at it, they become great against their inclinations, and still have to fight the urge at every step.
This doesn't just apply to tech, look at it in politics. How much time do we spend in politics debating policy? Which of our different political philosophies lead to different tradeoffs that are a deep expression of our values? How much time in politics do we spend debating state capacity? A state that can do none of the things, it's irrelevant which of the things we want the state to do. A state that can do more things, almost always just the sheer option of things they can do means they can find a better policy.
And yet policy arguments are just so fun while state capacity arguments are incredibly wonky and depressing.
But I've drawn that same diagram on 100 different whiteboards in front of 100 different juniors and said the same words of "before we figure out how to do the best thing, let's just see if we can even do a thing first" and it seems to have gotten through to an appreciable number of them enough that it feels like a high converting meme.
Great article. That bit about having 20% free at the start to 90-100% free time towards the end of a project, to address issues, is very insightful. I stumbled a lot in a project recently because I suspect I had it the other way round.
I agree that every project needs one person who has a laser focus on getting the project across the finish line. There are always dozens of slightly annoying and unrelated blockers.
I call it my volleyball theory, after playing on a mid team back in college. If the ball is coming down near a large group, no one gets it. If you're the only one there, you'll get that ball one way or another!
this is a reality in organisation: " it’s paradoxically often better for you if there is some kind of problem that forces a delay, for the same reason that the heroic on-call engineer who hotfixes an incident gets more credit than the careful engineer who prevents one." Its ironic in many areas. A leader (Project or Political) who ran projects and shipped smoothly is less valued than a leader who created a mess and got them fixed ...who is more celebrated. :(
most of this makes sense but I think what would make it more useful is more focus on topics that mainly fall under project management: building relationships with key stakeholders, establishing early+clear communication channels, building in timeline slack, how to navigate management politics etc. i dont think shipping in big tech cos is significantly different to shipping a large construction project in terms of project management, but rather there is a tendency for managers in big tech to be significantly less tenured and hence ignorant of basic management priniciples that lead to easily avoidable blunders.
> I have shipped a lot of different projects over the last ~10 years in tech. I often get tapped to lead new ones when it’s important to get it right, because I’m good at it.
Why not list some of them that I can actually try out? How should I give this any credence if I cannot evaluate them?
Articles like these really speak to me. In a lot of ways, developers that claim to take ownership might be a few small cognitive steps from actual ownership aka have shipping in mind. Lots of take aways to prevent headaches and pain points.
It’s a great post and often what I see folks doing.
Hot take though: why do we need so many people and processes and mysticism around software these days? It seems to me the reason why it’s so difficult to ship anything, let alone anything good and useful, is because there are so many people involved who don’t actually participate in engineering. They manage time lines and meetings and emails and talking to customers. And somehow they’ve made themselves essential to the process so that developers can’t ship code anymore.
Like… I know my users on a first name basis. I know how their business works. We’ve talked about the button and we want to run an experiment.
As a developer I might want to just try it out with them. Maybe the change will be a good fit for the product as a whole, maybe it will be a solution that only works for this one customer, maybe it means we need to be able to configure which workflow is used.
If I can’t just ship that and get the feedback I need without involving a handful of other people and being completely fearful that I could take down our service… what is wrong here? Is it that there aren’t enough “shippers” on the team or is it because the organization has made it difficult to ship anything?
That being said, this is the world we live in, and the article has a lot of great advice. I’m definitely used to/conditioned to think this way.
Excellent piece. My observation in less technically-oriented but still innovative companies: shipping also occurs when you post about it on LinkedIn and can show with a concrete example (a graph or a small video)
I've also "shipped" projects at big tech companies and this level of bootlicking really doesn't deserve to be called "shipping" it's just delivering for your management. Using the term "shipping" is a great scissor statement because it muddies the water.
The mentality from the article is another symptom of the many issues Software Engineering faces as profession.
Namely that a significant portion of us think of ourselves not as engineers, who need to tell management to get fucked occasionally, but as optimizers for accolades from whatever group can dole out rewards. This starts off in academia and continues into the professional world.
Certain folks just want to build a technical fiefdom for themselves, or get headpats from people who are above them in any given hierarchy.
Yeah this is "how the game is played". Playing this game eventually leads to the death of your organization and is why we have a corporate life cycle in the first place.
Eventually people like this eat an organization from the inside, pushing out anyone with an actual opinion or who optimizing for actually getting things done.
Instead it's just pure mercenary behavior for your managements attention.
Is it just me. Or has the term "ship", "shipped", only become a (specific to software production) industry-used term asn of relatively recent times? Specifically as Sam Altman has been using it a lot I have also noticed. Ever since his increased use of the term, I've seen it used more and more. This could be one of those times where I am noticing it because I am thinking about it, but it really doesn't feel like that.
I have always associated new software versions with with my the word "release", "released". While shipped is not an incorrect term to use for software. Release just feels more appropriate.
Has anyone else noticed this?
All of that said, I am not condemning the practice in the least. Just interested in the etymnology of one of my favorite areas of interest. Hopefully draw some more knowledgeable folks on the question.
The practical advice in here resonates very strongly with my experience, I like the analysis a lot.
The philosophy about what "shipping" is... Presupposes that your motivation for working in big tech is to get to L9 and make 7 figures or whatever. Dunno about that. (Also I suspect there's a ceiling on where you can get by being really good at shipping).
I like to think I do this shit on my own terms, I have my own goals. Pleasing leadership and getting good ratings is very often instrumental to those goals (and money is great) but I think I still get to decide what shipping means to me.
Me myself, I joined a project that shipped on time. The lead developer, a contractor, begged for just three more months to make it all actually work properly.
Nope.
We shipped, and then it took two more years to finally get it functional.
And the contractor was banned from ever working there again (a WA state agency).
Shipping for your leadership instead of users is so very American and explains how serial blunderers fail from one big project to another while the employees get the short end of stick. Happy if you makes you happy that some big wig on the leadership like you, but is that really all you want to achieve in life?
> Shipping is really hard and you have to make it your main priority if you want to ship
This feels like it’s written by AI by a junior level “prompt engineer”
> Shipping doesn’t mean deploying code, it means making your leadership team happy
Aka, corporate glazing. Gotcha. Over sell your corporate accomplishments and polish your turd with a clean deck. Gotta make sure the font is just barely legible and explain it quickly before they ask follow up questions
> You need your leadership team to trust you in order to ship
yea that’s why I get _paid_ to do work, not glaze c-level executives and appeal to their fragile egos
> Most of the essential technical work is in anticipating problems and creating fallback plans
Falling back is a loser mentality.
> You should asking yourself “can I ship right this second?”
no, I should be asking myself, “why am I still at this dead end job that asks me to constantly ‘ship’ a broken product. Leadership doesn’t want to encourage delivering value to the customer”
> Have courage!
You don’t need courage at a “big TeCh” job. Just be slightly competent and you are already light years ahead of your colleagues
How I ship projects at big tech companies
(seangoedecke.com)1361 points by gfysfm 11 November 2024 | 375 comments
Comments
> Shipping is a social construct within a company. Concretely, that means that a project is shipped when the important people at your company believe it is shipped.
A lot of the ideas in here resonated with me a lot. This is the kind of article I wish I'd read before I spent a few years leading an engineering team at a medium-sized tech company!
EDIT: clarified the ethical question.
- Have endless arguments about what to name the product
- Have industrial design team enforce the latest design language
- Create product variant SKU numbers, get them loaded into ordering system. Get buy-in from sales on these variants.
- Create part numbers for the FRUs and CRUs.
- Create multi-level BOM for all the variants.
- Buy replacement parts and push them into parts depots
- Calculate reliability so that warranty department knows how much to charge you
- Get agency approvals
- Create packaging
- Test packaging
- Translate any words on the package into french (for Canada)
- Get vendors to sign your big company environmental requirements document
- Write legal notices and get it translated into many languages
- Work with documentation team to get printed (usually short) manual translated into many languages
- Get safety engineer to sign off on products, usually means you need to make a bunch of stickers and include them in the BoM.
- Figure out the part numbers for the power cords needed for each country
- Figure out the part numbers for the rack mount rail kits for various types of racks that you support
- Write announcement letter
- Create videos and documents for sales team training
- Write troubleshooting guide
- Train repair center
etc..
Rarely in large enterprise environments are projects greenfields. The new system is often there to replace some existing Line of Business system.
Inthese cases the pervers but common shipping strategy is:
1. freeze all maintainance and evolution of the existing system. Seems reasonable until you realize that these enterprise projects are multi year calendar endeavours and the business will not stop changing during that time. The idea is not realy to halt all development, just to inctease the friction of getting anything done on the now declared 'legacy' system above an administrative pain threshold.
2. Commandeer all the IT staff to serve on the 'new' system's project team and overwhelm them with tasks. Make sure they remember any time spend on the 'legacy' (keep hammering in that term) codebase is basically waisting time assuring you'll end up at the bottom of the corporate dinosaur tar pit at best.
3. Force they 'new' system into production long before it is ready. Use all the leverage you build from point 1 to promise 'going forward'. When essential gaps to support the business are pointed out en mass week1, reverse the tables and intigate a very heavy administrative process demanding proof that the lacking functionality is actually required with a full ROI breakdown coutering your handwoven 3 minute overinflated guestimate for implementation. Again, you know essentials are missing, you are just trying to gain time as the longer you can stay deployed the higher the chance you will.
This process is very destructive to not just the business, but also moral of most involved. Most projects taking this scorched earth shipping strategy fail regardless. This is no small contribution to the cynicism oftem found in veteran battle hardened corporate IT shops.
You’ll never ship a perfect product; no such thing exists. But if you ship early (and incomplete), users will be delighted—and often change their minds about what they want. That’s great, because you didn’t spend forever on their initial (flawed) requirements.
It sounds to me like this person is good at getting paid no matter how it goes for the shareholders or the rest of the team.
Yes that is pretty much what I see as well, I see how things go to nothing with wrong people who think they can make bunch of requirements and dump them on dev/qa - then be surprised nothing works - and also how much stuff I try to hand over ends up not happening despite people claiming that they are very motivated to pick up a project. If they are motivated and I see nothing happens in 3 weeks I just take it over again and don't even talk to the guy anymore.
I am also quite often seen as an asshole because I don't ask "everyone" for permission and mostly care about my actual boss.
Generally in my book projects should only be assigned to teams, and managing that project to completion should be part of that team's way of working.
Doing this in an agile manner generally means smaller projects, one at a time.
Well now, hold on a second. You didn't tell me this was about consulting.
If we're being paid like pirates and fightin' like pirates thats one thing.
> Project competence. You want to aim for something like a NASA mission control vibe
Having read a little about early Apollo-era (and earlier) mission control this was a fun comparison. The astronauts and flight controllers had no idea what they were doing! There was no textbook – they were performing a human first. They were generally young and very cocky people hired out of university, but were successful anyway, for some reasons, including:
1. They were further culled through harsh adversarial simulation training. This made sure only those that could cut it came out the other end.
2. They approached each new mission phase with a trial mission, each time extending the envelope in a controlled manner. (The cowboy moon touchdown on Apollo 11? It's often described as the first moon landing, but it really wasn't. It was just a test to see if they could reach the surface and get back up. The real first precision landing came on Apollo 12. Oh, but that was also just a rehearsal for the first scientific mission on Apollo 14, etc.)
3. They were fluid in how they organised themselves, and allowed mission needs to dictate which positions existed, rather than trying to impose a hierarchy onto a mission.
Going back to shipping, we can squint and look at this as:
1. Fake it, either until you make it or get tossed out.
2. Prefer to ship many smaller things than one big thing. This builds confidence in your abilities, gets you experience shipping things, and allows you more feedback to adjust to what the real project is.
3. Build coalitions across department boundaries, and draw on their support to ship.
Analogies are like scissors: more fun when you run with them.
I could really relate to this. Having been part of incidents, I fucking hate them and try to anticipate them with various safe deployment strategies. To leaders though it apparently seems like Im not doing anything because “nothing ever goes wrong” lol.
This article is not about “shipping”software.
It’s about pleasing upper management. Those are not the same things. The article even says it explicitly - if your users hate it and the market laughs, but executives like it, you’ve shipped. If you’ve given users the best software ever but your executive management doesn’t know, or even actively hates it, you haven’t shipped.
Blech.
Specifically then, as a technical person, this means you don't ignore the requirements and just think about the code. Instead, you need to communicate with both your higher-ups and your end-users, in order to make sure the code you're writing matches their needs - in my opinion, requirements is the part that most developers are weakest in, but if they strengthen would make the biggest difference. Get it as close as possible to being right on the first time (it's very rare that this happens but focusing on doing really good requirements increases your chances that you don't create the wrong system).
Definitely. This can work very much to your favour.
There have been times when I have happily shipped a product containing issues and bugs that would be catastrophic for any customer using it, because I know that no customer actually ever will. The product is shipped and released, everyone is happy.
The catastrophic issues won't ever come up because that shipped product will never be used by anyone.
Big people are happy because the product is shipped.
Team are happy because big people have stopped causing so much friction in their desire to ship.
When the measurement becomes "have we slapped the label shipped on it", then that's what you get. A label.
Pigs can fly. I know this because if it's got to ship and it's a pig, then sufficient thrust will be brought to bear to achieve flight.
Shipping is some liminal space between sheer will, recklessness, and tautology. It ships because it has to ship, because to not ship is to fall, and failure is not an option.
I have been shipping (as opposed to "writing") software, for pretty much my entire adult life.
A lot of it has involved holding my nose, and gingerly handing it off, dangling between my thumb and index finger, while wearing rubber gloves.
This chap has a point. "Ship" is in the eye of the managers.
Which is why good management is so important.
My biggest career mistake was a project that I had no way to know was the CEO's pet project.
Can't say where (non-disparagement agreement), but oh boy was that a lot of messy fallout — QA awarded me a prize for least bugs on launch, yet at the same time line manger also immediately went from giving me bonuses to putting me on a PIP for having too many bugs in launch.
(more amazing: 54 ships, but not a single leader-ship joke. that takes nerve.)
You're quite likely to be handed projects that make no sense. Bad ideas are created every day. Even if you ship them, you may be associated with their failure.
1. Focus on stakeholder management
2. Know and understand what you build end-to-end
3. Because of 2. you can quickly assess the impact of various options to resolve a problem and communicating this back to stakeholders, generating trust
I like the example of 3. In the blog post.
So many people run projects like they just coast around. Never a moment to pause, reflect and proactively thinking about potential risks and possible mitigations.
Oh my, we never saw this road block coming, how could we have known? Well..
Some of my observations:
- Often it is about taking a leadership & ownership of fixing if it fails in the production
- Often it is about, guiding team not to overengineer for the future scenarios
This becomes 500% difficult when you are building for someone else:
In our current scenario, we are working with a semi-government organisation who goes by the RFP fine print then an actual project needs.
“I ship at big tech” is a broad statement… as a reader, I need elaboration or I tend to question the rest of the article. Is this just a guy bloviating or does he really know his stuff? And bold claims need to be back up.
The bad faith interpretation is that this is corrupt and misaligned.
The good faith interpretation is that each leader has context and motivations that are important.
The right interpretation depends on the exact circumstance.
Getting leaders aligned on the right definition of “shipped” is a separate problem to the one described here, and it’s not the project lead’s direct responsibility. It’s the responsibility of the leadership team, the CEO, and the board.
But the realism proposed in the post is useful for project leads. Meeting leadership’s expectation is not necessarily corrupt, and if it is corrupt, it can be improved by directly discussing and negotiating what it means to ship a project.
In general when I ship projects in the context of the org I with for, I _do_ meet the requirements of higher-ups/stakeholders, but not because that's my goal -- but because we're aligned on those requirements. Shipping to make management happy feels kind of incestuous, you should be shipping because you and management both agree on what the end goals of the project are.
This might seem like a subtle distinction, but it can create issues. If you're using your management as your metre-stick without truly understanding and believing in what they want, you'll likely over-optimize for the project's goals that they've stated, but miss the target of actually making the project successful.
This approach is fine if you're primary goal is to get paid, get promoted, please your managers, etc. But if your goal is to create great things, I think you need set your metre-stick based on the project not on the management. From the outside these approaches look very similar -- which is why I largely agree with the "destination" and a lot of the tips in this post, otherwise.
And you will feel the toxicity created by this kind of poor leadership by way of constant blame games, misalignments, and unnecessary stress over garbage.
> you have to get clear on what the company is looking to get out of the project.
Regardless of politics, starting with why we are doing this project at the first place is often overlooked.
Feature flags are a super weapon if you are trying to keep the path to production clear, especially if you have a diverse deployment or customer base. They don't have to be a constant binary value either. I've set up "flags" that were SQL queries which produced the actual boolean fact based upon customer-specific data and requirements.
These things do tend to accumulate, but cleaning them up is easy as long as you have a way to report on the current configuration across all deployments. If two or more instances have different settings but their users seem happy with this arrangement, then you go the other way and promote the flag into some kind of official configuration abstraction.
Customer and environment specific branches quickly become a death spiral if you are trying to grow a B2B/SaaS product. I'd much rather fight a spreadsheet of flags & config than rebase branches and resolve conflicts that have silently piled up for the last several months.
* Your first bug report that you file against yourself which you fix yourself. Your project is stable enough to be usable with a known bug, but you go back and fix that bug when you have time to dedicate to it. Contrast with nothing working until everything works.
* Your first time seeing someone find a flaw that lies so far out of your own mental model of the project that it causes a very jarring feeling, hopefully followed by excitement that the project is now taking on a life of its own outside you / your team as the sole developers and users. An example of this might be someone filing a bug that image rendering is missing from a git visualisation tool where you’d never even considered using it for projects with version controlled image assets.
I feel like there’s an analogy here to a baby’s first detectable heartbeat and a child’s first original that-never-occurred-to-me thought. There must be a lot more in between too, with the social acceptance of a project becoming shipped as the child entering the adult world as its own person.
True, but that doesn't mean you have a good leader. They are hard to find, but not impossible.
This is not universally the case, and I've been much happier when I find teams and orgs where this is not true.
Good leaders understand the details. Find them. Be one.
We call this role something like "lead developer" and it's as much about people, relationships, logistics, and documentation as it is about technical systems — but like TFA says, it's also your job to have the best holistic understanding of the technical system. A lot of more junior engineers are likely to work on a large project, and they will inevitably need guidance to satisfy the constraints of other parts of the system. You are responsible for knowing those constraints and, as much as possible, preventing mismatches when different components come together.
Meanwhile, the non technical staff (whether it's a manager or someone from the sales team, documentation team, customer support team, etc) are going to need someone who can explain the technical system to them in plain language. "Can it do this? How fast? Why can't it do this?" You tend to become their main interlocutor.
It's kind of your job to anticipate problems before they happen, to look into the future so you see what's coming and solve it in advance, and then to show up all the time when something needs help around launch, which it always does. It's very undefined but there's always something that needs doing.
This role doesn't sit well with some engineers who think their job is basically about coding and staying within their lane — those engineers don't get to be lead dev too much, because they don't do well with the organizational side of things. (It depends, of course, whether there is a hands-on project manager who can pick up a lot of that slack, but where I work, we don't always have that.)
(Context: I have worked in a large tech company a little more than 3 years and shipped some things. And before that I shipped a lot more things in smaller places – I like to think that shipping in small shops was a good prerequisite for shipping in large ones.)
The ego is just bleeding out of this sentence.
He's a SSE at GitHub.
In the before times, we called this an architect. We're too agile for that now, apparently.
Be critical about adding new services or complexity.
Ask yourself can this be solved in a way that does not require changes to the shipping process. If no, evaluate possible alternate approaches.
A fanatical dedication to monitoring the production system is beneficial. The faster you can catch that the last deploy screwed something up, the less stress and down time there will be.
The shipping process should also include the ability a rollback the latest deployment. Again less stress and possible downtime there will be.
Achieving these goals is the best way to win the trust and influence on the people up the chain.
There’s whole books on this subject. I think this blogpost does a great job of summarizing the goals for large tech company shipping — well written — nicely done
Launching a significantly complex product is HARD, even if it is an MVP.
Nobody understands how hard, even if they've done it before. It works in staging, must be ready! Then users start trying to signup and actually use the product and the wheels come off.
I've usually been the person that understands it all and launch week is always hell. Or depressingly boring because you have no users.
It's hard and boring, but it's also rewarding.
And then there are most people - who just check out when it's sort of done. I don't like those people.
This may not be great for user experience but great for getting things up and running. Although, upper management will not be happy to see slow rollouts.
<doing a poor job> --- <doing a average job> --- <doing a good job>
Whereas the more accurate mental model is:
<doing nothing> --------------------------------------- <doing a thing> -- <doing the best thing>
The former mental model is intellectually satisfying because you get to infinitely yak shave over every little tradeoff and doing nothing isn't even on your mental radar.
The latter is deeply humblingly mundane and doesn't get to center us as masters of the universe which is why we all initially resist it. Even for the people who become great at it, they become great against their inclinations, and still have to fight the urge at every step.
This doesn't just apply to tech, look at it in politics. How much time do we spend in politics debating policy? Which of our different political philosophies lead to different tradeoffs that are a deep expression of our values? How much time in politics do we spend debating state capacity? A state that can do none of the things, it's irrelevant which of the things we want the state to do. A state that can do more things, almost always just the sheer option of things they can do means they can find a better policy.
And yet policy arguments are just so fun while state capacity arguments are incredibly wonky and depressing.
But I've drawn that same diagram on 100 different whiteboards in front of 100 different juniors and said the same words of "before we figure out how to do the best thing, let's just see if we can even do a thing first" and it seems to have gotten through to an appreciable number of them enough that it feels like a high converting meme.
I call it my volleyball theory, after playing on a mid team back in college. If the ball is coming down near a large group, no one gets it. If you're the only one there, you'll get that ball one way or another!
Design by managers who simplistify everything into a manufacturing process in their "factory"
Those same simplistifiers managers would love "facilitators" who are now concerned with a function of shipping in their "factory".
Why not list some of them that I can actually try out? How should I give this any credence if I cannot evaluate them?
Hot take though: why do we need so many people and processes and mysticism around software these days? It seems to me the reason why it’s so difficult to ship anything, let alone anything good and useful, is because there are so many people involved who don’t actually participate in engineering. They manage time lines and meetings and emails and talking to customers. And somehow they’ve made themselves essential to the process so that developers can’t ship code anymore.
Like… I know my users on a first name basis. I know how their business works. We’ve talked about the button and we want to run an experiment.
As a developer I might want to just try it out with them. Maybe the change will be a good fit for the product as a whole, maybe it will be a solution that only works for this one customer, maybe it means we need to be able to configure which workflow is used.
If I can’t just ship that and get the feedback I need without involving a handful of other people and being completely fearful that I could take down our service… what is wrong here? Is it that there aren’t enough “shippers” on the team or is it because the organization has made it difficult to ship anything?
That being said, this is the world we live in, and the article has a lot of great advice. I’m definitely used to/conditioned to think this way.
The mentality from the article is another symptom of the many issues Software Engineering faces as profession.
Namely that a significant portion of us think of ourselves not as engineers, who need to tell management to get fucked occasionally, but as optimizers for accolades from whatever group can dole out rewards. This starts off in academia and continues into the professional world.
Certain folks just want to build a technical fiefdom for themselves, or get headpats from people who are above them in any given hierarchy.
Yeah this is "how the game is played". Playing this game eventually leads to the death of your organization and is why we have a corporate life cycle in the first place.
Eventually people like this eat an organization from the inside, pushing out anyone with an actual opinion or who optimizing for actually getting things done.
Instead it's just pure mercenary behavior for your managements attention.
I have always associated new software versions with with my the word "release", "released". While shipped is not an incorrect term to use for software. Release just feels more appropriate.
Has anyone else noticed this?
All of that said, I am not condemning the practice in the least. Just interested in the etymnology of one of my favorite areas of interest. Hopefully draw some more knowledgeable folks on the question.
The philosophy about what "shipping" is... Presupposes that your motivation for working in big tech is to get to L9 and make 7 figures or whatever. Dunno about that. (Also I suspect there's a ceiling on where you can get by being really good at shipping).
I like to think I do this shit on my own terms, I have my own goals. Pleasing leadership and getting good ratings is very often instrumental to those goals (and money is great) but I think I still get to decide what shipping means to me.
> "Ship It! A Practical Guide to Successful Software Projects", by Jared Richardson, Will Gwaltney, Jr
> "The Pragmatic Programmer, 20th Anniversary Edition, your journey to mastery", by David Thomas, Andrew Hunt
> "The Libertarian Coder", by Adam Ard, at https://rethinkingsoftware.substack.com/p/the-libertarian-co...
Me myself, I joined a project that shipped on time. The lead developer, a contractor, begged for just three more months to make it all actually work properly.
Nope.
We shipped, and then it took two more years to finally get it functional.
And the contractor was banned from ever working there again (a WA state agency).
Yippee!
Deploy often. Every day. Keep momentum. Ensure you have a chat box to talk with users so they can complain. Profit.
This feels like it’s written by AI by a junior level “prompt engineer”
> Shipping doesn’t mean deploying code, it means making your leadership team happy
Aka, corporate glazing. Gotcha. Over sell your corporate accomplishments and polish your turd with a clean deck. Gotta make sure the font is just barely legible and explain it quickly before they ask follow up questions
> You need your leadership team to trust you in order to ship
yea that’s why I get _paid_ to do work, not glaze c-level executives and appeal to their fragile egos
> Most of the essential technical work is in anticipating problems and creating fallback plans
Falling back is a loser mentality.
> You should asking yourself “can I ship right this second?”
no, I should be asking myself, “why am I still at this dead end job that asks me to constantly ‘ship’ a broken product. Leadership doesn’t want to encourage delivering value to the customer”
> Have courage!
You don’t need courage at a “big TeCh” job. Just be slightly competent and you are already light years ahead of your colleagues