How to Organize an Engineering Team So Security Work Happens: Part Two

Resources and Work

This is the second part of two part essay on how to get security work done when you’re starting a new security program done at a company.  In the first part, which you can read here, we focused on executive decision-making and the process of managing technical work.  In this part, we’re looking at the properties of the technical ecosystem and the team that each have the biggest impact on security program outcomes, the different categories of security work, and how they’re best staffed and organized.  First, though, a word on “capacity”.

Capacity

This is the thing that CTOs tend to worry about when looking at starting a security program — do they have the resources, both in dollars and engineer-hours, to do the work that this entails.  Sometimes, the answer is no.  We encourage teams to not start too early — if you don’t have product-market fit, you should be thinking about the security you need to be an ethical guardian of the data, money, and trust of your employees and early customers and to meet your legal obligations, but not more.¹  Every company is different, but a good starting point is to wait until you’re around the ten engineer mark, but no later than the thirty engineer mark (and preferably twenty).  You will need at least one person on the team dedicated to infrastructure (out of ten; more if you’re bigger), and it’s great if someone else can be at least 50% on IT work for laptops and the like.  Budget-wise, while you can spend much more, expect a minimum of about $200/employee/month across both IT and infrastructure/development tooling.  These are ballpark numbers and will vary by industry, technical stack, and strategy, among other things.

That said, capacity isn’t always the bottleneck.  Yes, security tools get expensive and the work can eat a lot of time.  You can trade time for money and vice versa, but only so much — more on this later.  You can also hire or fundraise your way out of a capacity problem, but this won't necessarily fix the constraints on getting security work done.

Mutability

At least two thirds of the initial work we put on clients’ roadmaps is basic engineering work, getting their production environment, CI/CD, development practices, and IT management up to par.  That can be quite a lift, but it pays the team back with more reliable systems and, usually, an environment that’s nicer and faster to work in.  The biggest resource the team needs for a successful security program is the ability to change their systems.  The clients we’ve had who have struggled the most with getting security work done have had legacy systems they’ve inherited from an earlier stage in the company’s life or critical components built by people who left no documentation.  Your fancy all-Terraform and Control Tower AWS organization doesn’t help if your legacy account has all the critical data in it and you’re 40% into a migration effort that’s already taken two years of work from 20% of your engineering team.²  Adding people to a late project famously doesn’t make it faster and may make it slower.

The bigger the system, the harder it is to understand and secure.  Think about every line of code you write as a liability, not an asset — it needs to be patched, maintained, refactored, monitored, and responded to.  You spend code to buy features.  Code quantity limits system mutability and thus system securability.  Keeping your system as simple as possible to solve the problems you actually have is key.  Why are you rolling out a clustered or sharded database when you haven’t profiled your database queries?  Is your 99th percentile database load really too big to fit with room to spare on a single AWS db.m9g.48xlarge instance with almost a terabyte of ram across 192 vCPUs?  Almost certainly not, and yet by clustering you’ve bought a system that will be more fragile and complex and will be harder and slower to change.

Technical debt adds complexity just as fast as early optimization and scale.  If there are two ways to do something because you never finished a migration, you have to check both paths for vulnerabilities and make sure they keep working.  Keep things as simple as you can for as long as you can and finish every migration you start.

Slack

The next resource CTOs should worry about is slack — how much free time your technical staff have.  If your answer is “none”, you’re accumulating technical debt in ways that you don’t understand and likely have a growing list of staff who are single points of failure.  If you have a serious, sustained emergency — and eventually you will — the recovery process is likely to burn people out.  When they quit, they’ll take irreplaceable system knowledge out the door.

Slack is linked to system mutability and the capacity to do security-focused work.  Every process of system development and maintenance will have unplanned work.  Developers will notice code that smells bad or come across weird behavior unrelated to their current work.  If they have time to go back after they finish that ticket, great.  If they don’t, these things add up and make the code base fragile and harder to change.  Other things that often end up in this category (but shouldn’t) include writing new tests, documenting behavior, and teaching other engineers about parts of the system they understand.  In a healthy development organization, this will be between ten and twenty percent of an engineer’s time.³  This kind of work is also key to the feelings of pride and ownership that prevent burnout.  If engineers can’t do this, and worse if they’re working overtime or not taking vacations, emergencies are more likely and will be harder to recover from.

Security is, unfortunately, a source of emergencies.  Making them less likely is the core of the work, but emergency patches to upstream vulnerabilities and incident response are unavoidable facts of life.  Part of building an effective security program is making sure the team has the capacity and resilience to absorb predictable emergencies without harm.  As an executive team, you need to understand that this overhead is necessary and resist the temptation to overload the engineering team with product work.  Between slack, meeting time, and regular security work, it’s unlikely that a development team can sustainably spend more than 80% of their working hours on product work, and even 70% is usually aspirational.

Slack is not a capacity issue.  Adding more engineers in the same culture with the same incentives to their line and project managers and the executive team does not mean more slack, it means more churn on the product.  Sometimes, especially pre-product/market fit, this does mean success for the company — that’s often where bad habits start.  Engineering quality and resilience are hard to measure and their impact on the bottom line is slow.  Even if you build a culture that values them across the engineering team, if that change doesn’t reach the rest of the company — especially product, sales, and marketing — it will be eroded.  Enforcing rules about slack at the CTO level can help, but nothing beats culture in the end.

Security-Related Work

Now that we’ve talked about the less-expected resources, let’s talk about what security work itself is.  There are five categories of scheduled security-related work:

1. Detection — finding and triaging potential security incidents⁴
2. Security-specific infrastructure work — building tools needed for other security work
3. Hardening, patching, and architecture changes — work on production or IT systems done for security purposes
4. Security testing and analysis — penetration tests, code review, etc.
5. Security governance — managing the overall security program and processed

Detection and Security-Specific Infrastructure

The first two categories have available time/money trade-offs that most small companies should take advantage of.  Especially as the speed of exploitation is increasing, you should have 24×7 coverage for detection if you have online infrastructure.  The smallest sustainable team for that is six people.  While you need the coverage, companies with less than a few hundred engineers do not need this much in-house capacity unless there are industry-specific extenuating circumstances.  Yes, these folks can do other security-related work on the side, but being good at detection and investigation is a different skill set than being good at developer enablement, compliance, building infrastructure, or any of the other relevant tasks.  Good generalists are more expensive, harder to find, and in more demand.  Most of them also don’t want to do shift work on detection.  It’s more effective to outsource that work to teams of people who do nothing else and are happy about that.  Detection also isn’t just a matter of having the bodies and a SIEM.  Building out intelligence channels, response and review automation, tracking, processes, etc. means you need senior talent, not just a pile of juniors, and can take a year or two of work across the whole team.

Many, many companies would now like to sell you AI-driven solutions that they claim eliminate the need for humans in the detection process, but in reality, they at best reduce it.  There are some kinds of anomaly detection that are difficult without flexible automation, especially when done in real time, and agentic or LLM-based systems can do more than conventional automation in some ways, but also cause false positives that are more plausible, are often no better at understanding the human context of actions, and also add additional failure modes.  A human in the loop in the detection process, especially a smart human with the experience to catch subtle things, remains indispensable.  Among other things, when you have orchestrated remediation actions, the human can keep your detection and response tool from itself being used as an attack vector.

As with detection, security-specific infrastructure is also better consumed as a service, especially at small companies.  A reputable SIEM vendor is going to be better at both running their service and securing the ecosystem around it than most small to medium companies will be.  They also exist outside of your internal trust boundaries — an attack that compromises your systems is unlikely to also compromise their systems.  This means that an attacker is less likely to be able to delay detection by suppressing alerts that might tip you off and you’re less likely to be flying blind during the incident response process as the attacker can’t delete logs.

Building additional pieces of critical infrastructure internally is expensive if you’re not already in the building lots of critical infrastructure business, which few startups are.  When you’re pricing SaaS tools versus in-house hosting, you need to get a realistic understanding of the work required to maintain the tool and price in the cost of the folks who will manage it, the people who will manage them, etc.  Talk to people who have seen the tools you’re looking at go wrong, not just the happy path deployments.  A single Elastic Search cluster that goes wrong enough at scale can burn out an entire team.⁵

If you do decide to build security infrastructure in house, it should be built by your regular infrastructure team, not by the security team.  Security infrastructure needs to work in the same way and follow the same rules as everything else in your ecosystem, including the deployment process, run books, documentation, and anything else common across all of your systems.  Security engineers are not infrastructure engineers, even though they may have a lot of common skill sets.  Security teams are incentivized to build infrastructure in a hurry and cut corners so they can get back to their real work, and can even be the worst offenders on things like patching.  For work scheduling reasons, you may want to take part of your infrastructure team and second them to the security team to get infrastructure built, but folks there should be rotated through so everyone gets to know the security systems, and that team should maintain the same standards and practices they have in their normal work.  If you have a mature engineering culture, none of this may matter, but this is rare in small companies.  If you have a sensitive system that needs to be compartmentalized away from the rest of the engineering team, it should still be built in this way but handed over to the security team once it’s in place (with sufficient documentation for them to maintain it independently).  If your security strategy depends on the rest of the engineering team not understanding the architecture of a security system, your strategy is wrong and you need to start over.

Hardening, Patching, and Architecture Changes

This category is core product work, the stuff that we talked about scheduling in the process section of part one.  When you start your security program, there will be a long list of this work.  It’s not uncommon to come out of a whole-organization initial security review with twenty or thirty epics of work that would take one or more years for the entire company to work through.  Now, obviously the team is not going to do nothing but security work for a year.  Much of this initial work will be general engineering work, getting the system into a state where changes can be made quickly, paying down technical debt, and getting things to a place where the system can be secured.  With that caveat, it’s likely that security-driven work will take between a third and a fifth of team bandwidth for the first two or three years of a security program.  In some cases, it has been more efficient, especially for infrastructure teams, to have one or more quarter-long security pushes where they get larger pieces of work done more.  However, in most cases the key is tight coordination with product planning so security work can be streamlined in with product work that touches the same systems.  The point at which the company can afford to spend time on engineering quality is also when a security program should be started — you can do it earlier, but you shouldn’t wait longer.  There’s a lot more to be said about the details of this work, but it’s more about the technical nature of the work than how to get it done, so that will wait for a later essay.

Architectural changes, at the level of the team doing the work, are just like any large development project, but security input to architecture merits a specific note.  Good security architecture decisions need someone who has seen a large variety of systems and who can work at the systems design level.  A surprising number of even staff and senior staff level engineers are, surprisingly, not systems thinkers as such — they’re just good developers.  Folks in this category are often decent at building systems that handle the type of requirements they’ve dealt with before, but less good at synthesizing architectures around new types of requirements.  Companies doing novel technical work with significant security exposure should hire in the architectural talent they need in-house, as this will be a significant part of the company’s core business.  For small teams building traditional systems where security is a primary driver of architectural change, bringing in a CISO who can also do architectural work can be a better balance, whether they’re full-time or fractional; more on this when we talk about governance below.  If you do not already have a process for how architectural decisions are made across the engineering organization, or the process you have doesn't provide a way for people outside the team doing the work to give feedback, it's time to change that.  Some variation of architecture decision records are often a good fit, with the caveat that there should be somewhere central (Slack is fine) where people post when they're starting the ADR process.  This ensures that folks from the security side can get involved early if they need to, rather than needing to be the fun police overruling a decision everyone else is already happy with.

One thing that companies sometimes do to handle the tension between the product roadmap and the security roadmap is to make the security team responsible for fixing security issues in production.  This is always a bad idea.  The team responsible for a system is the one who knows it, and having someone else come in and start changing that system is inefficient (they don’t know the system, so they’re slower), dangerous (the team that owns the system now doesn’t know some parts of it), and unreliable (work conflicts between the security team and the owning team can make the whole thing take longer than if the owning team just did the work).  Whoever owns the system needs to be the one fixing it.  A corollary to this is that every system must be owned by some team, and that team must have the bandwidth to fix that system, even if they’re doing development work on other systems.  Failing this means you cannot change your systems and thus cannot secure them.

Security Testing and Analysis

This category of work covers a few different areas — penetration testing, application security testing, security code review, and cryptographic analysis,.  Different aspects of this are handled differently, by a mix of tools, short-term consulting, long-term advisory support, and direct hires.  In small firms, most or all of this work will be outsourced or handled by tools.  As the company grows larger and matures, some of these functions will be brought in-house.  In some cases, compliance regimes or market requirements mean that work must be outsourced — PCI, ISO27001, and SOC2 audits are always external, for instance, but themselves also require external penetration tests.⁶  Even without requirements for external review, small companies don’t have enough of this work to make sense to do it in house.  It’s unusual to see folks focused on penetration testing on a security team smaller than a dozen people.

Application security review and security assistance to developers is the first set of testing work that should be brought in-house, especially for teams that write significant amounts of high-risk code.⁷  It’s often difficult to find application security engineers focused on developer enablement rather than just finding bugs and tossing them over the wall.  We have had luck work with taking security-interested developers, either who already work at the company or new hires, and teaching them the security testing part of the job.  The goal of this job is helping other engineers get better at finding and fixing bugs themselves and steering development practices toward safer patterns and libraries, along with the occasional code audit.  This engineer may end up writing some code — things like high-risk data validation libraries are a good example — and will fix some bugs as a way of demonstrating the pattern for the fix to others.  However, it’s critical for the sustainability of a security program that engineers do their own bug fixes.  Secure development training is useful, but secure development practice is much more instructive.

A single application security engineer can’t be everywhere, so tools fill in the gap.  Static and dynamic security testing tools will catch many bug classes, especially when combined with fuzzing.  Bringing SAST in early will pay dividends, as developers find bugs early rather than replicating them throughout a code base where fixes will take far longer.  AI-assisted fuzzing is also showing good results, even in complex code bases.  Despite the hype, other types of AI testing have a smaller improvement in practice — about twice as many true positives and half as many false positives is a good rule of thumb, but the human review process often takes longer than it did with conventional tools.⁸  LLM-based testing is also often no better at things like logic issues in business code that require understanding the human intent of the system.  External human review remains the gold standard for security-critical code.  Two things that may not be obvious to folks who haven’t both written code and done application security review professionally are that reviewing code for errors is harder than writing it and that being good at writing code doesn’t make you good at reviewing it.  For high-risk systems, it’s best if someone who does only code security review takes a swing at it.

Unless you have a staff cryptographer, meaning someone who has a doctorate in the subject, cryptographic reviews will always be outsourced to someone who does.  If you're just implementing an existing, well-documented cryptographic protocol, it should still must get external code review, but if you're implementing a novel cryptographic protocol or an existing cryptographic primitive, it needs both code and cryptographic review.  If you are implementing a novel cryptographic primitive in commercial code, you have without exception made a terrible decision and need to re-architect your system so this is not required.

Governance

Governance work covers a lot of different areas.  At the top level, this means determining the company’s threat landscape, setting company strategy around security, communicating these and the company’s current level of exposure to the rest of the executive team and the board, handling the risk acceptance process, and ensuring that security work is adequately resourced.  At the operational level, this includes everything from process building and policy writing, vendor security reviews, decisions on risk, and compliance tracking to managing external security audits, scheduling and tracking security-driven technical work, and communication about security with customers, regulators, and vulnerability reporters.  Some technical work is also effectively governance work — making decisions about security architecture and running incident response processes.  There’s also the job of working with the CTO to make sure security work gets done — helping to set up planning processes if they don’t exist, advising on team structure and talent gaps, and negotiating joint policies and processes like the SDLC and code review.

In a large company, while the executive and CTO liaison work stays with the CISO, the rest of this work (along with some of the previous categories) is split across a security department.  On small teams this is often all on the CISO.  However, some parts of this work can be split off at different points even in smaller teams.  If the engineering team has a project manager, they can handle the tracking of security work alongside the rest of the engineering work.  If there is a dedicated IT staffer, they can handle the first pass of vendor reviews and data gathering and pass the results off to the CISO for the final sign-off.

Compliance regimes can also be split off, but this requires care.  If a small company is going through ISO 27001 or SOC2, there are three roles that need to be filled.  The first is the named security officer responsible for executing the security program, as per the standards.  This will always be the CISO.  The second is the person who is gathering the compliance evidence to show to auditors.  This is time-consuming work that requires a good deal of process compliance, but not seniority or — as long as there’s an easy escalation path to the CISO — much technical security experience.  The third role combines, before certification, advisory work as to the requirements for certification, policy completeness, etc., and, immediately before and ongoing after certification, internal auditing prior to each external audit.  The evidence gathering and internal audit roles are frequently combined into a single compliance manager position, but this carries risks.

Compliance-first security can have a serious impact on product velocity, company culture, and staff morale, and usually doesn't result in secure technical outcomes.  It’s important that the CISO, not compliance, remains responsible for security strategy and architecture decisions that serve the technical and business needs of the company, including when to push back against narrow compliance mandates that are harmful or incoherent in production systems as they exist.  If compliance runs strategy and no one else on the team has a security background, it’s easy for a useless checklist-driven approach to look effective until there’s a major incident.  If there is a compliance manager, it’s best that they report up to the CISO, rather than being a peer.  This ensures that the goal is security and that compliance is a tool to get there, rather than the other way around.  Alternately, keep evidence gathering in-house, possibly run by the engineering project manager until that becomes too much workload, and outsource the compliance guidance and external audit function.

The broad nature of governance work presents a hard problem.  When a company has between about thirty and a hundred engineers, combining all the strands of work mentioned at the start of this section results in just enough work for one full-time role.  Unfortunately, most folks who have the background and seniority to handle that scope of work well and who want an in-house job want to be running a team.  Folks who don’t want to be running a team are usually missing some crucial skill areas — either they’re great engineers but don’t understand or want to do governance or policy work, or they understand governance but don’t have a technical understanding of how security works.⁹  There are unicorns here, but they tend to know their worth.  Just because someone who could command a seven figure salary as a CISO wants an individual contributor role within a smaller team doesn’t make them that much cheaper.

This dilemma is why Systems Structure Ltd. exists as a consultancy — we saw a lot of small teams that needed our depth of experience, but none of them had enough work to justify a full-time role, especially when there are easy options to shift the parts of the role that don’t need that level of experience away.  If a firm has fewer than a hundred engineers, they are generally better off with a long-term advisor filling the CISO role than bringing the role in-house.¹⁰  This is true even if they have security requirements that drive a higher degree of security complexity or assurance or more regulatory burdens, but the threshold may be lower in those cases — somewhere around the fifty to seventy engineer range.

Growth speed and regulatory burden both impact this.  If you have a forty engineer team but you’ve just closed a funding round and are planning to quadruple that team in the next eighteen months, hiring a permanent CISO is the right choice — although you may still want someone to get things moving and bridge the gap during the executive search process.  On the other hand, a stable company with over a hundred engineers that’s more of a “lifestyle business”, i.e. one that’s not trying to grow exponentially, may find long-term outsourcing still makes sense.  The same can apply at a large firm that does development or technical work for other companies and has a small and stable internal technical footprint.  Fintech, medical, or defense companies may want to bring governance roles in house sooner, and may need a CISO in house before they can even bring a product to market.  They still face the challenge of being able to hire someone with the right background into a small team, and they may find that even if they have an in-house role that’s in the CISO position on paper, long-term external guidance is crucial to growing that person into the role they’ll need to be in as the company grows.

Incident Response

We’ll close with a note on incident response, also known as the unscheduled security work that we excluded from our list at the start of this section. Like detection and penetration testing, incident response is best outsourced at small and medium companies.  Even with a large engineering team and a ten or twenty person security team where you’ve brought all your security detection back work in house, you’re unlikely to have significant digital forensics capacity.  You can respond to small incidents, but if the entire company is offline with ransomware, you need someone to call.  It pays to know who that will be and to have an agreement signed beforehand — this is the difference between an eight hour response time and a $500/hour rate and a 72 hour response time and a $2,000/hour rate.  Many companies that offer penetration testing service will also sign a master services agreement where, for a minimum annual spend on testing services, you get a fixed rate and response time for incidents with responders who (in theory) already know your systems.  You insurance provider may also bundle incident response with your coverage, and may require you to use one of their covered vendors if they're going to cover incident response work.  Make sure you’re comfortable with their vendor, rates, and process before your next insurance renewal.¹¹

Conclusion

Across this and the last essay, we’ve gone through all of the areas where we’ve seen major blockers for teams getting security work done.  We’re happy to answer questions from teams facing challenges around this sort of thing — you can book an hour of our time to chat here.  Our next pieces will cover security metrics, early choices you can make to make your life easier around security, things to look at in an initial review, and the secure use of coding agents.  There’s also a series of essays on different security design patterns in the works.  If you’ve got other ideas of things you’d like to see here, let us know.

Systems Structure Limited is a boutique security consultancy, providing fractional CISO services to startups.  Over the past nine years, we’ve built security programs for fifteen different companies, and our work has been responsible for hundreds of millions of dollars in capital raised by our clients.  If you’re reading this essay because you’re getting ready to start a security program at your startup, let’s talk!  If you're a startup considering starting a security program, you can book an hour of our time to talk through issues like these here.

---

Footnotes

1. There are choices you can make early on to make your life easier later, but that’s another essay.
2. Entirely out of scope for this essay, but AI is as often going to be harmful as helpful here — you cannot secure a system you do not understand.
3. More, likely much more for people in less-legible but often critical glue roles.
4. The incident response side, especially for large incidents, is less scheduled; we’ll talk about this later.
5. Yes, and no I don't want to talk about it.
6. But not, for some unclear reason, application security reviews.
7. Code written in memory-unsafe languages, cryptographic implementations, and anything safety critical, at least, should all get dedicated security review, either in-house or external.
8. Given the many hundred-fold increase in computational complexity, it’s reasonable to suggest that these tools are underperforming.
9. Both is also sadly an option.
10. In my biased opinion, obviously — but I’ve seen the inside of a lot of small engineering teams and seen all of the failure modes of in-house solo security teams repeatedly, both while coming in as a fractional CISO and before in other roles.
11. Incident response coverage is in most cases the primary reason it makes sense for startups to purchase cyber insurance.

How to Organize an Engineering Team So Security Work Happens: Part One

Decision-Marking and Process

Once you’ve decided that it’s time to start a security program and you’ve figured out what work you’d like to start with — either from your own expertise or an outside review — it’s time to figure out how to get the work done.  For some companies, this is easy.  If the technical founder or CTO is security aware, comfortable balancing security risk versus product and market needs, and the team is small enough that they can manage everyone’s work streams, that’s pretty much that.  We’ve done security reviews for a few clients in this sort of situation, and they’ve gotten great outcomes.  Most companies, though, run into some complications when they try to put a security program into practice.  In this essay and its sequel (coming soon), we’ll look at the key features an organization needs to successfully roll out and execute a security program.  In this piece, we’ll focus on executive decision-making and the project management process.  In the second part, we’ll look at what’s needed in the company’s technical ecosystem and engineering team structure, the different categories of security-related work, and how they’re best staffed and organized.  Throughout this, we’ll talk about the CISO as the security decision-maker; think of this as a placeholder for whoever is doing that work, as long as it’s someone who isn’t the CEO or CTO.  In part two we’ll also talk about the different shapes that role can take and what makes sense when.

Executive Decision-Making

The way the executive team makes decisions is the single largest make or break issue for security program success.  If it’s aligned with the fundamentals of how security works for organizations, things are easy, but if it’s not, nothing else that happens in the rest of the organization will save the security program.  It may still look good on paper, but if, or when, it’s put to the test for real, the technical outcomes will not be good.  And it’s not just security that can be impacted here — you may see an impact across the entire business from a security program that’s badly run at the executive level.  We can break what’s needed down into five key components:

Trust

The first and most important requirement is trust, both toward the other members of the team, including the CISO, and self-confidence on the part of the executive team.  Few CEOs and COOs arrive at the point of starting a security program already understanding the ins and outs of security, even if they have some technical background; even many CTOs don’t.  This means they’re going to be spending a lot of time making consequential decisions about subject matter that’s a little foreign to them.  Good explanations always help, but trust is the core of it.

One of the most important skills for an executive is being comfortable with not knowing things — and with other people knowing that they don’t know something.  This takes confidence, but it’s the starting point of all decision-making processes.  If you aren’t comfortable with what you don’t know, then working with experts to fix that gap is going to be hard.  During the process of starting a security program, you’ll learn a lot, but you’re still going to be both leaning on explanations from other people and making decisions with partial knowledge.  If you trust those people and you’re comfortable in this process, things can go smoothly.  On the other hand, if you don’t trust the people giving you technical and security advice, the process is all-but-guaranteed to fail.  Find a team that you trust.¹

One of the best ways to build trust is to treat the CISO as a full member of the executive team.  Many companies have the CISO report up through the CTO, and we see this as a mistake, as it keeps the CISO from understanding the full context of the business and from having a deep relationship with the rest of the executive team.  As the company grows, this becomes less important, but in small companies, the CTO is often product-focused.  To be clear, this is good!  It’s critical for a company that’s finding product-market fit.  As companies get larger, the role of the CTO moves toward shepherding the entire engineering organization, and it’s easier to take on a broader perspective.  In our experience, most firms under 200 technical staff are better having the CISO as a peer to the CTO.

Market and Threat Awareness

So, how much security is enough?  Every CEO or CTO has some version of this question, and the answer is never as clear as anyone would like.  A lot of things feed into this — what level of security do potential customers in your industry demand?  How regulated is your industry, and what do the regulators demand?  How targeted is the company’s line of business — are you just dealing with “being a company on the internet” risks, or does your sector have specific risks?  What is your tech stack like — have you made it easy on yourself, or are you in a position where getting to an appropriate level of security is going to take much more work?  Finally, who’s after you?  You may be just running a chain of fast food restaurants, but if you’re doing it in Kyiv today, you have a different risk profile than you do if you’re in London.

As a CEO or CTO, you should have some understanding of what your customers and regulators want, at least, but the rest of this may be new to you.  Getting to a shared highlevel understanding on market demands and security threats that everyone on the executive team is confident in is critical, as is understanding that new information can change the rest of your security program and impact products too.  We’ve had multiple clients who started out selling to mid-sized companies and had a real shock with their first few big enterprise clients.  If a shock like that happens to you, take advantage of it — the world has just delivered you some expensive information and the sooner you learn from it the better.  It’s not unreasonable for the executive team to want some additional information from sources other than their CISO — talking to their peers at other companies in the same industry, for instance, can make sense.  At the end of the day, though, every company is different.  There may be factors that put your firm in a different place than peers that seem similar to you.  There’s a fine line between information gathering and second guessing or trying to negotiate with reality, and this is where trust is critical.

How deep of an understanding of market demands, threats, and security strategy each member of the executive team will need depends on their exact portfolio.  In general, the CTO, CEO, COO, and General Counsel are most involved, but other folks should still have a general understanding.  Depending on the roles you have, more people may be on that list.  If you have a Chief Product Officer, for instance, they may need a deeper understanding than the CEO, even.  Talking through the structure of how work is split up should be an early part of framing the CISO role and their interactions with the rest of the executive team.

Strategy, Risk, and Trade-Offs

Once you understand what security looks like for your firm, the next step is figuring out a technical strategy for security and how that interacts with the technical and business strategy of the company as a whole.  This is where trade-offs need to be made, and the core of those trade-offs is the risk tolerance of the company.  If you’re in a heavily regulated or security conscious line of business, have solid product-market fit, and are on your way to delivering well without massive growth pressure, you may have a low tolerance for security risk.  Things are going your way and the only thing taking security risks is going to do is mess everything up.  On the other hand, if you have a high risk tolerance in other areas of the business, it may be the case that your security risk tolerance should match.

Security risk is complex, however, despite what many risk register vendors would like to tell you.  You’re not just taking a risk for the company as an entity, you’re also taking risks for your staff and your customers, personally.  Even the least risk-averse firm has some basic ethical (not to mention legal) obligations here.

Some companies also make trade-offs that come from the founders.  We’ve had clients with high bars for staff privacy, even with respect to automated tooling, who were willing to accept risks to the entire firm to maintain that privacy.  Similarly, we’ve had clients who valued independence from third-party vendors more strongly than SaaS-heavy industry norms, leading to technical trade-offs on security risk (not to mention product velocity).  It’s your firm, and you get to make those calls.  The critical issue is that everyone making relevant decisions understands the company’s risk tolerance and has a good rubric for the way the executive team expects decisions to be made.  If the folks doing the work don’t have a solid idea of what solutions will be acceptable, conflict is inevitable.

Most of the time, doing something for security means you can’t do something else with the same time or money.  Ideally, you don’t just have a long list of waiting security tickets, but a technical strategy for the way security outcomes will be achieved for the company that acts as a through-line across all the security work.  Explaining this strategy is on the CISO and CTO, and it’s worth talking through it until e.g. the CEO can recognize it when making relevant decisions.

For folks without a technical background the through-line of a security strategy can be hard to see, as in practice security can be kind of like trying to put your finger on all the holes in a colander.  That said, trying to negotiate with technical reality does not work.  There is a bar for security that comes just with being a company that exists on the internet, and that bar doesn’t change even you stare at it really hard.

If the CISO comes to the CEO and says they need two engineers for a quarter each for patching known vulnerabilities and system hardening, saying they can only have one doesn’t make the risks to the business from the other project go away.  If as a CEO you’ve expressed a risk tolerance for the company as a whole and the CISO is giving you a plan that achieves that risk tolerance, they’ve made the professional judgement call you’re asking them to make.  If there’s context that makes it impossible, they should have it — and then work with you to get to a plan that is possible.  The right time to handle this is up front, making sure that as the CISO is starting to develop the company’s security strategy, they have a clear understanding of what resources are available, what the constraints and go-to-market or product strategy is, and even what investor relationships are like.  It’s possible to hold all this information close to your chest, but then you can’t complain when you get a strategy based on what the CISO does know.  Reaching out to your board to help understand these risk trade-offs can be useful, and including the CISO in that conversation is helpful.  Remember also that as a CEO, you get to frame this conversation — if you choose an us versus them framing between security and product, you will get the result you asked for.

In reality, these trade-offs often aren’t zero sum.  A good CISO and CTO should look at how the security strategy interacts with the technical and business strategies of the company.  In some cases, the work needed for security can open up possibilities for the whole company’s technical strategy, and in the best case even create business opportunities that weren’t possible before.  This kind of strategic synergy² requires that the CEO, CTO, and CISO all understand the high-level technical and business strategy, including the pressures, market demands, and structural options that the business may be exploring or facing.  The business side of this is sometimes seen as a long way away from the CISO’s domain, especially in less technology-centric companies.  While the CISO doesn’t need to hear every detail of the sales team’s current challenges, they should have access to as much of that information as is practical.  A good CISO is a deep partner of the CTO and their work will impact the business as a whole.  As before, trust is key.

Commitment and Decisiveness

Some executive teams are good at making decisions and sticking to them, even to a fault.  Some aren’t.  A security program at a company that hasn’t had one before is almost without exception the largest set of work the company has undertaken that’s not driven by product-market fit, if we exclude non-security regulatory demands in highly-regulated industries.  It will impact the rest of your roadmap, and it will impact your ability to implement last-minute product changes.  This can be terrifying, and for some executives, it means making a kind of decision they haven’t faced before.  Making the wrong decision and sticking with it until you get information that demonstrates that you were wrong is often less bad than being indecisive.  Now, if the wrong decision was made, the company still has to weather the event that reveals this — in some cases, hindsight might show a much brighter alternative world.  When it happens, though, the only thing to do is re-plan and move forward.  Failing to make a decision, or worse, getting into a loop of starting, getting cold feet, doing something else, and then getting corrected back to the original plan can be far worse than just sticking to the wrong decision.  The cost of security work is hard to estimate ahead of time (more on this below), but stopping and starting or reprioritizing is a sure-fire way to make it more expensive.

Cultural Leadership

As goes the executive team, so goes the company.  If the executive team expects everyone else to comply with a security policy but lets themselves be the exception, everyone else will try to get out of it — don’t be that guy.  Starting a security program is often one of the first changes from being a startup to being a big-boy company, and one of the failure modes is when security end up being the fun police who no one wants to work with.  While there are some things that a security team can do to mitigate this, the executives are the ones who set the tone here.  If they make clear how things are going to change — and here’s a place where it’s useful that the whole executive team understands how security work will impact people’s work, especially on the IT side — and why it matters for the company, showing that the security team has their trust and approval, outcomes are better for the whole firm.  This is why making it clear that they’re following the same rules as everyone else is so critical.  It’s a big transition for the whole company, and it’s a lot easier if everyone is on the same side going through it.  IT and security can handle all the day to day communications, but they need the CEO to make it clear that the changes are coming from the top.

As an aside, there are two choices you can make early on as a founder that will make your life easier when it’s time to start taking security seriously, especially if you know the company will have a low risk-tolerance.  First, set norms that company devices and accounts are for work, and that while folks can do some personal stuff on them, they shouldn’t think about these as personal devices.  Partially, this is just a good idea overall — anyone who has ever dealt with the discovery process around a lawsuit against their employer is happy to keep everything in their personal life as far away from work devices as possible.  Primarily, it means it’s less of a shock to the company culture when devices start getting hardened or monitored more closely.  The other choice is to be very, very, very careful about never using the access you have to company systems to check up on employees except in cases where you’re trying to rule out malicious action³ — and even then, it’s best to be up front about it to the rest of the team, even if it’s after the fact.  Building a security program unavoidably means that the security team will have access to tools and data that could violate the privacy of employees.  If staff can’t trust that that access will be used with the utmost care and won’t be used for any purpose other than technical security outcomes, there will be much more conflict around a security program rollout.  If you’re worried about one of your employees slacking off or working their side-hustle on company time, do not use Workspace SuperAdmin access to catch them.  Go have the uncomfortable conversation instead.

Process

Once the executive team is onboard with the work, knows how to talk to each other and make hard decisions, understands the risks and threats, and is showing the company that they care about security, it’s time to get the rest of the company moving.  In small companies, the CTO says go and both developers get to work and the whole problem fits in everyone’s head — but teams this small are going to be hard-pressed to start anything that one would call a security program.  The team size where starting a security program makes sense is around the 10-20 engineer mark (more on this in part two), meaning there are at least two different streams of work happening, and maybe more like a dozen.  Security is often the first stakeholder needing to get engineering work done that’s not driven by the product roadmap or vision.  Individual engineers or teams may have had internal work that they’ve done alongside their product-driven work — tooling they needed, quality of life improvements, bug fixes, etc. — but this is driven bottom up; they’re the customer for their work.  Now that there’s a second top-down stakeholder, the way decisions are made needs to change.

Clear Planning Decisions

The process by which the team makes decisions about what work gets done does not matter as long as a) it’s not too onerous on the team and b) both the CISO and CTO understand what the decisions are, have a say in the decisions line with the priorities set by the executive team, know how the resulting work is going as it happens, and are both involved when the work gets off track or priorities need to be changed.  Above all else, how and when decisions are made and what decisions have been made needs to be clear to everyone involved.  This is more difficult than it sounds.

The first requirement in planning work is that there be a plan.  It’s not uncommon to see early teams where there is no plan beyond what folks will start working on next Monday.  Sometimes this is because design and development is happening just-in-time, in response to customer requests day by day.  Sometimes there’s a grand plan in someone’s head and there hasn’t been a need to write it all out.  Starting a security program is often a driver in moving from a sprint planning plus vague notes in a ticket backlog to at least limited quarter-scale planning.  Once a plan exists somewhere, there’s a place for the CISO and CTO to write down the decisions they make and a way for the rest of the team to understand those choices and the priority they have.  Security work is often the definition of important but not urgent (until it would have needed to have been done last week), and without a plan, urgent product work can keep cutting in line.

Once there’s some kind of plan (and this can be a whiteboard or a text file in source control), there needs to be a way that everyone agrees on for how decisions get made.  It’s also necessary that decisions are made⁴.  If the CTO, their two managers, and the two most senior engineers get together every Monday and make those decisions, it’s easy — make sure the CISO is there too, call the group back together if the plan changes on Wednesday, and you’re solid.

In some cases, especially in fast moving teams weighted toward senior- or staff-level engineers, the CTO may be steering by expressing some general intentions and then expecting their team to go make the actual decisions and do sensible things.  This is great, until someone else comes along and also expresses some general — or worse, specific — intentions.  If the team is used to executing without checking back in — say there are 25 engineers and the first manager starts next month and the CTO is out of bandwidth⁵ — this can result in surprises.  Even if the end result is good, the surprise itself causes problems.  An organization like this isn’t going to spin on a dime to epics and Gantt charts, but they don’t need to.  The CTO, with input from the CISO about their needs, needs to make it clear to the team what kinds of things they have the authority to make calls on, what they want to be notified about, and what decisions they need to be involved in.  They also need to provide a channel where the team can get decisions back in a timely manner, and make sure the CISO has visibility too.⁶

Whatever structure is used, it needs to apply to everyone.  If the CEO gets a critical request from the biggest client, it needs to go through the same process — even, and maybe especially — if there’s no way that won’t be the top priority.  Even if all the other plans get swept off the table (it happens, and it’s often the right call), everyone knows what happened and why.  If there’s a process, it needs to be the process.  If the process is too heavyweight to follow in a case like this, it’s the wrong process for the company at this stage.  See if there’s a lighter, faster version that still does what’s needed.

In larger development teams, we’ve seen issues where there’s a plan on paper, but it doesn’t relate much to what happens.  For instance, if a team has a big product project and a big security project and both of them are supposed to get done this quarter, but the product project has their quarterly bonus riding on it, the security work isn’t getting done.  You get the work you incentivize, and if you’re going to try to use incentives to drive team performance, they should map to the priorities you agree on.  This problem gets worse when it’s not clear to all stakeholders what work is even happening.  If you end up with a decision process that only has one cycle per quarter at the CTO/CISO level for prioritizing work and work isn’t done to plan… well, good luck.

Estimation

If you’ve got two pieces of work that are at an equal priority, it would be useful to know how long they’ll take when you’re deciding what to work on.  Unfortunately, guessing ahead of time how long it will take to get a nontrivial piece of software into a specific state is a fool’s errand⁷.  Instead of deciding ahead of time that an unknowable amount of work will take 29 and ¼ days, ask the team that owns the system in question what kind of a job they could do on a given feature in a given amount of time, and what the contingencies look like.  Work will expand to fill the time available, but it will sometimes also contract.  For instance, you might be able to replace the vulnerable pattern that you’ve been using for some set of operations with a safer library so new development can go forward in a better state and then pick five random legacy instances to fix in the time you have, but you can’t also fix all of the other places the old pattern is used.  Fine!  Now you have a better instinct for what doing the rest of it will look like, you’re no longer adding to the technical debt pile, and the next time this work comes back up the priority stack, you can get further.

Now, as a security person, this pains me — an incomplete migration does mean more tech debt because there are two ways of doing things, and parts of the system are still vulnerable.  However, this is better for me than either never getting that work scheduled because I can’t get exclusive use of that team for an entire quarter at once, or starting the work with an estimated two weeks scheduled and then hard blocking two other teams when they run a month over — meaning I’m definitely not getting their time back to finish the job later.

Early in the team’s security journey, time-boxing work so you can get some improvement in a bunch of different places while doing other work is often more useful than trying to get things perfect in one place.  There will always be more security work that could be done than can be done, even if you do nothing else.  That said, sometimes you will have specific fixed security requirements, the same as any other product requirement.  Most often this is either a direct compliance requirement, or a compliance requirement that your customers face.  Here, it’s important that you treat these as what they are — requirements for your product to find market fit.  Yes, your security team is going to be taking the lead on converting what you’re hearing from your customers into specifications, but it’s important to distinguish between work the security team selects to reduce the risk to the company and work the company does so they can sell to customers (even if it will also reduce risk to the company).

Interleaving Work

There is a cost to make a change to a given part of a system — any change at all.  An engineer may need to remember how that service gets deployed, relearn the structure of the code, or go familiarize themselves with a piece of the tech stack they haven’t touched before.  If you’re going to pay that price for one thing, you might as well get two things done at the same time.  If a system is getting replaced, adding some additional architectural changes beyond the one that drove the upgrade is often cheaper.  While you’re in there reworking how we do all the billing tracking, how about we also stop passing all the account data around in big JSON blobs so we don’t need to parse risky customer-supplied data everywhere.  This kind of interleaving of work is a big part of making security changes cheaper when you have legacy systems, and finding opportunities for it is key to a cost effective security program.  This is why the CISO doesn’t just need to know the plan for security work and how it’s going, but also everything else happening in engineering.  The whole team (in small organizations) should know the plan too, so they can help spot these chances.

End, Part One

In this part, we’ve gone over what you can do to make sure your new security program will be successful within the executive team and in terms of how you plan and decide on work.  In the next part, we’ll talk about the kind of resource limits you may run into when building out a security program, the nature of the five different kinds of security-related work, and how best to take care of each one.

Systems Structure Limited is a boutique security consultancy, providing fractional CISO services to startups.  Over the past nine years, we’ve built security programs for fifteen different companies, and our work has been responsible for hundreds of millions of dollars in capital raised by our clients.  If you’re reading this essay because you’re getting ready to start a security program at your startup, let’s talk!

---

Footnotes

1. Getting comfortable with not knowing things and self-confidence in general is outside of the scope of this essay, but therapy is a great start.  Just saying.
2. Ugh, I can’t believe I just typed that, but that is literally what we’re talking about here.
3. If you’re tempted to do this, talk to your lawyer first and tell them I sent you.  They’ll thank me.
4. You would think this was obvious, wouldn’t you?  But a lot of teams have issues here.
5. I have never met a CTO who didn’t have too much on their plate.
6. Even if the CTO does have bandwidth and there is a planning process, talking through with the team where the limits and expectations of agency are for them will pay dividends, especially in emergencies.
7. Many people will tell you differently, especially before it turns out they were wrong.  These people are rarely the ones doing the work