Social Engineering Defense in the Age of Voice Cloning

Oct 12

This essay will take a bit of a different turn, digging into a specific problem and looking at it both from the perspective of our corporate clients and as something that families and communities need to deal with — in part because most of the solutions are the same in both cases. More broadly, this is an interesting context to look at how these problems and solutions tie into security theory and how it gives us practical answers about what to do in the real world. This way of thinking underlies all of our work.

The Problem

LLMs have made it easy to copy someone's voice from even just a few minutes of sample audio and to make that voice say whatever you're saying, even adapting turns of phrase. This can be done live — say over the phone — and the same thing can be done with video. It's not perfect, but it's good enough in many cases for people to be convinced, especially when under stress and in the absence of any contradictory signals. Voice cloning has been noticed by both more serious organized exploitation organizations and, more lately, the ordinary scam groups that prey on private citizens. Both cases play out similarly.

The attack starts with the attacker finding a likely victim where they can get audio (and ideally video) of them speaking, and then doing enough background research to find out enough about their life to impersonate them at least briefly. If you're an executive who gives public talks the former is trivial, and it's not hard for many people with public social media presences either, especially somewhere like Instagram or Tiktok. As an attacker does surveillance work, figuring out the victim’s phone number, spouse’s or co-workers’ identities and phone numbers, etc., they can call people and get voice samples that way.

For a financial attack, the attacker needs to find two people who have a sufficient trust relationship to move money for each other — a couple, especially with a shared account, or a CEO and someone on their finance team who's likely to be able to send a wire transfer. They call one of the people (in a corporate context the person who's lower status) pretending to be the other and come up with a plausible story. This story will be something that conveys urgency because the attacker needs the transfer done immediately, before people cross-check anything. Ideally it’s something that’s not too outlandish so no further checking will be done. If the attack is done via a phone call, they'll use callerID spoofing so it looks like the call is coming from the real person's phone number.

This can all be done remotely. Depending on the country, a week of work that results in a single successful thousand dollar payoff may be enough of a return on investment to justify the work, risk, and expenses, even if there are also a dozen failed attempts that week. The barrier to entry for this kind of an attack has also gotten lower as LLMs have made automating open source intelligence easier, especially as data breaches makie illegitimate access to personal information ever easier. Before we talk about how to handle this in your organization (or your personal life), a brief detour into theory.

Theory

Familiar security problems have familiar solutions. Sometimes the familiar solution is easy but not as effective as it could be or causes other problems. Sometimes you get an unfamiliar security problem. In both cases, you may need to go back to the drawing board to figure out how to (better) solve the problem. We don’t start from zero though, because the drawing board has a bunch of templates — principles for how to design secure systems and generic forms of well-known problems. This is our toolkit for how to think about new ways to solve problems.

The generic name for the social engineering attack we’re discussing is the “confused deputy problem”. Some entity (the deputy) has the ability to conduct an action on behalf of others, and that action might aid an attacker in accomplishing their goals. The deputy has a list of actors or types of actors whom it is supposed to act for, but the attacker is able to confuse the deputy into taking action for them, without being on that list. Critically, the action is taken in the name of the deputy, not the attack; the deputy is using their own authority to take the action.

For example, you have a group of engineers who need to be able to update a file on a server. One of them builds a little service that listens on the network in the engineering lab at your office. To update the file, you send the service the new file and where you’d like to put it. This is accepted because the lab network isn’t connected to the Internet or the rest of the company’s networks. One day someone at a competitor gets a job at a cleaning service, comes in to clean the lab, and plugs their laptop in to the lab network. The service, not having any way to know who they are, obediently uploads the file. Now the attacker can overwrite critical files on the server.

This ability to take a wide variety of actions (or an action on any of a large set of objects) in a system is called "ambient authority" — the deputy is trusted and has a significant latitude in the system without downstream systems checking anything other than that the action comes through the deputy. Ambient authority is a common design pattern,because it’s easy and in many environments a default. There are several possible ways to mitigate the issue.

If the deputy doesn’t actually need all the authority it has, you use the principle of least privilege and constrain it to only do the things it needs to do. In our example, it might be that the file is the list of whose turn it is to bring in donuts on which week, and if the service can only overwrite that file — not because it internally limits which paths it will write files to, but because it actually doesn’t have permission to write anywhere else — then the problem goes away. Your competitors can engage in a denial of donuts attack, but that’s about it. In most cases this is only a partial solution — there was a reason the deputy had the broad privileges it had, so we need to look for other strategies.

If the service needs to be able to write to every file but there are rules about who’s supposed to be allowed to write to what, we need to figure out where the rules live. There are three options: inside the file (or rather, the filesystem) being written to, inside the service, or inside the request.

If the rules live in the filesystem, then the deputy only needs to identify the user and correctly pass the identity on to the filesystem. If our example service were intended to be on the company’s normal internal network the user could log in via SSO, and the filesystem would have permissions tied to SSO users. This keeps the deputy simpler, but moves complexity into the filesystem — it has to be linked to the SSO ecosystem, which adds central management of permissions and other overhead. The users could also log in with a local credential, which trades less system complexity for more local management work and the risk of using a real operating system credential for a one-off web app.

If the rules live inside the service, the deputy still needs to identify the user, but the identity used for this service doesn’t matter for anything else. If the rules are simple and the system is stand-alone, this is a winning combination. However, if instead of managing donuts for a few people in one lab this was one of tens of thousands of nodes in a global donut logistics network with millions of users, getting the access rules onto the system and making sure that they’re kept up to date becomes a challenge.

Putting the rules into the request would solve the update problem, but it sounds backwards — if the request is coming from the person who might be the attacker, what’s to stop them from writing a rule that lets them do whatever they want? Here, we use the magic of cryptography, where Alice, who owns the donut list, writes a statement saying that Bob can update the donuts list in such a way that the statement can only have come from her. The service can see from the file that Alice owns it, and because Bob can prove that he’s the Bob in the statement that Alice wrote, when he passes the statement in along with his maple bar order, the service knows it’s legitimate. If someone else got ahold of Bob’s statement but didn’t have Bob’s keys, they couldn’t prove that they were him. Now, when we scale this to tens of thousands of systems, only Alice and Bob’s keys need to be distributed to everyone, not every rule. This is a simplified description of a capability system. They can be harder to reason about than access control list systems, but they can work without any central system at all, and can express complex rules concisely, like “only the person who first created this message can edit it”.

Each of these three strategies does the same thing — it allows the service to still be able to do a wide variety of things, but provides an effective set of constraints so it only does the right thing in the right context. By thinking about this from a theory frame, we can go back to the practical problem and see which of these strategies are useful.

Systems and Tools, not Blame

It’s a good idea to approach design problems from the perspective that the people involved are trying to do a good job to the best of their abilities. Yes, sometimes people end up being in a job that they wouldn’t be able to do with any system or under any circumstances, but that’s a problem of hiring or training, not an issue inside the system — we’ll discuss malicious insiders later. It’s more common that the way their job and the systems they use to perform it were not well-designed for the abilities that they, as a competent human, have. For instance, we know that tired pilots are less safe because humans, when they get tired, are more likely to be distracted, will miss cues, and have slower reflexes. Instead of blaming the human for reacting in the way that their physiology dictates, we set limits on how long they can fly for and how much they must have slept beforehand. If this is true at work, it’s much more true with the people you love.

If we want a specific outcome from some work, we need to ensure that both the tools used in the work and the goals and incentive structure for the person doing the work line up. If the only tool we give the user is a hammer and their goal is to whack every cute little stuffed animal that pops up through a grid of holes the instant they see it, that’s what they’re going to try to do. If one in a thousand of those cute stuffed animals is a real mouse, the mouse is going to have a bad day. This is not the fault of the person with the hammer, it’s the fault of the person who let the mouse into the whack-a-mole box. Blaming the person with the hammer more is not going to make the mouse safer. If we can’t keep the mice out, then we do need to warn the person with the hammer, but we also need to change their job so they have time to look at each head that pops up and think about it for a few seconds, and maybe also install some better lighting. Otherwise, we will keep getting the same result.

We see versions of this blame game play out in various contexts. For instance, there is a scale of how much the system is responsible for in self-driving car automation. If a car is fully autonomous, the person in the car can ignore what’s going on. At one step lower, the vehicle might need human assistance, but it’s expected in all cases to be able to ensure that the vehicle will end up in a safe state if the human can’t take over. Functionally, these are the same level, — the human has no obligation to interact with the system until the request happens and there’s no set timeframe for when they need to be able to take over, so the car can never be in a situation where there is no safe automated option. The level below that is dangerous, though — it’s called conditional automation. Here, the car is responsible for all control and monitoring of the situation but the driver, who has nothing to do otherwise, must be able to take over from a situation the car sees no way out of or cannot understand on less than a second of notice. In theory, this is for things like “a construction worker on the road is making a hand gesture that the car doesn’t understand”, but in that context, the car could still slow down and stop until the driver has time to figure out what’s happening. In practice, it’s been used in cases where the car figures out that a collision is inevitable and hands control back to the driver, who is now to blame for the collision that happens a tenth of a second later.¹

If the job requires vigilance, vigilance needs to be a significant part of the job and the worker needs to be sufficiently engaged in the process of vigilance and in making meaningful decisions to be able to maintain both attention and situational awareness. A requirement for vigilance that is orthogonal to all other requirements placed on the worker is meaningless. If everything that your employee does requires them to click on emails, the details of the email that aren’t related to what they care about will not be salient to them and they will click on emails.

There are less extreme versions of this, too. There’s a common pattern where a job has a set of rules that must be followed in order to prevent accidents plus a set of mandatory performance metrics that the worker must achieve. As workloads shift, workers will change their practices to accommodate the workload required, reducing the safety margin in the process. It’s common to have situations where this is in practice both mandatory (or the worker will be fired) and forbidden (correct process must always be followed or the worker will be fired). Management may not even be aware that this is happening — work as done always diverges from work as imagined, and the current safety margin may not be clear even to the people who are doing the work, let alone supervisors. Because the system still seems to have spare capacity, workloads are increased further or resources are reduced. When an accident happens, the employee is responsible for not following the rules, even if it was never possible.

If we do not want failures, we must ensure that we are designing a system where they are not inevitable. This is harder in adversarial systems where the enemy gets to redesign some parts of the system without notice. Back to the practical side.

Reduce Ambient Authority

So from our theory, our first step is to reduce the ambient authority of the person receiving the cloned call if possible. This is only relevant in corporate environments — you're not going to reduce your life partner's access to their own bank account. In a business context, though, this can mean reducing the number of people with the ability to make transactions. This reduces the size of the attack surface and increases the approval throughput per person, meaning they can make vigilance a greater part of their job.

We can also add a second approver for payments, reducing their authority from “approve and make a payment” to “approve a payment”, and creating another role “inspect planned payments for fraud and then make them”. Now when the "CEO" calls someone in finance directly asking for money to be wired to an account, someone else in the department less-influenced by having been on the phone call and cued to be suspicious can look at the transaction before it's out the door. This is like our first strategy above — we’ve moved enforcement of the rules away from the deputy who may be confused.

While the company can’t get away with not making wire transfers or ACH payments, other high-risk payment mechanisms can be regulated, reducing authority that way. If the company can ban or limit purchases of gift cards (a common tool for small dollar frauds), requiring the recipient, authorizer, and reason to be put down in writing first, a request by phone to someone in finance to buy a thousand dollar gift card to use for a spontaneous give away at a conference will look much more suspicious. Unlike reducing access or adding an additional approver this can be implemented outside of work — just make a rule in the family that you’ll never ask each other to buy a gift card, and get your birthday present shopping for the cousins done earlier.

Our options for reducing ambient authority reduce the risks, but they don’t fix the underlying problem.

Identifying the Caller

There are two ways of identifying the person you're on the phone with if you don’t think you can believe your own ears. The easiest version is to switch to a trusted channel and ask for verification there. An attacker who hasn't already compromised the user's login account or laptop/phone can't respond via Slack or Signal. A policy (at work) or an agreement (at home) that any request for money or for elevated access permissions that comes via an untrusted channel will be verified by a trusted channel will stop most of these attacks. This approach has a similar dynamic to our last strategy above, the capability system. In that context we use the encrypted statement in the request to effectively create a trusted channel proving the request was allowed; here we do the same thing via an existing trusted channel.

Another example of channel verification, specific to phone calls, is to hang up the call and then call them back. Caller ID can be spoofed, but doing so doesn't let the attacker reroute incoming calls, only change what the recipient sees. A SIM swapping attack, where the attacker convinces the victim's carrier to issue a new SIM for a given number, will let the attacker take over a number, so this isn't perfect but it does raise the bar.² The more expensive you can make the attack, the less likely it is for it to make sense for the attacker. Attackers may also call from random numbers and then claim (as the person they're impersonating) that they've lost their phone, or it's been stolen — anything to get you to not try to cross-check their story. It's worth trying to cross check anyway, though.

Laptops and phones do get compromised and stolen, however, so just using a trusted channel that's always logged in on a device may not be enough — the attacker may already be using the trusted channel, or the victim may have actually lost their device and genuinely not have access to the trusted channel. The next step is to have a passphrase set up between people. When a request that would need the passphrase comes in, regardless of whether or not the request seems suspicious, the called party (the one being asked to act) asks the caller (the potential attacker) for the passphrase; if the caller doesn't have it then the request is not accepted. Again, attackers will try to talk their way around this, claiming they forgot it, etc. This is a deviation from the script they expect, though, so it's likely that you're going to get some hesitation or stumbling in their immediate response, that may still give you enough information to make a decision. You should use a different passphrase with each person. Like with passwords, a random set of four or five words makes a good passphrase. These passphrases should not be written down — these are intended to be quick to use in sensitive situations, and are less effective if both parties don’t have them memorized. When the attacker says they can’t remember, it needs to be implausible enough to throw major red flags. If they must be stored, they need to be in a password safe.

Challenge response pairs are even better than just a passphrase, because the called party doesn't make it clear what the intended response is. Just saying, in the middle of a call, "banana whistle widget foxtrot" doesn't tell the caller that that's a request for a passphrase, so you're more likely to get a "What are you talking about" than a smooth "oh crap, I forgot the response phrase". Challenge phrases can also be something that sounds normal and innocuous or can be woven into a conversation. Ideally, they're also things that won't come up in regular conversation. For instance, if both parties are vegan, telling the other party that you'll cook them a steak dinner when they get home is going to be obvious to the real person, but harder for an attacker. Even if they've seen in the process of their research that the person they're impersonating is vegan it's unlikely to be top of mind, and if it is, their response will be something that takes the suggestion literally in an attempt to stay in character, making the gap more clear. If a challenge phrase has been used in a context which turned out to be fraudulent, it should be changed immediately.

Proof of life questions are another version of this strategy. They’re used in contexts where someone is likely to be a kidnapping victim, such as for reporters or humanitarian aid staff entering conflict regions. They should be questions that only the person in question will be able to answer, but which they will also be able to reliably answer the same way under stress. Four to six questions are written down, enough to be used over the course of a protracted rescue operation, and then they're sealed and put in a safe until needed. These should be questions about things that happened to the person that no one else knows about and that they've never told anyone about — anything that could be a security question from a website is the wrong kind of question. Proof of life questions could be used in the place of a passphrase inside a couple, but they're more trouble to use. The obscurity and specificity that makes them useful also means the answers are not going to be remembered by the called party without checking. Keeping them written down somewhere easy to refer back to when out and about (and not in the safe) makes them vulnerable to compromise. They can also be quite intimate, making them less appropriate for use at work — although they are also used by executive protection teams in specific cases.

Rehearsals and Stress Innoculation

While work processes requiring secondary approvals are easier to train and enforce, passphrases are harder to remember and the process of asking for one is either going to feel awkward or be forgotten if they're not practiced. If you're going to institute a system like this, you need to rehearse it. A rehearsal like this can be fast, so making a point of doing it at least once a month won’t be a burden. At home, maybe bring it up on date nights when you're alone together; with your parents or other family, on calls if you don't see each other monthly. At work, if you have a 1:1 meeting one of you should bring it up at least once a month — make it into an in-joke between the two of you.

At work you might end up with passphrases with a number of people, some of whom — like people on the finance team — may be distant from your day to day job. This can be handled by policy — any direct requests for payments must go through the CFO or the head of accounting if they're not available (adjust as makes sense). This reduces the number of trusted people and increases familiarity, making it more likely something off will be noticed. When someone random on the accounts payable team receives a request, the answer is that they're not able to approve the transaction. If they see that the CFO and head of accounting are both online, that's also an immediate red flag.

As companies get large, this starts to break down — there are too many people with signing authority for transactions, too many people who can approve them, too many people with access to critical information who could be giving a talk somewhere they urgently need to get into their email account for, and too many IT staff with the ability to unlock accounts and reset passwords. Larger companies also have the capacity to add more process and work at a slower cadence, so narrowing down who can do tasks for whom is possible. Needing to set up passphrases with people you don't interact with regularly (unless they're e.g. a dedicated support engineer for the executive team) is a sign that you should be working on process, rather than trying to remember more than four or five passphrases at most.

Remembering the passphrases is the first step, but being comfortable using them under stress takes more practice, especially in work contexts where power imbalances exist. As part of an annual security training (or quarterly, for folks in high-risk finance positions), it's useful to run a social engineering exercise. This doesn't need to be unannounced or a test with consequences for failing — that just wastes time and makes people distrust the security team — but it should be high-fidelity, with the attacker role played by someone who's quick on their feet and willing to lean into the acting. Try making a voice clone for the test, and remember to make a separate passphrase for the exercise. This process is called stress inoculation – when you're in a high-stress situation, you fall back on your training and whatever feels familiar, even if you can follow a process when not stressed. If you rehearse under stress, you're more likely to be able to act as intended when the stress is real. It's standard practice for firefighters, paramedics, and other safety critical professions.

Managing Malicious Insiders

Social engineering is easier when you have help. In more serious corporate social engineering attacks, it's not uncommon for an insider to be either convinced or coerced into helping the attackers. This is usually not an issue inside families, except for the very rich, but equivalent situations can exist in e.g. domestic violence or child abuse situations. An insider can help the attacker by providing private information about the target and the person to be impersonated, such as contact information, nicknames, internal recordings of them talking, or their calendars and location. They can also provide information about internal policies and procedures. There's a delicate balance in handling insider threats — treating everyone with suspicion is poisonous to company culture and can make defection more likely as there's less of a sense of loyalty, only fear of getting caught. Ignoring the problem entirely works great, until it doesn't.³

As a first step, it's important to have good audit logging across internal systems — including otherwise innocuous things like who's looked at whose calendars. This will not help prevent the attack, but it can be useful in detection, either in the aftermath or if the attack is recognized as what it is and stopped. The security team should make it clear to everyone that this information will only be used to detect compromises or in cases where actual malice is suspected, and management is responsible for making this believable by never abusing it.

Organizations are fine with people knowing e.g. that everyone has a password to log into their email; they only care that the password itself doesn't leak. However, the existence of the passphrase system itself should not be known to anyone not involved in the process or documented in standard security policy. If it's impossible to not have a documented policy that details the passphrase process for compliance or regulatory reasons the policy must also be secret; it can be shown to auditors if required, but a copy should not be provided. If a malicious insider knows about the passphrase, the attacker will be able to be smoother in blowing off the request and may even preempt it with their excuse, making it more likely that a stressed victim will acquiesce. This same approach is sometimes taken with other high-value internal secrets, like root keys, where the exact protections used are also secret.

While the policy only exists in a company context, it's best to hold the existence of passphrases closely outside of work too. Relatives, especially if there's tension in the family, don't need to know that you and your spouse or child have passphrases with each other. With parents, siblings can probably share a passphrase if they're all close, but that's not always the case. In some cases, it may make sense for you and your partner to have separate passphrases with your child, if they're old enough to remember more than one. If you're estranged from your partner, they do not need to know about these kinds of precautions.

Lowering the Reporting Barrier

Eventually, an attack will succeed even with the best preparations. How bad it is depends on what happens immediately after someone realizes that something is wrong. In a well-run organization (or a happy family), folks should feel safe reporting or talking about something that has gone wrong. As soon as someone is afraid of reporting a mistake, it's less likely that the consequences can be prevented. We see this with most phishing training.

A user is trying to do their job, and their job includes clicking on emails. They click on an email, and it turns out to be a trap, and now they have to waste an hour or more of their day on a training that they've already taken — which doesn't give them any tools to improve — because their job is still clicking on emails and not reading email headers. The message they learn from this is that interacting with security will waste their time and make their day worse. If the company has a policy that staff who commit security violations (unintentionally) will be disciplined the effect is magnified. If someone has messed up and knows that if they report it they will be punished and doesn't know for sure that it will be noticed if they don't say anything, they will not say anything. This is the vigilance and performance trap we described earlier. The first step toward fixing this is a policy that ensures that staff will not be disciplined or fired for a mistake, even it leads to a breach⁴, but that also makes it clear that failing to report a mistake that might have security consequences once they’re aware of it will be treated as malice and may get them fired. This reverses the incentives.

Beyond this, we want to train and reinforce reporting. Yes, people can and should learn more about how to identify forged emails, voice cloned calls, and all other kinds of social engineering attacks. However, they should learn that when they're engaged and in a position to learn, not randomly when they're trying to do other things — unless their job is only about detecting those things happening. Sending someone an email clearly from the security team that asks them to report it as phishing and has no chance of leading to punishment is neither bad nor disruptive. If there’s a reward for some number of people who successfully report the email, it can feel positive even if there isn't also a feeling of intrinsic satisfaction in helping to keep the company safe. For folks in high risk positions who are expected to be social engineering targets, you should consider doing the same thing with phone calls and other contact channels, once they've been taught the mechanism for how to report things. For emails, the disruption is small enough that doing it monthly isn't a big deal, and even for phone calls, once a quarter isn't unreasonable.

For calls and directed social engineering you need more than just a quick place where someone can click to report it. You still need that part, so the initial alert gets out fast and response can start immediately while it may still be possible to cancel pending payments or contain other kinds of breaches. It should be followed with a longer form that prompts them to write down as much information as they can about the attack while it’s still fresh in their memory. You should have a playbook for each type of action that could be the result of a social engineering attack, and test them at least annually as part of your incident response drills; possibly more often, especially if the company is changing finance platforms or IT management tools. This part goes for individuals too — making sure (especially with e.g. older parents) that they know exactly how to look up payments that have been made and how to cancel or challenge each type.

At home, the blame and shame that we talked about earlier can be more ingrained — even if people do trust each other, shame may make them hide what’s happened. This can be especially true for kids who may not yet fully understand what’s happening, teenagers who are trying to establish their own identity separate from that of their parents, or (especially older) parents who are ashamed that they’ve let their family down, don’t feel like they understand technology, or don’t understand how an attack like this could have happened. In all of these cases, the best thing you can do is to talk about this beforehand. If everyone understands the risks, knows (again) that they won’t be blamed, and has a collective plan for how to handle something like this if it does happen, it’s less likely to go terribly wrong.

Conclusion

Hopefully now you’ve got a couple more ideas in your toolkit for dealing with social engineering attacks, a better idea of how the threats are changing, and have a bit more of an idea about how one goes from problem to solution via theory. Our next pieces will cover security metrics, early choices you can make to make your life easier around security, things to look at in an initial review, and the secure use of coding agents. There’s also a series of essays on different security design patterns in the works. If you’ve got other ideas of things you’d like to see here, let us know.

Systems Structure Limited is a boutique security consultancy, providing fractional CISO services to startups. Over the past nine years, we’ve built security programs for fifteen different companies, and our work has been responsible for hundreds of millions of dollars in capital raised by our clients. If you’re reading this essay because you’re getting ready to start a security program at your startup, let’s talk! If you're a startup considering starting a security program, you can book an hour of our time to talk through issues like these here.

Footnotes

This is called a “moral crumple zone”
For a social engineering attack where someone claims to be calling from your bank, the bank will most of the time have a mechanism for you to figure out if a call is genuine, sometimes via an identifier that you the person gives you that comes up in your bank app and sometimes just by calling in to customer service yourself. Banks are surprisingly bad at getting this stuff right, though.
That said, many small companies whose line of work doesn’t put them at industrial espionage can mostly ignore insider threats for a long time, especially if in tight-knit organizations with good wages and reasonable demands on employee time.
Consistent failures are a performance problem — staff might be fired for poor performance if they have a consistent pattern of security mistakes, but they’re being fired for the pattern, not any single mistake.

Eleanor Saitta