Sonia Ben Ouagrham-Gormley on Barriers to Bioweapons

Sonia Ben Ouagrham-Gormley is an associate professor at George Mason University and the deputy director of their biodefense program. Sonia has written extensively on the proliferation and non-proliferation of bioweapons, being one of the key voices to have emphasized the challenges that organizations, tacit knowledge, and other factors have caused for states and terrorists that have attempted to acquire weapons of mass destruction.

In this episode we talk about:

• Misconceptions around the ease of bioweapon production — how and why bioweapons programs face unique challenges compared to nuclear weapons programs
• The crucial role of tacit knowledge in bioweapons production
• Will AI make bioweapons much easier to develop, or will human expertise remain a major bottleneck?
• Case studies of bioweapons programs, illustrating the practical difficulties and failures encountered even by well-resourced state actors.
• (How) has Biological Weapons Convention prevented bioweapon proliferation?
• Do global political trends point towards proliferation, even without AI?

Resources

• Barriers to Bioweapons: The Challenges of Expertise and Organization for Weapons Development by Sonia Ben Ouagrham-Gormley
• The Anthrax Diaries: an Anthropology of Biological Warfare by Slava Paperno, Kathleen Vogel, Sonia Ben Ouagrham-Gormley, and others
• Phantom Menace or Looming Danger?: A New Framework for Assessing Bioweapons Threats by Kathleen Vogel

Transcript

Note that this transcript is machine-generated, by a model which is typically accurate but sometimes hallucinates entire sentences. Please check with the original audio before using this transcript to quote our guest.

Luca

Hi, you’re listening to Hear This Idea. In this episode, I speak to Sonia Ben Ouagrham-Gormley, who is an associate professor at George Mason University and the deputy director of their biodefense program. Sonia has written extensively on the proliferation and non-proliferation of bioweapons, being one of the key voices to have emphasized the challenges that organizations, tacit knowledge, and other factors have caused for states and terrorists that have attempted to acquire weapons of mass destruction. Sonia argues policymakers still often underestimate just how difficult the steps to building a bioweapon are, and that should really matter a lot for people who are looking to also design effective strategies that aim to prevent their spread. Given the recent attention that the intersection between AI and bioterrorism has gotten, and the prospect that AI could lower the barriers to bioweapons, I was really curious to hear what Sonia thinks about all the recent discussion. So in this interview, we talk about where the belief that bioweapons are easy to make came from and why it’s been difficult to change, why transferring tacit knowledge is so hard, and in particular, the challenges that rogue actors face. And lastly, it’s mentioned what Sonia makes of the AI bio discourse and what types of advances in technology would cause her concern. I found Sonia’s work and perspectives very informative. I especially appreciated hearing her get into the nitty gritty details of what the existing bottlenecks in her domain are. And using that as a way to reason about what it would take for a technology to be transformative in it. That evidently requires a lot of expertise, but it just strikes me as a really fruitful approach to reason about AI and other trends more generally. Often trends that involve fuzzy concepts that are hard to pin down and go at loss then. So I’m excited about this approach, not just in bio, but in other areas too. And without further ado, here’s the episode.

Sonia Ben Ouagrham-Gormley

So these days, I deal mostly with the effect or the impact of emerging technologies on security, and particularly biosecurity, and a lot of it also affecting biosecurity issues in certain regions of the world, including Russia and China.

Luca

So… Maybe to frame our conversation, I think we’ll be discussing, especially in the first half of the interview, a lot about your book, Barriers to Bioweapons. And in that book, I think that you really critique the view that making bioweapons is very easy, that all you need, I think, to quote your book, biomaterials, scientific knowledge, and equipment. Maybe to start things off, where do you think that view came from? Or why do you think that view was so persistent, such that you writing this book, I think, really added to the discourse?

Sonia Ben Ouagrham-Gormley

I think there are several sources. I can think of three major reasons. The first one is the fact that we think about bioweapons and bioweapons nonproliferation by using a framework of analysis that was primarily developed for nuclear weapons. So nuclear weapons cannot be produced unless you can have access to the fissile material. So highly enriched uranium and plutonium-239. And to do that, since these materials do not exist in nature or are not enough to make a bomb. Countries or terrorist groups would need to get the technology to enrich uranium or reprocess plutonium. And that technology is very expensive. It’s very specific to weapons development. And it creates a clear signature. And there are barriers to acquiring that technology. So for nuclear weapons, if you don’t have access to that, to that technology, and it’s because it’s specific and because there are barriers, it’s very difficult to access the technology, then it becomes very difficult to produce a weapon. And so this framework of looking at proliferation from the material point of view has been adapted to biological weapons. And that doesn’t work because biological weapons. are not conditioned by the access to material and technology because that technology is dual use. So the idea is, so because technology related to biodevelopment is dual use, it means that it is easily accessible, commercially accessible. And therefore, from there, we… …reached the conclusion that bioweapons are easy to produce. Because anyone can have access to the technology, therefore, those weapons are easy to produce. So I think that’s the first mistake. And that… Using the nuclear framework of analysis implies that we don’t pay attention or don’t pay sufficient attention to understanding what is the barrier in bioweapons development.

Why is tacit knowledge so hard to transfer?

So if it’s not technology access, then what is it? And that I found out myself by conducting a research project. That was an oral history of the U.S. and Soviet bioweapons program. So I interviewed with a colleague of mine, a former U.S. and former Soviet bioweapons scientist. And when I say scientist, I mean technicians, scientists, biologists, people who are specialized in different areas of bioweapons development. So production. testing, weaponization, etc., as well as managers and administrative personnel in order to have a broad idea of what is it that conditions bioweapons development. And so through these interviews, I found out that the key barrier to bioweapons development is access to knowledge. Because… Civilian knowledge in biology does not translate easily to weapons development. And to produce a bioweapon, countries have to, or scientists have to develop a bioweapon-specific expertise. That’s not the same as the expertise they had if they are, you know, very well-known virologists or bacteriologists. In the civilian world, that expertise does not immediately apply to bioweapons development. There are specific expertise or there is specific expertise that needs to be developed. We saw that in case studies related to the Soviet program and the U.S. program. That, for example, in the case of the Soviet program, there was a new facility created in the 70s called Vector, which is one of the core. facilities of the Soviet bioweapons program. And when it was created, it was created bringing in people from the local university of Novosibirsk. And these were people who were very, you know, at the top of their expertise in virology, in smallpox. So they knew, you know, they were very competent experts in their field, but they had never worked on bioweapons development. And that facility, and it took them about five years to develop that bioweapons expertise. So for five years, the first five years of the life of the facility, the personnel just did experiments to learn how to work with certain agents in a certain biosafety environment, how to learn certain techniques and certain processes that are used only in the bioweapons field and that do not exist in the civilian field, in regular laboratory work. And so it took them five years with people who were really at the top of their game. The director of that facility was a very famous smallpox expert. And they were able to do it within five years because they were trained by bioweapon scientists from a different Russian facility. So it’s unknown if they didn’t have access to those bioweapons experts, it would have taken them much longer. And in the American program, for example, when I interviewed American bioweapon scientists, former bioweapon scientists, they told me that it took them about two decades before they felt that they knew enough about the agents they were working on to really use them as bioweapons. And the U.S. program started from scratch with no prior bioweapons expertise. So you see the, that’s, you know, and the U.S. program also had access to lots of people from universities, from MIT to Harvard to Cornell, I mean, high-level scientists. And yet, according to them, it took them about two decades to really develop that expertise, that bioweapons-specific expertise.

How could AI change biorisk?

Luca

Yeah, maybe just to recap or to kind of stand it back to make sure I’m understanding this. It sounds like a lot of the initial policy framing is in terms of like nuclear weapons, whether it’s this big emphasis on material access. For biology, that doesn’t seem to really hold because, as you mentioned, many things are dual use. There’s many good reasons to have access to biomaterials such that they can’t be regulated or kind of limited in access in the same way. But still, we have this puzzle that despite there being perhaps easier material access, if we like. Look at the world, there aren’t that many bioweapon programs. And as you mentioned, even when the US tried and had a lot of resources, there was a lot of difficulty in getting this. So there must still be some barrier here. And you point to this idea of knowledge. And this transferability, especially from the, I guess, more consumer or civilian use case to the weapons and terrorist or misuse case as being a really difficult step. I’m curious if that is something specific about biology that makes that transferability of skills really hard in contrast to, say, nuclear energy to nuclear weapons. Or if it is just the case that there are many hurdles, and whilst with bio we might overcome the initial hurdle of material access, we still falter at one of the key ingredients that you need. Or if there’s something very particular about biology here that makes this transferability and this need of very minutiae detail very important or very essential.

Sonia Ben Ouagrham-Gormley

It’s not specific to bioweapons. So there is one thing that is specific to bioweapons, but the issue of acquiring the weapons-specific expertise is also a problem in the nuclear field. There was research previously in the mid-90s that also based on interviews of U.S. nuclear weapons scientists who indicated that… It takes about five to 10 years to transform a physicist, civilian physicist into a nuclear weapons physicist. It’s not this idea of, or this problem of acquiring weapons specific expertise is not specific to bio. What is specific to bio and that you can’t find in nuclear or the chemical weapons field is that the, uh, Raw material that is used in bioweapons is living microorganisms.

Why is tacit knowledge especially important in biology?

And there are a variety, all kinds of viruses, all kinds of bacteria, toxins. And each of them requires different types of expertise. While in the nuclear field, there are two materials that you need to produce in a nuclear weapon. And in the nuclear field, you have this issue of acquiring or developing the weapon-specific expertise, but there’s the problem of access to material, which is complicated. So you have two barriers. In the bioweapons field, you have one barrier, which is just the acquisition of knowledge, but that… This acquisition of bioweapons expertise is made complicated by the fact that the material is really, really unpredictable. And it’s not one expertise, it’s multiple expertise that’s needed.

Luca

Yeah, and I think this maybe nicely leads us onto this other concept that you really emphasize in your book around tacit knowledge. So because, you know, Living organisms are so unpredictable and you have this like really complicated mapping of needing all of these very specific inputs, like getting those right in order to like get the output right. That like often when it’s just like humans doing this, we don’t really know like what is going on or can find it really hard to express what you actually need to do right in order to get, you know, a certain cell to work in the way that you wanted to. This is a great blog post by Erica D. Benedictus. who says that like one of these terms in like biology is scientists talking about like what works in their hands because even you know just changing the person doing this changes all kinds of factors about like how hard you are crushing the cells or how you’re angling your pipette and stuff and that all of this just creates this like real complicatedness. Yeah can you flesh out a bit more about what you think kind of counts under this like term of tacit knowledge? And how sometimes maybe as well it gets misperceived with what you’re concretely pointing to here, of what makes it difficult to acquire this expertise.

Sonia Ben Ouagrham-Gormley

So tacit knowledge is the kind of knowledge you develop through experimentation, first-hand experience and experimentation in the laboratory. And that’s when you… Conduct a process which might be very well defined, clearly explained in a book, for example, or something that is taught in schools, but is really difficult to master because it relies on visual and sensorial cues that allow you to determine whether you’re doing the right thing. So, you know, for example, when scientists talk about cells, sometimes they say they look really happy. What does it mean to say that a cell looks happy? So these are things that you learn through experience and experimentation. And these are things that are difficult to verbalize. You can’t explain what is a happy sale, right? And so tacit knowledge is formed of knowledge that one cannot be verbalized or cannot be easily verbalized. And because it can’t be verbalized, very often… The person who owns that knowledge doesn’t even know that they have it. They don’t know that they are doing things in a certain way that allows the experiment to proceed successfully or a task to be successful. So that’s one thing about tacit knowledge. But as far as bioagents are concerned and work in… Biology generally, not for bioweapons only, but in the regular university laboratories, there are also all kinds of variables that cannot be measured. So, for example, the humidity level, the pH of the water, the ambient air circulation, etc. All these can affect bioagents because bioagents are very sensitive to the environmental conditions as well as to how they are manipulated. And that’s where the tacit knowledge becomes important. And so this is tacit knowledge is really all of the expertise, all of the knowledge, the know-how, the skills that you develop through experience and through firsthand experimentation.

Luca

And then to clarify, a lot of these… inputs or like kind of requirements specific to a certain biological pathogen as well so for example if i’m like very good at like doing uh or let’s say like synthesizing horsepox for example does that also generalize to like anthrax or is it because i need to hold my um pipette differently or i need to you know have a different humidity or a different like whatever thing that there’s a lot of variance amongst living organisms despite them all being living you

Sonia Ben Ouagrham-Gormley

Right. So there are great distinctions in behavior between viruses and bacteria and toxins. And even within one family, you know, within the virus family, a smallpox doesn’t behave the same way as, let’s say, COVID-19. Right. So you have to have that specific expertise related to the agent itself. And so knowledge with anthrax does not transfer to knowledge with smallpox or another virus or even another bacteria. If you work with anthrax, your expertise does not help you much when working with plague. So that’s unique. The expertise is specific to the agent and does not translate to another agent. And that’s the complication with bioweapons development because, and we’ve seen it, you know, with state bioweapons development and terrorist bioweapons development or attempt at developing bioweapons. When they fail with one agent and move on to another agent, hoping that the next agent will be easier to deal with. In fact, they’re creating more problems to themselves because they’re starting from scratch. And they need to acquire that new expertise related to the new agent they select. And that’s why terrorist groups have been usually completely… unsuccessful in bioweapons development, but other state programs. There’s really only two states that have succeeded in developing bioweapons. And even then, the success was rather limited. And these are the US and Soviet Union. Yeah. The others have been unsuccessful or didn’t go beyond the research and development phase.

Luca

Yeah, I really want to get into these case studies deeper in just a minute. But maybe just to kind of go back to the very initial question I asked of why is it that there is this like belief amongst many policymakers, especially kind of in the post 9-11 period, that making bioweapons is like so easy. I’m still, yeah, kind of like interested in like, why that environment existed, or like why that like narrative existed. And why it is maybe also important from a policy perspective to correct this belief.

Why do people think building bioweapons is easy? Why does it matter?

Sonia Ben Ouagrham-Gormley

Yeah. So one is we’re using the framework developed for nuclear weapons that is based on material. So nuclear weapons are conditioned by the access to the proper material and technology. That’s not the case with bioweapons. So applying that model to bioweapons doesn’t work.

Luca

Is that then just because like nuclear weapons, like… Why would it be the case that we would generalize so heavily from this model to this other domain?

Sonia Ben Ouagrham-Gormley

That leads me to my other reason, and that we don’t have an ongoing bioweapons program. So in the nuclear field, we still have a nuclear weapons program. There’s still one in the US, and the five countries still have nuclear weapons. Even if they don’t design new weapons or produce new weapons, those programs are still in existence. And so there are experts who can say, no, that’s not possible, right? Or this is possible. And this, you know, to clarify and correct our understanding of what makes nuclear weapons development difficult. In the bioweapons field, there’s no authority. We don’t have a bioweapons program in the United States. It was ended in 1969. In Britain, it was ended in the 50s. France, same thing. The most recent is the Soviet bioweapons program, which was ended in the early 90s. So it’s still several decades already. So we don’t have today an authority who can say. What you’re talking about is feasible or not feasible in bioweapons development. So that’s one thing. The lack of expertise, the lack of authority that can arbiter the debate that we’re having on bioweapons does not exist. So everything is possible. And the other problem is that since the 80s… There has been this narrative that has been repeated by government officials, by the intelligence community, including in the United States, that says that bioweapons are easy to produce. You just need to acquire the three main ingredients, which is material, technology, and information, scientific information. And if you have that, then you can produce a bioweapon. And this narrative has been developed already. We find statements from the American intelligence community of the mid-80s. And this has been repeated over and over and over again by government officials on different sides, not just Republicans, but Democrats as well, the intelligence community, experts or biosecurity experts. So this has been repeated so often that today it’s believed that’s truth. Everybody believes that bioweapons are easy to produce. And it’s very hard. to correct the records, even when the information comes from former bioweapons scientists themselves.

Luca

Yeah, that’s a really interesting point that you made there as well, that maybe because there aren’t many bioweapons programs, we kind of like lose a sense of like just how hard like making this thing is in a way that’s like maybe different to nuclear weapons. It does make me wonder, and maybe this is like too particular a question. But how the policy mood is like maybe different in Russia or in like former Soviet countries, where you maybe have like more of that like recent experience to like draw from about like how complicated or like fine grained a lot of these issues can be.

Sonia Ben Ouagrham-Gormley

It’s hard to say. Yes and no. So when I was interviewing some former. Soviet bioweapon scientists and from Russia, from Uzbekistan, from Kazakhstan. I remember one interview with a Russian scientist who… I asked explicitly, what do you think? Do you think that, and that was soon after the anthrax letters. So, and I asked him, so what do you think? Do you think a terrorist group would be able to produce an anthrax based bioweapon? And his first reaction was, of course, it’s easy. You know, you can, an anthrax is so much easier to deal with than viruses, etc. And then I said, well, can you walk me through the process, you know, from A to Z, from acquiring anthrax, processing it, scaling it up, you know, and then weaponizing it and then disseminating it?

Could you go through the different stages of that weapons development and tell me how a terrorist would go about doing this? And so he went through those stages in detail. And at each stage, he would say, oh, yeah, but they would have to know this, or they would have to know that. And they would have to get that type of technology. And they would have to have this variety of expertise, which one person couldn’t have. So they would need to create a team, etc. And so by the end of the conversation, he concluded, well, I guess it’s not that simple. And I had the same exercise with former US bioweapons scientists. And it was the same first reaction. Of course, it’s easy. And then when you ask them to go through the process in detail, they realize that’s not that simple. That’s much more complicated than we think.

Luca

I think this point is a really good and important one. um yeah often when like push comes to shove you realize just how difficult things are and that’s also just from the vantage point of like you as an expert right i don’t know the exact like qualifications that they had but they might realize like oh yeah there’s one particular thing that i know that actually ends up being really difficult and then still neglecting that like all the things that they don’t know a ton about probably also ends up being that complicated um challenging um yeah and maybe you want to conclude this like um introduction section again with like asking um what some of the like policy relevance like implications like here are you So I can draw like maybe like two distinct reasons already, like why we might care about this question of like how hard bioweapons are. One is around like prioritizing. You know, I’m sure that like governments have like limited resources with like how they could combat terrorism or like rogue state actors. And you want to have a good calibrated sense on where the biggest risks are likely to come from. Also within the life sciences themselves, I’m aware that because everything is like dual use, there are like real tensions between say confining or like limiting scientific knowledge for the purposes of like safety versus like accelerating progress, which also has like a lot of upside. And that’s all I think like within this like prioritization bucket. But then there’s also this like other question, which is like, you really need to understand like what the… bio threat is and what the hard steps are in order to best combat them and towards the end of your book you give this like really interesting example about when it came to disarming the russian bio weapons program and limiting some of this like brain drain that this could actually have some really important implications and i was wondering if you could like yeah maybe talk listeners through uh yeah like what your what your reasoning there was yeah i mean i i will uh just reiterate what you just said that mischaracterizing the threat implies.

Sonia Ben Ouagrham-Gormley

that uh we develop poesies that not that are um that do not deal with a real threat, and we ignore the existing threat. So after 9-11, the U.S. has developed a biodefense program that cost over $60 million, and that was geared primarily to deal with the terrorist threat. Well, we know today that terrorists cannot, or at least… So a lot of money was wasted in an effort, wasted in areas that diverted our attention from the real problems. And in the biofield, natural outbreaks are probably much more likely than a threat from bioterrorism. Another… Danger of mischaracterizing the threat, and that’s relevant today, is this issue of disinformation and misinformation. Russia has been conducting a campaign of disinformation about U.S. bioactivities in the former Soviet Union for a long time. But more recently, they’ve sharply increased the disinformation messages that they’ve been… Spreading and basically the idea is that the U.S. is what they try to make people believe is that the U.S. is using its support to other countries in the bio. Research and biosurveillance field as a cover for bioweapons development. And they spread stories that make no sense at all and that contradict each other. But they really don’t care because they know that people are going to believe. Whatever story is out there, because there’s no understanding of what it takes to produce a bioweapon. So they have stories about super soldiers that are being produced in Ukraine to fight in Russia or during the war with Russia. They are talking about developing the U.S. and Ukraine developing a genetic weapon against the Russians that would target only the Russians. And even Russian scientists came out and said, that’s not possible. There’s no gene that makes you a Russian. So that’s another problem, you know, being a victim of disinformation by other countries. And in this case, it’s in Russia. The other challenge is that if we don’t understand how bioweapons are… or were produced when we deal with or try to develop nonproliferation policies, as was the case after the break of the Soviet Union, where the policy was primarily to fund scientists so that they stay in their former bioweapons facility. So we know where they are. That way they don’t go to Iran or North Korea and help them produce bioweapons. So support projects at the former bioweapons facilities and often support projects that are related to the agents that were in the past used in bioweapons development. And we found that when we studied through the oral history project that I did with one of my colleagues, when we studied the different type of expertise and different types of knowledge that were created in different facilities, we found that this approach was perfectly fine for some facilities. But for other facilities, that became a problem because, in fact, it was helping the scientists maintain the bioweapon-specific expertise that they had developed in Soviet times. And that’s here briefly that there are some, depending on how people work in different facilities, some of them develop personal knowledge. that is unique to the person. And some of them produce mostly communal knowledge, that is knowledge that is created by a team of scientists working together. And that is not owned by anyone, but owned by the whole team. So when it comes to personal knowledge, if you support someone to stay in their facility, to do work, even related to agents they used to work with in the past. What happens is that they may be maintaining their personal knowledge, but not the communal knowledge. Well, if you try to maintain teams in the same facility where the main knowledge is communal, Then you’re in fact maintaining that communal knowledge about bioweapons development. And that is more threatening than personal knowledge, because we know that to produce a bioweapon, you need a variety of expertise. So if you maintain the teams together in the same place and support work related to bioagent that they used to work with in the past, Then they maintain that expertise that can be threatening if they decide to go as a team in another country, or if Russia decided to recreate their bioweapons program. So they have still that expertise available.

Luca

Yeah, I find that such an interesting and initially counterintuitive point as well. Yeah, when the Soviet Union collapsed and people were rightfully worried about how some of this knowledge might proliferate, there’s such an instinct just to keep people where they are so that they don’t spread the knowledge. But in keeping entire teams there, you’re keeping that tacit knowledge alive than you would if you were to more systematically try to break people up. Because as you noted, things are so specialized and you require… A lot of different people without like very specific synergy or something like being in the same room or continually working in the same lab.

Sonia Ben Ouagrham-Gormley

And I have to say that was also my instinct in the early 2000s. Well, you know, we need to keep these people where they are in order to prevent them from proliferating. But it’s only after studying really specific expertise, specific facilities and learning how they work that you understand the distinctions. And when it becomes important to have a policy that’s much more flexible and not just a cookie cutter type of policy that is applied the same way to everyone and to every facility.

Luca

Let’s maybe turn to how tacit knowledge, especially in the life sciences, actually spreads and what kind of proliferation then to worry about. Maybe one framing here is you’ve emphasized before that the life sciences are like very distinct in that they’re dual use. You know, there’s also civilian and a lot of like commercial applications here. And that kind of makes me think as well that there must be a lot of like interest and also like built up know-how to know how to like actually spread this tacit knowledge. If I’m a pharmaceutical company or if I’m a biotech startup, then I like want to make sure that like, you know, knowledge gets transferred and like find effective like ways to make that happen. All of science is also like based on this idea that people can build on top of each other’s knowledge. So given, you know, these big interests actually find effective flows of tacit knowledge, why is it that in the like bioweapons, like angle specifically, this thing is hard? And how much harder is it than in, say, like your average commercial or, you know, university scientific civilian use case?

Sonia Ben Ouagrham-Gormley

Tacit knowledge transfers the same way, whether you work on bioweapons or in the civilian field, you know, tacit knowledge is a… area of study in economics, and it’s been a long area of studies for a long time. When I was a student and I did my degree in economics, tacit knowledge was something we studied because every time an individual with specific expertise left a company without training the person who was going to replace them, that created all kinds of problems that can be very costly. So Industry businesses have understood this importance of not just letting people go, but making sure that the new people are trained with those who have those tacit skills. And in the bioweapons field or in science in general, tacit knowledge is acquired through experimentation, right? Personal experimentation. And it is transferred when people work together side by side. And it can be in a model of, you know, master-disciple type of relationship where… The lead scientist takes under his wing or her wing a postdoc and work with them in the laboratory. And they may not know exactly what is being transferred, but by mimicking or by working side by side and doing things the same way. That expertise is transferred, sometimes without anyone knowing that some kind of important knowledge was transferred. Sometimes it can be visual. So an individual will look at someone else carrying out a task in a certain way, and slowly they mimic them and they try consciously. to do the same thing. It doesn’t happen or it doesn’t succeed often right away because seeing something and repeating it, doing it are two different things. So they have to, as you were talking earlier about crushing cells, you have to make sure you know that you need to crush the cells, but how strongly, you know, how much pressure you impose or you use. How do you hold your pipette and things like that have an impact on how knowledge is transferred.

Luca

Yeah.

Sonia Ben Ouagrham-Gormley

But basically in the laboratory, knowledge is transferred this way through direct interactions between people, but also long-term interaction, not just a couple of days or a week together. And there’s a lot of research in the field of science and technology that studies that, that looks at people working together in the lab. And they show that it takes about a year, several months before someone can acquire an expertise or tacit skills that cannot be verbalized.

Luca

Yeah, that’s interesting. I’m curious, like how that specifically gets measured. Like one thing I’m imagining here is like how long it takes for a postdoc to be able to do a certain experiment independently and like how long these things take. And I guess also how much variation there is here, even within the life sciences or something, of, oh, this particular step takes X days or Y weeks or something to acquire. Being able to do the whole experiment independently, or maybe experiment is the wrong word here, but synthesizing this specific pathogen from step A to step C, that takes, say, how long?

Sonia Ben Ouagrham-Gormley

It’s hard to say. There’s no, I mean, if we could, if we could really identify or measure how long it takes to learn specific expertise and then repeat it, that would be, I mean, a big deal because I’ve seen that and I’ve tried. to work on that, to identify how long it takes to lose that expertise. And it really depends on the environment. So I’ll give you an example. There was a friend of mine, Kathleen Vogel, wrote a paper about the synthesis of the poliovirus that took place in the early 2000, 2002, I believe. And she found that a lot of the, I mean, Much of the experiment hinged upon a specific task which aimed to create the cell extract that would be used to grow the virus. And to produce the cell extract, they used a Down’s homogenizer, which was used to crush the cells. And they found that… that task, which seems very simple, and again, it’s taught in schools. There’s no secrets about it. The secret is how you do it. And in the laboratory, there were people who could do it better than others. Some people had even found that if they bought a Down’s homogenizer from a commercial provider, and they were able to do it better. That equipment doesn’t work very well, so they had their own equipment made, custom made, to fit their own purposes. Hmm. But what’s interesting is that there was a case of a postdoc who was sent to the laboratory in New York from Belgium to explicitly learn that technique of crushing the cells in order to be able to produce a cell extract that could be used later to grow the virus. He spent about a year at the laboratory in New York. And after a year, he was able to produce a cell extract that could be used to grow the virus. But when he went back to his laboratory in Belgium, he was not able to reproduce his work. And simply because there are other variables that could not be measured. And… They think that perhaps the water, the pH of the water was different. So that had an impact on producing the cell extract. Even in the laboratory in New York, they found that the bovine serum that they use to produce the cell extract can have different characteristics depending on the season and depending on the supplier. And that affects their own work. So they decided to always ensure sameness in everything. So always use the same supplier. Make sure they get the bovine serum during the right season and not in the wrong season. Because all this… Even in their laboratory where they know how to make the cell extract, sometimes they failed because of those variables that cannot be measured and will affect their work anyway.

Luca

Maybe one thing to ask you is like how expensive or like costly and not just in like dollar terms, but also in time terms, like is a failure? If like, because one thing I could imagine is like, look, every time I like try to crush these cells, it’s going to be like essentially semi random. Like if it works or if it doesn’t work and the odds are like stacked against my favor. So should I just like try and do like a thousand batches and then like hope that like one of them works? Or if I like fail at like one step. Does that mean that I need to go all the way back to the beginning? And because, you know, organisms like take time to like grow and like stuff like these things actually like very costly. Like, yeah. How should I be thinking in terms of how difficult are like setback at a given steppers?

Sonia Ben Ouagrham-Gormley

Yeah. So that’s an interesting question because that introduces another issue in science that we always forget. And that’s the issue of trust. So scientists have to trust that something is possible. in order to achieve the right results, I mean, the positive results. And that means that there needs to be a demonstration first. So in the case of the cell extract, that postdoc from Belgium knew that it was possible to produce a good cell extract because he was trained by people who knew how to do it, even though sometimes they failed and they had to start all over again, and can be weeks of work that’s wasted that way. It can be months. It really depends on what’s the process we’re dealing with. If that postdoc had dealt with someone who was not as successful, or if he had been trained by someone who didn’t demonstrate that it was possible, he may not have persevered enough to eventually succeed himself. And that reminds me of a story about… An experiment that was described in a publication about… the development of laser technology. And in order, there’s a process in laser technology where scientists have to use sapphire and have to measure the quality of sapphire. And they do that by putting, hanging sapphire at the end of a thread and that thread is greased with animal grease. And that’s very simple. Again, nothing very complicated about the whole process. But the whole story is that… During the Cold War, the Soviets always achieved measurements that were much higher than the measurements obtained by European and American scientists, to the point where Western scientists thought that the Soviets were just lying, making up results. And it’s only after the breakup of the Soviet Union that a former Soviet scientist came to a laboratory in England. And showed his colleagues, British colleagues, how they did it in Soviet times. And the trick was, instead of using animal grease, they used human skin oil. So they would take that piece of thread and run it behind their ears or under their nose to gather skin oil. And of course… Sometimes the skin oil, I mean, someone has dry skin, someone else had too oily skin. They really had to find the perfect. It’s funny.

Luca

It’s a funny job description. Yeah.

Sonia Ben Ouagrham-Gormley

And so he showed that. And while he was in Great Britain in the British laboratory, he showed that they, and he obtained measurements that were as high as those obtained in the Soviet Union. So British scientists were suddenly. So it was demonstrated to them that those measurements, as high as they were, were actually possible. So they experimented with this technique for about a year. And after a year, they were able to themselves achieve those high results that they didn’t believe were possible during the Cold War. So the point that I’m making here is that in order to… There are all kinds of problems associated, particularly with the bioweapons field, because of the external variables that cannot always be measured. But if you have a demonstration, if a scientist knows that a certain result is achievable because they saw someone else achieving those results, Then they will try and try and try and try until they get it. And sometimes, again, scientists will tell you, they try and try and try and try. They think they are doing exactly the same things all the time. It fails, it fails, it fails until it succeeds. And they don’t know why they are suddenly successful. Well, they’ve been doing the same thing over and over again.

Luca

On an emotional level, I do kind of like struggle with it, where because it seems that things are just like really random, like both in terms of like for a given pathogen, like how hard is it to do? For a given person, how hard is it to acquire the tacit knowledge? Given a certain professor, how good are they at transmuting that knowledge? It’s hard to rule out, like, no, it is like impossible that… A person with X months and Y dollars will be able to make pathogen Z, which when you’re then worried about like terrorism or like biological misuse is like really scary. But it does just like fundamentally seem to be an argument about like likelihoods or something that you need like many things to go right. So you’re not just like rolling one dice, you’re rolling like thousands. And you need to like really be worried that like they all come up like snake eyes or something. But then like given, you know, like specific risks or something, that might just like still be like really worrying. But the point is, is that there’s just like many, many steps here, many more than like people often think.

Why could bioweapons be especially hard for rogue actors?

Sonia Ben Ouagrham-Gormley

Yeah. And I think we need to have a more nuanced analysis of what’s possible and for whom. If you’re thinking about terrorist groups who have no prior expertise, they start with no knowledge at all. So to get to the point where they can manipulate agents, bioagents, and make them do something specific, it will take them, the learning curve will be so steep and will take them so much time that it might be just impossible for them. But if you’re taking scientists who have had a career in the civilian, you know, biology, the learning curve will be shorter in the sense that they have the basics of biology, right? Now they have to acquire the bioweapons expertise. It may take them years too, but that it will be. Less complicated, they start at a level that’s much higher than someone with no expertise in biology at all. And if you’re taking now former bioweapon scientists who have stopped, let’s say, former Soviet bioweapon scientists, who haven’t worked on bioweapons for the past 30 years, but are asked to produce a bioweapon today. Their expertise is still there and it will take time to come back, you know, especially those tacit skills. But it might be easier for them, right? So the assessment should take into account the starting point, what kind of knowledge exists at the start. And then you have to think about the local culture. And that’s what we found interesting also in this oral study project. That to be successful on a regular basis, in spite of all the variables that cannot be controlled. In laboratories, scientists developed a certain local culture that is specific to that environment, to their laboratory. And you can find evidence of that also in biology, civilian biology, where there are some tasks that are complicated. You can see that the local culture is very specific and that allows to be more successful. Then another laboratory may have experts as well, but doesn’t have also that local culture.

Luca

The first point you raised there, I think, really stuck with me from your book as well. There’s this concept of leapfrogging, that you can maybe skip certain technologies or certain steps if they’re handed down to you. So to take an everyday example, many countries developing today don’t need to have landlines because mobile phones work, so they can leapfrog that particular step. But I think your book really emphasizes that there are limits to how mobile phones work. Ben Ouagrham-Gormley, Barriers to Bioweapons, Hear This Idea. Yeah, and then turning to this second point about local cultures, this is maybe a good transition here to then talking about some of these unsuccessful attempts by both state and non-state actors to acquire biological weapons, or I should say failing to do so, where it seems that in many cases when you talk about Iraq and South Africa and Aum Shinrikyo, it seems that the local cultures in these cases were extraordinarily bad, in many cases because there’s a really bad self-selection effect between people who want to do something illegal. and then how well they’re able to execute on this, given certain constraints. Yeah, this is kind of, I guess, just teeing you up here, but do you want to expand on that point?

Sonia Ben Ouagrham-Gormley

Yeah, and that, I think, allows to introduce other variables that are never taken into account when we make assessments about bioweapons development. And that’s the issue of organization and management, which are important in any weapons program. All right. And in any scientific and technological endeavor, it doesn’t have to be weapons related. Management of a team, of a program and the organization of a program affect how work proceeds and how successful it can be. And that’s even more important in the bioweapons field because the key barrier to bioweapons development is the acquisition of knowledge. And we know that if you don’t have the right organization to allow knowledge, one, to allow the creation of knowledge. then the transfer of that knowledge to the right people within a team or program and the effective use or efficient use of that knowledge, then the program does not succeed. And that’s even more complicated for countries or groups that are trying to develop weapons in a covert environment. Because a covert environment… The priority for a covert program is to make sure that the program is not detected. And in order to prevent detection, they have to adopt an organization and a management model that contradicts completely the needs for knowledge creation, transfer, and use. And that implies compartmentalization and fragmentation of a program. creating barriers between people through either security clearances or by limiting what different people know about what they are doing or the purpose of what they’re doing. And by creating those fragmentation, that creates fragmentation in terms of knowledge transfer. So one person at the beginning of the process may be creating important knowledge. But because of the organization that prevents free communication between people, that knowledge may not transfer to the right person.

Luca

Yeah, yeah.

Sonia Ben Ouagrham-Gormley

And so, and if there is expertise somewhere else that could help them solve a problem somewhere else in the program that could help solve a problem at a specific stage, they, you know, scientists A wouldn’t know who to ask, you know, within the program because they are not, they don’t communicate. And they are, sometimes they don’t know who else is involved in the program. So that, that’s, um… In effect, it’s a good thing, right, for programs that they create barriers to knowledge transfer. And that affects all programs, whether they’re chemical, nuclear, or bio, but even more bio because the key constraint is knowledge development.

Sonia Ben Ouagrham-Gormley

I want to maybe like draw out what feels to me like a distinct factor. So there are like challenges that have to do with just the environment being covert. So to take the example of like on Shinrikyo, they like, you know, had to hide being detected by like the Japanese police. And that creates a lot of problems because you have to like be in secret and compartmentalize and all the things you say. But then there also seems to be this like other self-selection thing, which is that like most of the actors to date who have tried to acquire biological weapons because they are like massive taboo. Even on like a state like level, they seem to be, you know, either corrupt actors or like easily corrupt actors, autocratic dictators, or like cults. And like those in terms of like, you know, local productive, like truth seeking cultures don’t seem to be like the best fit here. And that like worries me somewhat as well. It feels like really hard to like break down how much of it has to do with like just general covertness versus this like self selection for like bad kind of local cultures. But to the degree that a lot of it has to do with, like, specific ineptitudes of, like, certain people, if, like,
only, you know, Iraq had, like, a better, like, manager or something. Then I’m, like, really worried of, like, yeah, like, what if, like, the next bad actor just ends up being, like, really good at, like, management rather than it being innate problem of creating bioweapons, which is this, like, you know, you have to be covert. And even if you’re the best manager in the world, this is going to be, like, a really difficult obstacle. But it doesn’t seem to me at the moment, at least from having read your case studies, you will know this much better, that the people, like, making these weapons were the best scientific, like, managers out there. And they seemed in many cases to be, like… Very bad. And like, kind of not in a, you know, to actually pull it off. But they had like other incentives such as money or prestige or just like people pleasing.

Luca

Right. But also, I mean, in all of the countries that tried and terrorist groups that tried to develop bioweapons, none of them had sufficient expertise, right? From the beginning, they start with nothing, right? And in the case of Aung San Suu Kyi, for example, the people who were involved in their bioweapons development had absolutely no expertise in biology. The head of the program, I think, had some expertise in an area of biology, but certainly not applicable, or that would have allowed them to really understand the agents that they had selected for work, which were anthrax and botulinum toxin. So in most cases, even state programs like the Iraqi program. They may have had some expertise, but that expertise was not adapted to what they were trying to do. So in the Iraqi program, for example, there was an expert in cancer, another expert in… some kind of area of virology, I believe, but non-adapted to the agents they had selected. So they also selected anthrax and botulinum toxin bacteria. So one, they don’t have knowledge or they don’t have expertise that’s adapted to what they selected to work on, which means that they have no knowledge from beginning. So here you already have the learning curve, which is long. But the other challenge of covertness is that… And it would be the same with people who are experts, who are competent scientists. Let’s not say experts, but competent scientists. Yeah. Covertness implies that you appoint people in certain positions that you trust. So people who are not going to defect or who are not going to be easily turned by a foreign. intelligence service. So people who are loyal to the manager or the person in charge. And loyalty sometimes doesn’t go hand in hand with competence in a certain field. And you can see that if you look at the Iraqi bioweapons program, a nuclear weapons program, originally, when they started, they had people who were competent, probably not with knowledge applicable to bioweapons or nuclear, but they were competent in their field. And those people were replaced by people who were loyal to the government, simply because there was a need to ensure that covertness. And loyalty, again, doesn’t mean that they have the expertise. And we saw that also in the South African bioweapons program. At some point, there was… A dentist who was head of one of the facilities of the South African bioweapons program, who had no expertise in biology, in bioweapons, but he was loyal to the head of the program. And that’s why he was appointed there. So I think that’s the challenge with covertness that we really don’t appreciate enough, that loyalty… is more important than competence. And that creates challenges to weapons development.

Sonia Ben Ouagrham-Gormley

Just an off-the-cuff thought, but there seems to be another channel that you can add here as well, which is if the loyal person being employed knows that they’re not competent enough to run this thing. How that then also in turn changes their incentives of how they want to be stepping things up and things. It maybe links again to this idea of what you were saying that scientists also need to really believe at the core that they are able to pull this off in order to be willing and to put the grit or manage people to put the grit in to succeed here.

Luca

Or they might increase secrecy within the program to cover themselves, to cover their failures. And that also works against making any progress.

Sonia Ben Ouagrham-Gormley

So, yeah, one thought, and this kind of leads us into the second half of the conversation I want to be having, which is this kind of question of looking ahead. Correct me if I’m wrong here, but I think all of the examples here, so Iraq, South Africa, like Aum, Shonrikyo, are from the 20th century. And I’m sure that you must hear this question a ton, but… It is a really important question, which is to ask, how much has changed? Like there have been such breakthroughs, especially in like synthetic biology, but also in like a ton of other domains. And now there’s like a lot of discourse around artificial intelligence and how that might like intersect with like bioweapon programs or bioterrorism. Yeah, how much do you think that these things fundamentally lower the barriers? That like even if they like mostly act on like material constraints rather than on like tacit knowledge and… We can get to that like ladder scenario as well. Even if like the main effect here is like to just like lower the like material entry to barrier. Maybe there are like ways in which this like does affect tacit knowledge, such as you no longer need like as big a team because you can automate some steps. Or just because like more people have access to this now, the likelihood that you find somebody who can make it work like increase. I guess like there’s a ton to get into here. But maybe like the headline question is like, how much more worried are like you now than you would have been like 20 years ago?

Luca

Well, you know, this issue of new technologies and how that might facilitate or speed up a process is very interesting because… One, I think the mistake that’s always made is that to assume that the technology will work by itself, right? So, you know, a PCR machine will work by itself. Another kind of machine or AI will work by itself. The tool is a tool and it can help facilitate some tasks or speed up some process, but that does not eliminate the need of the involvement of the individuals, of people. Because even with the PCR machine, which is used extensively everywhere, some former student of mine wrote her dissertation about new technologies and how they affect the use or the intervention of individuals? Do they work by themselves? And she found that even with a PCR machine, there are only really one step that is automated, and it’s the temperature part. And at the end of the process, humans are still involved to solve problems that are created by the machine. And there were other research projects that showed that the challenge with new technologies is that… They create new problems. They may facilitate one aspect of a process or automate a process, but they create new problems. And in order to solve those new problems, the individual using the new technology needs to, one, have sufficient expertise to understand that there is a problem. Where is the problem? At what stage of the process is it happening and finding ways to solve that problem? Someone who doesn’t have that expertise or prior knowledge would not be able to understand whether the failure is due to the fact that they did something wrong or is it because the technology has created a new challenge. And even the kits that are used to facilitate some work or some tasks in biology, they are used extensively. And if you look at sites where scientists communicate with each other to ask questions, etc., you’ll see that all the kits… Create new problems. And in order to solve those problems, they ask their colleagues and their colleagues say, OK, I did this. This is how I solved that problem. And that scientist will try to do the same, but it doesn’t work for him or for her. And they have to come up with their own solution that is appropriate in their own environment. And so my biggest… My critic, when I hear that new technologies are going to facilitate or speed up the development of bioweapons, my biggest critic is that a technology does not work on its own. We need to study the technology in the environment where it is going to be used, which means that we need to understand the technology as it affects the user. And how the technology was developed by the developer. Because we found also, and again, there are all kinds of research on that, that the technology is in fact a result of the assumptions and the tacit knowledge of the developer himself. And often a technology is… AI is the same thing. If you don’t know anything about the tool, you don’t know whether the tool is giving you solutions that are feasible. And in the bioweapons field, you know if something is feasible if you produce it, if you try to produce it. And that’s a key challenge that we don’t understand. Even in former bioweapons program, scientists themselves faced new challenges when they would tell you that the biggest problems start when you start making the stuff. Because there are unexpected problems that… Yeah. And so they have to go back to the drawing board and start the process all over again.

Sonia Ben Ouagrham-Gormley

A lot of this feels like it has this fundamental echo of the tacit knowledge discussion that we had before. In your book, you mentioned that a very common mistake of postgrads is to always use the newest equipment. But in many senses, that’s actually a mistake because it means that the tacit knowledge that they acquired from their professors who didn’t work with that new equipment no longer applies. Similarly, now incorporating new tools such as AI or such as other things. might again mean that like things actually end up breaking down and you don’t know exactly what it is and like how to fix it. I will say that it sounds like a lot of this is like more an argument for like the transformative effect that such technologies might have just ends up being slower to like kind of diffuse out or like for people to like learn that it’s not the case that like AI gets developed today and then you know a year later or something people suddenly know how to make bioweapons but that there’s a lot of tinkering and like experimentation that needs to happen. But it doesn’t comfort me in the sense that it tells me that, like, no, this, like, new equipment, you know, there’s no way that people can, like, find out to, like, actually use it in order to, like, get through these barriers or in order to acquire TASA knowledge or in order to acquire knowledge that was, like, previously TASA. Does that feel kind of right to you or would you, like, reject, like, more strongly? And I’d be really keen to hear if you do. But then it is more that, like, oh, these things just take time and maybe in 10 years I’ll be really worried. But, like, right now, I don’t think it’s going to, like, change overnight.

Luca

Yeah, I mean, again, the technology does not work in a vacuum. It works in a specific environment. And so I can’t say for sure, you know, AI will never be used to produce bioweapons. I can’t say that. But what I would say is that if we’re worried about AI, we need to understand the environment in which it is used. And… When we say used, what do we mean, right? I think last year there was a group of scientists from North Carolina State University, I believe, who also have a private company that designs, using AI, they design new molecules for the pharmaceutical industry, so to produce new drugs. And as an experiment, Instead of asking their tool to develop beneficial molecules for drugs, they asked them to develop molecules that could be used for chemical weapons. And that became, you know, they wrote an article. They didn’t describe what they found. I mean, they didn’t list what they found, but they described the experiment showing that it is a threat. And… When I read that, my view was, so what they found was that the tool, the AI tool came up with about, I think, 30 or 40,000 molecules that could potentially use for chemical weapons. Some of them were very close to VX, very dangerous chemicals, chemical weapons. And my reaction when I read that was… Sure, the AI came up with a list, but they didn’t produce anything. They don’t know if any of those molecules can be produced, right? And anyone in the pharmaceutical industry will tell you that an idea on paper is very different from a drug. And sometimes even an idea that, you know, when they start developing it, there are some challenges. And transforming the idea into a drug, but even if they do… Then the drug can fail at any stage of development, including at the very end. After years of development, they find that the first stage of human testing, the drug flops. And that’s 15 years of work that are wasted. And so the point that I make every time I see something like this is that There’s a big distinction between an idea or thinking about what a tool could do and actually making things. Because this is where the problem, the biggest problem is. And then my other critique is that… The tool without the user is just useless. You know, if you don’t understand who is the user, you will not be able to determine whether the tool can be threatening. So the tool in itself… is threatening only if it’s used by people who can actually use it and transform what the tool gives us into something that can be threatening, and which implies understanding who are these users, what kind of expertise they have, what kind of team do they have, what kind of organization do they have, all these things that are not addressed. when we make assumptions about new technologies.

Sonia Ben Ouagrham-Gormley

And also, I guess, counterfactually adding knowledge or capabilities here as well also matters. If your tool is really good for the, I’m going to make up a number here, but 10 people who could already independently synthesize Anthrax or something, then this extra tool isn’t that counterfactually important. It matters that it is useful to the very specific group that with the tool. could develop the weapon, but without the tool, couldn’t. And that is the group that you’re worried about. Yeah.

Luca

And I’ll give you an example that’s probably more understandable to the general public. It’s the case of Iraq and enrichment of uranium, or Libya. When we found out that they got, for example, Libya got centrifuge technologies from the CAN network, which was a black… Market network of nuclear technology. We thought, oh my God, they have centrifuge technology. They can enrich uranium. Same thing for the Iraqis. When they got centrifuge technology from a German supplier, they… We thought, okay, they have centrifuge technology, which means that they can enrich uranium and produce HEU, that’s weapons grade, to produce a weapon. It turns out that in both cases, the centrifuge technology was found unpacked in the crates, delivery crates, because they had no one who knew how to use it. And they also had some parts that were missing. that they needed to produce themselves, but they didn’t know how to produce those parts because they didn’t have the industrial and knowledge capability to produce those parts. So again, you know, if we look at just the technology and what it could do, we can come up with the worst possible, you know, most dangerous threat assessments. Yeah. But if you look at the technology in combination with the user, you know, the technology in combination with the user, Then you have a more nuanced assessment. And that’s the kind of message I would like to deliver with this issue of new technology and bioweapons development, that we really need to have a more nuanced assessment of those technologies based on what are the characteristics of the user and the environment in which it is used.

Sonia Ben Ouagrham-Gormley

Yeah. One thing that seems really important here is like… being specific and talking like object level. You mentioned before, like, I don’t know if like AI is going to be used for a bioweapon. And I’m like, yes, like, I don’t think anybody knows. Like, what do you mean by AI? What do you mean by bioweapon? And like, who do you mean is like using it? Like pinning these things down is really important. But then yeah, that makes me want to ask, and you know, feel free to like forget the whole concept of like AI. But like, what would be a technology that like, if it were to come about in the next 10 years would really worry you in terms of So like one, you know, toy example that I can think of is just like an automated like lab helper where I can like show them my like test tube or like my Petri dish and they can tell me, your cells look healthy, your cells don’t look healthy. And here are like some steps or like some common issues that like might be wrong here. Like that to me feels very scary based on the discussion that we had before. I’m curious if that like also scares you and if you see that as like possible or yeah, if there are like other versions of this. Where you would say like, yes, if this thing was possible, then I would be really scared. And then I think, yeah, like policymakers should really be paying attention here.

Luca

That specific technology about whether, you know, a machine telling you whether the cells are healthy or not, I think it would imply that tacit knowledge can be encapsulated in a technology that can recognize those things, which I don’t think that’s really possible, right? You always need a human at the end of the chain or somewhere in the chain. What would worry me more, considering that the whole challenge of bioweapons development is, or a large part of the challenge, I wouldn’t say the whole challenge, but a large part of the challenge is to make, to work with agents that are unpredictable, that unpredictable naturally, but also even more unpredictable in the environment. So if there is a technology that can make the agent more predictable in their behavior and that they do not change behavior characteristics once they are outside in the air, that would be a technology that would, I would say, facilitate weapons development because that reduces a challenge. That’s specific to bioweapons development that you can’t find in the chemical nuclear field because nuclear material or chemical agents are more stable and they do not vary as much because of the environment. And they are not leaving organisms, so they don’t change themselves. So that would be a technology that can be a resource. Because it would solve one major challenge of bioweapons development. But again, it would not solve all of the problems, right? So even in the nuclear field, you know, HEU doesn’t change properties because it’s raining outside or because it’s too hot, right? It’s HEU. But using HEU to produce, to develop a nuclear weapon is complicated. And even in the US program, the Manhattan Project, the biggest challenges were more in engineering problems and not so much nuclear-related issues.

Sonia Ben Ouagrham-Gormley

I do feel like there is an importance in maybe arguing for that. A given technology doesn’t need to suddenly make everything possible or every step. It does feel that if a technology can really help me with one step. Ben Ouagrham-Gormley, Barriers to Bioweapons, Hear This Idea.

Luca

If we had this kind of discussion or this kind of nuanced analysis, that would be great. But today, the challenge is every time we have a new technology, we go from zero to 100. There’s this belief that this new technology is 100% going to allow or facilitate or accelerate some kind of development. It’s only after 10 years. We had the same discourse about CRISPR. 10 years ago, when it was developed, that was the same kind of discourse. You know, the technology suddenly is going to help produce designer babies and super soldiers and all kinds of agents that are going to be used for bioweapons and, you know, genetic diseases, etc. And 10 years later, we’re still trying to figure out how to, we’re still learning about the CRISPR system. We find out that there are all kinds of casts. Yeah. So I think that’s the kind of nuanced analysis that we need to introduce in the debate. Every time there’s a new technology, we need to look at it carefully and say, by how much is it going to help? Is it going to be just half a percent? Is it going to be nothing? Or is it going to be 10%? And even if it’s 10%, that’s, as you said, a lot. And what can we do to prevent that? Yeah. And we don’t have that nuanced analysis right now. It’s always 0 to 100.

Sonia Ben Ouagrham-Gormley

Yeah, and then to add another extra layer of complication, it’s not just the likelihood of it succeeding, but also if it succeeds, how bad is it? There’s a big difference between something from 10 people to 1,000 people to being in the next million death pandemic and what can I have you. You mentioned CRISPR there. I’m curious about another technology, which I actually don’t see much discussed in the bioweapons realm, which actually in and of itself feels informative. And that is like how the internet has like transformed like any of the like bioterrorism, like risk assessment and like what have you. It seems that especially with AI labs today, a lot of emphasis is being placed on making sure that large language models don’t tell people protocols about how to say step by step make smallpox, which maybe previously was only accessible in some random PDFs, or you had to kind of like construct from one paper and then another paper in order to like synthesize and find together. And at least this like particular. The risk that people are discussing feels super analogous to how I would imagine that the internet transformed things like in the 90s and like early 2000s. But I wonder like if it did, or like if people now like see the internet as like having been, you know, I’m not saying 0.1 to 10%, but at least in like relative terms, a meaningful step in allowing more actors access to misusing biology than before.

Luca

So here there are two issues. The first one is what you can access through the internet or transfer through the internet is only explicit information. That is the kind of information that can be translated into words. But not tacit information. So all of those skills, that’s the, every time we talk about information in threat assessment, it’s always explicit information. And explicit information is information that can be transferred easily, including through the internet. But it’s incomplete information. It doesn’t give you, tell you everything you need to know to reproduce, even if it’s a guide to, you know, a protocol, how to make something. that protocol does not include the tacit knowledge that the author of the document owns and the tacit knowledge that is essential in reproducing that protocol. And again, we’ve seen that through case studies in the Soviet Union where within the same program, the smallpox weapon or the anthrax weapons, were designed in different, in separate entities. And there were the protocols, the sample of the agent that was developed, were sent to another institute for production, or to develop the production process. And each time, the second institute received the protocol, and detailed, in some cases, for the anthrax weapon. They received 400 pages of documentation, plus the sample of the agent. And they were never able to produce it. It took them two years. They tried for two years. They were not able to produce it. So they had to ask the authors of the weapon to come and work with them. And even with the authors of the weapon present, On the site, it took three more years to eventually produce that anthrax weapon. And the anthrax weapon that was produced was a different one. It was not the one that was developed originally because that’s another problem in replicating the work. of somebody else in a new environment, or even one’s own work in a new environment, there’s a process of translating that new technology or that work in a new environment. Because the environment is different. And again, that’s valid for any kind of weapons, not just bio. The environment is different. The local expertise is different. The technology they use is different, etc. And there’s a need to adjust to that new environment. And the adjustment leads to a reinvention. You have to come up with a new process, a new protocol that is adapted to this new environment. And that’s why every time you see this kind of transfer. Even if it’s done by the authors of the technology or the weapon or the process, the translation process leads to a new type of protocol, a new type of process that is adjusted to that specific environment. So that cannot be transferred through the internet. These are things… The internet, publications, you know, even the methodology section of scientific articles just tells you, provides limited and incomplete explicit information. It does not include any of the tacit form of knowledge that is required to reproduce the work. That’s why even in science, there’s so much… So many problems with reproducibility because you don’t have everything that’s needed. The documents do not provide all the information required. And when I say information, I mean tacit knowledge.

Sonia Ben Ouagrham-Gormley

Yeah, and I do wonder if there’s like, you know, more diffuse stories that we can tell. So there’s like one emphasis, which is like, you know, did the internet… Like, help people learn what the protocol for, say, smallpox is, and were they then able to, like, make it? And the answer there is, like, no. Like, even if you, like, knew what the protocol was, you probably wouldn’t be able to synthesize it. And in the case of smallpox, you probably wouldn’t be able to acquire the materials either, because people are, like, really looking out for this. Like, that’s wrong. But there are, like, other stories I can tell here as well, where I think this, like, dual-use thing starts becoming, like, a real problem. But it’s, like, look, if I want to learn biology, I’m sure that the internet, like, must have just, like, really helped. In order to like let people take online courses, study the basics of biology, maybe even like acquire in like some very specific domains a lot of like pretty close to like expert like knowledge here from being able to access papers, being able to find out like where suppliers are or even like applying to labs and just like, you know, letting people partake in like the scientific community much more so than before the internet. But there I can like I think tell more of a story of like, yes, like this technology does enable. I brought a part of like the human population to take part in science and to take part in like really incredibly like valuable activities. But of course, the downside of that will also be that like more people can now do bad things with these like technologies as well. And it’s a less like clear cut story, but it is a story of the internet. inadvertently like also increasing the biofuel landscape even if that like channel is like way to diffuse to take any um like direct or like their policy action from um i wonder yeah like if you make anything of that well again again uh one the information will be um

Luca

incomplete two you always need uh very much like new technologies you always need to assess uh the information in relation to the user If the information provides you, again, I’ll use the case in the nuclear field because we have examples. The Libyans received from the CAN network designs of a nuclear weapon that the Chinese had provided Pakistan and Pakistan, the Pakistani network. This CAN network was led by a Pakistani. provided those designs to Libya. And there were notes in English and in Chinese. So there were some, you know, markings here and there. So the design, you know, if you say, oh, my God, they have a design, a nuclear bomb design, that’s dangerous. You know, they have it. The challenge is, how do they translate this design into an actual bomb? And for the Libyans, they didn’t have any expertise at all. They didn’t have a nuclear technology, a nuclear industry. They started with no knowledge. So in order to be able to use those designs, they had to first create that whole expertise in physics and mechanics and all that. Now, I’ll give you the same example, but in the Soviet case. So in the Manhattan Project, there was a spy called Klaus Fuchs who provided information about the American bomb to the Soviets. And the Soviets had already a program in place and they had the design in place. But when they received the information from class folks, they decided to drop their own. And in the hopes that, you know, using, having access to something that was already identified and, you know, that they worked on, etc. would save them some time. Yeah. And they wouldn’t have to do all the calculation, they didn’t have to do all the work necessary to come up with this design. And then they realized that it doesn’t save… time, because in order to make this design work or this information work in this new environment, they had first to check it. And they ended up, the Soviets ended up having to redo the calculations and the work, the laboratory work that they were hoping to save by getting information through spies from the U.S. nuclear weapons program. So, and the Soviets had You know, they had a, you know, very brilliant people in physics and they had already set up their nuclear weapons program. So they were, that was very different from the Libyan case. But at the same time, a design or a protocol. are, again, explicit information that do not include all of the tacit knowledge, but also all of the reasons why some decisions were made. Why do you have this specific design? And why do you have this specific protocol? And all of these decisions are made based on tacit knowledge and experimentation in a specific environment. Hmm. And that’s not replicated in the explicit document. And that’s what we’re missing. That, you know, that the document, the Internet is just a way to make information more easily accessible. Doesn’t mean that having access to information will allow, will facilitate its use by any kind of users.

Sonia Ben Ouagrham-Gormley

I think that’s a really important and interesting point. Maybe just a final question to kind of like wrap up the section here then is, again, when we think about AI and especially like large language models, are there like conditions where you would be like, oh, I think here this is like a route that they could transmute tacit knowledge? Maybe one direct option, which I’ve heard and I’m curious for your reaction to, is that there are now just a lot more online lab notebooks where people and scientists document what they’re doing day by day or what experiments they’re doing. And this feels just a lot richer in order to have these types of insights or have a basis to synthesize such insights from than just the methodology sections of papers, which are much shorter. And then there are also some indirect routes, I can imagine, which is rather than… chat GPTN telling me uh like what that like tacit knowledge is um it’s telling me like which lab professors uh I should like reach out to in order to like chat and who are like maybe also like willing to speak to me um or which kind of like labs are like the relevant things that I should then interact like in the real world with um I think a perfectly legitimate answer here is like neither of these things scare me and like I just think this is like a really big barrier uh but I’m curious yeah if there are like conditions here or like things that you would want to have policymakers uh keep a like lookout for where you think they should be focusing their attention?

Luca

I’m going to disclose. I’m working on a new project and I’m still developing it. But the idea is to look at AI in relation to the user and the developer. Because you need to, in order to understand how AI is going to help different types of users, particularly if you’re thinking of threat issues, threat. agents, etc. You need to understand how the users, the developers have developed this new technology. So, for example, when you look at AI and ChatGPT, I mean, you’ve probably read all of those articles of people testing ChatGPT to, for example, figure out what a certain person has said about someone who is known in a certain field. What did they say about specific issues? And Chuck GPT would come up with a narrative that makes a lot of sense with footnotes and quotes, etc. And someone who doesn’t know that person, and there was an article in the bulletin of the Atomic Scientist who, A nuclear expert, nuclear weapons expert, used Chad GPT to find out what Chad GPT would find out about himself and about what he said and what he did, etc., about certain issues. And he found that the narrative makes a lot of sense, but it was filled with… Made up information. Sometimes the quotes were completely made up. Sometimes the sources were made up. But… On the surface, when you read it, he felt like he could have said this, he could have said that, but he didn’t. And there was a case where Chad GPT quoted him while he was testifying before Congress. He never testified before Congress. The whole thing was made up. So the question is, and that relates also to the issue of big data. Big data was another… You’re a source of worry in terms of weapons development, that if you have access to a lot of data, then you may be able to do something threatening with that information. And the question is, what’s the quality of the data that is input in the technology? So for Chad GPT, there was a problem of the quality of data and the fact that… The user had to tell ChatGPT, don’t make up stuff. I want you to just find things that are, and ChatGPT is going to find information out there, but we don’t know what is the quality of that information out there. So the user becomes important, right? So if you don’t have the ability to evaluate the information that is given to you. Then you may be led towards a path that is going to waste all of your resources, which in a way is good in the field of non-proliferation. So that’s what I’m trying to do. I’m trying to figure out, you know, with these different tools, to what extent they can help certain users and what are the assumptions and the tacit knowledge that is of the developers that is introduced in the machine that can make it or the… the technology that can make it hard to use by certain types of users.

Sonia Ben Ouagrham-Gormley

Yeah. Yeah, no, that sounds like a really interesting project. And I’m super excited to see when your results come out, if they are public, to see what you find there. I think that we maybe need to begin wrapping up the interview. So we have some final questions that we always ask guests, just to finish things off. One feels very similar to what we talked about. Just now, which is for people early on in their careers, imagine 20 or like kind of early 30 year olds. What are some projects that you would love, you know, people who are like keen to do stuff in this space to like work on? Are there like any questions that are like on your mind or that you think are like great opportunities here? And the field being doesn’t have to be just AI and bioterrorism, but just like bio, like weapons and bio misuse overall.

Luca

Yeah, I think one very interesting study would be to look at, go back several years. Maybe decades. And look at what was said about new technologies. And then measure the development of that technology over time. And was it able to do what people expected it to do? Right. So for example, in the nuclear field, when nuclear energy was developed, and nuclear weapons were tested. At the time, we thought, or there were assessments, even by people, you know, scientists, not just either general public who didn’t know anything about this. But there were assessments that soon we’d have cars powered with nuclear energy, our watches would be powered with nuclear energy, our homes would be all powered with nuclear energy, and none of this happened. Right, so… So looking back in time, you know, how much of the expectations associated with the new technology have actually materialized? And are there any patterns that we can see in terms of technology, actual? Yeah.

Sonia Ben Ouagrham-Gormley

Yeah, yeah, that is interesting. I can think of some counter examples like top of my head. Like one is I used to work on more like kind of climate change stuff. And the famous graph there is like solar PV prices and International Energy Agency always assuming each year that like now we’re going to hit the plateau of solar is not going to get any cheaper than this. And like year on year, it always like the expectations and I think it’s like continuing to decline. Yeah, I mean, AI also feels like, you know, not that dissimilar. A lot of people said it would peter out. And at least for now, like scaling laws still seem to be holding. But I think this is really interesting, especially in the specific case of like how technologies might be misused and whether threats… The things that people were worried about, about how these technologies might be misused early on, ended up generalizing to what things turned out to be seems really important and really good. I know that especially post 9-11, I think there was a lot of forecasts around bioterrorism included. And I know that a lot of these ended up not looking very great.

Closing questions

Luca

So, yeah, I think this is a really interesting set of questions that people can for sure pick things from. The penultimate question is, yeah, can you recommend three resources for people to learn more about things that we’ve talked about here? Anything that you found maybe particularly useful in your own journey, but also just general reading recommendations?

Sonia Ben Ouagrham-Gormley

Uh, read my book. The challenge is that there’s very little Information gathered in one source or in one place. When I did my research, I had to go in different places, interview people. So there’s a lot of information that’s not out there accessible. But I do have a project with, again, Kathleen Vogel, where we, for this oral history project, we interviewed and filmed. Some of the scientists that we, Soviet and American scientists, and we started a website where we put in… some of the interviews and the transcripts of the, of the interviews and the website is called the anthrax diaries. And it’s housed by Cornell university. And, you know, it’s just the beginning. We had a grant to do that work and we ran out of money. So we haven’t been able to put in all of the interviews, but we’re still trying to get funding in order to, to, you know, putting all the interviews and the transcripts, etc. That’s a source that doesn’t exist anywhere because you don’t have live interviews of former, actual former bioweapon scientists, not just people who know biology, right? And talking about different issues, technological problems, social problems, organization, you know, ethical issues. How do they… What I found fascinating was that most of them had a, a lot of them had a medical degree. So how did they reconcile their oath to do good and not do harm with the development of bioweapons that were going to kill people with a lot of, a lot of pain? You know, it’s not a weapon that kills instantly. So there was a lot of things like that that I think should be out there in the public. And we started to do that. Hopefully, we’ll get funding to continue the project and put all of this data. Information out there so people can hear from the horse mouth this information, not just from people, you know, experts like me, but those who actually made those weapons or attempted those weapons.

Luca

So this one, this project, the Anthrax Diaries is accessible, you know, just Google Anthrax Diaries and you can find some of the interviews already out there. But Kathleen Vogel wrote a book also called Phantom Menace.

Sonia Ben Ouagrham-Gormley

Okay, that’s a good name.

Luca

And there she looks at also biology and the challenges, several of the controversial experiments that were done in the early 2000s and that were also… viewed as threats because they would help. They were viewed as, you know, giving a blueprint to terrorists to develop bioweapons. And that includes the synthesis of the poliovirus, all of the work that the Venter Institute did, and other kinds of research project. And where she went and talked to the scientists who did the research, and she found that, you know, there’s a big distinction between the two of them. What can be, what the estimate of what could be done with those, with this work and what actually could be done. And she emphasizes again, this issue of tacit, explicit knowledge and how an experiment is usually, even though it’s… It’s presented as something that’s easy because it relies mostly, like the poliovirus experiment or synthesis of the poliovirus, relies on information of the internet, on technology that’s commercially available that anyone can have access to. But the whole experiment hinged upon this issue of producing the good cell extract. And that’s something that doesn’t use high tech. It’s completely, you know, experimentation, learning how to do it, etc. And she found that that was the same issue, and that even though an experiment was presented as happening very quickly. And easily, it in fact relied on expertise that had been developed over decades. And that’s specific to that laboratory. So that’s another good…

Sonia Ben Ouagrham-Gormley

Great. And then, yeah, just the very last question to wrap things up is, where can people find you and your own work online? What’s the best resource?

Luca

So George Mason University, the biodefense program in Arlington. But, you know, just Google my name and you’ll find all my publications out there.

Sonia Ben Ouagrham-Gormley

Fantastic. Yeah, great. And we’ll add links to both the books that you mentioned and also your academic website and everything to our transcript. But yeah, then the very last thing to say is, Sonja, thanks so much for joining us and the great conversation.

Luca

Yeah, thank you very much. I’m always happy to talk about this, especially with people who ask questions that very few people ask generally when we do a threat assessment. So thank you.

Outro

Luca

That was Sonia Ben Ouagrham-Gormley on Barriers to Bioweapons. As always, if you want to learn more, you can read the write-up at hearthisidea.com forward slash episodes forward slash Ben Ouagrham-Gormley. There you’ll find links to all the papers and books referenced throughout our interview, plus a whole lot more. If you enjoyed this podcast and find it valuable, then one of the best ways to help us out is to write a review on whatever platform you’re listening to. You can also give us a shout out on Twitter, we’re at hearthisidea. We also do have a short feedback survey, which should only take you somewhere between five to ten minutes to fill out. We read every submission, find them tremendously useful. And as a thank you, you’ll also get a free book from us. A big thanks as always to our producer, Jason, for editing these episodes.
And thanks very much to you for listening.