Contents

1. Disclaimer
2. Introduction
3. Problem Space
4. Solution Space
5. Concluding Remarks

Disclaimer

This article contains subject matter that requires careful discussion. While the general topic is not new, this might be unsettling for some subsets of readers. It may result in feelings of disbelief in one subset, and perhaps feelings of dread in another. I kindly ask that we, including myself, keep an open mind so that we can all think clearly and truthfully about the nuances in the problem and solution spaces.

To borrow and adapt phrasing from François Chollet’s post on a similar topic from 2018: This article contains my own personal views. I do not speak for my employer. I am writing this as a member of civil society for readers (e.g. cyber security practitioners) in their capacity as members of civil society. If you reference this article, please have the honesty to present these views as what they are: personal, speculative opinions, to be judged on their own merits.

Introduction

In my last post, I discussed what it looks like in commercial cyber when market incentives and long-term national security incentives seem to pull in opposite directions. It gives strategic adversaries more breathing room to operate because we have been too busy maximizing capital. In this post, I’d like to elaborate on a major blind spot that demonstrates how far along they’ve come and how little we’re doing.

It is becoming understood that various actors are able to create digital twins of ourselves and society to ultimately control us in ways that reduce our privacy and volition. The competition between defenders and adversaries has extended beyond computers into our minds. I contend that we are experiencing a failure of imagination about adversaries at a scale that has proven lucrative for companies while simultaneously fraying our cognition and social fabric. This knot poses threats to both democracy and human security. Protecting our minds has become as important as protecting our computers.

There is a way out of this, but it requires sustained effort and creativity. Specifically, I believe it requires: (1) a way to think clearly about the modern threat landscape (where vendors have lacked commercial incentive¹), (2) the motivation to correlate our daily actions to the bigger picture, and (3) the courage to make decisions that help improve collective well-being with the agency each of us already has. With more awareness and collaboration, I’m cautiously optimistic things will improve.

In this essay, I will discuss both the problem space and the solution space as I see them. Here’s the article structure so you’ll know what you’re getting into:

• In the problem space, I’ll introduce the PRC version of “parallel intelligence,” and how methodologies like that can advance what is known as “cognitive warfare” (at a technical level). Throughout, I’ll emphasize the threat that cognitive warfare poses to democracy and human flourishing.

• In discussing the solution space, I’ll start with the principles and mindset shift required to address the threat. I’ll expand from there with the pillars that most efforts could fall into. This should help scaffold a mental model for response and stimulate ideas into action.

If this sounds like your cup of tea, then grab it and start sipping. It’s going to be a dense read. While I hope it will be enlightening, it will be an active reading exercise where I’d suggest taking breaks between sections or paragraphs to pause and reflect on what I’m trying to convey. Feel free to print it out or load it into your e-reader. If you’re feeling nerdy, check out the footnotes for more nuance and speculative views.

As for the length, I wanted to weave most of my current thoughts together into one essay for those that are interested in thinking deeply and laterally about the subject matter. While the nature of AI summaries will obscure exactly what I’m trying to convey, if you need to take that route, then I’d suggest using Google’s NotebookLM and generate a “deep dive” podcast with the link to this article.² In the future, I might spotlight topics from this essay with smaller, more consumable, posts.

I circulated drafts of this essay among several peers for their feedback before publication. Many of them contributed to it with insightful discussion and reading materials. Several aspects of the discussions made me pause and forced me to refine my thinking. I am deeply grateful for their support.

Problem Space

I’ll begin with describing the problem space. Not the one that yields solutions for defenders, but the one that yields solutions for authoritarian adversaries. Understanding it comes with a learning curve that we‘ll have to build up to. I’ve tried my best to minimize its steepness at the loss of some nuance.³ Once you see adversary activity through this lens, it can be difficult to unsee.

To a particular “demographic” that might be reading this: You have probably noticed the term cognitive warfare. Perhaps you may not yet have the vocabulary to articulate what it is, but you can already feel inklings of it. If what I’m about to describe clicks in a horrifyingly intuitive way, some side-effects include: uncontrollable critical thinking, spontaneous or reflexive correlations on the topics of perception and persuasion, and existential dread about hyperobjects which does not go away.⁴ (Also, feel free to reach out to me as a fellow member of civil society. It helps to talk about it.)

Cybernetics 101 and Parallel Intelligence

Consider the humble thermostat as a cybernetic system. It has sensors which measure the current temperature, actuators which enable heating or cooling, and a controller that processes sensor data to make decisions on activating heating or cooling based on a user-defined goal. If sensors are the eyes, and actuators are the hands, then controllers are the brains of the operation.

When we apply this mental model to modulating computer security, we can get AV/EDR software. Their sensors include a wide array of real-time OS telemetry and inferences of that data. Their actuators include the ability to cloud-up normalized telemetry and prevent code execution. And their controllers are the relevant logic that connects sensors to desired outcomes with actuators based on administrator policies. A “computer security thermostat,” if you will.

What if consumer behaviour could be modulated in this way?⁵ In the social media context, sensors could collect a variety of human telemetry (messages, interactions, geolocation, purchases, biometrics, etc.) and make levels of inferences from that data. Actuators can come in the form of feeds and notifications. Controllers can construct real-time quantitative and qualitative profiles of users with sensor data. These profiles can then be used to make subtle but optimal decisions on ads, as well as the sequencing of feeds and notifications. When compared to platform owners, end-users are severely limited in their ability to configure the desired outcomes in controllers. A “revenue thermostat” is a first-order way to think about it.

These examples start to demonstrate the applicability of cybernetics in a variety of fields. In what many might call the “age of agentic AI,” modern approaches to modelling and simulation as well as the Chinese form of intelligence called Parallel Intelligence (PI or PARINT) could be seen as advanced versions of traditional cybernetic systems.

Proposed by polymath Fei-Yue Wang in 2004 (and envisioned as early as 1994)⁶, the core of Parallel Intelligence is the ACP methodology: artificial societies (A), computational experiments (C), and parallel execution (P).

It looks like a complex topic but one way to start thinking about this is to first imagine the actual society we live in. In this physical world, there is a wide array of sensors that can track what people do online and offline with increasing levels of fidelity. This includes data from interactions with phones and social media.

Now imagine computerized versions of that world where data from those sensors flow into them in real-time to make increasingly accurate models of us and our environment. These are “artificial societies.” For initial illustrative purposes, it might help to imagine higher-resolution versions of simulator games like The Sims or Cities: Skylines. These worlds could be configured to have desired objectives (like maximizing party loyalty or revenue). Simulated scenarios or “computational experiments” can run against them to make descriptions, predictions, and prescriptions relevant to those objectives.

Indeed, these simulations can yield a variety of interventions to introduce into the real world, and this is done in the effort to make the actual society converge to the desired artificial society. This is what “parallel execution” is about: artificial societies are continually enriched with sensor data, and actual societies are continually managed toward a desired outcome with quantifiably optimal interventions. A “social control thermostat”, if you will.

Although Parallel Intelligence spans a variety of benign applications (transit, agriculture, healthcare, etc.), competitive implementations present risks to societal well-being within democratic societies and overall human flourishing. According to DEMOS (UK think tank), democracies are suffering an epistemic collapse, which is “a deeper breakdown in the basic conditions that allow societies to establish truth, debate what matters, and hold power to account.” As I’ll describe in more detail, the ACP methodology could be used to automate these breakdowns.

In my current thinking, it seems prudent to recognize that with technological approaches similar to ACP, there are a variety of actors who are interested in developing and using them with different intents in mind:

Commercial Entities. Technology like this can be useful for any sales and marketing department selling running shoes, private jets, pop concerts, and more. It can be equally useful for frontier labs supporting the development of public-facing services like self-driving cars. Within this ecosystem, players can range from marketing start-ups to hyperscalers. In a way, I currently think this is where the technology underpinning Shoshana Zuboff’s concept of surveillance capitalism is going. In this concept, personal data is collected and commodified to serve the profit-making incentive.

Political Consultancies. The most obvious public example of this was the controversial work of Cambridge Analytica. In the 2017 talk From Mad Men to Math Men, chief executive Alexander Nix described how they collected vast amounts of data to build detailed psychological profiles of voters, using them for highly persuasive messaging. The end of this talk surfaced the tension between what free and fair elections mean at political versus psychological levels of analysis. In his book Mindf*ck, former employee Christopher Wylie expressed his opinion that “America is now living in the aftermath of the first scaled deployment of a psychological weapon of mass destruction.” He further wrote:

Like so many people in technology, I stupidly fell for the hubristic allure of Facebook’s call to “move fast and break things.” I’ve never regretted something so much. I moved fast, I built things of immense power, and I never fully appreciated what I was breaking until it was too late.

Authoritarian Governments. According to researchers like Jason Bruzdzinski (former MITRE Principal) and Anthony Vinci (former NGA CTO), Parallel Intelligence increasingly plays a role within China’s global system of social control. In The Fourth Intelligence Revolution, Dr. Vinci wrote:

Artificial intelligence can drive intelligence even farther into much more sophisticated forms of analysis. The technology can be used for modeling and simulation by creating AI agents that can model the thought processes and actions of adversaries. Scenarios can be planned and played out to see what will happen. Indeed, some have argued that China is already headed in this direction through the creation of parallel intelligence. In this approach, the Chinese surveillance system is used to collect as much open-source data as possible and then to model out the U.S. military, economy, and more to eventually develop countermeasures to our systems.

Bruzdzinski’s open-source analysis suggests that adversaries like China and Russia are working on what he calls a mega model. This can be “at the level of large and complex cities like Manhattan, Houston, Los Angeles, etc.,” and can be developed “with a shocking degree of fidelity, in some cases down to the square metre of resolution.” He also suggests that modelling of people and organizations in societies is already happening. He continues to say, “I think that it’s really important to get after this and to get after it very quickly, because the failure to do so could have very profound implications.”

One of the implications to me is that even if the CCP hasn’t fully realized the vision of PARINT, they already have the whole-of-society capacity to improve their networks of sensors, actuators, and controllers beyond what any other Western actor can achieve alone.⁷ Their advantage in these areas increases fidelity of their models and simulations, and therefore the power of their panopticonic social control system (which is becoming global).⁸ This would give them the means to influence Western commercial and political entities in ways that efficiently achieve the goals of those entities (i.e. more capital, more engaged voters), while ensuring that the second-order effects ultimately serve CCP interests.

To ground this all with an example, we should consider that China is working relentlessly to shape Western public opinion to oppose support for Taiwan. Increasingly comprehensive digital twin models of citizens⁹ can be created from multi-disciplinary inferences which, in turn, rely on ubiquitous technical surveillance. Relevant sensors can be sourced from state-linked entities (Huawei, TikTok, Temu, etc.), bulk purchases from Western entities (e.g. data brokers), and data breaches. Scenarios can be simulated on these models to prescribe intervention portfolios, tailored at both individual- and group-levels.

Individual-level interventions can include the creation of digital twin personas of U.S. Congress members to maintain a presence wherever they browse social media and create engaging content intended to diminish their support for Taiwan. As discovered by researchers Brett Goldstein and Brett Benson, it turns out that a Chinese company called GoLaxy was already attempting to do this:

What sets GoLaxy apart is its integration of generative A.I. with enormous troves of personal data. Its systems continually mine social media platforms to build dynamic psychological profiles. Its content is customized to a person’s values, beliefs, emotional tendencies and vulnerabilities. According to the documents, A.I. personas can then engage users in what appears to be a conversation — content that feels authentic, adapts in real-time and avoids detection. The result is a highly efficient propaganda engine that’s designed to be nearly indistinguishable from legitimate online interaction, delivered instantaneously at a scale never before achieved.

Group-level interventions can include targeted “butterfly effect” supply-chain disruptions which could raise the cost-of-living for broad segments of the population, as companies make rational business decisions to pass increased costs onto consumers. As a second-order effect, this could support narratives for domestic allocation of funding that would otherwise be used to assist Taiwan, e.g. “Why are we spending money defending Taiwan when I can’t afford groceries?”

This example starts to demonstrate how broad the sensor and actuator options can be, and how they can be orchestrated through ACP-like systems. Cyber intrusion operations might not be necessarily required, and attribution can be difficult when the CCP can parasitically exploit the business practices of commercial entities.

I think it’s important to expand on this parasite analogy, even though it might be disorienting for commercial practitioners. The phenomenon of parasitical asymmetry is described in more detail in Chapter 4 of Anthony Vinci’s book. The insight I had when reading this was that adversaries in commercial cybersecurity are conventionally framed as classes of storms or animals, but none of those reflect the behavioural characteristics of parasites. I think this conventional framing has, along with other incentives, conditioned us to focus more on the threats which are immediately antagonistic to business.

As we begin to understand that competition has extended beyond computers into minds, the adversary’s most durable strategy is not to breach computers but to make our rational commercial behaviour ultimately serve their interests. It used to be about adversaries charging into the fortress through the perimeter. Now it’s about parasites making you unwittingly serve adversary interests from within. Once you see it this way, the action isn’t necessarily in cyber telemetry. It’s in the business decisions and the externalities they produce.¹⁰

Artificial Intelligence. My speculation is that the arms race of ACP-like AI systems between the actors above will exacerbate human security issues as an unintended consequence (i.e. the loss of privacy, volition¹¹, competence, trust, patience, integrity, struggle, inefficiency, and serendipity).¹² Assuming this is not terminal, it will probably take at least a generation to recover from. Novels like Daemon¹³ and, perhaps eventually, Optimal describe the societal impacts of this possible future. Fei-Yue Wang’s collaborators openly acknowledge that “many scenarios that were once thought to only appear in science fiction films such as ‘Source Code’, ‘Westworld’, and ‘The Matrix’ have now become a reality.”

I should reiterate that Parallel Intelligence and the ACP methodology does have a variety of benign uses. When implemented collaboratively and humanely (instead of competitively and efficiently), I think it can help strengthen democratic societies instead of eroding them. But the ongoing arms race of competitive implementations has concerning impacts on society that can emerge from effects on cognition. Serious solutions require wisdom (i.e. higher-order cognition), which has been under deliberate attack. So we need to understand how this arms race of ACP-like systems contributes to something called “cognitive warfare”, which is not well known to most people. The next section aims to form a baseline understanding.

Cognitive Warfare

Cognition is being considered by NATO as the sixth domain of warfare. The basic meaning is in the name: it treats cognition as another domain of warfare. If the cyber domain targets computers, then the cognitive domain targets minds. Yes, that includes biological brains. Anthony Vinci dedicated a chapter to cognitive warfare in his book, and there are a few choice quotes worth referencing as a way to introduce this topic:

• It is a means to control how people think and what they think in such a way as to influence them to do what you want, whether that’s a private surrendering on the battlefield … or an entire population being for or against a war.

• In other words, cognitive warfare is about actively changing or even controlling how we think and act, i.e., mind control, and this is not science fiction, it is very real.¹⁴

• Where LSD failed, silicon has succeeded.

While information operations (IO) have traditionally focused on the data that shapes “what” individuals and groups think, cognitive warfare (CW) is an evolution that also shapes “how” individuals and groups think and the meaning derived from data.¹⁵

This includes effects like degrading: attention spans, absorption of nuanced information, and rationality. For any number of topics, it also includes making people think reflexively instead of reflectively, exploitatively instead of exploratively, and decreasing bit-depth and second-order thinking.¹⁶ We can start to think of these areas as having controls that can be dialed up or down. By the way, have you noticed changes in these areas over the past decade or more?

Metaphorically: is decision making bit-depth increasing in machines, while decreasing in humans? This question was inspired by David Krakauer's (Santa Fe Institute president) mental model of complementary and competitive artifacts, as well as thinking about how human- and machine-learned decision making processes could be understood through the lens of cyclomatic complexity.

In this style of warfare, targets scale up to populations across generations. Yes, population-scale includes you, your family, friends, and neighbours. Methods could intentionally trigger fractal-like effects rippling up from the neuronal-level to the organizational-level.¹⁷ I think with more introspection, it can sometimes feel like a spell has been cast over all of us. Perhaps we are under a “persistent state of cognitive manipulation” as Dr. Jake Bebber describes it (as a scholar for the Hudson Institute):

Thus far, policymakers in the United States have been slow to diagnose and react to cognitive warfare not only because of its novelty but also perhaps because the American public has remained under a persistent state of cognitive manipulation, which has debilitated the people.

This is probably not what most people want to hear, but these effects will likely take decades to recover from.¹⁸ Fact check interventions with nuanced white papers just aren’t that useful for anyone whose brain has been conditioned to rely on AI summaries or spend hours at a time flipping through short-form videos.

To me, it is actually the orchestration of interventions that requires nuance, not necessarily in the content of what’s presented. There is still a time and place for longer-form content, but knowing when and where to present them matters just as much. To borrow a sports analogy from Dr. Sean Guillory: “think of how golfers assess the lay of the land on the golf course and choose their clubs based on where they want to move the ball.” In order to assess the lay of the land, we first need to sharpen our understanding of the systems that could shape the terrain and automate these effects.

Let’s sketch out what the automated orchestration of these effects could look like.¹⁹ Central to any domain of warfare is a competition on cybernetic capabilities for sensors, controllers, and actuators.²⁰ I see ACP as a cybernetic vehicle for enabling any domain of warfare, and it has influenced the way I think about how cognitive warfare can be operationalized. We’re about to get into the weeds here, but reading through it will help build an appreciation for what’s at stake. This is an area of ongoing research for me, and I try to oscillate between the forest and trees. Consider the following as speculative, directional, and open to change:

Data Collection. This starts with the comprehensive collection of various streams²¹ of personally identifiable information and behaviour (PII and PIB) associated with citizens. As mentioned earlier for the PRC, the sourcing of this information can go beyond breach records into data collected from state-linked entities (Huawei, TikTok, Temu, etc.) as well as purchases from Western entities. Various privacy policies provide a starting point for understanding the data types that can provide value (e.g. biometrics, messages, geolocation, video, audio, purchases). Maintaining a hierarchical taxonomy of data types would be useful for organizing both offensive and defensive efforts. This can help catalog which organizations are relevant per data type, and help identify collection gaps.

Quantitative and Qualitative Inferences. This is the process of developing frame-dependent²² sensors that infer knowledge from raw personal data or other inferences. For example, participating in a video call may yield a variety of inferences including but not limited to: meeting attendance frequency, interruption frequency, talk-time share, LIWC scores, Flesch-Kincaid grade level, other LLM-based natural language processing methods²³, heart rate estimation, and fMRI brain activity. These inferences can be used together to create digital twin representations of targets.

I currently see representations as collections of quantitative and qualitative knowledge. Quantitative knowledge can come in the form of scalars and vectors that can combine to form high-dimensional, per-individual, feature vectors. Qualitative knowledge can come in the form of LLM-generated, per-individual, personas²⁴ where some can be frame-specific and others can serve as a consolidation of frames. Both of these should be computable as a function of targets and time windows (which can be sliding). These representations can be used to surface vulnerabilities or biases that can be exploitable in a context-dependent way.

Common Operating Picture. There should be ways to visualize the cognitive state of individuals and groups, in a manner that provides utility beyond conventional geographical representations. It makes sense to me to have visualizations that span across physiological, psychological, and sociological levels. A small set of examples include: weighted social network graphs that include parasocial relationships²⁵, persona projections²⁶, and belief landscapes²⁷. See the footnotes for more details.

Entropy could be interesting as a metric from information theory that cuts across integrative levels, and as a quantitative measure for cognitive security. Brian Russell and John Bicknell have described understanding relative system behaviour as “the coin of the realm”, where entropy serves as a measurement. I’m not yet sure how the measurement would be implemented at each level.

Control Centre. I’m imagining this to be the interface that commanders or system operators use to both define and manifest desired outcomes (i.e. self-fulfilling prophecies in the context of Merton’s Laws). If warfare could be understood abstractly as zero-sum competitions between groups for welfare maximization, then I have to imagine that this interface is where the flourishing and languishing of targets can be parameterized.

I currently use the word “campaign” to describe the specifications of desired outcomes. Every campaign could be scoped toward particular individuals or groups, with the option to impose a deadline. From a quantitative standpoint, desired outcomes can be defined in terms of changes to feature values (e.g. decreasing daily hours spent sleeping, increasing daily hours spent in parasocial relationships). Desired outcomes could also be qualitatively defined as imperative language (prompts) which can be used to continually revise personas and feature values for convergence.

Simulations could be recommended and scheduled as a way to mine interventions for parallel execution. There should be controls on whether mined interventions should be introduced into the real world automatically (human-on-the-loop), or if manual human validation is required (human-in-the-loop). This would depend on the stakes and scale of those decisions. Furthermore, campaigns should be ranked to ensure that the proposed interventions of lower-priority campaigns do not impede the success of higher-priority campaigns.

Modeling and Simulation. While LLM-based digital twins of individuals serve as a starting point to simulate how targets might respond to various situations, simulations in ACP can go far beyond that. As suggested in an earlier figure, modeling a city would involve not just the individuals, but perhaps also business processes across enterprises, transportation systems, and so on. Simulating these additional systems can increase realism by constraining what’s possible and likely. They can also increase realism by modeling the second-order effects that occur between systems (e.g. how the outcomes of simulated supply chain disruptions can impact opinion polling on digital twins). At a more technical level, I find it useful to study frameworks (e.g. Concordia and AgentSociety) and simulation games (e.g. Universe Sandbox and Democracy) through the lens of ACP-based PI.

Control Vectors. Although the social media methods for disinformation campaigns are known to be prominent in this domain, I think they only start to scratch the surface of what’s possible. Cognitive warfare interventions would be most effective when they span across and combine all instruments of both national power and parasitic asymmetries. In the ACP context, each campaign can benefit from various interventions being used together. Consider the following examples across media, economics, and physiological health:

1. Bots on social media, gaming, and betting platforms can amplify or generate media to serve a variety of campaigns. In media, both content and form can produce effects. Content is the subject matter and meaning that creators direct our attention toward. Form is the aesthetic and design techniques which are often invisible when we’re not looking for them.
   
   When it comes to content, social media bots can amplify real posts about local business closures, local crime, social inequalities, and empty store shelves. While these raise awareness about true grievances that need to be addressed, it can also serve the interests of adversaries as a way to distract target populations. Bots today can also fabricate a firehose of mostly false, but internally consistent accounts of current events. This would serve efforts to make epistemic security mentally exhausting. When combined with algorithmic news feeds, research suggests this traps people in “information cocoons” with homogenous viewpoints, further aggravating social polarization. Together, these ultimately erode the conditions for functioning democracies.
   
   When it comes to form, consider the design techniques (or “TTPs”, if you will) that are often used competitively to present content for algorithmic feeds. These include the short provocative sentence on a LinkedIn post, as well as the frenetic and visually fragmented TikTok with word-popping subtitles. Now consider how the presentation of content changes as its length decreases. A calming 30-minute episode of Martha Cooks versus a 15-second supercut of the same recipe. A gentle incandescent turn signal versus its dazzling LED counterpart in some modern cars. Research suggests that increased consumption of short-form videos is associated with poorer cognition. Even when the content is actively determined to be truthful, passive processing of its form risks degrading cognition. Adversaries like the CCP don’t need to generate these “payloads” on their own. They just need to induce capitalistic competition among Western media platforms and content creators (influencers, marketing departments, etc.).²⁸

2. Various actuators may produce economic effects that exacerbate income inequality while increasing GDP. One example is exploiting dynamic pricing algorithms to induce artificial demand and raise prices (flights, ride-sharing, concerts, freight and logistics, etc.).²⁹ Another is inducing friction in supply chains (i.e. weaponized interdependence) in ways that raise prices for consumers as companies make rational decisions to protect their profit margins. These economic effects can have second-order cognitive effects. When it comes to the PRC deterring public support for Taiwan, the effects might discourage support from the affluent top 10% so they can protect their lifestyles and income streams. Meanwhile, the bottom 90% might become increasingly focused on their psychological and financial survival in a way that reduces the cognitive bandwidth to think about seemingly abstract topics like foreign policy.

3. Another set of actuators could indirectly produce lower-level physiological effects that can ripple upward to degrade critical thinking and induce societal incohesion. To think clearly about these, we need to regard cognitive processes as inseparable from their physical substrates.³⁰ One example for highly targeted influence could involve the ability to generate noise from personal devices for precisely-timed sleep disruptions (e.g. spam calls or tampered notification settings). The production and dissemination of content that discourages healthy eating and exercise could be relevant for less targeted influence, but it’s not clear to me how much deliberate effort a traditional adversary needs to invest toward this end of the causal chain when it can happen organically through commercial incentives. Consider the incentives for ultra-processed foods and binge-watching shows at night.³¹

From a cyber security perspective, it’s remarkable that these actuators don’t always require “initial access” into networks or sweeping administrative privileges (e.g. Domain Admin). Researchers Paul Thompson and Sean Guillory provide a useful perspective:

Unlike traditional cyberattacks at the time focusing on breaking systems, cognitive attacks work by distorting how humans perceive reality and make decisions. Rather than taking down a server or corrupting data, a cognitive attack shifts the interpretation of information. The attacker does not need to control the system, they only need to control the conclusions the system’s users draw from it.

Academic Researchers. Underpinning innovation in all of the areas above would be researchers providing perspectives from a variety of disciplines. By applying frames within them, they could aid the development of sensors, controllers, and actuators.³² Some examples include helping build sensor automation for quantitative and qualitative inferences, and determining frame-specific parameters for actuators. The diversity of perspectives would make the overall cybernetic system stronger in a way that is consistent with Ashby’s Law of Requisite Variety. In other words, a larger set of perspectives yields a larger (and more amorphous) action space for influencing target populations.

While I’m certain that what I’ve started to sketch out requires further refining, the potential of such a system becomes apparent. Let me phrase it this way: Is a “value system” a set of held beliefs? Could a “value system” also be understood as a system of capabilities for entrenching those beliefs into reality?

It fascinates me to think about how raw personal data can be collected so that they could be used to run simulations that are distilled into optimized courses of action in the real world. I’ve come to realize that what starts with collection ultimately ends with control.

It’s commonly understood that in the 20^th century, propaganda took shape in the form of broadcasted communications (pamphlets, rallies, etc.). In the 21^st century, I believe propaganda will increasingly be understood as the systemic orchestration of personalized, ambient realities.³³ As Fei-Yue Wang and his collaborators suggest, you can think metaphorically about how The Matrix shapes ambient reality and creates a “prison for your mind.” I believe this poses various threats to democracy and individual human flourishing. I also think that the ACP framework offers a methodology for orchestrating this.

In the pursuit of thinking clearly about difficult topics, this is probably a good time to start considering two uncomfortable views: First, technological innovation in social control and marketing have trended in similar directions. Second, marketing departments across many companies seem to unwittingly advance the cognitive warfare interests of authoritarian adversaries, as a second-order effect of the profit making incentive.

This is a tough economic knot to untangle, but the first step is to recognize that it’s there. Our value system makes it harder to recognize that what many people see as a feature in our society (i.e. market economics), is probably seen as an exploitable bug by authoritarian governments. In turn, they might view the closed and controlled nature of their economies as a necessary defense to the vulnerabilities they’re able to (easily) exploit in ours.³⁴

Overall, we seem to be losing a battle that the average citizen doesn’t realize we’re in. We haven’t quite had a cognitive “9/11 moment” in a way that will tangibly and emotionally resonate with most citizens. It seems logical that authoritarian adversaries want to avoid such a flashpoint because it would mobilize more citizens to fight them. As researcher Elizabeth Anderson said about cognitive warfare: “The supreme art of war is to subdue the enemy without fighting. Our adversaries understand this, and they’re betting that we won’t recognize the war until after we’ve already lost.”

If you’ve gotten this far, I like to think it’s because you’re starting to make the choice to wake up and see the writing on the wall more clearly. Democracy isn’t just a state of territory, it’s a state of mind. Maybe like me, you probably don’t like where the status quo is going and are mulling over what it takes to do something about it. The next section covers how I’m currently thinking about the solution space, and I encourage you to keep reading.

Now here is another note to that particular “demographic” I mentioned near the top of this post: It is very easy to feel disillusioned by what‘s happening and turn inward, because what‘s the most one person can do anyway? I get it, I’ve been there and was stuck there for years. Earlier, I wrote about the negative side-effects that come with understanding the breadth and depth of the problem space. Let me talk about the positive side-effects that come with engaging in the solution space. They include: (1) improved meta-cognition as a way to defend your own mind, (2) a “centering effect” that comes from feeling compassion for those that you disagree with, and (3) a sense of agency, purpose, and even creativity in tumultuous times. Lastly, I want you to know that hope can be strategic. Even when you don’t feel it, hope can still be a useful tool.

Solution Space

Principles and Mindset Shift

By engaging with the problem space, we widened our understanding of the technological apparatus and the variety of actors incentivized to use it in ways that can undermine cognitive security and democracy. We also understood perspectives from national security practitioners warning us about the threat.

Over the past couple of years, I’ve had many conversations with diverse perspectives on what should be done about this. For many cyber security practitioners who “get it,” a common feeling is that this is anxiety inducing. It is anxiety inducing to think about how far behind we are in cognitive security, how frivolous many of today’s cybersecurity practices seem in light of this generational challenge, and how many companies unwittingly exacerbate the problem through their action and inaction. This also piles onto other equally pressing concerns like climate change, kinetic wars, and the uncertainties of AI. And all of this is happening while we are moving through our day, keeping our heads down, trying to survive the next round of workforce reductions.

It has become so easy to think that making a difference is beyond our control. “Maybe it’s above our pay grade, or maybe the real solutions aren’t profitable, so maybe it’s only a problem for government or civil society (non-profits) to solve.” This line of thinking will just make us feel even more anxious as time passes and inaction abounds. What we need to understand is that cognitive warfare is intended to cause paralysis within ourselves and between our sectors. Adversaries are counting on us to not step up before it’s too late.

What we’re actually dealing with is called a collective action problem. It’s where we would all be better off working together but fail to do so because our individual incentives discourage joint action toward the common good. This sounds to me like something adversaries would want to induce as part of a divide and conquer strategy.³⁵

When it comes to the relations between government, industry, and civil society, I think a more productive framing is to see ourselves as members of a basketball team, with our positions representing these sectors. In this team, we should be working together to score wins for (little-d) democratic principles. This includes convictions for objective truth, consensus reality, compassion for all, freedom of speech and thought (cognitive liberty), and the rule of law. These are part of the basic conditions that allow democracies to function effectively. We are competing against players who are promoting autocratic principles. They are getting stronger and are currently winning.

Seeing things this way helps us understand that it’s going to take a whole-of-society team effort, and that we need to get to know our teammates and their positions better.³⁶ Specifically we should be thinking about: What can governments do? What can companies do? What can civil society do? And how can we form bridges for all three to work together?

As players on “Team Democracy,” we need to think of ourselves as having trusted influence with each other (often in fluid situations). This helps us work toward shared outcomes that benefit us all. Sometimes it’s a social process when you get stuck and realize you need to pass the ball. This helps build relationships. Sometimes it’s a mental process as you alternate positions from playing a commercial role to playing a civil society role, reflecting on how your actions within both roles can serve the greater good. This helps build empathy. When you start to see things this way and find like-minded people to work with, it will open a door to a wider “action space” that may have been difficult to see before (despite hiding in plain sight).³⁷

Of course, this all sounds like extra effort. But the cure to anxiety is action. You can use the agency you already have to make small and creative choices driven by the principled conviction of how democratic societies are normally supposed to function. Everyone has something small that they’re good at, that they can contribute to (even when it’s not obvious). But when more people do this, it gets easier and adds up like compounding interest.

And along the way, others and even your own inner monologue will try to convince you to keep your head down and stay in your lane. While there’s the action space within your typical responsibilities, there’s also the action space that promotes democratic principles. It turns out that these are not mutually exclusive, and the middle part of the Venn diagram is larger than you think. Figuring out exactly what that “middle part” is within each of our roles is the fun and creative part about converting anxiety into little meaningful actions.

My crude depiction of the Venn diagram analogy.

As adversaries continue to induce the cognitive version of “death by a thousand cuts,” coarsening the bit-depth of our decision making, we have to realize that no action on its own will be decisive. Instead, we must recognize the need to start with what I would call “flourishing by a thousand small wins.”

To some, this might sound like a tall order, but we seem to be admittedly late to the cognitive security game and should start somewhere. To others, focusing on “small wins” might not sound like enough. I am all for ambitious projects, and we need motivated leaders who are willing to dedicate their careers to them. That said, they will only be effective with a culture of collective effort.

In any case, I’ve been thinking about how to categorize potential efforts. Currently, it seems to me that any meaningful action will likely fall into at least one of the following three pillars: (1) degrade adversary capabilities, (2) build cognitive resilience, and (3) foster communities of interest. In other words: impose cost on adversaries, become wiser, and spread the word.

These are intentionally broad and I’ll define them in the proceeding sections. Within each, I’ll discuss what I think meaningful actions could look like from my perspective (as a starting point for a more constructive debate).

It’s likely that a subset of tactically-minded readers might be expecting specific and decisive recommendations on what to do. I’m not sure if that’s the right question to ask at this early stage, and I would be skeptical of quick and comprehensive-sounding commercial solutions because of what the profit motive currently incentivizes. Instead, I can share my opinion on what the outcomes should look like as well as some guidance on what it takes to get there. This is a journey, and it’s up to all of us to act within what we can control and expand the middle part of the Venn diagram in our own, unique, ways.

Pillar 1: Degrade Adversary Capabilities

We now have a baseline, cybernetic, understanding of what capabilities for social control looks like. When you start to see things this way, degrading adversary capabilities ultimately involves:

1. Limiting and distorting the flow of personal data ingested into sensors.
2. Blunting hardware and software advances in controllers.
3. Hardening various processes that could be exploited for their actuators.

Meaningful action here will be diverse and cut across all sectors of society, with examples ranging from export regulations on AI chips, to everyday citizens installing ad blockers, to having commercial cyber security departments and vendors realize they’re on one of the frontlines and adapt to this challenge. Each action will appear imperfect when evaluated on its own, but together they will help push the needle.

Adapting to cognitive threats is what cyber security organizations are supposed to be doing, in the way physical security organizations continue to adapt to cyber threats. I contend that market incentives and the parasitic exploitation of them have made adaptation difficult. To stimulate progress, I’ll spotlight one idea³⁸ that I think is fundamental and worth building on across sectors: proactive threat intelligence for maligned cybernetic systems.

We’ve become so lucratively good at building systems that identify and react to adversary IOCs and TTPs, but not so much on systems that anticipate their operational objectives and helps orchestrate layered defenses. Given that actors are operating downstream from the technology available to them³⁹, adopting a cybernetic perspective helps orient on threats from ACP-like systems.

Behind every sensor, controller, and actuator are sets of exploited data or processes, conceptual frames, and motivations. Surfacing these relationships require skills that are ultimately better suited for different types of analysts. Persisting these vetted relationships into a graph database would help provide proactive direction and productivity to a variety of downstream consumers. This includes but is not limited to: threat hunting, product security, offensive security, business continuity, and procurement. For some more details, read the caption of the figure below and this footnote.⁴⁰ I may consider dedicating a post to this later (as I refine this concept), but wanted to plant a seed for now.

An ontology for the proactive and speculative mapping of cybernetic systems related to cognitive warfare. This can be used to surface targets for exploitable data and processes. The ability to query this data could help prioritize hardening systems and influence procurement decisions.

It’s well understood that actions to degrade adversary capabilities are necessary and will help. However, my main criticism with this pillar is that despite companies having some muscle memory in this area, focusing mostly on this aspect would be dangerously incomplete. It’s a bit like solving malaria by hunting mosquitoes and disrupting their breeding grounds. That’s just not enough. We also need to think about how to inoculate ourselves. We all need to become wiser.

Pillar 2: Build Cognitive Resilience

Whether it’s cognitive warfare, climate change, or the risks of AI, we all need to become the most mature version of ourselves in order to solve collective action problems.⁴¹ We need regular people in society to gain resilience.

Cyber security practitioners are used to thinking about computers as endpoints that need to be protected systematically. And it’s obvious today that computer security is fundamentally more than Kensington locks or the physical security of data centres. We now need to realize that the focus has shifted, and treat human minds as the ultimate “endpoints” that systematically need resilience and protection. This means meeting the cognitive domain on its own terms and not fall into the trap of treating it as an extension of cyber.

So what does resilience look like at a high level? To me, it’s about dialing up our attention spans, our ability to think rationally, to think reflectively instead of reflexively, and raising the “bit-depth” of our decision making processes. As people work together toward dialing these up, it will allow us to reclaim declining aspects of our humanity like trust, social cohesion, curiosity, creativity, serendipity, patience, competence, privacy, and volition.

Both Anthony Vinci and Jennifer Ewbank (former CIA deputy director) make generally similar arguments that the path forward is for regular people to:

1. Understand the world through the lens of intelligence practitioners.⁴²
2. Learn practical ways to navigate this world with critical thinking.

There can be a lot of creative and situation-specific ways to introduce both of these. It ultimately comes down to choices you can make to raise awareness about the situation and to induce critical thinking. These choices can happen at work, when you’re in the company of friends and family, and in your inner monologue. The opportunity to choose is everywhere, you just have to see it.

Consider this limited set of examples to stimulate ideas for what you can do:

• Individual Choices: Remember that you should see your own mind as an “endpoint” that requires protection. Get familiar with your hardwired cognitive biases, so that you can improve resisting them. Verify information through operational testing and triangulation (e.g. checking multiple news sources). Learn about mental models⁴³ and structured analytic techniques. Read more books. Think in continuums instead of binary categories. Surround yourself with individuals who respectfully challenge what you think.⁴⁴ Use AI-enabled systems (e.g. feeds and chatbots) carefully and with healthy skepticism. Improve your sleep, diet, and exercise. This all takes effort and it’s worth acknowledging the level of privilege it takes to have the time and resources to work on these.

• Social Choices: Start conversations that circle around cognitive security across media types. For documentaries, consider The AI Doc and The Social Dilemma. For movies, Idiocracy, WALL-E, Leave the World Behind, and Good Luck, Have Fun, Don’t Die. For TV series, The Undeclared War, 3 Body Problem, and Pluribus. Perhaps you can talk about cognitive warfare at your next PowerPoint Party, or consider reading The Fourth Intelligence Revolution for your next book club.
  
  You can also be more subtle (or ambient) by providing grit to people’s views in a civil way. This involves politely asking questions like, “Why do you think that?” and “What is the evidence behind that?” This can take effort and be uncomfortable at first. But above all, meet your friends and family where they’re at, be empathetic, and actively seek out their perspectives with an open mind. When this goes well, everyone changes their mind in ways that builds consensus and strengthens relationships.⁴⁴

• Workplace Choices: Everyone with a job has some skill that they are paid to be good at, that they can start to see differently and leverage toward collective cognitive security. Maybe this is the product manager who can enable more humane design choices, or the course designer who can induce critical thinking through form (not just content), or the red team manager who can set exercise objectives to address subtle cognitive threats.
  
  Perhaps instead of infusing cognitive security into your deliverables, it becomes a more subtle part of how you shape workplace culture. In a corporate environment, that could mean encouraging thoughtful writing and allowing time for silent reflection within meetings. Helping to empower the meaningful actions of others can also go a long way. Think about it as morphing your organization instead of fighting it.

For many of us, expanding this middle part of the Venn diagram might seem unusual at first. This is normal and takes some getting used to. In a cybernetic sense, I now see expanding the middle part for this pillar as a fun mini game of discovering “actuators” that induce critical thinking (e.g. word choice), and then invoking them at the right time and place for the right audience. It’s a subtle art.

Now as rewarding as this pillar might be, it can sometimes be challenging in different ways requiring careful navigation. Here are some insights I’ve learned:

• Learning Curve. As you may have noticed in reading through the Problem Space section, there’s a real learning curve to this subject matter and it requires careful discussion. Speaking about “mind control” as a matter-of-fact to an audience that’s not ready for it might lead to counterproductive misunderstandings. It’s important to realize that there’s a path (or gradient) to understanding the extent of cognitive warfare and you have to meet people and sectors within their Overton window. For example, most companies might not actively exercise the foresight needed to defend Taiwan and Ukraine, but many of them care about topics like insider threat and workforce competency. Always remember to meet people where they’re at on this learning curve.

• Non-Partisan Solutions. I know this is tough, but for every sector, I think it is critically important that introducing cognitive warfare must be done in a non-partisan way. Moral foundations theory recognizes the psychological differences between liberals, conservatives, and libertarians. The inherent diversity in these perspectives can provide the necessary grit for developing solutions that bridges divides in ways that are difficult for adversaries to polarize. Figuring out how to lower the temperature soon⁴⁵ will help get these mindsets to work together. At a personal level, learning about the cognitive domain and the human brain has resulted in a “centering” and “de-polarizing” effect for me. A clarifying effect that I didn’t realize I needed.

• Cross-Sector Coordination. It’s not clear to me that any particular sector can productively “solo” efforts in cognitive security without dealing with hard limitations or ethical dilemmas. For government and industry, soloing risks further democratic backsliding. The risks of becoming what we oppose should emphasize the importance of trust and knowing when to pass the metaphorical basketball. I’ve previously argued that market incentives and long-term national security incentives seem to pull in opposite directions. If this resonates (as I’ve heard from others), then it’s important to be wise enough to impose some self-restraint on what commercial entities should do, despite what they could do.

Pillar 3: Foster Communities of Interest

“Teamwork makes the dream work.” Put another way, I see this last pillar as the engine that breathes life and creativity into the first two pillars. The Irish poet Pádraig Ó Tuama views creativity as a form of community building. He writes, “it is in the vulnerability and risk of cooperation that we find ourselves alive.”

Fostering communities of interest can come in different forms:

• Explore Existing Initiatives. The cognitive security community is growing. If you’re looking to learn more, I would highlight the Cognitive Security Institute (talks) and the Information Professionals Association (podcast). The MAD Warfare podcast has a fun-but-educational way of making expert perspectives accessible to a general audience. Although not explicitly focused on cognitive security, the Center for Humane Technology (podcast) promotes similar outcomes. Collecting a variety of perspectives from reputable sources like these helps stimulate ideas for degrading adversary capabilities and building resilience. Engaging this way can also provide a “refuge” for those who work in industries which are less incentivized to recognize cognitive warfare.⁴⁶

• Trusted Brainstorm Groups. In my experience, talking about cognitive security can sometimes be a delicate and vulnerable process depending on who you’re discussing with and where they are on the learning curve. But not talking about it can impede creativity and meaningful progress. My suggestion is to find some people you know who “get it” and talk with them to: build mutual and grounded understanding, trade ideas, and coordinate actions. Eventually, you might find yourself in a variety of multiple groups which helps foster creativity and avoid echo chambers.

My crude depiction of trusted brainstorm groups. The dotted lines represent their porous nature.

If the above are done right, the process of engaging will help you practice the aspects of humanity that enable functional democracies. This process won’t be smooth or efficient, but together, you’ll be able to search for objective truth, deliberate, and build consensus. There is value in allowing space for serendipity to happen. Sometimes you’ll find unlikely allies. Expect to meet people from different professions, sectors, educational backgrounds, economic status, and political mindsets. Sometimes you’ll need to build a shared language to pursue goals together (like Grace and Rocky from Project Hail Mary).

This pillar will be important for rebuilding the trust networks that allow us to pass the metaphorical basketball between each other and sectors. At a time where most people are being conditioned to act inwardly, meaningful action requires us to resist that and act for the greater good.

Concluding Remarks

If you managed to read all the way to this point: thank you. I’m deeply grateful for your time and your willingness to grapple with the subject matter.

In this essay, we’ve come to understand that competition has shifted into the cognitive domain, where our minds are literally the targets for sensemaking and control. We learned how witting and unwitting adversaries to democracy are using cybernetic principles to build digital twins of our societies, putting them on the path to orchestrate personalized ambient realities and collectively degrade our cognitive “bit-depth.” Finally, we arrived at what I see as the three pillars in the solution space and the necessary mindset shift it will take to get there.

Serious defence requires a whole-of-society effort to solve this collective action problem. We should simultaneously impose cost on adversaries (degrading their capabilities), actively become the most mature version of ourselves (build cognitive resiliency), and rebuild the trust networks that allow us to cooperate (foster communities of interest). All of this is going to take effort, creativity, trust, open-mindedness, and courage. Practicing these will allow us to take back what it means to be human in democratic societies before it’s too late.

When I was first starting to learn about cognitive security as a cyber practitioner, I was overwhelmed with how late to the game we are. As a cog in the system, I assumed meaningful action was somewhere out there beyond my control. It took some time, but I can now recognize the choices I can make within each pillar. We’ve had control all along, but we just might not have realized it. It is because we are cogs in the system that we can make unique choices no one else can.

If you work in the commercial sector, let this post serve as an updated reminder:

Regardless of what role you play in our commercial ecosystem, the decisions you make on a daily basis matter. Whether you are an executive or a frontline worker, you have the agency to make choices and those choices will signal something about your disposition.

Will your choices maximize capital in the status quo way that furthers the interests of authoritarian adversaries in their mission to erode democracy and our cognitive security? Or will your choices help further the viability and cohesion of free and open societies—even when it’s the harder thing to do?

What is your north star?

About Me

The following self-description probably diverges from the conventional “conference talk” style of writing these, but given the subject matter I’d like to write more authentically.

I’m grappling with some form of imposter syndrome. Professionally, I portray myself as an “offensive security engineer” as a way to fit in. Personally, I tend to see myself as more of a pattern-seeking social scientist at heart (who happens to know how to code). I can’t seem to avoid noticing the psychological effects adversaries produce through their technical actions.⁴⁷ The effects that many practitioners in commercial cyber security are not incentivized to notice.

As some of you know, in school I majored in Public Policy and double-minored in Psychology and Computer Science. While I’m not an expert in any of these, what this means is that I’ve had several years to ponder about: (1) the differences between autocratic and democratic systems, (2) how autocratic societies and their citizens are structurally configured to think in a non-WEIRD way, and (3) how that influences what they might see as their “offensive” use of computers. Together, I’ve found these helpful to avoid “mirror imaging.”⁴⁸

As events have unfolded over the past decade, it’s been simultaneously fascinating and frustrating to work in the cyber security industry through these lenses and I wanted to use this essay as an opportunity to develop and share some of my insights. They draw on various sources, and might be painfully obvious to some, while being newer to some of us.

I have been working on versions of this essay for the past couple of years. Given that the subject matter might sound controversial to some, I wanted to deepen my understanding and wait for the subject matter to start to become more mainstream. The spark for this research direction came around 2020 when I started to learn about OODA loops during the pandemic. Since then, I’ve published posts that should somewhat prime regular readers of this blog for the subject matter in this essay.

Lastly, you can find my interpretation of the times in which we live, here.