The notion that we are moving in the direction of artificial general intelligence looks more like an article of faith than the conclusion of any argument or the implication of any set of observations. It was an assumption present at the birth of modern artificial intelligence, planted there by a variety of brilliant theorists and science fiction writers. Its profound influence on the culture of Silicon Valley has meant that it has played an important role in the conceptual evolution of the field. But it is not obviously justified by the evolution of the technology itself in recent years, and seems at odds with the general direction of that evolution.
The best piece I’ve read on AI in a while, probably because it confirms an inchoate suspicion that has been nagging at me. AI has been impressive in fields where progress is easily verifiable (like coding and math), and recursive self-improvement looks more tractable there. For ‘softer’ or more tacit kinds of knowledge, there’s less reason to expect the same kind of progress, unless someone finds a way to verify it. So why is the AI story presented as (and research seemingly geared toward) reaching general intelligence?
If we remain skeptical of how AI will evolve (maybe it’s AGI, maybe it’s more domain-specific intelligences), then many of the threat models and utopian hopes often ascribed to AI abstractly would need rethinking.
From what I gather, China doesn’t seem to conceive of AI competition as a race to AGI. Their approach seems much more practical and immediate, and thus better suited to domain-specific AI. Maybe the U.S. is playing the wrong game, and maybe for largely faith-based reasons. OCT 9 | REPLY
Recent open-source AI releases have crossed an important threshold of disruptive capability. On September 29, Anthropic reported that Z.ai’s GLM-5.3 approached the capability of its Mythos model at certain tasks related to cyber exploitation. Like Anthropic had done with Mythos, Z.ai at first delayed the release of GLM-5.3 to provide time to shore up cyber defenses.
But since August 28, the model has been out in the wild—its weights are freely downloadable on HuggingFace—with built-in safeguards that can be easily bypassed with creative prompting or removed altogether through a process coined “abliteration.” Some commentators have speculated this model may have been the one used to develop “WeWorm,” a critical exploit, first reported in September, capable of reaching the 800 million users of China’s most popular messaging app. […]
Beijing has ample reason to keep promoting widely available AI. Xi Jinping personally endorsed China’s open-source leadership at the World AI Conference in July. TC260’s governance framework is notably “nonbinding” and explicitly endorses an open-source AI ecosystem. But these priorities can coexist with restrictions on more dangerous capabilities—and we are likely to see these restrictions manifest soon. The likely outcome is a bifurcation in Chinese strategy: continued diffusion at the low end of the AI value chain, with Chinese labs increasingly withholding systems that Beijing regards as threats to social stability or state control.
Given sufficiently advanced cyber capabilities, I’d expect restrictions on open-weight models to become hard to avoid. There are already calls for this in the U.S. It will be interesting to see how this plays out constitutionally. My concern is that, given the national security implications, courts might rethink the First Amendment protections that now extend to code and data from cases like Bernstein and Sorrell. That could knock over a lot of the 30-plus years of ‘code is speech’ precedent. OCT 6 | REPLY
So I read the 40-page summary by the independent panel of its decision in the Manchester City case. A couple of things stood out to me that I haven’t seen discussed elsewhere.
First, there is the often-repeated line by Manchester City and its executives that it’s innocent and “a comprehensive body of irrefutable evidence exists in support of all of its positions” in the case. That struck me as odd, and I gather I wasn’t alone. What are they talking about, exactly? Well, from within the four corners of the decision summary, Man City’s explanation for the sponsorships the panel found to be shams seems to be that ADUG (the company that owns Man City) did not subsidize the payments to Man City by the sponsors. In City’s telling, that is a misunderstanding by the League. What happened, you see, is that
[Abu Dhabi (AD)] Sponsors had from time to time applied for financial assistance from the AD Government towards their liability to pay the Recorded Sponsorship Fees. Such applications had been granted, and financial assistance had been provided to the AD Sponsors by the AD Government via the [Crown Prince Court of Abu Dhabi]. In each case the level of financial assistance provided by the AD Government to the relevant Sponsor had been in the amount that the PL wrongly contended represented a Tagged Sum from ADUG.
So, it’s not that ADUG (owned by the brother of the crown prince) paid most of what the sponsors owed; it’s that, completely separately, the sponsors sought government handouts to cover their costs and those were properly granted in accordance with AD law. Of that, it seems Man City is saying, it has irrefutable proof.
Well, even if they have proof of that, that hardly exonerates them of the breaches for reasons too obvious to state. Regardless, the panel found the Crown Prince excuse to be untrue and “concluded that it was an ‘explanation’ that the Club had concocted well after the event in an attempt to obscure and conceal the realities of the Disguised Funding Scheme.”
Second, there’s an idea I’ve heard expressed, that by punishing Man City too harshly the Premier League would be shooting itself not so much in the foot as in the head, and so the clubs that comprise the league will find a way to settle the case in a relatively measured manner. I don’t know. When you read the kind of conduct they engaged in, and the scale of financial doping at stake, I can’t imagine City’s rival clubs will be able to restrain themselves. And the scale gives a clue as to why it may not be a problem.
I heard Kieran Maguire say recently that before the Abu Dhabi purchase, Man City’s average league position was 14th, and within a decade, after close to a billion pounds of illicit funds were injected, its average position was 2nd. What that suggests to me, and it’s what critics have long charged, is that City is closer to a Potemkin club than a organically built one. If City is relegated, are millions really going to stop tuning in? Wouldn’t that turn of events actually be quite an exciting twist in the narrative that is the EPL’s real product? I just don’t see there actually being substantial blowback on the league if they set fire to City and, on the flip side, a slap on the wrist would set a very bad precedent. I feel very bad for OG City fans whose club has been abused this way, but I think the end result is going to be pretty severe. SEP 30 | REPLY
In war, it might seem as if wealthy states always benefit from new military capabilities. But the reality is more complicated. If a new system is expensive, it indeed favors rich incumbents. But if it is cheap and demands large organizational changes to adopt, other countries can take advantage just as quickly—if not faster. In fact, the most powerful militaries in the world often struggle in the face of required disruptive organizational change, as their prior success has created bureaucracies and organizational biases that make adaption harder. As a result, they have a hard time integrating new capabilities that cannot be plugged into their existing structures.
Precise mass is just such a capability. The most conventionally powerful militaries are organized around small numbers of exquisite, expensive, hard-to-produce weapons and platforms such as Tomahawk missiles and F-35s. Their units train and plan around these systems. They thus cannot easily adjust to precise mass capabilities, which require different kinds of training and entirely new conceptions of how to fight. Exquisite systems, for instance, are scarce, so units must take great care not to waste systems and to minimize the risk of destruction. With precise mass, by contrast, units are supposed to move quickly and not make the perfect the enemy of the good. This helps explain why the United States has struggled to adapt to precise mass—even though the country invented precision guidance and boasts the best military in the world.
“Precise mass” weapons are low-cost, technologically advanced offensive systems that combine high-accuracy guidance with cheap, scalable manufacturing to overwhelm adversaries with volume. This is a good overview of the challenge the U.S. faces given the advent of these weapons. But I’m shocked (not shocked?) that the article treats it completely as a bureaucratic, organizational, and military-cultural issue. It doesn’t once mention the incentives of the producers and dependents of the “exquisite” and incredibly expensive weapons platforms. SEP 28 | REPLY
The problem is not that the EU and its national governments are not doing enough to spur competitiveness. It is that they are doing too much to impede it. Through the slow accretion of layers of labor and tax policy, red tape, and technology regulation, they have erected barriers to the creation of the kind of large, vibrant startup ecosystem currently thriving in the United States. This ecosystem has been the cradle of many major innovative high-tech companies, as well as those sparking innovation and growth in other sectors. If the EU does not create the conditions for such companies to develop, then the bloc’s decline may become irreversible.
The “slow accretion of layers” bit reminded me of demosclerosis and Mancur Olson’s theory of interest group rent-seeking leading to the decline of nations. So, I wondered who were the groups in favor of the EU AI Act, which the article discusses as a new millstone around Europe’s neck. Who lobbied for it and what are they protecting. Well, it seems that the only groups who were enthusiastically for the law were the European institutions themselves–the Commission, the Parliament, and Brussels-based nonprofits. Incumbent industry, and not just the tech sector, sought to soften the thing. At a certain scale I guess it’s possible the rent-seekers are the bureaucrats themselves. It makes sense that when there’s a deficit of democracy, as in the EU, the stationary bandit can forget how its bread is buttered. SEP 28 | REPLY
After the FDA drafted language to outlaw the manufacture of the mint-flavored cigarettes in 2022, it asked for public comments — and was hit with an avalanche of 175,000 messages over three months. …
Such campaigns usually have little effect on rulemaking because agencies typically consider the substance of comments rather than the volume, said Steven Balla, co-director of the Regulatory Studies Center at George Washington University. “You can imagine a case where there are a million ‘I hate this rule’ comments,” Balla said. “But the agency’s response will mostly be to the handful that are substantive.”
What’s novel about the menthol campaign is that the tactic seems to have worked.
In this case, the Biden administration was looking for an excuse to withdraw the rulemaking, which had upset the black base it needed in the upcoming election, but that just proves the strategy is worth the shot. And it seems to me that with AI, it should be relatively easy for interest groups to help citizens draft completely unique, well-argued, and (most importantly) substantive letters. Of course, on the receiving end, the agencies will want to use AI to triage a massive influx. I wonder if courts would let them since it wouldn’t be quite the same thing as using computers to count identical form letters. SEP 27 | REPLY
A decade later, we are doing it all again with artificial intelligence. We have the demands for aggressive regulation, the imperative of global coordination, and the obstacle of an untrustworthy Chinese Communist Party. Pope Leo released an encyclical. And Donald Trump is insisting that predictions of AI doom, like predictions of climate doom, are a “HOAX.”
Of course, neither climate risk nor AI risk is a hoax. But in each case the pitched battle between catastrophism and complacency has left too little room for the pragmatism with which we humans might manage to muddle through, as we typically do. Much of my own early policy work addressed this dynamic in the climate context, analyzing the underlying causes of overheated rhetoric and making the case that one could, and should, accept the physical science of rising temperatures while rejecting the poorly constructed causal chains that led toward “Hell on Earth.”
This is exactly how I feel about AI. There’s plenty about which to be skeptical, and the accelerationists discredit themselves when they dismiss all concern. But the doomers driving the anti-AI activism (and thus the public conversation) are nuts. SEP 26 | REPLY
The past six months have seen a big uptick in media attention about the dangers of artificial intelligence, and growing political concern over its consequences. This began with Anthropic’s release last April of its Mythos model, which could reportedly break into a wide range of government computer systems thought to be secure. This was followed by the Hugging Face incident, where an OpenAI test of an AI agent escaped control of its human organizers, broke into other websites, and organized a swarm of other AI agents to cover their tracks. All of this occurred against the backdrop of growing opposition to data centers, and the emergence of anti-AI wings in both the Democratic and Republican parties.
I saw much of this dynamic in crypto policy. So many shape rotators who saw every political question as an engineering problem. Either no concept of, or too literal a view of, group interest and political incentives. SEP 23 | REPLY