Ever since the reelection of President Obama, bureaucrats have been behaving badly. Conservatives may have steeled themselves to expect bad performance from bureaucrats at all times; but even fans of federal authority should be concerned about recent bungling and abuses.

The Affordable Care Act (“Obamacare”) required millions of Americans to sign up for health-insurance policies, but last year’s rollout of the federal website failed disastrously and has been limping ever since. The Veterans Administration turned out to be falsifying records to conceal long waiting times and poor care at hospitals for military veterans.

When it was revealed that the Internal Revenue Service was singling out conservative advocacy groups for denial of tax exempt status, even President Obama expressed concern. But his promised internal investigations—and seemingly relentless probes by House oversight committees—have failed to determine how this happened or who should be blamed. Two years of internal investigations and House oversight have left the same sort of fog over who should be blamed for the State Department’s failure to protect our ambassador and his aides from being slain by terrorists who penetrated the U.S. diplomatic compound in Benghazi.

Perhaps this should have been expected. Starting in 2011, House committees tried to find out why the Bureau of Alcohol, Tobacco and Firearms had provided guns to Mexican drug cartels. After two years of stalling and prevarication from Attorney General Eric Holder (who held authority over the ATF), the House of Representatives voted to hold him in contempt in 2013. It made no apparent difference to his conduct or his career.

Alas, there’s more. Last year, we learned that the National Security Agency allowed a very junior contractor, Edward Snowden, to gain access to millions of secret records. We still don’t know how that happened or who is to blame. But then we still don’t know how, the year before, the State Department’s secret cables could be delivered to Wikileaks by Bradley Manning, an Army private (and one so personally troubled that he sought a sex-change operation while in the Army). At the time, it was acknowledged to be the largest release of classified information in U.S. history. No senior officials have been held responsible for either of these extraordinarily damaging security failures.

Now President Obama says U.S. intelligence services were “surprised” by the emergence of the Islamic State, which, earlier this year, seized Iraq’s second largest city and much of its oil-producing territory. Perhaps the White House did receive adequate warnings. But if the President does have grounds to blame the intelligence community, he doesn’t seem to be very disturbed. In fact he seems very patient about bad advice. In 2011, Director of National Intelligence James Clapper assured Congress—as he presumably had told the President—that the Muslim Brotherhood was a “group” that was “largely secular” and had “eschewed violence.” When the Snowden leaks proved that Clapper had misinformed Congress in saying the NSA kept no records of phone calls by American citizens, he explained his false testimony as “the least untruthful answer” he could give at the time. He remains Director of National Intelligence – presumably still giving the least inaccurate guidance he can.

Not even those charged with guarding the President’s person have performed reliably. Last year, we learned that Secret Service agents had engaged in drunken revels in foreign capitals while they were supposed to be guarding the President on overseas trips. The revelations did not prompt an effective managerial shake-up, and recently Secret Service guards failed to stop a crazed intruder from climbing the White House fence and entering the executive mansion through the open front door. As more details came out, it became clear that the man had actually run through much of the building before he was finally captured by an off-duty agent who happened to be in the right place.

The first response of the Secret Service was to suggest that it needed more resources. For almost two weeks thereafter, White House spokesmen insisted the President had “full confidence” in Director Joan Pierson—up until the day she was finally removed from office (joining VA Secretary Eric Shinseki as the only top-level official to be removed for poor performance).

What’s going on? To figure this out, James Q. Wilson’s classic book Bureaucracy is still a good place to start. Its subtitle, What Government Agencies Do and Why They Do It, reflects the author’s patience with the subject. This year marks Bureaucracy’s 25th anniversary. A second edition, launched in 2000, remains in print. Alas, there is unlikely to be a revised, third edition. Wilson died in 2012 after a long academic career at Harvard, UCLA and, in his last years, at Pepperdine. Probably no successor will venture to update a book reflecting such a vast range of personal inquiry and attentive reading, and covering so many different government agencies and programs, state, local, and federal.

It’s worth saying at the outset that Bureaucracy doesn’t give much attention to “scandal.” The one index entry for that term takes readers to a passage about the Food and Drug Administration’s reluctance to approve drugs that might turn out to have dangerous side-effects and so generate public denunciations of the agency. You might infer that agencies used to be more concerned about angry comment on their performance,

Still, Wilson’s book lays out the matters that help us grasp why federal agencies have faltered so in recent years and why such dysfunction has so rarely affected the careers of the people charged with running them. Wilson’s main theme is that managing government programs is challenging. That may seem too obvious to belabor. Corporate boards, after all, approve huge salaries to managers of business operations because it makes a huge difference whether they can recruit the most capable people for such jobs.   But managing government programs is harder, in some ways, because (as Wilson emphasizes) expectations for government programs are more diverse and often more contradictory and these competing claims are often reinforced by competing lines of accountability.

This was not the traditional view of social scientists. Senator Daniel Patrick Moynihan (D-NY), in a review quoted on the book’s dust jacket, proclaimed that “Wilson is our Weber.” But the important thing about Wilson’s book is how little it owes to Max Weber, the German polymath often regarded as the founder of modern sociology. Weber considered bureaucracy a modern machine, compared to which all other governing devices are like medieval handcrafts. As Weber depicts it, bureaucracy is far more powerful, far more precise and reliable in its operations than any alternate control mechanism. But like a machine, a bureaucracy runs by its own internal logic, not as the immediate user might prefer. So the user—the purported political master—adapts to the machine rather than vice versa.

Weber’s account emphasizes the contrast between forms of government in different eras and different cultures, making “bureaucracy” appear as the inevitable expression of “rational-legal authority” in the modern West. In Weber’s version, the problems of “bureaucracy” are emblematic of what he called “the iron cage of modernity,” locking individuals into precisely defined roles.

Wilson wastes no time on such vast civilizational contrasts. He focuses on American agencies in the here and now. And he continually emphasizes choices and trade-offs, even within the givens of our contemporary democratic setting. His one chapter on foreign comparisons looks at bureaucracy in modern parliamentary democracy, with particular reference to Britain—to show that there are important differences even within broadly similar settings. (Summary: The British version relies more on professional self-direction, so is often more impervious to political challenge.) Avoiding world-historical abstractions of the Weberian sort, Wilson continually directs attention to concrete cases. Where he sees contrasts, these are unexpected differences noticeable between this and that case.

So Bureaucracy starts with three examples: armies, prisons, schools. Wilson doesn’t bother to emphasize differences among these categories. That’s blindingly obvious, if you’re not fixated on “bureaucracy” as a Weberian “ideal type.” Instead, Wilson highlights the great (and fatal) differences between the operating patterns of the French and German armies in 1940. Contrary to the national stereotype, the Germans emphasized tactical adaptation at combat level units, enabling their troops to maneuver into positions where the French least expected them. Regarding prisons, Wilson emphasizes that institutions in some American states remained peaceful and orderly while otherwise similar prisons in other states were beset by violence and mayhem. Among schools, Wilson pointed to examples where the same individual school yielded very different results, as measured by performance on standardized tests, under different principals. Institutions with comparable resources can function very differently depending on how they are run.

If this point is lost in Weberian abstractions about “bureaucracy,” it is often missed by economists, as well. They usually view government agencies as analogues to private firms, whose internal activities can be understood from the outset by a simple model of external incentives. But firms ultimately seek to maximize profits and operate under competitive pressure. It’s not easy to say what agencies “maximize” nor with whom they are, or perceive themselves to be, competing.

Wilson tries to treat government agencies on their own terms. He distinguishes the different things that different agencies are expected to accomplish—their output. Some really are expected to produce a measurable volume of individualized, self-contained tasks, such as the Social Security Administration’s delivery of checks to beneficiaries. In such “production agencies” (as Wilson calls them) managers can readily track performance. But that probably creates incentives to misreport what’s been done, if managers aren’t alert, as illustrated by the recent scandals at the Veterans Administration.

Other agencies can be readily judged by overall results, but not by toting up daily output statistics. The clearest example here is the military. Armies have to win in the field or at least not lose. But it’s hard for outsiders to judge how well they are doing from day to day or even to judge whether costs (in casualties, ours and civilian casualties on the other side) are acceptable, given the assignment. Wilson calls such agencies “craft agencies” because they rely on the specialized understandings of professionals. He makes the sensible point that more discretion is usually accorded to managers at times when achieving the main mission is of higher priority. Thus do military commanders usually loosen peacetime rules in the stress of actual combat.

Finally, there are agencies that perform jobs that are hard to measure even in results. Wilson offers the State Department as an example. Most of what foreign service officers do is observing and reporting; it’s hard to judge, for any particular officer, how the reports contribute to particular results. Wilson calls such agencies “coping agencies.” He might well have considered intelligence agencies to belong in this category.

Wilson notes that the most effective Secretaries of State—he cites Henry Kissinger as a preeminent example—often achieved their diplomatic breakthroughs by circumventing the routines and dispositions of the career bureaucracy. Perhaps our intelligence chiefs failed to give the President more reliable guidance because they were too content to summarize what staffers compiled for them. Perhaps they failed to win the President’s trust—or didn’t exert themselves to get his attention.

Wilson’s more general point is that there is usually some reason for standard operating procedures within agencies. Successful agencies develop a sense of mission and a corresponding agency culture. It makes the manager’s task much easier because it means operatives already have an implicit sense of how things should be done and where it is acceptable to take short-cuts and where it most definitely isn’t. Having a well-defined ethos also makes work easier, or at least more satisfying, for agency operatives since they then know when they are doing their job properly. So executives who try to change an agency’s sense of mission are taking risks—and often face considerable resistance.

What government agencies most often seek to maximize—if that term applies at all—is professional or institutional autonomy, Wilson says. That is, they seek to avoid provoking angry or hostile reactions, which risk triggering interventions from Congress or from higher authorities in the executive branch. Agencies tend to be self-protective and risk-averse, for good reasons and bad.

Wilson challenged the conventional wisdom of political science in the 1970s and 1980s, that agencies were most likely to adhere to the expectations placed on them by congressional oversight committees. Even specialized committees don’t usually have very well-defined expectations, he countered. And when they do, they may be resisted by other congressional committees. Wilson had no doubt Congress could get its way—but argued that it usually doesn’t know what it really wants. Often, Congress just wants to avoid blame. Constituents, or aggregates of diverse constituencies, press in different directions.

So Presidents do have considerable room to influence the operation of government agencies, but only up to a point. Congress stands in their way. The agencies themselves will resist too much White House interference. Wilson ends his chapter on Congress with the confident claim, “Congress retains enormous influence over the bureaucracy.” He concludes his chapter on Presidents with the warning: “Making the bureaucracy accountable to the president in any comprehensive or enduring way is impossible.”

Readers of Bureaucracy will gain insight into why the Department of Health and Human Services did so poorly in getting the federal health-exchange website up and running. The department’s previous duties gave it little experience with a project as ambitious as the website, which was supposed to make complex choice menus available to millions of users. Pressures from the White House accentuated the challenge. White House overseers evidently wanted the website to conceal or de-emphasize certain awkward facts that might disturb users, such as changes in the cost of insurance premiums and co-pay obligations. The White House was even less equipped than HHS to manage this kind of project.

Still, the underlying question remains: Why has there been so little consequence for poor performance? President Obama seems to have been disinclined to make too much of the botched website lest that lend weight to Republican criticism. But Republican critics were so angry at larger problems with Obamacare, it was hard for them to focus on the immediate incompetence of the HHS managers. Democrats, eager to rebut Republican charges on the larger effects of the new insurance mandates, didn’t dwell for long on the bungling of the website.

Something similar might be said of the NSA. The Snowden revelations triggered angry protests about threats to privacy. There was not much focus on the managerial incompetence that opened the way for Snowden’s internal spying. Perhaps Republican committee chairmen in the House, usually solicitous of national security programs, did not want to be seen making things harder for a beleaguered NSA.

But Republican House committees did try to fan public indignation over the ATF gun-running scandal, over the security failings at Benghazi, and over the IRS abuses. Committee hearings and reports fired up Republican partisans. But not many others. President Obama dismissed these investigations as partisan efforts to play up “phony scandals.” Most in the media decided the real story was the partisan struggle between Obama and the Republicans, not whether government agencies had or had not failed in their duties. President Obama stood by Attorney General Holder and Secretary Clinton and Director Clapper. Congressional Democrats stood by the President. Republicans barked and the media caravan moved on.

Wilson, I think, would have called attention to changes in the larger political setting.   Disputes that might have seemed to pit congressional prerogatives against executive overreach (or misconduct) are now cast as partisan or ideological, just more fussing between Fox viewers and MSNBC viewers. Perhaps that reflects the fact that the President’s party controls the Senate, while the opposing party only holds the House.   Since meaningful retaliation from Congress would require agreement between the two Houses of Congress, the President can shield agencies from paying a price for their derelictions.

It was also true in the 1980s that the President’s party controlled the Senate but not the House. That didn’t stop Congress from stirring up a lot of trouble for Reagan era officials—as Wilson’s 1989 book noticed. Perhaps the difference is that today’s Congress is not just divided along partisan lines but ideologically polarized, far more so than in the 1980s when a lot of Democratic seats were held by conservative Southerners and some Republican seats were held by relative liberals from Northeastern states. Today’s incumbents often won their seats through campaign contributions and support from out of state. Campaign donors are more often motivated by ideological commitments rather than the more concrete local interests that are more amenable to compromise.

The Obama White House has been overwhelmingly focused on managing “the narrative” or “the story”—and the story is always about Obama, not the particular agencies. The general media has embraced that script. In a curious way, conservative bloggers and talk-radio hosts in the alternative media have the same priority, which is Obama—that is, attacking him, without much real interest in actual policies or performance at the agencies. It’s been hard to get people to focus on that.

You can see the point if you look at the two exceptions. Eric Shinseki and Julia Pierson were, as I said, removed as Secretary of Veterans Affairs and director of the Secret Service, respectively. In both cases there was a rare bipartisan demand that action be taken to get these situations cleaned up. The political parties have much less agreement and the public less concern, it seems, on what they really want from HHS or the NSA.

Wilson’s Bureaucracy appeared in the same year as Taming the Prince, a study of executive power by his Harvard colleague, Harvey Mansfield. Wilson doesn’t use the same vocabulary but his account emphasizes Mansfield’s concluding observation: that a successful executive needs qualities we might call “virtue.” Wilson emphasizes that we need administrators with some sense of honor, but also some political agility and judgment. A political system that can’t secure even a minimum of administrative competence is in big trouble.

Bureaucracy is not a machine. An agency is instead a kind of community, which needs leadership, yes, but also an internal culture to reinforce norms. In the past, career civil service officials tried to restrain or blow the whistle on political appointees when the latter were thought to be mismanaging an agency. Career civil servants have less reason to try when a White House stands by the most abusive political managers, like Lois Lerner and her protective superiors at the IRS. Agency staff must be even more demoralized when White House cover stories are embraced by members of Congress from the President’s party.

It would appear that in the right political setting, federal agencies can “get away with” scandalously bad performance. The next administration will no doubt start with promises of a new beginning. But the distrust accumulated in the Obama era will hang over the next administration, be it Democratic or Republican. Who will believe the IRS is functioning honestly just because its new commissioner says so?

James Q. Wilson was, in broad terms, conservative, but not particularly libertarian. He supported anti-drug laws and emphasized the importance of cracking down on atmospheric crimes like vandalism. He was no populist, either. One of his first books, The Amateur Democrat (1962), warned against the undisciplined enthusiasms of “reformers” in politics. So while skeptical of great regulatory ventures, Wilson respected the difficulty of maintaining honest, competent administration.

Bureaucracy ends with “a few modest suggestions that might make a small difference.” Among other things, Wilson urged that agencies be given more discretion to order their own operations. The experience of Obama’s second term makes even that modest suggestion seem utopian. The last thing Obama’s successor will be inclined to do is trust career bureaucrats to clean up their own shops.

Quite apart from questions of law-breaking, federal agencies inspire less and less trust that they can do their jobs competently. We have given more and more power to federal agencies while expecting less and less in the way of competent or even conscientious performance. Congress and the President can’t agree on ways to halt crushing accumulations of debt. Nor even impart some sense of discipline to their own agents in the federal bureaucracy. Along with burdens on our economy, the Obama years have been imposing real costs on our nation’s “political capital”—the institutional capacity and trust required to get things done.