The relationship between art and architecture in recent decades has ranged from sympathetic complicity to outright hostility.

The artist Gordon Matta-Clark's most famous instance of what he called “anarchitecture” amounted to cutting through an abandoned house, as one would a cake, and then tilting the two halves slightly so as to reveal a slim vertical “V” of empty space between them. However physically destructive his actions toward the built environment might have appeared, it was more semiotic (exploring the interweave between positive and negative space) than condemnatory—along the lines of the Surrealist Louis Aragon's more extreme statement that, artistically, the best architecture might be achieved by applying dynamite to a church.

Though it is a stretch to say whether contemporary artists who work with architecture have, in one way or another, followed Matta-Clark or Aragon, the relationship between architecture and art over the past half-century or so—beginning in the early days of Minimalism—has become more complex and more adversarial.

The root reason is that architecture is an art form in which clients hire architectural firms to create buildings that may be functional but also embody identities and agendas. And by its very nature, architecture tends toward the dictatorial, in the sense that, once built, we can't not see it, or even, when an offending or inconvenient building rises in our own neighborhood, avoid interacting with it. Art, on the other hand, is usually proudly avoidable, tucked inside galleries and museums and, most importantly, useless. Or at least its aesthetic quality has no impact on practical utility. After all, a bad painting will cover a crack in the wall just as well as a good one.

Early in the history of modernism, however, hope for utopian collaboration between art and architecture filled the air. “Painters and sculptors, champions of the art of today . . . join together so that we might rebuild the cities,” Le Corbusier exhorted artists in 1923. “Then your works will find their place within the framework of the era, and, everywhere, you will be accepted and understood. Tell yourselves that architecture needs your attention. Be mindful of architecture.”

Such optimism is still around in places. An intensely colored wall drawing by Sol LeWitt may have a cold, algorithmic rationale behind it, but it sure does brighten up the lobby of a Manhattan apartment building. Even the politically and racially charged faux-antebellum silhouettes of Kara Walker, no matter how arresting, refrain from punching holes in the museum walls on which they're installed.

Yet art—particularly sculpture—has been getting pricklier in its attitude toward architecture. It didn't mean to, at least not in a programmatic sense. It was just that in wrenching itself away from 19th-century academicism, sculpture radically simplified the ornate (and, ironically, architectural) pedestal into simple blocks and cylinders. Then, such early-modern sculptors as Brancusi made the pedestal an integral part of the sculpture itself; successors such as Anthony Caro did away with the pedestal altogether. With the dawn of Minimalism in the mid-1960s, sculpture took on as much of the quality of modern architecture—large scale, simple forms, theatrical siting—as it could without veering into functionality. The late, rough-hewn, tons-of-steel works of Richard Serra arguably shape space in the manner of architecture with their feats of engineering.

Some contemporary artists have given themselves license to practice architecture (although precedents go back as far as 1919, when Kurt Schwitters started on his Merzbau, his house-as-assemblage masterpiece in Hanover, Germany). Rachel Whiteread built a short-lived structure-as-relief with House (1993–94). The full-size cast of the inside of an about-to-be-demolished residence in East London commented with a kind of wistful strength on the destruction necessarily preceding much new architecture; the work still paid homage to the visual dignity of the built environment. And if any of James Turrell's Skyspaces (precisely crafted ceiling/roof openings that crop the sky into crisply bordered, meditative shapes) imply that, in the end, the materiality of conventional architecture is metaphysically vulgar, they simultaneously work as beautiful examples of architecture itself.

With some notable exceptions—such as Light and Space works by Turrell, Robert Irwin, Doug Wheeler, and others, which have a plays-well-with-architecture vibe—both contemporary architecture and contemporary art have seen much of their residual utopianism evaporate in subsequent decades. Another leitmotif began to waft through the dialogue between art and architecture: hostility crept in—gently, at first. The spectator experiences, for example, Dan Graham's glass-and-mirror “pavilions”—frequently installed on the roofs or in the lobbies of museums—as being, according to the artist, “somewhere between architecture and television.” A bit of snark lurks in that last word. It implies that architecture lies on a media continuum, and can get (if it has not already gotten) coolly trivial, bereft of any lingering socially reformative drive. Graham's pavilions are International Style architecture intentionally miniaturized and made into walk-in perceptual puzzles for a few people at a time, with no real function for their form to follow.

In 1974, for his art exhibition at the Claire S. Copley Gallery in Los Angeles, Michael Asher simply removed the transverse wall in the gallery—the one visible barrier between the gallery's empty exhibition space and its small rear office. He then repainted the walls and repaired the carpet. I was there: the effect was astonishing. The gallery came visually more alive, with the clarity of being inside a simple storefront box somehow invigorating. It was like those yellow-tinted sunglasses that supposedly make everything appear sharper (at least in the infomercials), except that the colors in the gallery—people's skin, their clothes, the catalogue spines in the office, etc.—were true. Though Asher's working philosophy was almost as apolitically phenomenological as Turrell's, an implied Post-it for architects could be spied in the piece: “Folks, take away some of what you do and you can actually improve what's left behind.”

Four years earlier, Robert Smithson had attacked architecture more forcefully, albeit as a sidebar to the main concept he was exploring: Can making itself constitute sculpture? In Partially Buried Woodshed (1970) at Kent State University in Ohio, Smithson—with more than a little of Aragon's attitude—piled dirt on an abandoned shed until it partially collapsed. Today, when Smithsonesque gestures are fairly common in the edgier precincts of contemporary art, there's a term for this semi-guerilla technique: “buildering,” a combination of “building” and “bouldering,” the risky, rope-and-harness-free method of rock-climbing.

In 2004, the Spanish artist-provocateur Santiago Sierra had a Belgian museum's doors and windows taken out (after obtaining permission and removing all the works of art), so that the galleries were exposed to the elements. That same year, he strained the structural integrity of the upper floors of an Austrian kunsthalle with 300 tons of bricks, requiring additional pillars on the lower floors and, naturally, a restriction on the number of viewers.

Not all of contemporary art's antagonism toward architecture arises from a deep critical impulse. Much of it comes from the simple fact that just putting flat, rectangular paintings and moderately sized sculpture on the walls and floor of the traditional white-cube gallery won't get many artists noticed, exhibited, published, and collected in today's hyper-market-driven, showbiz-like art world. More and more, art is expected to do something directly to the architectural setting in which it's shown. Sometimes it's just cheerful, temporary augmentation with easy cleanup; Sarah Sze's intricate suspended sculpture is an example. At other times, it's not a mere usurpation of the architect's function, but an actual reassertion of art's ultimate authority over architecture, saying, in effect, that the first figurative painting on a cave wall was more important than the cave itself. In the years to come, serious architecture will be tested by how it responds to these kinds of challenges from artists.

Artists are architecture critics on the ground, operating (like architects themselves) with space and physical material instead of words after the fact. If contemporary architecture is to be more than just practical—if it is to have aesthetic and theoretical resilience—it will need to come to terms, however fractious, with art.

Peter Plagens is a painter and art critic. He was a senior writer at NEWSWEEK from 1989 to 2003.