MILES P. BERGER has just bought one of the biggest empty buildings in Newark. Soon he will take possession of the one next door. Then Mr. Berger will have two large empty buildings, only a fraction of the city's ample inventory.

But skeptics who think he has stumbled into the urban equivalent of investing in swampland should think again. Mr. Berger and his partners may be conspicuous in their daring, but they are not greenhorns. They know Newark wants to promote, at almost any cost, the physical transformation of a distressed city that produces too little tax revenue.

They also know what's taking shape just across Broad Street: the New Jersey Performing Arts Center, the state's $200 million response to Lincoln Center.

"Location, location, location," said Mr. Berger, reciting the real-estate developer's mantra and emphasizing three points.

One: All highways, railroads and flight paths in New Jersey lead to Newark, the state's largest city.

Two: The larger of his two empty buildings, the former Hahne's department store on Broad Street, had the best location in Newark when it was built in 1901, and it will again.

Three: The arts center, with a 2,700-seat concert hall and a 500-seat theater, is scheduled to open in 1995, and so is No. 609 Broad Street, the former Hahne's.

"It's no accident that Hahne's decided to build there," Mr. Berger said. "This is truly the center of Newark."

The Berger Organization -- Miles, his brother, Bruce, and his cousin Sol -- paid $2.5 million for the Hahne's building, a four-story pile of brick and limestone with an architectural style that can only be called fin de siecle mercantile. Each floor, a forest of support columns, is nearly two football fields in size. A skylight obscured by tar paper and the four-floor atrium it illuminated will be uncovered to brighten customized office space. Retail shops are planned for the first floor. The basement will accommodate 150 cars. Cost of the renovations: $25 million.

Next door, the 15-story Griffith Piano Company building, acquired at a tax auction, will provide more office space, and its recital hall will be reborn as a nightclub or theater. "A place to go after the concert," Mr. Berger said. "This is a project that will invite people outside and over here."

But "over here," a short stroll across Military Park, is the city center that did not hold, as the middle-class exodus and poor economy vacuumed commerce from its once-thriving core. Macy's finally abandoned its 16-story building on Market Street last year, joining a long and dispirited caravan of merchants who led the way out of town in the 1960's and 70's. Hahne & Company, founded here in 1858, closed its doors in 1986.

Mr. Berger is betting this will change, the way that Lincoln Center and other urban renewal projects on the Upper West Side of Manhattan helped reclaim neighborhoods and promote commerce, a process that began in the 1960's just as Newark's retreat became a rout.

That's a long time to wait, even for a methodical investor like Mr. Berger, whose company successfully restored the landmark Robert Treat Hotel and the handsome 1920's Firemen's Insurance Company building, just a brief stroll from his Broad Street buildings.

In the short run, Mr. Berger said he would do what seemed to work best in a city where private investment still relies on public money: he will seek tenants from county, state and Federal agencies, which have helped stabilize the rental market here. He will also take advantage of a tax-abatement program that has so far proved a double-edged sword for Newark and the restoration program it calls its renaissance.

Abatements have helped cultivate a swath of glass-walled office towers on the eastern edge of the original business corridor. The result is a trade-off: a patina of urban prosperity, but a citizenry still yearning for education, jobs and amenities that wealthier places take for granted.

Among these are retail stores, an endangered species here, despite the thousands of potential shoppers who fill the new office towers.