A gold mining boom followed by the discovery of quartz saw the town grow significantly from the 1850s, with the population supporting 12 hotels and three banks. This boom eventually subsided with all mining operations ceasing in the very early 1900s.
A legacy to Chiltern's mining days are the well-preserved historic buildings which are built around the town's main road junction of Main Street and Conness Street, including the Federal Standard newspaper office (built in 1860), the Star Hotel (1866), Dow's Pharmacy (1860) and Gilmour's Corner Store (1890). The Chiltern Athenaeum Museum, originally the town hall and council chambers, presents a large collection of historical books and paintings.
Further north along Main Street, north of Crawford Street, is a group of historic government buildings, situated just outside of the town's commercial centre. There is a post office (built in 1863), masonic hall, the former court house (1865) and an old police lock up which was erected in 1874 at the rear of the court house.
Martin Park is located on the corner of Main Street and Alliance Street. It features a playground, pool, music bowl, BBQ facilities and shady lawns. The park extends to the shores of Lake Anderson which is an ornamental lake that was formally opened in 1876. This picturesque lake is a haven for a range of birdlife and is surrounded by open lawns, sporting facilities and a caravan park. Overlooking Lake Anderson in Victoria Street is Lake View House which was built in 1870. It was the childhood home of novel writer Ethel (Henry) Richardson and these days the residence and surrounding gardens are open to the public.
There are a number of scenic drives on the outskirts of Chiltern, many of which traverse the Chiltern-Mt Pilot National Park which surrounds much of the town. North-west of town is Donchi Hill Road which leads to a scenic lookout over the surrounding plains and passes through areas of old mining sites. South of Chiltern, on the road to Beechworth, is the turn-off to Mount Pilot where commanding views can be enjoyed from the mountain's peak. To the west of town is the Chiltern Valley where a drive along Chiltern Valley Road includes views of pastures, farmland and a vineyard.