Many of Port Fairy's early buildings remain from its days as a port for sealers and whalers back in the 1800s, and many of those buildings can be viewed in the town's commercial centre along Bank Street and Sackville Street. Port Fairy is home to Victoria's oldest licensed hotel, the Caledonian Inn, which dates back to 1844.
The main focal point of Port Fairy is the Moyne River as it approaches the coast. The Fishermans Wharf area along the river is lined with boats and fishing craft, and good views of river activity can be enjoyed from the footbridge over the Moyne. Another good viewing spot is from the historic fortifications at Battery Hill which is located the southern end of Griffiths Street. Views can be enjoyed over the Moyne River and to the ocean.
Griffiths Island is situated at the mouth of the Moyne River and is linked to the coast via a pedestrian causeway. The island is home to a large colony of mutton birds, while at the eastern end of the island is the Port Fairy Lighthouse. A walking track circumnavigates the island.
Port Fairy features three main beach areas. Along Ocean Drive and fronting the Southern Ocean are pockets of sandy beaches, while running parallel to the Moyne River and accessed via Beach Street is the patrolled surf beach which faces Port Fairy Bay. Secluded beaches can also be found on Griffiths Island.
Several attractive parks and gardens exist in Port Fairy including the Botanic Gardens which were first established in 1858 and the expansive Southcombe Park which fronts a significant area of the town's southern coastline. A characteristic of Port Fairy is extensive rows of Norfolk Pines which line many of the streets and the Moyne River.