There are many places worth visiting in Sydney for history.
The Victoria Barracks and Sydney Hospital are worth visit. The Barracks were designed by a forger and built competently, while Sydney Hospital was designed by a professional and built poorly. It is still a beautiful building and worth wondering through on your way to and from the domain.
It is also lovely to walk through the courtyard of the main building at the University of Sydney with its lovely Jacaranda tree. But University of Sydney also houses a great museum in the historical Macleay Building and Museum.
A walk up to Sydney Observatory is worth doing for their lovely museum and the fact that on some nights you can look at the heavens through their historical telescope.
Parramatta has a number of lovely historical buildings including Elizabeth Farm, Experiment Farm Cottage and Government House. I loved walking past these places when I lived at Parramatta.
The Queen Victoria Building is a must see, though it dates from the late 19th century and built in the art nouveau style, it is still a lovely place to shop.
You also have the opportunity to walk around, drink, shop, eat and party in the well preserved rocks district.