The building, located at 11-13 Randle Street, was built in 1912 as a factory for hat maker RC Henderson, who sold women's headwear around Australia and overseas.
Yesterday's fire wasn't the first to impact the factory. According to a City of Sydney heritage listing, a blaze in 1927 burnt down a nearby building and led to repairs on Randle Street.
"On 11 July 1927, the nearby Morton's printery in Chalmers Street burnt down, causing some damage to RC Henderson's millinery factory," the listing says.
"The damage and subsequent repairs mostly appeared to relate to the adjoining site at No. 7-9."
A few years later, it was the site of a crime, when a woman was arrested for robbery, including stealing the "exclusive Paris model" of a Henderson hat.
Henderson himself died in 1949, and his company went public the following year before collapsing in 1954. BB & B Ltd bought the building in 1957 before subdividing and leasing out the individual floors.
The factory continued to be used to make hats and other clothing into the 1970s, with "each floor seemingly having a different tenant", according to Sydney City archives.
Ownership changed hands once again in 1978, when Hanave Investments purchased it.
Under its ownership, one of the levels became a gym, another was partitioned, while another became an architects' office.
More recently, the building was empty but heritage listed.
However, there were plans to put the building to use once again.
In 2020, a $40 million proposal to turn the building into a 123-room hotel, first lodged in 2019, was given the tick of approval.
According to the application listing on the City of Sydney website, the development would have also included two restaurants, a small bar and a cafe, and would have also included neighbouring addresses 7-9 and 15 Randle Street.
Rubble and destruction the day after 'once in a decade' blaze
Sign up here to receive our daily newsletters and breaking news alerts, sent straight to your inbox.