It's interesting to note that previous versions of the Fabricate spell did not specify raw material: "By means of this spell, the wizard is able to convert material of one sort into a product that is of the same material." Though the examples given were the same, it could be argued then that worked materials were usable to power the spell.
In 5e, the Fabricate spell actually specifies the term raw material: "You convert raw materials into products of the same material." So what is a raw material?
According to Wikipedia: "A raw material, also known as a feedstock or most correctly unprocessed material, is a basic material that is used to produce goods, finished products, energy, or intermediate materials which are feedstock for future finished products. Here, the specification is unprocessed; however, lumber (wood that has been processed into beams and planks) is considered a raw material in the Wikipedia article.
This corresponds to BusinessDictionary's definition of raw material: "Basic substance in its natural, modified, or semi-processed state, used as an input to a production process for subsequent modification or transformation into a finished good." Here, semi-processed is one of the accepted states.
It seems like bricks and planks could be argued to be raw materials so long as they had not been fashioned into a finished product, like a wall. You might also be able to argue that a pile of loose bricks retrieved from a ruined house would also be acceptable as raw material since they are no longer part of a finished product.
The nature of the fortification in your question would be important to determine whether this spell would work. Is it a natural cliff face? Yup, Fabricate away. Packed earthen ramparts? Maybe, up to DM intepretation (it's a finished product, but the earth is still raw and unprocessed). Castle wall? No, not in 5e.