For example, as in the phrase "instability factor", here why is the noun form instability used instead of the adj. form unstable?
That is because "instability" isn't really an adjective (although several grammars I have seen would have indicated it as such). There is a very simplistic test for real adjectives that also work in many cases in English: if you can't use the suspected adjective attributively with no or only minimal change and with no change in meaning, then it aint a real adjective -- then it is often part of the noun.
For example, is "wooden" an adjective in "This is a wooden table"? Ask yourself if "This table is wooden" would have exactly the same meaning. If not, it aint an real adjective in the first sentence. As it happens, saying "this table is wooden" would be acceptable English but it doesn't have the same meaning as "this is a wooden table". Okay, that may be difficult example, because the attributive use still makes sense. Let's try an example in which the attributive use is non-sensical. Take "ballroom" in "This is ballroom dancing". Can you say "This dancing is ballroom"? No? Then it aint an adjective.
This is an instability factor x this factor is instability. Not an adjective.
This is an unstable factor x this factor is unstable. Yes, an adjective, but this doesn't mean the same as "instability factor".
And why isn't a construction site referred to as a constructing site instead?
The term "constructing site" would actually make sense, but the fact of the matter is that the other term, "construction site", is far more common (in fact, the former is also acceptable English but it is so uncommon that people will be baffled by it).
Is there any linguistic rule to follow or are they just arbitrary set phrases and we have to memorise each of them respectively?
There are rules, and there are exceptions to the rules. And there are also rules of thumb (the above rule is an example of a rule of thumb -- very simplistic, and it might or might not work in a few, some or most cases).
Looking forward to your kind replies.
My reply isn't kind. But this is just another example to show that language isn't logical, isn't it? You use the word "kind", even though you don't mean kindness.