How about this - the house's appeareance would be affected, not it's constitution. So a storm would like rip away parts of thr thatching, or demolish a window. The next time the house is repaired, that would get fixed. Not really logical, but after all, we're talking graal here
