The Trouble with High-Level Understanding

According to SilasBarta22 on LessWrong, at the highest-level of understanding:

At this stage, not only do you have good, well-connected models of reality, but they are so well-grounded, that they "regenerate" when "damaged".  That is, you weren't merely fed these wonderful models outright by some other Really Smart Being (though initially you might have been), but rather, you also consistently use a reliable method for gaining knowledge, and this method would eventually stumble upon the same model you have now, no matter how much knowledge is stripped away from it.

The trouble with this is you might be fooling yourself into thinking you have such a deep understanding and then derive conclusions based on your erroneous model that you don't think you have to check.

For example, I've read that part of the initial reaction to Starship Troopers by Robert Heinlein came from people who noticed that the book was very pro-military and then deduced from that that Heinlein must have been pro-conscription. That turned out not to be the case. I suspect that the deductions were done by people who thought they had a well-grounded understanding of “militarists.” As a result, they were confident about regenerating Heinlein's opinion of conscription. (I have earlier mentioned other examples of the same phenomenon.)