The rich are different from you and me.
A letter to the Editor in the Boston Globe points out that
. . . the opponents of wealth and inheritance taxes usually overlook the common practice of taxing working-class wealth (home ownership) every year through property taxes — even when that wealth is owed (rather than owned) as mortgage debt.
Any tax in isolation looks unfair to someone.