I think defense is one of the few stats you can actually think about like real basketball. You want to have a good defender on the other teams top scorers so you want flexibility. If your PF has poor defense then you want to have a good defensive C, if your PG is weak defensively then you want a SG that can shift up to defend, etc. Some players can shift around on defense better than others so it's just something to consider just like you consider their positioning % when building your offensive depth.
You can definitely tell the difference in overall team defense. If you have a team of all good defenders you will see the difference and your team will limit others offensively. You can use the SimMatchup to confirm this. As with any stat though you have to consider dollars spent and not leave yourself with other weaknesses that are too many or too severe.
That is my opinion but I don't have a ton of experience. Though I have done MANY SimMatchups.