I noticed that in a few of my Mikrotik devices I can select a higher CPU frequency to boost performance. However, when I do that I get a big "Error: CPU frequency not at default" message and so maybe it's a bad idea to do this. Is it really? I haven't noticed much of a temperature change if at all.

Well if you really don't need that extra performance out of your CPU specifically, i don't think its worth it, as it ultimately shortens the lifespan of the device.

I mean, if the CPU isn't high on temperature you won't kill it (in a few months) or anything like that, but for example when i overclock the CPU on my hap ac³, the router gets noticably hotter and i don't think that's really good for its longevity.

It's fine to run CPU overclocked as long as it's stable and doesn't get too hot. Testing it may be a bit tricky, as ROS doesn't have any memory test or benchmark.
Usually it will just not boot if you set it too high, but as always with overclocking, there is a danger of rare data corruption to occur when you are just on the edge of stability.
But as you don't have any control over CPU voltage, danger of directly damaging it by just increasing the clock is very very small. If you just need a bit of more CPU power to do what you need, it's one more option to try. Just be prepared to netinstall the device or reset it to defaults if things go wrong.