There are action=add-src-to-address-list and action=add-dest-to-address-list in NAT, Mangle and Filter.

How can I remove an address from a list as an action too?

Wouldn't it be nice to have also action=del-src-from-address-list and action=del-dest-from-address-list?

explain the purpose of which you want it and you should be able to achieve it with other firewall rules in combination.

Most importantly - add-src can have timeout so it auto-removes after a time.

I am already using the time out option, however I was considering using some port knocking scenario to shut down access in addition to open them.

Yes, I think this would be useful for port-knocking, or when detecting that a system outside has successfully established a connection (logged in), it can be removed from the blacklist.

I do think it can be worked aorund by instead adding the address to a whitelist (and then possibly having a script come around every x minutes checking the whitelists and deleting the whitelisted entries from the blacklists). But that would be much more complicated and likely error prone