I say this from personal experience; I had a crank come loose like that, and when it does- it sounds just like rod knock.
One wall of the woodruff keyway was all worn down, and the key itself was all boogered up.
However, I was able to get away with what I consider a very easy repair(compared to pulling the crank out).
I removed the pulley, the rad and anything else that was in my way. Then used a MIG welder to build up the deformed/worn
wall of the keyway. Then used a die grinder to re-cut the wall to a nice vertical. Put in a new key, and never had a problem again.
This however, is just my one experience. your mileage may vary.