Collect what you like, I consider the coin to me collector acceptable but it will never grade. I have a 31S that looks similar, I love it....but actually has some luster you can't see in my pictures.
If I understand what you are saying, I think you are on the right track. Die polishing will change the surface of the die, which may affect metal flow. However, remember that luster is primarily from the movement of metal during striking. After a die has been polished, the luster may be slightly different, but should still be present. A more extreme form of die work is known as "basining." In this process, the entire surface of the coin is refinished or polished. This caused many of the PL Morgans you see. Basining a die would be much more likely to affect the curvature of the fields, as you mention. It would still make a cartwheel luster pattern when it strikes a coin, however, because the luster is from the movement of metal during the strike. On your coin, I still think the hairlines you see, and the lack of luster, are from cleaning and not from die polishing.
Thanks for the additional insights @BadThad and @physics-fan3.14. And I'm OK with that. It's going into a set in 2x2's anyway, so I don't want the bigger slab to store. And it turns out I have a lot more understanding of the coin than if I sent it in and just got "Unc Details" as my only feedback. Now I have a coin I can pick up and understand, at the same time I appreciate the strike and coloring. This is something that will help me that will help me understand what I see. I don't really want cleaned coins, but I have a couple of hundred that are not. I don't want them all to look the same, and prefer RB to Red for that reason. I also think this is collectible and this discussion just makes it even more interesting.