Getting better I think. This with a new "white" light and on a white background. I think I need a second lamp. I have increased the brightness with Photoscape but not changed the colour cast. Still not quite happy with the focus. I increased the lens distance a bit and used the autofocus on the camera. Maybe better under stronger light?
The white light has already made a huge difference. I agree that you need a second one. It looks like the coin might be tilted a little bit, which is going to hurt the focus. Notice that the top half of the coin appears in better focus than the bottom half. Make sure that the coin is exactly perpendicular to the lens. Moving it a little farther away should help as well, although your image will be smaller.
White isn't always "white." The definition of "white" varies, and the resource you use called "white" may or may not be more "bluish" than the next. On the Kelvin scale, by which color temperature is measured, "white" isn't at one end as it is in RGB ("black" is the total absence of light, "white" is all three colors of light equally and 100%, and every color lies between). On the Kelvin scale once you go past white you proceed into bluish tones, and some hint of blue is often added to what purports to be "white" in order to make the result appear subjectively "brighter" to the observer. "18% grey" is an absolute on any scale, and any shade of grey is easier on the camera. All that aside, I do White Balance using pure white - a stack of whatever printer paper I have laying around - and do not worry about the minor difference between the paper color and RGB 255, 255, 255. It gets me close enough to not notice the difference on a monitor which displays color rather capably, and gives me a baseline for exposure correction (I try to get the coin not washed out while still keeping the "white" visible on the screen as "white" as possible). In @PaddyB's pics above, the "white" background is kinda grey, and I attempt to make it whiter than that. Then I shoot the images you see on a black background anyways, because that's how I like it.
All that being said (and true), the original comment was about *shooting* with a neutral gray background, nothing said about setting a white balance against that. I think that's why the questions came up. That aside, the most recent photos can certainly be color-adjusted in a photo program if they were shot on an essentially white background. The background in these images looks a bit beige to me, so pulling a bit of yellow out of the image should get it very close to accurate.
There are only two reasons for any choice of background color - ease of White Balance, and subjective opinion of how you like your image to look. I based my answer on the first (and @Kentucky's reply to @Dougmeister), and you obviously based your answer on the second.
I have heard a third reason stated, that the background color will "bleed" onto the coin image. Not so sure I believe this one, but it's an interesting concept.
Everything you see in the image was reflected into the lens, and since the lighting by definition cannot be absolutely vertical, some of that background is going to reflect onto all parts of the sensor. The majority of the light might bounce off the coin, but light isn't a point source and there will be light arriving from a slightly different origin point which will bounce from the background into the lens at the exact same angle as light from the part of the bulb which illuminates the coin. The background won't "bleed into" the image, it will be an integral part of it.