Writing scripts to parse Heritage completed auction results

I absolutely hate sorting through completed auctions at Heritage to try and figure out what a particular coin has been selling for (if there are recent ones).

So, I thought I'd write a program to help.

Apparently, Heritage has had things like this before. They detected something (too many keystrokes too quickly? "Use of developer or inspection tools"? Unsure.

ANYWAY... I highly doubt that they would be willing to give me permission, but...

1) IF THEY WOULD... can anyone think of a scenario that they might agree to?
I can guarantee that I'm qualified.

2) If they WOULDN'T... does anyone have a better way?

Back before I learned the tools that would accomplish this, I tried copying & pasting into Excel and running VBA macros to clean up the data. I was hoping for a better, more elegant solution.

Your help would be appreciated. Thanks!

 

Log in or Sign up to hide this ad.

I have nothing useful to offer, but a lot of web sites have issues with third parties scraping data from their sites. I use an online google spreadsheet to track my portfolio and the sites I use to pull in real time data sometimes arbitrarily change their formatting, presumably to mess with freeloaders like me scraping their data.

I spend some time to fix it and carry on.

 

What are the capabilities of the programming language you are using? Is it a full-capability language like Java or C#?

Try putting a timer in between each network command to slow it down to a human speed. If you are simulating typing into a page, you may need to put some sort of Sleep() timer between keystrokes, for example. Or, if you are packaging up all the input into a few packets, make sure to wait a bit extra after receiving a response back, just as if you were a person.

I’ve tried automating a few things over the years, but not recently. I can look around through my saved code if you want to give more detail about what you are doing.

Once you slow the interactions with the web site down to a human speed, you can wander off and have some coffee, or just let it grind away in background. It doesn’t sound like it matters how long it takes to complete.

 

Python.

Good idea on the timing thing.

Thinking about reaching out to Heritage. If they could sell a service like this, would they? Would it benefit them or hurt them?

 

We know they make their data available to NGC and PCGS, who report auction prices in their population reports.

 

@Burton Strauss III , but I'll bet the TOGs pay a pretty price fur that data, or so there is some quid pro quo of some sort going on, yeah?

 

Doubt TPGs pay for prices realized data from any company. I believe some auction companies send data to TPGs free of charge. There is a bit of quid pro quo though. Auction companies often use images from TPGs in description of auction lots. Once the data is publicly available, anyone can copy and republish numerical data and the gist of text descriptions. Publicly available data can’t be copyrighted. What is proprietary and restricted is the exact format of the data and photos. When I’ve wanted to use photos from auction companies for magazine articles, they were very generous. I could use them freely as long as the companies got credited.

Heritage sued Christie’s over downloading prices realized AND images which were then used for its own business purposes. However, Heritage claimed the downloading was accomplished in part by implanting malware on its servers. If true, that would be illegal. I recall that several years ago, Heritage’s website was down for at least one day. That outage might have been to eliminate malware and institute better security. Case description:

https://ipwatchdog.com/2017/01/03/heritage-auctions-sues-christies-copyrights/#

Case was forced to arbitration because of the wording of Heritage’s own user agreement. According to Google AI, arbitrator required Christies to pay $1.8 million. Then they squabbled over payment of lawyers’ fees:

https://www.loeb.com/en/insights/publications/2018/01/heritage-capital-corporation-v-christies-inc

Not the first time Heritage went after Christie’s legally. There was an earlier lawsuit over employee poaching. These boys just can’t get along!

Auction companies probably welcome their prices realized being broadcast far and wide. It’s free publicity. Some, like Great Collections, don’t send their data to TPGs because they want folks to visit their own website. I’ve complained about this to GC because having as much data as possible in one place, like PCGS, makes it easier to do research. But they don’t want to change. However, if the TPGs (or anyone) wanted to collect the data and publish it in their own format, there’s no legal reason not to. Of course, if a big company brings a lawsuit against you, your defense can cost a lot even if the suit is baseless.

There is no reason to write scripts to obtain lot descriptions and prices realized one by one from company websites. Most of the large companies have their catalogs and prices realized in PDF format which are offered to anyone to download. They’re also available on the Newman Portal. I have digital copies of all the Heritage and SB catalogs going back decades. If I were inclined to, I could write programs to extract any of the data from the catalog and PR files that exist on my own computer. This would be much faster than trying to obtain the data over the internet.

Mike