Forum
I should also have said thank you for elaborating in such detail about the painstaking work involved in the creation of the data. 30-plus database tables must have been hundreds of hours of work. Are they mainly, as you said, similar to the US album sales examples you gave on the site? In other words based on market size for, say, somewhere like France and using that Top-France data as a basis? There were not many award schemes outside of US and UK in those days so that info is priceless. I’m not sure what you mean when you talk about ‘receipts’. Can you explain?
That link was a bit insecure but once I’d got in it looked fascinating. He has TWWK Sinatra EP as 100,000-plus not 120k, and well short of the 135k you have settled on. Presumably that is the best you can do without any other guidance and you have made an assumption that it was 120k on their part? They don’t say so. And your chart estimate is then 35% above the Top France figure which is kind of a big mark up when you think of it and there is the danger it gets repeated too often and the final outcomes impact badly on the CSPC results. I think sticking to 100k is better practice.
Finally, the screenshot illustrating those mid-60s US album sales for Sinatra and his daughter are intriguing. You previously mentioned that you have adjusted the methodology from the guide article you wrote on the site some years back. What has changed particularly? More allowance for certain ranges such as Top 10 or Top 40 positions or whatever? It needs updating if it is different as that was/is a core ‘Understanding’ article on Chartmasters.
Sorry for all the extra queries and questions, I just like to understand the detail - which I guess most looking at sites like this are keen to do.
You're welcome! I'll try to give hints on every topic, sorry in advance if I forget something!
I’m glad you have noticed how, like so many other popular selling websites, the traders, dealers and businesses have moved in en masse. I’m not so sure that you are quite appreciating how that is very much an issue when you are trying to establish where a record was originally bought though. Like a message in a bottle bobbing about on the ocean, nobody knows where it went during its journey until it finally washed up on a shore somewhere. But if it is found they can at least open it and find out where it came from originally.
A better comparison would be thinking about 100 bottles thrown on the ocean from Barcelona, 100 from Lille, and 100 from Gothenburg. If 90 arrive in Egypt, given the flows of water and the dates, you would most certainly do know where they came from. Of course, 10 will arrive in Cyprus, Malta, Tunisia or so, but the core of the pack will follow the organic route.
The same happens here. You mention that records can swap hands many times over. It is true when you think at it on a micro level. There were billions of records sold through, even if in absolute terms there were many international 'exchanges', in relative terms they were a tiny percentage of the market. Out of 100 records purchased in France in 1967, the huge majority remained in France.
As I said I hate the 'believe me' arguments, so I don't blame you for the conclusions. What you say in general is understandable, and without hard evidence it's difficult to be convinced that something that can theorically happen, isn't a real thing when compared to the macro jigsaw. I've a little more time this time so I went on to check examples to make it clearer.
You referred to those releases with 100s of sellers usually all coming from the country of origin. Well that is indeed only natural the bigger the number. Sinatra has less interest these days, so I thought I’d look at one of his biggest 60s hits, and in-keeping with TWWN I kept to France and looked at the huge hit the year before that was ‘Strangers In The Night’ to see if it gave any better clues as to the provenance and history of its original owners from its sellers.<a href=" removed link "> removed link
I found this EP with 1,788 owners and trusted that would have lots of sellers. It sure did, 200 copies were available to buy of which only 84 were in France with the balance of 116 elsewhere. I am absolutely convinced I am not wrong about the implications of this as you say I am. It means, as I’ve maintained, a popular collectable like this EP - and they were one of the most interesting and more affordable formats compared to LP’s for people with not too much money - was an exotic purchase for Sinatra fans around the world.
From your example it does look that way - but you picked one with the same specificities than TWWK. If, as you said, records often changed hands, and ultimately it's impossible to trace their origin, we would see this situation happen in every relevant release. I went on to check all the big releases of Strangers in the Night.
73 sellers from the US, no other country on more than 3. A clear case of a record sold in the US only, not even in Canada. The few sellers elsewhere are spread over enough for them to be expatriates, or copies sold through online outlets in recent years. Of course, due to shipping costs, when may think it doesn't prove that the country of origin is 'visible' from sellers as we talk about the US version. Let's go to European ones.
A whopping 160 sellers from the UK, no other country on more than 5. Here it's becoming clear that the situation you explain, while it feels very plausible, isn't representative. There are no crazy amount of copies exchanged internationally through time as we can see.
95 sellers from the UK, no other country higher than 2. Same conclusion.
59 sellers from Spain, no other country higher than 3.
89 sellers from Italy, next up is on 2!
As you can see, we can clearly the country of origin of all these releases. More examples:
63 sellers from Germany, but 10 in both Denmark & Switzerland, 7 in Italy, 6 in Hungary, 5 in Austria, etc. More on that below.
The one you mention, 82 sellers from France, but also 32 from Portugal, 19 from Belgium, etc.
Have we demonstrated that sellers do not tell where records were purchased? Quite the opposite. Before the EU versions that arrived in mid-1980s, some countries were responsible for filling in the holes, as not every local industry had their pressing plants. These were France, Germany, and the Netherlands. You will find countless records reflecting that situation.
Strangers in the Night was a hit all over Europe, yet the only Swiss versions you can find are from local clubs. And next to no single ever pressed by Sinatra in Austria. Same in Hungary. The reason is obvious, these countries were served with German pressings. It's no coincidence that they pop up among the sellers for that one.
Also no printed version of SITN in Portugal until 1969. Also only later reissues in Belgium. Why? Because these countries were served with French pressings. In this case we also have the French specificty regarding EPs. This was (with Portugal) the last European country to get rid of this format (Mexico and Brazil lasted longer though), yet there were still consumers interested in them across Europe. From 1966-1967 we start to see many cases of France being the lone producer for EPs, with their copies shipped across Europe.
I can understand your concerns about data being corrupted by professionals, but as you can see, these sellers are indeed a real gold mine, a representative panel of the life of all these records / versions. As someone who checked hundreds of thousands of them, there are very few exceptions, very few cases that you struggle to explain. They always fit incredibly well to who sold what / where, giving amazing insights of the globalness of artists/hits.
By the way, this conversation / example is funny to me as it brings me to when I learned the life trajectory of my parents. I'm born and raised in France from Portuguese parents, and went there every summer vacations (and still do). When I was a teenager I find their old vinyls. I was surprised because we had no record player in France until my older brother brought one, and I though my parents never ever listened to music. In truth my father was in love with Disco music when he was younger (with the J5 haircut and all), and my mother a Pink Floyd fan. So I saw all these ABBA, Bee Gees and Floyd records and saw that they were all either from France or the Netherlands. At first I though they had bought them in France, but was left bugged about the Dutch records. That's how I asked and learned that they moved in France in the 80s only. These LPs remained in Portugal ever since they left. They were all bought in Portuguese local stores, in the modest city of Viana do Castelo. They weren't imports, collectors or whatever, just the traditional releases available in every store!
Before I go, it just occurred to me that for many years Japanese records were the most revered in collecting circles for sound, packaging and so on. I haven’t looked but I would not be surprised if original Beatles releases from there weren’t very widely spread around the globe - but it won’t only mean exports, will it?
Just answering this specific question, and I'm already forgetting the details, but we can see early 70s versions kinda widely spread already elsewhere, but it's from the 1976 reissues that we clearly notice that The Beatles' Japanese records were sold in large quantities abroad (by that I mean you could find them on some retailers in other countries, not some obscur imports through unknown fan clubs or so).
I should also have said thank you for elaborating in such detail about the painstaking work involved in the creation of the data. 30-plus database tables must have been hundreds of hours of work. Are they mainly, as you said, similar to the US album sales examples you gave on the site? In other words based on market size for, say, somewhere like France and using that Top-France data as a basis? There were not many award schemes outside of US and UK in those days so that info is priceless. I’m not sure what you mean when you talk about ‘receipts’. Can you explain?
That link was a bit insecure but once I’d got in it looked fascinating. He has TWWK Sinatra EP as 100,000-plus not 120k, and well short of the 135k you have settled on. Presumably that is the best you can do without any other guidance and you have made an assumption that it was 120k on their part? They don’t say so. And your chart estimate is then 35% above the Top France figure which is kind of a big mark up when you think of it and there is the danger it gets repeated too often and the final outcomes impact badly on the CSPC results. I think sticking to 100k is better practice.
Finally, the screenshot illustrating those mid-60s US album sales for Sinatra and his daughter are intriguing. You previously mentioned that you have adjusted the methodology from the guide article you wrote on the site some years back. What has changed particularly? More allowance for certain ranges such as Top 10 or Top 40 positions or whatever? It needs updating if it is different as that was/is a core ‘Understanding’ article on Chartmasters.
Sorry for all the extra queries and questions, I just like to understand the detail - which I guess most looking at sites like this are keen to do.
Great that you discovered and liked Top-France! The numbers are rounded to the lowest 50k (25k under 100k). Ironically, the annual rankings give more precision to sales. You can see that tracks from 33 to 54 are in excess of 100k, with Sinatra ranked at 47. So it could be a little lower than 120k, a little higher, but that's the closest you can get. Sticking to 'just' 100k would grant you to be always incorrect. Also, as I said, Top-France numbers must be taken as minimums. Receipts I mentioned are net shipments data from labels, who often do not cover the entire life of the record (at most 3 semesters included, often less than that).
When digging with these somewhat obscur releases, we need to think about ballparks, and indicators. If you try to look at the real exact verified value, you will only distort the truth. That's what I was doing years ago, like keeping cells blank in countries where a record failed to chart. That's how I got almost 100m sales too low for the Beatles. In the case of TWWK, we have 3 distinct methods (Discogs' indicators, Top-France receipts, chart run coonverted) which all suggest the same ballpark in France. Then we have Discogs sellers to reflect on sales in other markets. By the way, these aren't converted with France number, but instead compared to sellers of his other hits in each country. This EP charted at #2 in the Belgian Wallonia single chart, thanks to its 2 French versions (19 owners for the EP, 32 for the single). We lack chart history from Portugal, but it was clearly a big hit with this amount of sellers, and then you have every other country as well. It's possible that this EP sold 180k, or 220k. Around 200k is the correct ballpark given the available data. If you start being 'conservative', as I said you will only distort the reality, say it's just 100k in France, be conservative again and say 10k in Belgium, 0 in Portugal as we lack data, and 0 elsewhere ignoring Discogs sellers because we never know this can be corrupted. Ultimately you will give a figure that will be off 100% of the time!
About the US album sales screenshot I'm not sure to understand you question? Anyway the method is thiner than that, I've multiple sets of 'position weights' (depending on the format, the era, etc), that will say that at #27 you sell around xx% of the market, at #28 yy%, etc. Similarly, each calendar week has a specific week (the real weekly market value when available like in the UK or some years in the US, based on patterns when real data is missing), this enables the script to account for boosts (Valentine's, Mother day, Father Day, of course Christmas) and lower periods (January, or the summer). And if a real figure is known (like a Soundscan figure), it replaces the calculated figure, with the script only calculating sales for the rest of the run. It was indeed a lot of work to set up, but it saves me a lot of time now!
Terrific responses Guillaume, so much thought has gone into the way you see Discogs data and I think you did cover everything I queried. Nevertheless, I just need to go back on one thing you said a few posts ago here:
Also, different formats aren't equally collected, EPs survived much less than singles, an equal numbers of owners for 1 EP and 1 single from the same year and the same artist would suggest about two times more sales for the EP for example. So it is by no means easy, and it requires a lot of experience. Luckily we have been digging this matter for many years now, enough to get a decent clarity out of these indicators!
Supplementary observation: how is it reasonable to assume one EP owner and one single owner establishes the EP sold double what the single sold? EP’s were far more interesting to collectors in later years because they were neat little packages usually with photos, often in colour, plus contained more music.
An equal number of owners does not mean anything other than there are the same number of items for sale. In my experience, and my preferred attitude towards collecting, because most people would prefer to keep hold of the EP than a single in a plain bag - or sometimes none at all if it didn’t survive. If I could keep something from my collection, or one or the other between a single and EP, I’d always prefer to hold on to the EP as it just looks better. Let’s face it, playing these collectibles is not really necessary these days!
It is a struggle for me to read anything into the overall sales mix from what people have for sale on trading sites. For instance I’d wager as many copies of ‘Jailhouse Rock’ as an EP are for sale than the 10-inch 78-rpm single, but clearly the latter heavily outsold the former. This is because the EP was more of a keepsake and the 78 prone to break, scratch and generally not survive.
It is reasonable - and it isn't 🙂
We do have a complete set of ratios for 25+ formats, like cassette being worth roughly 10 times more than vinyls, box sets 5 times less, etc. That's based on an empirical approach, through many observations.
But that's still only a default system that we use to draft the skeleton of estimates. As every artist has a specific audience, based on how collected he is, where he was the most popular, the age of his consumers, etc, we re-caliber it all for every record. As I said in some previous message, every piece of discography that we estimate (say album sales in Germany), we start by estimating records for which we have information (certifications, runs, or any additional insights), and then start comparing these records' discogs presence with more obscur yet similar (year, country, format) records to caliber these albums.
Also important to notice that these observations evolve through time, as Discogs get more popular in some country, or a format getting newly hyped. That's why we always need to adjust, it's a system based on the balance of the discography, individually indicators from Discogs are pretty worthless!
Ok, I think I get it now. It is very much a case of trial and error which leads to a better understanding of the data as a fluid whole rather than working on individual items. As the information builds up it becomes more intuitive, almost like AI in some kind of way?
I’m always keen to learn more. You say you have complete ratio sets for lots of formats and then explain how a cassette can be worth roughly 10 times more than the vinyl. In Sinatra’s prime it was of course mainly cartridges in the US, so you have a ratio for that too then? How does it work exactly? If you have a total of, say, 14 cassette/cartridge owners on Discogs you kind of make that the equal of 140 vinyl?
As you say, that is an ‘empirical approach’ based on thousands of observations and is, I presume, of secondary use if you have better data like good and trusted chart runs and/or awards to use first and foremost? One of the standout compilation breakdowns you have presented was of the 1966 ‘Live At The Sands’ album, but surely your US total is in error at 1,545,000 when, in your chart run attachment example you gave a few posts ago you have the release calculated at 368,235? Something is wrong here, isn’t it, or is the seeming discrepancy explained by the Discogs data ‘presence’ as you describe where more obscure releases better calibrate the result over the many years that follow?
Fascinating as always.
You are spot on with all your questions! It's indeed some kind of machine learning, 14 cassettes/cartridge would (by default) be worth in sales in the real world about as many as 140 LP owners, and correct too about the fact that we first use primary data before filling in the holes with these indicators and patterns from Discogs!
I do not have all the details in mind anymore, but I do remember being impressed at how good catalog sales of Live At The Sands were. At times depending on how a discography is built, live albums work as compilations (an obvious example used to be AC/DC's Live), and it seems LATS somewhat got that role as well. Its run do reflect 368k sales, although as I said Sinatra's sales were underrated by Billboard's charts at that point, which can also be verified by this release going Gold in late 1967.
The run / cert suggest half a million shipped (and virtually sold) by the start of 1968. Then it's when Discogs starts helping. The initial versions combine for 4,173 owners, while vinyl versions issued from 1968 onwards combine for 1,711, so we are looking at a bit more than 700,000 sales in the vinyl era. Its CD version was released before Soundscan, in 1986. It happens that thanks to several leaks of data through the years, we have a very good estimate of its Soundscan-era sales: 660,000 copies sold since 1991. Its pace of sales during the 90s (20k/year) also enable to compare Discogs' indicators to estimate its sales from 1986 to 1990 - in that case 175,000 units.
Add to that a tiny extra from club sales (0.3%, as shown by Discogs too), and we get 705 (LP era sales) + 175 (CD 86-90) + 660 (Soundscan) + 5 (Clubs) = 1,545,000 copies sold in the US. As a pre-1982 double LP, it is eligible to 3xPlatinum.
This is actually a good example at how totals are constructed, adding every piece of information together to be as comprehensive and accurate as we can!
I follow the totals construction process up until the CD era Guillaume. You say 660k is the Soundscan era total from 1991 then refer to it being 20k a year on average in the 90s, which doesn't compute! I could see how you get the Discogs ownership data to provide an estimate for the vinyl era after 1968, but try as I might I can't get the CD ownership numbers to fit in the same way.
I found various reasonable-sized ownership entries for the CD of 535, 162, and some undated ones of 92 and 267, which total 1,056, and then there are 72 owners for the 1986 CD pre-Soundscan era release and 34 for the 1990 MC (which x10 = 340 perhaps?) so maybe 412, but that doesn't equate to 175,000 units from 1986-1991 - if anything it is more!
Sorry to be so dumb, but can you just walk me through this stage again? Especially as it is a good example of pulling all the information together. It is definitely the sort of thing you should explain in one of your useful decoding articles some time too!
