ChartMasters streaming tops and tools

We can’t control the time needed by APIs to retrieve streams or a disc information from Spotify. What we can do though is minimize the required number of calls to these APIs.

In April 2020, we’ve added a local database to the design of the tool. Basically, it gives the tool a memory, which opens the door to tricks improving performances.

Dealing with the versions of each record

Although on the front end you can see an expected number of releases for each artist, the Spotify database has an awful lot of versions of every record.

Imagines Dragons for example are up to 4 studio albums. The app shows 5 versions, as their debut Night Visions has both the standard and the deluxe version available. On their database though, these albums add for 37 versions.

When we set up an automatic process, all these are retrieved, and then streams for all of them are scanned. No need to say that since these releases have mostly the exact same tracklists, most of them bring zero information about streams, as different versions of the same song add to the same tally.

Our script now learns from your search. When someone looks for an artist nobody looked for, everything is scanned, to make sure we don’t miss streams. Then, the script memorizes which records brought relevant data, and which ones barely repeated the same information.

Versions adding no data are thus excluded from later searches, even from new users. Our exclusion list is up to nearly 600,000 entries.

These versions which can be safely ignored reduce the execution time by a factor 2 to 5 depending on how many repeated releases an artist has.

Limiting interactions

In November 2020, another major improvement was developed. The architecture was newly designed to limit interactions.

To make it simple, while previously ChartMasters was doing request to an API, now the API also queries ChartMasters. This prevents ins and outs as the playcount API reads the exclusions directly.

The original tool needed roughly 25 minutes to go through Justin Bieber‘s discography when we included the ‘Appears on’ releases. Now, it scans all the results in 25 seconds.

These performance limitations prompted us to exclude the possibility of searching for orphan songs.

These are all the songs listed only under the “Appears on” section, mostly features but also primary songs that belong to Soundtracks or similar products. Often, popular artists from our era have a handful of albums, 20-ish singles and no compilation, but they appear on hundreds of compilations.

So far, we only used this extended search internally when compiling CSPC studies.

Our engine can now scan them efficiently enough to go through the largest discographies in an acceptable time, under a minute for most artists. Consequently, we decided to switch on this functionality that has been requested countless of times!

A well known issue of our former tool was that it listed all songs from an artist page.

For most artists, it’s perfectly fine.

Then there are artists like Michael Jackson, Sting and Diana Ross who’s artist page includes compilations that feature hits from their former groups.

Worst are situations of artists who for an unknown reason got a Soundtrack on their artist page.

It’s not so surprising for Lady Gaga with the A Star Is Born OST since she strongly contributes to it.

Still, there are 5 songs which feature only Bradley Cooper, hits like Maybe It’s Time and Black Eyes that add for well over 200 million streams.

More extreme was the case of the soundtracks for 13 Reasons Why. The season 2 appears on pages of Selena Gomez, OneRepublic and YUNGBLUD.

They all contribute to 1 to 2 songs only, while the album contains a streaming hit as massive as Billie Eilish‘s lovely that will soon break over the 1 billion mark.

The lifted version of our tool now controls the id of all artists featured on songs from the artist page, ultimately removing from an artist results all songs which aren’t his songs.

When we select the discography view on our tool, it’s always annoying to get many folders for one album only, because of the numerous versions. It’s impossible to automatically regroup together all songs and versions which are related to an era. They have different IDs, often slightly different titles, different release dates.

Still, the tool now includes a bit of regular expression work that manages to group together some songs that were previously split over several versions of the same album.

If we use back our Imagine Dragons example, we can now see a clean 22-songs tracklist for their debut album.

Since the data is manipulated and compared in multiple ways through the code, it wasn’t possible to stick to the original ordering of the songs. In fact, as the album folder actually regroups songs from multiple versions, and as we may exclude the very first release for performance improvement since it rarely includes all songs, it breaks the tracklist ranking.

Still, we decided to rate the performance improvement and data quality, with albums totals now more comprehensive, over the order of the data representation.

At the top of the results, we now display how many followers the artist has. This information, just like the streaming numbers, is fresh, updated daily. Followers are fans who decided to be informed of new releases from an artist.

While there is no direct link between streaming numbers and followers, having a lot of fans obviously helps in getting large numbers, especially upon release. By experience, we can say that growing artists see their followers count increase dramatically. An the other end of the spectrum, one-hit wonders are unable to gain traction in terms of fans. This statistic thus appears key to gauche which artists are here to stay.

We already explained to you that the Popularity Index is way more relevant than the Monthly Listeners figure.

To follow how strong your favorite artist is going, this index, also updated daily, is now displayed at the top of streaming results too.

As it takes a lot of daily streams to smash, this indicator fluctuates slowly. In the long run, it’s absolutely fundamental still.

Artists can decrease in spite of consistent streams, this is when an artist performs so well that he raises the 100 points scale.