How it works — music growth

Architecture

Last.fm API read-only, key auth

↓ Python snapshots

Postgres self-hosted on Hetzner

↓ dbt staging + marts

Serving layer pre-joined api views

↓ Python export to JSON

Static site HTML · Tailwind · vanilla JS

A crontab runs the pipeline every Sunday at 9 AM UTC.

The dbt DAG

Staging views normalize raw tables. Marts do the analytical joins. The api layer is narrow, pre-joined, and pre-indexed — the app never queries marts directly.

staging

stg_artist_snapshots — one row per artist per week
stg_weekly_charts — chart page/rank
stg_genre_artists — artist ↔ genre tag

intermediate

int_artist_base — profanity filter applied once, at the source

marts

artist_growth_summary — first/last listener counts, total growth
artist_chart_position — chart page/rank stats, left-joined so uncharted artists still get a row
listener_growth — week-over-week deltas
genre_stats — per-genre aggregates

api (serving layer)

api_artist_profile — one pre-joined row per artist
api_artist_timeseries — listeners indexed to 100 at first observation
api_cohort_weekly — precomputed median/p25/p75 per cohort per week

Lessons learned

Next.js → static HTML

Originally, I had Music Growth's frontend entirely vibe-coded. Next.js was (and still is) a very popular framework at the time, so I chose that and had AI go to work. My goal was to create a fun and interactive frontend that anyone could use, not something boring like Power BI. However, issues and problems kept coming up, and the AI repeatedly had to implement more and more complex solutions that added on to one another.

Although the site eventually got up and worked just fine with Next.js, I had completely lost touch with the entire web directory of my codebase. I decided that I needed to simplify the frontend into something I could understand and personalize more for myself.

I had already used the HTML / Tailwind CSS / JS stack for my portfolio site (which fetched data from JSON files), so I figured I could do the same thing here (JSON export layer mentioned below). In the end, file count decreased 83% (53 → 9) and line count decreased 92% (9,226 → 721). The site runs smoother, looks the same, and most importantly, I understand it.

Neon vs. the JSON export layer

Since the site was originally dynamic, I was fetching my data from Neon, a cloud Postgres serverless database platform. Neon worked well, but had restricting free-tier limits. I decided that, since the size of the database was very small, I could just hold it in my Hetzner instance, creating a single box that hosts my site and holds the data. That still doesn't solve the static site problem, since I wouldn't be able to query any database, even the local Postgres one on my cloud instance.

So instead, after every weekly pipeline run, a python script generates JSON files from the local Postgres database for every request that previously required an API call. Javascript can then be used to read these JSON files from disk. This resulted in a much simpler, faster, and robust site.

What the data says

Listeners vs. scrobbles

Last.fm tracks both an artist's listeners and scrobbles. Listeners are unique users who have ever played that artist. Playcount/scrobbles are the total plays across all users for that artist. The data on this site uses listeners, not scrobbles. Last.fm's charts, however, use recent scrobble activity to rank artists.

So, when this site measures "growth," what it's really measuring is new users discovering an artist for the first time, not repeated engagements.

Listener counts are cumulative

This is the main problem that led to the creation of this site. Last.fm only displays cumulative listener stats. So the only way to monitor artist growth is to track stats over time (in this site's case, weekly).

Which artists grow faster?

Overall, the findings indicate what could be expected. As of week 19 data, the median % growth among the smallest artist quintile (1-28k artists) at 0.63% and the largest quintile (293k-9.1M) at 2.12%. Interestingly, the smallest quintile average % growth exceeds 50% while the largest quintile rests around 3%, meaning that most small artists stagnate, a few explode, and bigger artists grow more reliably.

As for genres, as of week 19 data, EDM, pop, and rap have the highest median growth %.

Chart position ≠ artist size

Last.fm has an API endpoint for their chart (chart.getTopArtists), which I originally assumed meant that it ordered by popularity. However, after reviewing data gained from chart placements, I learned that Last.fm's chart orders roughly by scrobble activity. This means that an artist with millions of listeners could rank lower, while buzzing smaller artists can rank higher.

After this discovery, I started grouping artists into size groups and quintiles before comparing statistics, and learned to never assume anything about data again.

Redacted artists

Certain artists have names that contain profanities. In the interest of keeping this site pg, I use an (admittedly very basic) REGEX to filter artist names before they're even included in the database.

Currently, almost 100 names have been redacted. If you feel you or your band has not been included in this database due to improper redaction, please let me know.

If you made it to the bottom of this page, thanks for sticking around! You may want to check out my other stuff.