1000 Words
So, I’m trying to get better at writing and I have heard that the best way to do that is by actually writing. So, the goal that I am going to work towards is writing at least 1000 words per night during the week on some topic. Tonight I am going to ramble a bit about Steam and the Web API that they provide.
Steam Web API
For those of you not aware, I built and maintain steamrecommender.com. I originally thought I had this brilliant idea because the recommendations within Steam were not that good and there were no other recommendation engines that were making use of the data that would come out of the Steam platform. So, I started working on an ingestion system that would crawl all of the Steam users (eventually). Valve conveniently developed a web api that I could make up to 100,000 requests per day.
My first idea was to generate a seed set of Steam users (which primarily consisted of my friends) and then make the right API calls to get their public details. In this case the public details consisted of all of the games that they owned with the number of minutes they have spent playing each and a list of all of their friends. The first call would be to get all of the games for a known Steam ID.
/IPlayerService/GetOwnedGames/v0001/
The second call would help me to grow the graph of Steam users that I could crawl.
/ISteamUser/GetFriendList/v0001/
The friend graph within Steam consisted of far more isolated subgraphs than I had expected when I first started looking for users. I had to quickly devise new plans to find additional users so that I could continue growing the number of players in my samples. I soon discovered the Steam communities where people could express interest in certain games and become part of their groups. At the time you could visit a page that contained a listing of every member of a group, so I would generate a seed set of popular games and wrote some code to scrape the Steam ids from these pages. Even with these additions growth was still slow. Note: Valve has since removed the listing of the groups.
Another reason that I faced slow growth in the acquisition of new Steam IDs was that I wanted to ensure that I was always working with fresh data. Things change over time, users acquire new friends and new games. Therefore, I needed to build in part of the system that would actually refresh a user every so often in order to have the latest snapshot. With the 100,000 requests per day limit some fraction of my requests were going to refreshing Steam IDs whose data needed refreshed and another fraction of these requests went to the backlog for Steam IDs that had not been loaded yet. That left very few queries for gathering new Steam IDs through the friends API.
Despite these hurdles I was able to gather enough player information to build a item-item collaborative filter that generated reasonable recommendations. This allowed me to launch the website and begin playing with improving the recommendations.
Well hell. I only made it to 550 words on my first try. Hopefully with more dedication and effort, then tomorrow I can have a better turnout.