Scraping your Fitocracy score with capybara-webkit
Fitocracy is a great site for tracking exercise, one which manages to have both a very friendly culture and an impressively gung-ho attitude. But they’ve never gotten around to implementing any kind of official API. If you want to look up your Fitocracy score from inside a script, you need to jump through a surprising number of hoops.
What we need is a generic web scraping tool like mechanize, but with the abilitity to deal with a rich JavaScript UI. It turns out the easiest way to do this is to use a headless web browser.
First, let’s create a Ruby Gemfile
. We’ll use capybara-webkit
, which is
normally used for testing Ruby websites:
source "https://rubygems.org"
gem "capybara"
gem "capybara-webkit"
# Optional, if you want to debug using save_and_open_page.
#gem 'launchy'
These gems can be installed using bundle install
.
Next, let’s include the necessary libraries in our Ruby script, and
configure Capybara as described in this handy article on using
Capybara in place of mechanize
:
require "capybara"
require "capybara/dsl"
require "capybara-webkit"
# Optional, if you want to debug using save_and_open_page.
#require "launchy"
Capybara.run_server = false
Capybara.current_driver = :webkit
Capybara.app_host = "https://www.fitocracy.com/"
Now, let’s use the Capybara API to drive the browser. This is slow, but acts just the way an actual user would, running JavaScript and clicking on UI elements. This saves us the trouble of figuring out the low-level HTTP API used internally, and we don’t have to set up any authentication cookies, either. If you just want to write a little personal automation, this is just fine—and it will be easy to fix when Fitocracy’s UI inevitably changes.
class FitoCrawler
include Capybara::DSL
def initialize(username, password)
visit('/')
click_on('Log in')
find('.login-username-link').click
fill_in('username-login-username', with: username)
fill_in('username-login-password', with: password)
find('#username-login-submit').click
end
def score
click_on('You')
# Want to see what's going on? Enable 'lauchy' to use this.
#save_and_open_page
# Grab the score, filter out ',', and convert to an integer.
find('#stat-points .stat-value').text.gsub(/,/, '').to_i
end
end
To use this, run:
score = FitoCrawler.new(FITOCRACY_USER, FITOCRACY_PASSWORD).score
Now that you know how to grab your score, you can easily integrate with
things like the Beeminder API or other data-tracking tools. For
example, using the beeminder
gem:
require "beeminder"
bee = Beeminder::User.new(BEEMINDER_TOKEN)
goal = bee.goal("fitocracy")
dp = Beeminder::Datapoint.new(value: score, comment: "Added automatically")
goal.add(dp)
Want to contact me about this article? Or if you're looking for something else to read, here's a list of popular posts.