Introduction

A week ago, I reviewed websites that offer automatic image tagging as a service. Today, I’m going to put some of that experience to practice by building a simple Rails photo gallery from scratch, which will automatically assign tags to uploaded images and present images by tags. I’ll use Clarifai as the tagging service.

There will be many code snippets provided, so you can experiment as you go along; alternatively, there’s a repository with the app already built and ready to play with.

The goal is to build an app as simple as possible in order to keep this post reasonably short. Specifically, the app will only allow for:

Submitting a photo and automatically assigned tags to it
Presenting the list of all tags
Displaying all photos marked with a given tag
Displaying a single photo, with all tags assigned to it

To make things simple, I’ll deliberately omit such features as editing or deleting images, manually editing lists of tags, user accounts and logging in/out, or tests. These are left as a—hopefully interesting!—exercise to the reader; you’ll find some of them already present in the repo.

I assume that you have basic experience with Rails, but not necessarily with Paperclip or computer vision. Seasoned Rails developers may wish to skip the initial sections and jump straight ahead to Automatic tagging.

Prerequisites

I’ll be using Rails 5 and Paperclip. These instructions will probably work on any version of Rails that’s compatible with Paperclip (which means at least 4.2 at the time of writing), but I’ve only tested them with Rails 5.0.0.1.

Paperclip is a popular gem by Thoughtbot that allows to easily attach files to your ActiveRecord models. It has especially good support for image attachments, being able to resize or otherwise manipulate images. You’ll need to install ImageMagick (and possibly the file program if you’re not on Unix); refer to Paperclip’s README for details.

Getting started

You’re now ready to generate our app. Open up a terminal and do: rails new gallery. Once the directories are generated, edit the Gemfile and add the gems that you’ll be using:

gem 'paperclip'      # for submitting and processing images
gem 'clarifai_ruby'  # for auto-detecting image features
gem 'pure-css-rails' # for a nicer UI

group :test do
  gem 'rspec-rails'        # we like testing Rails apps with RSpec
  gem 'capybara'           # for integration specs
  gem 'launchy'            # nifty for debugging
  gem 'factory_girl_rails' # for generating test data
end

Models

The model layer of the gallery is very simple. Create three models, Photo, Tag with self-explanatory names (each with a textual name or caption), and TaggedPhoto that will hold the many-to-many relation that binds photos and tags together. From the terminal, do:

rails generate model Photo title:string
rails generate model Tag name:string
rails generate model TaggedPhoto photo:references tag:references

In addition, the Photo model will hold the images themselves. These are managed by Paperclip, which has a handy way of generating the necessary migrations:

rails generate paperclip Photo image

You’re now ready to initialize your database and create the initial schema:

rake db:create
rake db:migrate

The remaining step is to tell Rails about the relations between your models. The TaggedPhoto model already has the required belongs_to annotations, so let’s start with app/models/tag.rb:

class Tag < ApplicationRecord
  has_many :tagged_photos
  has_many :photos, through: :tagged_photos

  default_scope { order('name ASC') }
end

Specifying the default_scope is a nifty way of ensuring that collections of tags will always be returned in alphabetical order.

Now turn your attention to the Photo model in app/models/photo.rb:

class Photo < ApplicationRecord
  has_many :tagged_photos
  has_many :tags, through: :tagged_photos

  has_attached_file :image, styles: { thumbnail: "320x240>" }

  validates_attachment :image, content_type:
    { content_type: ["image/jpeg", "image/gif", "image/png"] }
end

This tells Paperclip that:

the image field in your model (mapping to the four fields in the database: image_file_name, image_content_size, image_file_size, and image_updated_at) represents an image attachment,
in addition to storing the original, it should shrink the incoming images to 320x240 if they are larger, preserving the aspect ratio.

The validator ensures that the submitted files are actually of one of the given MIME types and is mandatory: without it, Paperclip will refuse to accept attachments.

Routes

You can now specify routes for your app in config/routes.rb. Given the use cases outlined before, you can restrict yourself to resourceful routes corresponding to photos and tags:

Rails.application.routes.draw do
  resources :photos, only: [:new, :create, :index, :show]
  resources :tags, only: [:index, :show]

  get "/", to: "photos#index"
end

Note that this yields tag paths of the form /tag/1 rather than the more readable /tag/outdoor. While it would be nice to have readable URLs, this would come at a cost of slightly increased complexity, so let’s stay with numerical tag IDs.

Submitting photos

At this point, I’ve sketched out the URL scheme for our app. You can now manipulate your photos from the console, but there are no controllers to handle them with. Let’s start with the photos controller. Construct one with rails g controller Photos and fill in the blanks in app/controllers/photos_controller.rb:

class PhotosController < ApplicationController
  def index
    @photos = Photo.all
  end

  def new
    @photo = Photo.new
  end

  def show
    @photo = Photo.find(params[:id])
  end

  def create
    @photo = Photo.new(photo_params)
    if @photo.save
      redirect_to photo_path(@photo)
    else
      render :new
    end
  end

  private

  def photo_params
    params.require(:photo).permit(:image, :title)
  end
end

Note that it suffices to supply title in the permitted params, not the individual Paperclip fields.

Let’s sketch out views rendered by these controller actions. First, the submit form in app/views/photos/new.html.erb; note the Pure.css classes for some eye candy and that you tell the browser to only accept images:

<%= form_for @photo, html: { class: "pure-form pure-form-aligned" } do |f| %>
<fieldset>
  <div class="pure-control-group">
  <%= f.label :title %>
  <%= f.text_field :title %>
  </div>

  <div class="pure-control-group">
  <%= f.label :image %>
  <%= f.file_field :image, accept: "image/*" %>
  </div>

  <div class="pure-controls">
  <%= f.submit "Upload", class: "pure-button pure-button-primary" %>
  </div>
</fieldset>
<% end %>

And the corresponding views that display a single image (app/views/photos/show.html.erb):

<figure>
  <%= image_tag(@photo.image.url, alt: @photo.title) %>
  <figcaption><%= @photo.title %></figcaption>
</figure>

…and all images (app/views/photos/show.html.erb), rendered as a simple series of thumbnail image tags:

<% @photos.each do |photo| %>
  <%= link_to image_tag(photo.image.url(:thumbnail), alt: photo.title),
              photo_url(photo) %>
<% end %>

You should now be able to launch the app, navigate to http://localhost:3000/photos/new, submit a photo and see it displayed both on the list of all images and on its individual page.

Displaying tags

Again, generate a controller and fill in actions in app/controllers/tags_controller.rb:

class TagsController < ApplicationController
  def index
    @tags = Tag.all
  end

  def show
    @tag = Tag.find(params[:id])
    @photos = @tag.photos
    render 'photos/index'
  end
end

Because the show action should render a list of pictures under a given tag, the structure of its output will be the same as the list of all pictures. Thus, we reuse the photos/index view and there’s no need for a separate view for a tag’s content.

Factor out the tag list to a shared partial so that you can reuse it in individual photo pages. Create app/views/shared/_tags.html.erb:

<div class="tags">
  <ul>
    <% tags.each do |tag| %>
    <li><%= link_to tag.name, tag_path(tag) %></li>
    <% end %>
  </ul>
</div>

Now app/views/tags/index.html.erb can be a simple one-liner:

<%= render "shared/tags", tags: @tags %>

And you can revisit app/views/photos/show.html.erb to include the tag list of a photo:

<%= render "shared/tags", tags: @photo.tags %>

Automatic tagging

At this point, you have a basic working photo gallery with tags. The only thing left is to actually implement a mechanism that will add tags to a photo upon upload. This is a self-contained operation that interoperates with an external service, so good practice calls for implementing this logic as a service object.

As mentioned, you’ll be using Clarifai as a tagging service provider, but let’s abstract away from it for now. Let’s assume we have a structure defined as:

ExternalTag = Struct.new(:name, :prob)

describing a tag coming from an external service, where name describes the tag name and prob denotes probability that the tag befits the given photo. Let’s also assume we have a method, get_tags, that returns an array of such objects given a local path to the photo.

We need to set some probability cut-off point, say, 0.8. Upon tagging a Photo object, tags assigned with probability below this threshold will be discarded, and the other tags will be instantiated and associated with the given photo. In terms of code (app/services/tagging_service.rb):

class TaggingService
  CUTOFF = 0.8

  def tag(photo)
    get_tags(photo.image.path).select { |x| x.prob > CUTOFF }.each do |external_tag|
      tag = Tag.find_or_create_by(name: external_tag.name)
      tagged_photo = TaggedPhoto.create(photo: photo, tag: tag)
    end
  end
end

Time to implement get_tags and get your hands dirty. Register on the Clarifai website and create an application. You’ll get a Client ID and Client Secret, which you’ll keep in secrets.yml under the clarifai_id and clarifai_secret fields, respectively. It is a good idea to hold the production values of these in environment variables and refer to them from secrets.yml.

As mentioned, I used the clarifai_ruby gem for interop with Clarifai. To configure it, create an initializer config/initializers/clarifai.rb with:

ClarifaiRuby.configure do |config|
  config.client_id = Rails.application.secrets.clarifai_id
  config.client_secret = Rails.application.secrets.clarifai_secret
end

Clarifai’s API documentation suggests that you can either upload the raw image data via POST requests, or submit a publicly available URL of the image via a GET request. Which one to choose? While in principle the image becomes visible to the outside world once processed by Paperclip, and so you could supply @photo.image.url to Clarifai, this is problematic. It will not work in development where you don’t want to expose our app to the outside world.

Instead, just POST the image data to Clarifai. How to do this with clarifai_ruby? Unfortunately, the gem’s README doesn’t explain it, and consulting the code corroborates the suspicion that it’s not supported by the gem.

Fortunately, monkey-patching it is easy. You just need to subclass TagRequest, following the original logic of the get method but checking whether a local path is supplied instead of an URI, and acting accordingly:

require 'clarifai_ruby'

module ClarifaiRuby
  class MyTagRequest < TagRequest
    def get(image_url_or_path, opts = {})
      if image_url_or_path =~ URI::regexp
        super
      else
        body = {
          encoded_data: File.new(image_url_or_path),
          model: opts[:model]
        }
        build_request!(body, opts)
        @raw_response = @client.post(TAG_PATH, body).parsed_response
        raise RequestError.new @raw_response["status_msg"] if @raw_response["status_code"] != "OK"
        TagResponse.new(raw_response)
      end
    end
  end
end

Now your get_tags becomes simple:

def get_tags(image_path)
  response = ClarifaiRuby::MyTagRequest.new.get(image_path)
  response.tag_images.first.tags.map { |x| ExternalTag.new(x.word, x.prob) }
end

To complete the picture, you need to invoke TaggingService#tag from the create action of our photos controller. Do it after the image is saved and processed by Paperclip:

def create
  @photo = Photo.new(photo_params)
  if @photo.save
    TaggingService.new.tag(@photo)
    redirect_to photo_path(@photo)
  else
    render :new
  end
end

And that’s it!

Conclusion

In a few lines of Rails code, we have built a prototype of a smart gallery that knows what is depicted on the photos you submit to it. I hope that you have enjoyed this little journey.

Rebased Team writing about tech we use.

Languages, frameworks, libraries, tools. Certified for 0% fluff.

A Photo Gallery that Automatically Tags Your Images

Daniel Janus