Classify images into ElasticSearch

Note: A more detailed version of this tutorial has been published on Elasticsearch’s blog

This tutorial sets a classification service that distinguishes among 1000 different image categories, from ‘ambulance’ to ‘paddlock’, and indexes images with their categories into an instance of ElasticSearch. For every image, the DeepDetect server can directly post and index the predicted categories into ElasticSearch. This means there’s no need for glue code in between the deep learning server and ElasticSearch.

The following presupposes that DeepDetect has been built & installed and that the tutorial on how to setup an image classifier has been completed. When done with the instructions below, you will find it easy to test other image classification models.

Setting up ElasticSearch

Unless you are using an existing instance, download ElasticSearch, untar and start with:

cd bin/
./elasticsearch

It should be now listening on localhost:9200

Categorizing images into ElasticSearch

DeepDetect supports output templates. An output template allows transforming the standard output of the DeepDetect server into any custom format. Here we use this capability to directly index the DeepDetect output into ElasticSearch.

Here is our first image:

Drawing

Let’s predict the categories for it and index them along with the image URI into ElasticSearch:

curl -XPOST "http://localhost:8080/predict" -d '{
       "service":"imageserv",
       "parameters":{
         "mllib":{
           "gpu":true
         },
         "input":{
           "width":224,
           "height":224
         },
         "output":{
           "best":3,
           "template":"{ {{#body}}{{#predictions}} \"uri\":\"{{uri}}\",\"categories\": [ {{#classes}} { \"category\":\"{{cat}}\",\"score\":{{prob}} } {{^last}},{{/last}}{{/classes}} ] {{/predictions}}{{/body}} }",
           "network":{
             "url":"http://localhost:9200/images/img",
             "http_method":"POST"
           }
         }
       },
       "data":["http://i.ytimg.com/vi/0vxOhd4qlnA/maxresdefault.jpg"]
     }'

and equivalently using the Python client:

from dd_client import DD
dd = DD('localhost')
dd.set_return_format(dd.RETURN_PYTHON)
mllib = 'caffe'
data = ['http://i.ytimg.com/vi/0vxOhd4qlnA/maxresdefault.jpg']
parameters_input = {'id':'id','separator':',',scale:True}
parameters_mllib = {'gpu':True}
parameters_output = {"best":3,"template":"{ {{#body}}{{#predictions}} \"uri\":\"{{uri}}\",\"categories\": [ {{#classes}} { \"category\":\"{{cat}}\",\"score\":{{prob}} } {{^last}},{{/last}}{{/classes}} ] {{/predictions}}{{/body}} }","network":{"url":"http://localhost:9200/images/img","http_method":"POST"}}
predict_output = dd.post_predict('imageserv',data,parameters_input,parameters_mllib,parameters_output)

which yields:

{
  "_index":"images",
  "_type":"img",
  "_id":"AVCvBfg7zqwAL3DK-gQ0",
  "_version":1,
  "created":true
}

which is the output of ElasticSearch, as reported by the DeepDetect server.

Let’s check that our image is within the index:

curl -XGET "http://localhost:9200/images/_search?q=helmet"
{
  "took":2,
  "timed_out":false,
  "_shards":{
    "total":5,
    "successful":5,
    "failed":0
  },
  "hits":{
    "total":1,
    "max_score":0.09492774,
    "hits":[
      {
        "_index":"images",
        "_type":"img",
        "_id":"AVCvBfg7zqwAL3DK-gQ0",
        "_score":0.09492774,
        "_source":{
          "uri":"http://i.ytimg.com/vi/0vxOhd4qlnA/maxresdefault.jpg",
          "categories":[
            {
              "category":"n03868863 oxygen mask",
              "score":0.225514
            },
            {
              "category":"n03127747 crash helmet",
              "score":0.209176
            },
            {
              "category":"n03379051 football helmet",
              "score":0.0739932
            }
          ]
        }
      }
    ]
  }
}

All good, the main category appears to be oxygen mask, the second one a crash helmet.

Note the two main parameters in the prediction + indexing call:

  • template within the parameters/output object: this takes a template in Mustache format. The available variables are those from the original DeepDetect output. Also, note the trick for controling the last comma;

  • network defines where and how the output should be sent. Here, the url holds the ElasticSearch resource and transport information, and http_method is set to ‘POST’. Another parameter is content_type, which default to ‘Content-Type: application/json’.

These simple two parameters can accomodate a variety of connections to external software applications, much beyond ElasticSearch.

Bulk categorization

Let’s improve on the categorization call above to categorize and index multiple images at once. We add the following image:

Drawing

For categorizing and indexing the two images at once, this time we use the ElasticSearch Bulk API:

curl -XPOST "http://localhost:8080/predict" -d '{
       "service":"imageserv",
       "parameters":{
         "mllib":{
           "gpu":true
         },
         "input":{
           "width":224,
           "height":224
         },
         "output":{
           "best":3,
           "template":"{{#body}} {{#predictions}} { \"index\": {\"_index\": \"images\", \"_type\":\"img\" } }\n {\"doc\": { \"uri\":\"{{uri}}\",\"categories\": [ {{#classes}} { \"category\":\"{{cat}}\",\"score\":{{prob}} } {{^last}},{{/last}}{{/classes}} ] } }\n {{/predictions}} {{/body}} }",
           "network":{
             "url":"http://localhost:9200/images/_bulk",
             "http_method":"POST"
           }
         }
       },
       "data":[
         "http://i.ytimg.com/vi/0vxOhd4qlnA/maxresdefault.jpg",
         "http://ak-hdl.buzzfed.com/static/enhanced/webdr05/2013/9/17/5/enhanced-buzz-1492-1379411828-15.jpg"
       ]
     }'
{
  "took":156,
  "errors":false,
  "items":[
    {
      "create":{
        "_index":"images",
        "_type":"img",
        "_id":"AVCvc16VzqwAL3DK-gQ8",
        "_version":1,
        "status":201
      }
    },
    {
      "create":{
        "_index":"images",
        "_type":"img",
        "_id":"AVCvc16VzqwAL3DK-gQ9",
        "_version":1,
        "status":201
      }
    }
  ]
}

Simple and fast! Let’s check on the index:

curl -XGET "http://localhost:9200/images/_search?q=hedgehog"
{
  "took":2,
  "timed_out":false,
  "_shards":{
    "total":5,
    "successful":5,
    "failed":0
  },
  "hits":{
    "total":1,
    "max_score":0.057534903,
    "hits":[
      {
        "_index":"images",
        "_type":"img",
        "_id":"AVCvco6uzqwAL3DK-gQ7",
        "_score":0.057534903,
        "_source": {
          "doc":{
            "uri":"http://ak-hdl.buzzfed.com/static/enhanced/webdr05/2013/9/17/5/enhanced-buzz-1492-1379411828-15.jpg",
            "categories":[
              {
              "category":"n02346627 porcupine, hedgehog","score":0.783433
              },
              {
              "category":"n02138441 meerkat, mierkat","score":0.0204417
              },
              {
              "category":"n02442845 mink","score":0.0182722
              }
            ]
          }
        }
      }
    ]
  }
}

That’s a hedgehog, fair enough.

So images can now easily be retrieved by keywords from ElasticSearch. Try it out on your own collections and applications.

Notes:

  • The deep neural net model used above has been trained on 1000 generic categories. In practice it is not accurate enough for specialized tasks. Train your own models relative to your own applications or get in touch if you have special needs. We provide a variety of other models that can suit your applications.

  • You can fill up your CPU or GPU memory with as many images as possible to categorize and index larger batches of images at once. This should allow to deal with large collections in reasonable time, especially by making use of a GPU.

  • While this short tutorial focuses on images, you can in practice rely on deep neural nets for other tasks without of much changes in the above setup and calls. Typical tasks may include text classification, sentiment analysis, data tagging, prediction & categorization, image segmentation etc…

  • There exists ways to build super-fast and accurate image to image similarity search based on similar deep neural nets, in the vein of what is described in http://www.iis.sinica.edu.tw/~kevinlin311.tw/cvprw15.pdf

  • It is possible there are better ways of crafting the ElasticSearch calls, let me know!

  • If you are interested in additional features, such as reading images directly from ElasticSearch, and other requirement for similar usages, speak up in the github issue section.

Hooking up Machine Learning / Deep Learning to fast storage and search backends such as ElasticSearch can power up many applications and pipelines in data driven tasks, let us know your thoughts, contact@deepdetect.com


DeepDetect documentation