Skip to content

/tag endpoint

The image tagging approach of Flow assigns descriptive keywords or tags to an image for easier searching and retrieval. It involves comparing the query image to an image database to find similar images and extracting their associated tags, which are then assigned to the query image.

Inspect vocabulary

You can inspect your current vocabulary available for tag predictions. This will give you an insight into the possibilities and limitations of tagging.

http://localhost:8983/api/cores/my-collection/terms?
&terms.limit=-1
&terms.fl=labels

The answer shows the available terms in connection with the number of images tagged with this term. The returned terms are the known vocabulary and only these terms can be predicted when tagging images.

The quality of predictions rely on your ground truth data. The more terms, the bigger the vocabulary. The more images associated with a term, the more visual examples there are for that term and the better the term's content is represented.

Example vocabulary
{
  "responseHeader":{
    "status":0,
    "QTime":8},
  "terms":{
    "labels":[
      "building",181,
      "facade",171,
      "food",119,
      "yosemite",113,
      "shoe",99,
      "computer",96,
      "ball",87,
      "misc",79,
      "fish",76,
      "space",72]}}

You can selectively add data to compensate for underrepresented terms or add new terms.

Single tag

If you want to predict the category (a single tag) of an image, you can use the following configuration. The query below searches for the most similar image with an assigned category in your collection and uses its category as prediction for the query image.

If there is no relevant image in your collection or no image with an assigned category, no prediction is returned.

Predict category
http://localhost:8983/api/cores/my-collection/tag?
&tagging.input.url=http://web.com/image.jpg
&tagging.field=category
Example response
{
  "responseHeader": {
    "status": 0,
    "QTime": 126
  },
  "response": {
    "numFound": 2,
    "start": 0,
    "numFoundExact": true,
    "docs": []
  },
  "tags": [
    {
      "tag": "beach",
      "confidence": 1
    }
  ]
}

Multiple tags

This approach is similar to the one used to predict a single tag, but is extended to identify multiple relevant tags for the queried image.

To achieve good results, you need a large and diverse image database with multiple descriptive keywords per image. This approach is most suitable for professionally curated image databases like stock photo agencies have.

The field referenced by the tagging.field parameter should be multi-valued as we need multiple keywords.

The keywords that are most common in the set of relevant images are determined.

Predict tags
http://localhost:8983/api/cores/my-collection/tag?
&tagging.input.url=http://web.com/image.jpg
&tagging.field=labels
&tagging.max=4
&tagging.inspect=5
Example response
{
  "responseHeader": {
    "status": 0,
    "QTime": 126
  },
  "response": {
    "numFound": 346,
    "start": 0,
    "numFoundExact": true,
    "docs": []
  },
  "tags": [
    {
      "tag": "nature",
      "confidence": 1
    },
    {
      "tag": "sun",
      "confidence": 0.7
    },
    {
      "tag": "tree",
      "confidence": 0.5
    },
    {
      "tag": "river",
      "confidence": 0.2
    }
  ]
}

Advantages

Controlled vocabulary

The image tagging approach used in Flow ensures consistency in the tags assigned to images. The predicted tags are selected from the indexed terms, which helps to reduce ambiguity and inconsistency in the tags assigned to images. This can make it easier to search and retrieve images based on specific tags or keywords. Additionally, a controlled vocabulary can help to improve the accuracy of the tags assigned to images by limiting the scope of possible tags to a predefined set of terms.

Language independent

Since the tags are selected from the indexed terms, they naturally reflect the language the indexed images are tagged with. This can be particularly useful in applications that involve international users where each user is building the vocabulary in its own language.

Adaptive to data

The tagging approach is adaptive to data, meaning it can evolve with a changing image collection. As new images are added or old images are removed from the database, the tags associated with each image may change. This means that the tags assigned to a query image may also change over time and improve as more data is indexed.

Limitations

The tagging approach is data dependent. This means that the accuracy and effectiveness of the approach rely heavily on the quality and scope of the image collection. If the image collection is not diverse enough or does not contain enough images with relevant tags, the tagging approach may not be able to assign accurate tags to a query image.

Furthermore, if the image database is small or contains a limited number of images, the tagging approach may be less effective or not work at all, as there may not be enough similar images with relevant tags to assign to the query image.

Tagging workflow

This section explains how to avoid analyzing an image more than once and how to index predicted tags.

graph LR
  A[Client]
  B[Flow /analyze endpoint]
  C[Flow /tag endpoint]
  D[Flow /update endpoint]

  A <==>|1. Analyze Image| B
A <==>|2. Predict Tags | C
  A ==>|3. Index Json| D
  1. Analyze the image via /analyze endpoint.
  2. Use the preprocessed json as tagging input tagging.input.preprocessed to retrieve tags from /tag endpoint
  3. Enrich the doc using the preprocessed json as value of the import field and add the predicted tags values for field label. Add the doc via /update handler.

Using the above workflow the image is only analyzed once.