django-elasticsearch-dsl-drf ¶

Integrate Elasticsearch DSL with Django REST framework in the shortest way possible, with least efforts possible.

Package provides views, serializers, filter backends, pagination and other handy add-ons.

You are expected to use django-elasticsearch-dsl for defining your Elasticsearch documents.

Prerequisites ¶

Django 1.8, 1.9, 1.10 and 1.11.
Python 2.7, 3.4, 3.5, 3.6
Elasticsearch 2.x, 5.x

Documentation ¶

Documentation is available on Read the Docs.

Main features and highlights ¶

Dynamic serializer for Documents.
Search filter backend.
Ordering filter backend.
Filtering filter backend (big variety of native- and functional- query lookups, such as gt, gte, lt, lte, endswith, contains, wildcard, exists, exclude, isnull, range, in, term and terms is implemented.
Geo-spatial filtering filter backend (the following filters implemented: geo_distance, geo_polygon and geo_bounding_box).
Geo-spatial ordering filter backend (the following filters implemented: geo_distance).
Faceted search filter backend.
Suggester filter backend.
Pagination (Page number and limit/offset pagination).
Ids filter backend.

Installation ¶

Install latest stable version from PyPI:

pip install django-elasticsearch-dsl-drf

or latest stable version from GitHub:

pip install https://github.com/barseghyanartur/django-elasticsearch-dsl-drf/archive/stable.tar.gz

or latest stable version from BitBucket:

pip install https://bitbucket.org/barseghyanartur/django-elasticsearch-dsl-drf/get/stable.tar.gz

Add rest_framework, django_elasticsearch_dsl and django_elasticsearch_dsl_drf to INSTALLED_APPS:

INSTALLED_APPS = (
    # ...
    # REST framework
    'rest_framework',

    # Django Elasticsearch integration
    'django_elasticsearch_dsl',

    # Django REST framework Elasticsearch integration (this package)
    'django_elasticsearch_dsl_drf',
    # ...
)

Quick start ¶

Perhaps the easiest way to get acquainted with django-elasticsearch-dsl-drf is to read the quick start tutorial.

See it as a guide of diving into integration of Elasticsearch with Django with very low knowledge entry level.

Testing ¶

Project is covered with tests.

To test with all supported Python/Django versions type:

tox

To test against specific environment, type:

tox -e py36-django110

To test just your working environment type:

./runtests.py

To run a single test in your working environment type:

./runtests.py src/django_elasticsearch_dsl_drf/tests/test_filtering.py

Or:

./manage.py test django_elasticsearch_dsl_drf.tests.test_ordering

It’s assumed that you have all the requirements installed. If not, first install the test requirements:

pip install -r examples/requirements/test.txt

Writing documentation ¶

Keep the following hierarchy.

=====
title
=====

header
======

sub-header
----------

sub-sub-header
~~~~~~~~~~~~~~

sub-sub-sub-header
^^^^^^^^^^^^^^^^^^

sub-sub-sub-sub-header
++++++++++++++++++++++

sub-sub-sub-sub-sub-header
**************************

Support ¶

For any issues contact me at the e-mail given in the Author section.

Author ¶

Artur Barseghyan <artur.barseghyan@gmail.com>

Project documentation ¶

Contents:

Table of Contents

django-elasticsearch-dsl-drf

Quick start ¶

The best way to get acquainted with django-elasticsearch-dsl-drf.

See it as a guide of diving into integration of Elasticsearch with Django with very low knowledge entry level.

Contents:

Table of Contents

Quick start

Installation ¶

Install latest stable version from PyPI:

pip install django-elasticsearch-dsl-drf

Add rest_framework, django_elasticsearch_dsl and django_elasticsearch_dsl_drf to INSTALLED_APPS:

INSTALLED_APPS = (
    # ...
    # REST framework
    'rest_framework',

    # Django Elasticsearch integration
    'django_elasticsearch_dsl',

    # Django REST framework Elasticsearch integration (this package)
    'django_elasticsearch_dsl_drf',
    # ...
)

Basic Django REST framework and django-elasticsearch-dsl configuration:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
    'DEFAULT_PAGINATION_CLASS':
        'rest_framework.pagination.PageNumberPagination',
    'PAGE_SIZE': 100,
    'ORDERING_PARAM': 'ordering',
}

# Elasticsearch configuration
ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'localhost:9200'
    },
}

Example app ¶

To get started, let’s imagine we have a simple book register with a couple of models.

Publisher model: The book publisher model. Each book might have only one publisher (ForeignKey relation).
Author model: The book author model. Each book might have unlimited number of authors (ManyToMany relation).
Tag model: The tag model. Each book might have unlimited number of tags (ManyToMany relation).
Book model: The book model.

To keep things separate, our Django models will reside in the books app. Elasticsearch documents and Django REST framework views will be defined in a search_indexes app. Both of the apps should be added to the INSTALLED_APPS.

INSTALLED_APPS = (
    # ...
    'books',  # Books application
    'search_indexes',  # Elasticsearch integration with the Django
                       # REST framework
    # ...
)

Sample models ¶

Content of the books/models.py file. Additionally, see the code comments.

Required imports ¶

Imports required for model definition.

books/models.py

import json

from django.conf import settings
from django.db import models
from django.utils.translation import ugettext, ugettext_lazy as _

from six import python_2_unicode_compatible

Book statuses ¶

books/models.py

# States indicate the publishing status of the book. Publishing might
# be in-progress, not yet published, published, rejected, etc.
BOOK_PUBLISHING_STATUS_PUBLISHED = 'published'
BOOK_PUBLISHING_STATUS_NOT_PUBLISHED = 'not_published'
BOOK_PUBLISHING_STATUS_IN_PROGRESS = 'in_progress'
BOOK_PUBLISHING_STATUS_CANCELLED = 'cancelled'
BOOK_PUBLISHING_STATUS_REJECTED = 'rejected'
BOOK_PUBLISHING_STATUS_CHOICES = (
    (BOOK_PUBLISHING_STATUS_PUBLISHED, "Published"),
    (BOOK_PUBLISHING_STATUS_NOT_PUBLISHED, "Not published"),
    (BOOK_PUBLISHING_STATUS_IN_PROGRESS, "In progress"),
    (BOOK_PUBLISHING_STATUS_CANCELLED, "Cancelled"),
    (BOOK_PUBLISHING_STATUS_REJECTED, "Rejected"),
)
BOOK_PUBLISHING_STATUS_DEFAULT = BOOK_PUBLISHING_STATUS_PUBLISHED

Publisher model ¶

books/models.py

@python_2_unicode_compatible
class Publisher(models.Model):
    """Publisher."""

    name = models.CharField(max_length=30)
    address = models.CharField(max_length=50)
    city = models.CharField(max_length=60)
    state_province = models.CharField(max_length=30)
    country = models.CharField(max_length=50)
    website = models.URLField()
    latitude = models.DecimalField(null=True,
                               blank=True,
                               decimal_places=15,
                               max_digits=19,
                               default=0)
    longitude = models.DecimalField(null=True,
                                    blank=True,
                                    decimal_places=15,
                                    max_digits=19,
                                    default=0)

    class Meta(object):
        """Meta options."""

        ordering = ["id"]

    def __str__(self):
        return self.name

    @property
    def location_field_indexing(self):
        """Location for indexing.

        Used in Elasticsearch indexing/tests of `geo_distance` native filter.
        """
        return {
            'lat': self.latitude,
            'lon': self.longitude,
        }

Author model ¶

books/models.py

@python_2_unicode_compatible
class Author(models.Model):
    """Author."""

    salutation = models.CharField(max_length=10)
    name = models.CharField(max_length=200)
    email = models.EmailField()
    headshot = models.ImageField(upload_to='authors', null=True, blank=True)

    class Meta(object):
        """Meta options."""

        ordering = ["id"]

    def __str__(self):
        return self.name

Tag model ¶

books/models.py

class Tag(models.Model):
    """Simple tag model."""

    title = models.CharField(max_length=255, unique=True)

    class Meta(object):
        """Meta options."""

        verbose_name = _("Tag")
        verbose_name_plural = _("Tags")

    def __str__(self):
        return self.title

Book model ¶

books/models.py

@python_2_unicode_compatible
class Book(models.Model):
    """Book."""

    title = models.CharField(max_length=100)
    description = models.TextField(null=True, blank=True)
    summary = models.TextField(null=True, blank=True)
    authors = models.ManyToManyField('books.Author', related_name='books')
    publisher = models.ForeignKey(Publisher, related_name='books')
    publication_date = models.DateField()
    state = models.CharField(max_length=100,
                             choices=BOOK_PUBLISHING_STATUS_CHOICES,
                             default=BOOK_PUBLISHING_STATUS_DEFAULT)
    isbn = models.CharField(max_length=100, unique=True)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    pages = models.PositiveIntegerField(default=200)
    stock_count = models.PositiveIntegerField(default=30)
    tags = models.ManyToManyField('books.Tag',
                                  related_name='books',
                                  blank=True)

    class Meta(object):
        """Meta options."""

        ordering = ["isbn"]

    def __str__(self):
        return self.title

    # The only publisher information we're going to need in our document
    # is the publisher name. Since publisher isn't a required field,
    # we define a properly on a model level to avoid indexing errors on
    # non-existing relation.
    @property
    def publisher_indexing(self):
        """Publisher for indexing.

        Used in Elasticsearch indexing.
        """
        if self.publisher is not None:
            return self.publisher.name

    # As of tags, again, we only need a flat list of tag names, on which
    # we can filter. Therefore, we define a properly on a model level,
    # which will return a JSON dumped list of tags relevant to the
    # current book model object.
    @property
    def tags_indexing(self):
        """Tags for indexing.

        Used in Elasticsearch indexing.
        """
        return [tag.title for tag in self.tags.all()]

Admin classes ¶

This is just trivial. A couple of correspondent admin classes in order to ba able to fill some data.

books/admin.py

from django.contrib import admin

from .models import *


@admin.register(Book)
class BookAdmin(admin.ModelAdmin):
    """Book admin."""

    list_display = ('title', 'isbn', 'price', 'publication_date')
    search_fields = ('title',)
    filter_horizontal = ('authors', 'tags',)


@admin.register(Author)
class AuthorAdmin(admin.ModelAdmin):
    """Author admin."""

    list_display = ('name', 'email',)
    search_fields = ('name',)


@admin.register(Publisher)
class PublisherAdmin(admin.ModelAdmin):
    """Publisher admin."""

    list_display = ('name',)
    search_fields = ('name',)


@admin.register(Tag)
class TagAdmin(admin.ModelAdmin):
    """Tag admin."""

    list_display = ('title',)
    search_fields = ('title',)

Create database tables ¶

For now, just run the migrations to create the database tables.

./manage.py makemigrations books
./manage.py migrate books

Fill in some data ¶

If you have followed the instructions, you should now be able to log into the Django admin and create a dozen of Book/Author/Publisher/Tag records in admin.

http://localhost:8000/admin/books/publisher/
http://localhost:8000/admin/books/author/
http://localhost:8000/admin/books/tag/
http://localhost:8000/admin/books/book/

Once you’ve done that, proceed to the next step.

Sample document ¶

In Elasticsearch, a document is a basic unit of information that can be indexed. For example, you can have a document for a single customer, another document for a single product, and yet another for a single order. This document is expressed in JSON (JavaScript Object Notation) which is an ubiquitous internet data interchange format.

Within an index/type, you can store as many documents as you want. Note that although a document physically resides in an index, a document actually must be indexed/assigned to a type inside an index.

Simply said, you could see an Elasticsearch index as a database and a document as a database table (which makes a Document definition in Elasticsearch DSL similar to a Django Model definition).

Often, complex SQL model structures are flatterned in Elasticsearch indexes/documents. Nested relations are denormalized.

In our example, all 4 models (Author, Publisher, Tag, Book) would be flatterned into a single BookDocument, which would hold all the required information.

Content of the search_indexes/documents/book.py file. Additionally, see the code comments.

Required imports ¶

search_indexes/documents/book.py

from django.conf import settings
from django_elasticsearch_dsl import DocType, Index, fields
from elasticsearch_dsl import analyzer

from books.models import Book

Index definition ¶

To separate dev/test/staging/production indexes, the following approach is recommended.

Settings ¶

settings/base.py

# Name of the Elasticsearch index
ELASTICSEARCH_INDEX_NAMES = {
    'search_indexes.documents.book': 'book',
    'search_indexes.documents.publisher': 'publisher',
}

settings/testing.py

# Name of the Elasticsearch index
ELASTICSEARCH_INDEX_NAMES = {
    'search_indexes.documents.book': 'test_book',
    'search_indexes.documents.publisher': 'test_publisher',
}

settings/production.py

# Name of the Elasticsearch index
ELASTICSEARCH_INDEX_NAMES = {
    'search_indexes.documents.book': 'prod_book',
    'search_indexes.documents.publisher': 'prod_publisher',
}

Document index ¶

search_indexes/documents/books.py

# Name of the Elasticsearch index
INDEX = Index(settings.ELASTICSEARCH_INDEX_NAMES[__name__])

# See Elasticsearch Indices API reference for available settings
INDEX.settings(
    number_of_shards=1,
    number_of_replicas=1
)

Custom analyzers ¶

html_strip = analyzer(
    'html_strip',
    tokenizer="standard",
    filter=["standard", "lowercase", "stop", "snowball"],
    char_filter=["html_strip"]
)

Document definition ¶

search_indexes/documents/book.py

@INDEX.doc_type
class BookDocument(DocType):
    """Book Elasticsearch document."""

    id = fields.IntegerField(attr='id')

    title = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    description = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    summary = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    publisher = fields.StringField(
        attr='publisher_indexing',
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    publication_date = fields.DateField()

    state = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    isbn = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    price = fields.FloatField()

    pages = fields.IntegerField()

    stock_count = fields.IntegerField()

    tags = fields.StringField(
        attr='tags_indexing',
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword', multi=True),
            'suggest': fields.CompletionField(multi=True),
        },
        multi=True
    )

    class Meta(object):
        """Meta options."""

        model = Book  # The model associate with this DocType

Syncing Django’s database with Elasticsearch indexes ¶

So far, we have a couple of Django models and a single (decentralized) Elasticsearch index/document (Book).

Full database sync ¶

The excellent django-elasticsearch-dsl library makes a good job of keeping the Book index fresh. It makes use of signals, so whenever the Book model is changed, the correspondent BookDocument indexes would be updated.

To simply run the full sync between Django’s database and Elasticsearch, do as follows:

Create Elasticsearch indexes:
```
./manage.py search_index --create -f
```
Sync the data:
```
./manage.py search_index --populate -f
```

However, in case if a Tag, Publisher or Author models change, the Book index would not be automatically updated.

Sample partial sync (using custom signals)¶

In order to keep indexes fresh, you will have to write a couple of simple lines of code (using Django’s signals). Whenever a change is made to any of the Tag, Publisher or Author models, we’re going to update the correspondent BookDocument index.

Required imports ¶

search_indexes/signals.py

from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver

from django_elasticsearch_dsl.registries import registry

Sample serializer ¶

At this step we’re going to define a serializer to be used in the Django REST framework ViewSet.

Content of the search_indexes/serializers.py file. Additionally, see the code comments.

Required imports ¶

search_indexes/serializers.py

import json

from rest_framework import serializers
from django_elasticsearch_dsl_drf.serializers import DocumentSerializer

from .documents import BookDocument

Serializer definition ¶

Simplest way to create a serializer, is to just specify which fields are needed to be serialized and leave it further to the dynamic serializer.

search_indexes/serializers.py

class BookDocumentSerializer(DocumentSerializer):
    """Serializer for the Book document."""

    tags = serializers.SerializerMethodField()

    class Meta(object):
        """Meta options."""

        # Specify the correspondent document class
        document = BookDocument

        # List the serializer fields. Note, that the order of the fields
        # is preserved in the ViewSet.
        fields = (
            'id',
            'title',
            'description',
            'summary',
            'publisher',
            'publication_date',
            'state',
            'isbn',
            'price',
            'pages',
            'stock_count',
            'tags',
        )

    def get_tags(self, obj):
        """Get tags."""
        if obj.tags:
            return list(obj.tags)
        else:
            return []

However, if dynamic serializer doesn’t work for your or you want to customize too many things, you are free to use standard Serializer class of the Django REST framework.

search_indexes/serializers.py

class BookDocumentSerializer(serializers.Serializer):
    """Serializer for the Book document."""

    id = serializers.IntegerField(read_only=True)

    title = serializers.CharField(read_only=True)
    description = serializers.CharField(read_only=True)
    summary = serializers.CharField(read_only=True)

    publisher = serializers.CharField(read_only=True)
    publication_date = serializers.DateField(read_only=True)
    state = serializers.CharField(read_only=True)
    isbn = serializers.CharField(read_only=True)
    price = serializers.FloatField(read_only=True)
    pages = serializers.IntegerField(read_only=True)
    stock_count = serializers.IntegerField(read_only=True)
    tags = serializers.SerializerMethodField()

    class Meta(object):
        """Meta options."""

        # List the serializer fields. Note, that the order of the fields
        # is preserved in the ViewSet.
        fields = (
            'id',
            'title',
            'description',
            'summary',
            'publisher',
            'publication_date',
            'state',
            'isbn',
            'price',
            'pages',
            'stock_count',
            'tags',
        )

    def get_tags(self, obj):
        """Get tags."""
        if obj.tags:
            return list(obj.tags)
        else:
            return []

ViewSet definition ¶

At this step, we’re going to define Django REST framework ViewSets.

Content of the search_indexes/viewsets.py file. Additionally, see the code comments.

Required imports ¶

search_indexes/viewsets.py

from django_elasticsearch_dsl_drf.constants import (
    LOOKUP_FILTER_TERMS,
    LOOKUP_FILTER_RANGE,
    LOOKUP_FILTER_PREFIX,
    LOOKUP_FILTER_WILDCARD,
    LOOKUP_QUERY_IN,
    LOOKUP_QUERY_GT,
    LOOKUP_QUERY_GTE,
    LOOKUP_QUERY_LT,
    LOOKUP_QUERY_LTE,
    LOOKUP_QUERY_EXCLUDE,
)
from django_elasticsearch_dsl_drf.filter_backends import (
    FilteringFilterBackend,
    IdsFilterBackend,
    OrderingFilterBackend,
    DefaultOrderingFilterBackend,
    SearchFilterBackend,
)
from django_elasticsearch_dsl_drf.views import BaseDocumentViewSet

from .documents import BookDocument, PublisherDocument
from .serializers import BookDocumentSerializer

ViewSet definition ¶

search_indexes/viewsets.py

class BookDocumentView(BaseDocumentViewSet):
    """The BookDocument view."""

    document = BookDocument
    serializer_class = BookDocumentSerializer
    lookup_field = 'id'
    filter_backends = [
        FilteringFilterBackend,
        IdsFilterBackend,
        OrderingFilterBackend,
        DefaultOrderingFilterBackend,
        SearchFilterBackend,
    ]
    # Define search fields
    search_fields = (
        'title',
        'description',
        'summary',
    )
    # Define filter fields
    filter_fields = {
        'id': {
            'field': 'id',
            # Note, that we limit the lookups of id field in this example,
            # to `range`, `in`, `gt`, `gte`, `lt` and `lte` filters.
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
            ],
        },
        'title': 'title.raw',
        'publisher': 'publisher.raw',
        'publication_date': 'publication_date',
        'state': 'state.raw',
        'isbn': 'isbn.raw',
        'price': {
            'field': 'price.raw',
            # Note, that we limit the lookups of `price` field in this
            # example, to `range`, `gt`, `gte`, `lt` and `lte` filters.
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
            ],
        },
        'pages': {
            'field': 'pages',
            # Note, that we limit the lookups of `pages` field in this
            # example, to `range`, `gt`, `gte`, `lt` and `lte` filters.
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
            ],
        },
        'stock_count': {
            'field': 'stock_count',
            # Note, that we limit the lookups of `stock_count` field in
            # this example, to `range`, `gt`, `gte`, `lt` and `lte`
            # filters.
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
            ],
        },
        'tags': {
            'field': 'tags',
            # Note, that we limit the lookups of `tags` field in
            # this example, to `terms, `prefix`, `wildcard`, `in` and
            # `exclude` filters.
            'lookups': [
                LOOKUP_FILTER_TERMS,
                LOOKUP_FILTER_PREFIX,
                LOOKUP_FILTER_WILDCARD,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_EXCLUDE,
            ],
        },
        'tags.raw': {
            'field': 'tags.raw',
            # Note, that we limit the lookups of `tags.raw` field in
            # this example, to `terms, `prefix`, `wildcard`, `in` and
            # `exclude` filters.
            'lookups': [
                LOOKUP_FILTER_TERMS,
                LOOKUP_FILTER_PREFIX,
                LOOKUP_FILTER_WILDCARD,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_EXCLUDE,
            ],
        },
    }
    # Define ordering fields
    ordering_fields = {
        'id': 'id',
        'title': 'title.raw',
        'price': 'price.raw',
        'state': 'state.raw',
        'publication_date': 'publication_date',
    }
    # Specify default ordering
    ordering = ('id', 'title', 'price',)

URLs ¶

At this step, we’re going to define url patterns.

Content of the search_indexes/urls.py file. Additionally, see the code comments.

Required imports ¶

search_indexes/urls.py

from django.conf.urls import url, include
from rest_framework_extensions.routers import ExtendedDefaultRouter

from .views import BookDocumentView

Router definition ¶

search_indexes/urls.py

router = ExtendedDefaultRouter()
books = router.register(r'books',
                        BookDocumentView,
                        base_name='bookdocument')

URL patterns ¶

search_indexes/urls.py

urlpatterns = [
    url(r'^', include(router.urls)),
]

Check what you’ve done so far ¶

At this point, you are one step away from a working example of integrating Elasticsearch DSL with Django.

URLs ¶

If you didn’t add the urls of the search_indexes example application to your project’s global url patterns, make sure to do it now.

from django.conf.urls import include, url
from search_indexes import urls as search_index_urls

urlpatterns = [
    # ...
    # Search URLs
    url(r'^search/', include(search_index_urls)),
    # ...
]

Test in browser ¶

Open the following URL in your browser.

http://localhost:8000/search/books/

Perform a number of lookups:

http://localhost:8001/search/books/?ids=54|55|56
http://localhost:8001/search/books/?summary__contains=photography
http://localhost:8001/search/books/?tags__contains=ython
http://localhost:8001/search/books/?state=published
http://localhost:8001/search/books/?pages__gt=10&pages__lt=30

Development and debugging ¶

Profiling tools ¶

Looking for profiling tools for Elasticsearch?

Try django-elasticsearch-debug-toolbar package. It’s implemented as a panel for the well known Django Debug Toolbar and gives you full insights on what’s happening on the side of Elasticsearch.

Installation ¶

pip install django-debug-toolbar
pip install django-elasticsearch-debug-toolbar

Configuration ¶

Change your development settings in the following way:

settings/dev.py

MIDDLEWARE_CLASSES += (
    'debug_toolbar.middleware.DebugToolbarMiddleware',
    'debug_toolbar_force.middleware.ForceDebugToolbarMiddleware',
)

INSTALLED_APPS += (
    'debug_toolbar',
    'elastic_panel',
)

DEBUG_TOOLBAR_CONFIG = {
    'INTERCEPT_REDIRECTS': False,
}

DEBUG_TOOLBAR_PANELS = (
    # Defaults
    'debug_toolbar.panels.versions.VersionsPanel',
    'debug_toolbar.panels.timer.TimerPanel',
    'debug_toolbar.panels.settings.SettingsPanel',
    'debug_toolbar.panels.headers.HeadersPanel',
    'debug_toolbar.panels.request.RequestPanel',
    'debug_toolbar.panels.sql.SQLPanel',
    'debug_toolbar.panels.staticfiles.StaticFilesPanel',
    'debug_toolbar.panels.templates.TemplatesPanel',
    'debug_toolbar.panels.cache.CachePanel',
    'debug_toolbar.panels.signals.SignalsPanel',
    'debug_toolbar.panels.logging.LoggingPanel',
    'debug_toolbar.panels.redirects.RedirectsPanel',
    # Additional
    'elastic_panel.panel.ElasticDebugPanel',
)

Debugging ¶

Although (the unbeatable) Kibana is strongly recommended for data analyses, there are other good tools worth mentioning. One of them is elasticsearch-head Elasticsearch 2.x plugin or a correspondent Chrome extension of the same plugin. You may find it very useful for quick data preview or testing Elasticsearch queries.

Filter usage examples ¶

Example usage of filtering backends.

Contents:

Table of Contents

Search ¶

Query param name reserved for search is search. Make sure your models and documents do not have it as a field or attribute.

Multiple search terms are joined with OR.

Let’s assume we have a number of Book items with fields title, description and summary.

Search in all fields ¶

Search in all fields (name, address, city, state_province and country) for word “reilly”.

http://127.0.0.1:8080/search/publisher/?search=reilly

Search a single term on specific field ¶

In order to search in specific field (name) for term “reilly”, add the field name separated with | to the search term.

http://127.0.0.1:8080/search/publisher/?search=name|reilly

Search for multiple terms ¶

In order to search for multiple terms “reilly”, “bloomsbury” add multiple search query params.

http://127.0.0.1:8080/search/publisher/?search=reilly&search=bloomsbury

Search for multiple terms in specific fields ¶

In order to search for multiple terms “reilly”, “bloomsbury” in specific fields add multiple search query params and field names separated with | to each of the search terms.

http://127.0.0.1:8080/search/publisher/?search=name|reilly&search=city|london

Filtering ¶

Supported lookups ¶

Native ¶

The following native (to Elasticsearch) filters/lookups are implemented:

term
terms
range
exists
prefix
wildcard
regexp
fuzzy
type
ids

term ¶

Find documents which contain the exact term specified in the field specified.

http://127.0.0.1:8080/search/books/?tags__term=education&tags__term=economy

terms ¶

Find documents which contain any of the exact terms specified in the field specified.

range ¶

Find documents where the field specified contains values (dates, numbers, or strings) in the range specified.

exists ¶

Find documents where the field specified contains any non-null value.

prefix ¶

Find documents where the field specified contains terms which begin with the exact prefix specified.

wildcard ¶

Find documents where the field specified contains terms which match the pattern specified, where the pattern supports single character wildcards (?) and multi-character wildcards (*)

ids ¶

Find documents with the specified type and IDs.

Functional ¶

The following functional (non-native to Elasticsearch, but common in Django) filters/lookups are implemented:

contains
in
gt
gte
lt
lte
startswith
endswith
isnull
exclude

contains ¶

Case-insensitive containment test.

in ¶

In a given list.

gt ¶

Greater than.

gte ¶

Greater than or equal to.

lt ¶

Less than.

lte ¶

Less than or equal to.

startswith ¶

Case-sensitive starts-with.

endswith ¶

Case-sensitive ends-with.

isnull ¶

Takes either True or False.

exclude ¶

Returns a new query set of containing objects that do not match the given lookup parameters.

Usage examples ¶

See the example project for sample models/views/serializers.

Additionally, see:

Basic usage examples ¶

Basic Django REST framework integration example

See the example project for sample models/views/serializers.

Contents:

Table of Contents

Basic usage examples
- Example app

Example app ¶

Sample models ¶

books/models.py:

class Publisher(models.Model):
    """Publisher."""

    name = models.CharField(max_length=30)
    address = models.CharField(max_length=50)
    city = models.CharField(max_length=60)
    state_province = models.CharField(max_length=30)
    country = models.CharField(max_length=50)
    website = models.URLField()
    latitude = models.DecimalField(null=True,
                               blank=True,
                               decimal_places=15,
                               max_digits=19,
                               default=0)
    longitude = models.DecimalField(null=True,
                                    blank=True,
                                    decimal_places=15,
                                    max_digits=19,
                                    default=0)

    class Meta(object):
        """Meta options."""

        ordering = ["id"]

    def __str__(self):
        return self.name

    @property
    def location_field_indexing(self):
        """Location for indexing.

        Used in Elasticsearch indexing/tests of `geo_distance` native filter.
        """
        return {
            'lat': self.latitude,
            'lon': self.longitude,
        }

Sample document ¶

search_indexes/documents/publisher.py:

from django_elasticsearch_dsl import DocType, Index, fields
from elasticsearch_dsl import analyzer

from books.models import Publisher

# Name of the Elasticsearch index
PUBLISHER_INDEX = Index('publisher')
# See Elasticsearch Indices API reference for available settings
PUBLISHER_INDEX.settings(
    number_of_shards=1,
    number_of_replicas=1
)


@PUBLISHER_INDEX.doc_type
class PublisherDocument(DocType):
    """Publisher Elasticsearch document."""

    id = fields.IntegerField(attr='id')

    name = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )
    info = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )
    address = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )
    city = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )
    state_province = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )
    country = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )
    website = fields.StringField()

    # Location
    location = fields.GeoPointField(attr='location_field_indexing')

    class Meta(object):
        """Meta options."""

        model = Publisher  # The model associate with this DocType

Sample serializer ¶

search_indexes/serializers.py:

import json

from django_elasticsearch_dsl_drf.serializers import DocumentSerializer

class PublisherDocumentSerializer(DocumentSerializer):
    """Serializer for Publisher document."""

    location = serializers.SerializerMethodField()

    class Meta(object):
        """Meta options."""

        # Note, that since we're using a dynamic serializer,
        # we only have to declare fields that we want to be shown. If
        # somehow, dynamic serializer doesn't work for you, either extend
        # or declare your serializer explicitly.
        fields = (
            'id',
            'name',
            'info',
            'address',
            'city',
            'state_province',
            'country',
            'website',
        )

    def get_location(self, obj):
    """Represent location value."""
    try:
        return obj.location.to_dict()
    except:
        return {}

Sample view ¶

search_indexes/views.py:

from django_elasticsearch_dsl_drf.constants import (
    LOOKUP_FILTER_GEO_DISTANCE,
)
from django_elasticsearch_dsl_drf.filter_backends import (
    FilteringFilterBackend,
    OrderingFilterBackend,
    SearchFilterBackend,
)
from django_elasticsearch_dsl_drf.views import BaseDocumentViewSet

# Example app models
from search_indexes.documents.publisher import PublisherDocument
from search_indxes.serializers import PublisherDocumentSerializer

class PublisherDocumentView(BaseDocumentViewSet):
    """The PublisherDocument view."""

    document = PublisherDocument
    serializer_class = PublisherDocumentSerializer
    lookup_field = 'id'
    filter_backends = [
        FilteringFilterBackend,
        OrderingFilterBackend,
        DefaultOrderingFilterBackend,
        SearchFilterBackend,
    ]
    # Define search fields
    search_fields = (
        'name',
        'info',
        'address',
        'city',
        'state_province',
        'country',
    )
    # Define filtering fields
    filter_fields = {
        'id': None,
        'name': 'name.raw',
        'city': 'city.raw',
        'state_province': 'state_province.raw',
        'country': 'country.raw',
    }
    # Define ordering fields
    ordering_fields = {
        'id': None,
        'name': None,
        'city': None,
        'country': None,
    }
    # Specify default ordering
    ordering = ('id', 'name',)
    # Define geo-spatial filtering fields
    geo_spatial_filter_fields = {
        'location': {
            'lookups': [
                LOOKUP_FILTER_GEO_DISTANCE,
            ],
        },
    }

Usage example ¶

Considering samples above, you should be able to perform the search, sorting and filtering actions described below.

Sample queries ¶

Search ¶

Query param name reserved for search is search. Make sure your models and documents do not have it as a field or attribute.

Multiple search terms are joined with OR.

Let’s assume we have a number of Book items with fields title, description and summary.

Search in all fields

Search in all fields (name, address, city, state_province and country) for word “reilly”.

http://127.0.0.1:8080/search/publisher/?search=reilly

Search a single term on specific field

In order to search in specific field (name) for term “reilly”, add the field name separated with | to the search term.

http://127.0.0.1:8080/search/publisher/?search=name|reilly

Search for multiple terms

In order to search for multiple terms “reilly”, “bloomsbury” add multiple search query params.

http://127.0.0.1:8080/search/publisher/?search=reilly&search=bloomsbury

Search for multiple terms in specific fields

In order to search for multiple terms “reilly”, “bloomsbury” in specific fields add multiple search query params and field names separated with | to each of the search terms.

http://127.0.0.1:8080/search/publisher/?search=name|reilly&search=city|london

Filtering ¶

Let’s assume we have a number of Publisher documents with in cities (Yerevan, Groningen, Amsterdam, London).

Multiple filter terms are joined with AND.

Filter documents by single field

Filter documents by field (city) “yerevan”.

http://127.0.0.1:8080/search/publisher/?city=yerevan

Filter documents by multiple fields

Filter documents by city “Yerevan” and “Groningen”.

http://127.0.0.1:8080/search/publisher/?city__in=yerevan|groningen

Filter document by a single field

Filter documents by (field country) “Armenia”.

http://127.0.0.1:8080/search/publisher/?country=armenia

Filter documents by multiple fields

Filter documents by multiple fields (field city) “Yerevan” and “Amsterdam” with use of functional in query filter.

http://127.0.0.1:8080/search/publisher/?city__in=yerevan|amsterdam

You can achieve the same effect by specifying multiple filters (city) “Yerevan” and “Amsterdam”. Note, that in this case multiple filter terms are joined with OR.

http://127.0.0.1:8080/search/publisher/?city=yerevan&city=amsterdam

If you want the same as above, but joined with AND, add __term to each lookup.

http://127.0.0.1:8080/search/publisher/?city__term=education&city__term=economy

Filter documents by a word part of a single field

Filter documents by a part word part in single field (city) “ondon”.

http://127.0.0.1:8080/search/publisher/?city__wildcard=*ondon

Geo-distance filtering

Filter documents by radius of 100000km from the given location.

http://127.0.0.1:8000/search/publishers/?location__geo_distance=100000km|12.04|-63.93

Ordering ¶

The - prefix means ordering should be descending.

Order documents by field (ascending)

Filter documents by field city (ascending).

http://127.0.0.1:8080/search/publisher/?search=country|armenia&ordering=city

Order documents by field (descending)

Filter documents by field country (descending).

http://127.0.0.1:8080/search/publisher/?ordering=-country

Order documents by multiple fields

If you want to order by multiple fields, use multiple ordering query params. In the example below, documents would be ordered first by field country (descending), then by field city (ascending).

http://127.0.0.1:8080/search/publisher/?ordering=-country&ordering=city

Advanced usage examples ¶

Advanced Django REST framework integration examples.

See the example project for sample models/views/serializers.

Contents:

Table of Contents

Example app ¶

Sample models ¶

books/models.py

import json

from django.conf import settings
from django.db import models
from django.utils.translation import ugettext, ugettext_lazy as _

from six import python_2_unicode_compatible

BOOK_PUBLISHING_STATUS_PUBLISHED = 'published'
BOOK_PUBLISHING_STATUS_NOT_PUBLISHED = 'not_published'
BOOK_PUBLISHING_STATUS_IN_PROGRESS = 'in_progress'
BOOK_PUBLISHING_STATUS_CANCELLED = 'cancelled'
BOOK_PUBLISHING_STATUS_REJECTED = 'rejected'
BOOK_PUBLISHING_STATUS_CHOICES = (
    (BOOK_PUBLISHING_STATUS_PUBLISHED, "Published"),
    (BOOK_PUBLISHING_STATUS_NOT_PUBLISHED, "Not published"),
    (BOOK_PUBLISHING_STATUS_IN_PROGRESS, "In progress"),
    (BOOK_PUBLISHING_STATUS_CANCELLED, "Cancelled"),
    (BOOK_PUBLISHING_STATUS_REJECTED, "Rejected"),
)
BOOK_PUBLISHING_STATUS_DEFAULT = BOOK_PUBLISHING_STATUS_PUBLISHED


@python_2_unicode_compatible
class Publisher(models.Model):
    """Publisher."""

    name = models.CharField(max_length=30)
    info = models.TextField(null=True, blank=True)
    address = models.CharField(max_length=50)
    city = models.CharField(max_length=60)
    state_province = models.CharField(max_length=30)
    country = models.CharField(max_length=50)
    website = models.URLField()
    latitude = models.DecimalField(null=True,
                               blank=True,
                               decimal_places=15,
                               max_digits=19,
                               default=0)
    longitude = models.DecimalField(null=True,
                                    blank=True,
                                    decimal_places=15,
                                    max_digits=19,
                                    default=0)

    class Meta(object):
        """Meta options."""

        ordering = ["id"]

    def __str__(self):
        return self.name

    @property
    def location_field_indexing(self):
        """Location for indexing.

        Used in Elasticsearch indexing/tests of `geo_distance` native filter.
        """
        return {
            'lat': self.latitude,
            'lon': self.longitude,
        }


@python_2_unicode_compatible
class Author(models.Model):
    """Author."""

    salutation = models.CharField(max_length=10)
    name = models.CharField(max_length=200)
    email = models.EmailField()
    headshot = models.ImageField(upload_to='authors', null=True, blank=True)

    class Meta(object):
        """Meta options."""

        ordering = ["id"]

    def __str__(self):
        return self.name


class Tag(models.Model):
    """Simple tag model."""

    title = models.CharField(max_length=255, unique=True)

    class Meta(object):
        """Meta options."""

        verbose_name = _("Tag")
        verbose_name_plural = _("Tags")

    def __str__(self):
        return self.title


@python_2_unicode_compatible
class Book(models.Model):
    """Book."""

    title = models.CharField(max_length=100)
    description = models.TextField(null=True, blank=True)
    summary = models.TextField(null=True, blank=True)
    authors = models.ManyToManyField('books.Author', related_name='books')
    publisher = models.ForeignKey(Publisher, related_name='books')
    publication_date = models.DateField()
    state = models.CharField(max_length=100,
                             choices=BOOK_PUBLISHING_STATUS_CHOICES,
                             default=BOOK_PUBLISHING_STATUS_DEFAULT)
    isbn = models.CharField(max_length=100, unique=True)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    pages = models.PositiveIntegerField(default=200)
    stock_count = models.PositiveIntegerField(default=30)
    tags = models.ManyToManyField('books.Tag',
                                  related_name='books',
                                  blank=True)

    class Meta(object):
        """Meta options."""

        ordering = ["isbn"]

    def __str__(self):
        return self.title

    @property
    def publisher_indexing(self):
        """Publisher for indexing.

        Used in Elasticsearch indexing.
        """
        if self.publisher is not None:
            return self.publisher.name

    @property
    def tags_indexing(self):
        """Tags for indexing.

        Used in Elasticsearch indexing.
        """
        return [tag.title for tag in self.tags.all()]

Sample document ¶

Index definition ¶

To separate dev/test/staging/production indexes, the following approach is recommended.

Settings ¶

settings/base.py

# Name of the Elasticsearch index
ELASTICSEARCH_INDEX_NAMES = {
    'search_indexes.documents.book': 'book',
    'search_indexes.documents.publisher': 'publisher',
}

settings/testing.py

# Name of the Elasticsearch index
ELASTICSEARCH_INDEX_NAMES = {
    'search_indexes.documents.book': 'test_book',
    'search_indexes.documents.publisher': 'test_publisher',
}

settings/production.py

# Name of the Elasticsearch index
ELASTICSEARCH_INDEX_NAMES = {
    'search_indexes.documents.book': 'prod_book',
    'search_indexes.documents.publisher': 'prod_publisher',
}

Document index ¶

search_indexes/documents/book.py

from django.conf import settings
from django_elasticsearch_dsl import DocType, Index, fields
from elasticsearch_dsl import analyzer

from books.models import Book

# Name of the Elasticsearch index
INDEX = Index(settings.ELASTICSEARCH_INDEX_NAMES[__name__])

# See Elasticsearch Indices API reference for available settings
INDEX.settings(
    number_of_shards=1,
    number_of_replicas=1
)

html_strip = analyzer(
    'html_strip',
    tokenizer="standard",
    filter=["standard", "lowercase", "stop", "snowball"],
    char_filter=["html_strip"]
)


@INDEX.doc_type
class BookDocument(DocType):
    """Book Elasticsearch document."""

    id = fields.IntegerField(attr='id')

    title = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    description = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    summary = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    publisher = fields.StringField(
        attr='publisher_indexing',
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    publication_date = fields.DateField()

    state = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    isbn = fields.StringField(
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword'),
        }
    )

    price = fields.FloatField()

    pages = fields.IntegerField()

    stock_count = fields.IntegerField()

    tags = fields.StringField(
        attr='tags_indexing',
        analyzer=html_strip,
        fields={
            'raw': fields.StringField(analyzer='keyword', multi=True),
            'suggest': fields.CompletionField(multi=True),
        },
        multi=True
    )

    class Meta(object):
        """Meta options."""

        model = Book  # The model associate with this DocType

Sample serializer ¶

search_indexes/serializers.py

import json

from rest_framework import serializers

class TagSerializer(serializers.Serializer):
    """Helper serializer for the Tag field of the Book document."""

    title = serializers.CharField()

    class Meta(object):
        """Meta options."""

        fields = ('title',)
        read_only_fields = ('title',)


class BookDocumentSerializer(serializers.Serializer):
    """Serializer for the Book document."""

    id = serializers.SerializerMethodField()

    title = serializers.CharField(read_only=True)
    description = serializers.CharField(read_only=True)
    summary = serializers.CharField(read_only=True)

    publisher = serializers.CharField(read_only=True)
    publication_date = serializers.DateField(read_only=True)
    state = serializers.CharField(read_only=True)
    isbn = serializers.CharField(read_only=True)
    price = serializers.FloatField(read_only=True)
    pages = serializers.IntegerField(read_only=True)
    stock_count = serializers.IntegerField(read_only=True)
    tags = serializers.SerializerMethodField()

    class Meta(object):
        """Meta options."""

        fields = (
            'id',
            'title',
            'description',
            'summary',
            'publisher',
            'publication_date',
            'state',
            'isbn',
            'price',
            'pages',
            'stock_count',
            'tags',
        )
        read_only_fields = fields

    def get_tags(self, obj):
        """Get tags."""
        if obj.tags:
            return list(obj.tags)
        else:
            return []

Sample view ¶

search_indexes/viewsets.py

from django_elasticsearch_dsl_drf.constants import (
    LOOKUP_FILTER_TERMS,
    LOOKUP_FILTER_RANGE,
    LOOKUP_FILTER_PREFIX,
    LOOKUP_FILTER_WILDCARD,
    LOOKUP_QUERY_IN,
    LOOKUP_QUERY_EXCLUDE,
)
from django_elasticsearch_dsl_drf.filter_backends import (
    FilteringFilterBackend,
    OrderingFilterBackend,
    DefaultOrderingFilterBackend,
    SearchFilterBackend,
)
from django_elasticsearch_dsl_drf.views import BaseDocumentViewSet

# Example app models
from search_indexes.documents.book import BookDocument
from search_indxes.serializers import BookDocumentSerializer


class BookDocumentView(BaseDocumentViewSet):
    """The BookDocument view."""

    document = BookDocument
    serializer_class = BookDocumentSerializer
    lookup_field = 'id'
    filter_backends = [
        FilteringFilterBackend,
        OrderingFilterBackend,
        DefaultOrderingFilterBackend,
        SearchFilterBackend,
    ]
    # Define search fields
    search_fields = (
        'title',
        'description',
        'summary',
    )
    # Define filtering fields
    filter_fields = {
        'id': {
            'field': '_id',
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_IN,
            ],
        },
        'publisher': 'publisher.raw',
        'publication_date': 'publication_date',
        'isbn': 'isbn.raw',
        'tags': {
            'field': 'tags',
            'lookups': [
                LOOKUP_FILTER_TERMS,
                LOOKUP_FILTER_PREFIX,
                LOOKUP_FILTER_WILDCARD,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_EXCLUDE,
            ],
        },
        'tags.raw': {
            'field': 'tags.raw',
            'lookups': [
                LOOKUP_FILTER_TERMS,
                LOOKUP_FILTER_PREFIX,
                LOOKUP_FILTER_WILDCARD,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_EXCLUDE,
            ],
        },
    }
    # Define ordering fields
    ordering_fields = {
        'id': 'id',
        'title': 'title.raw',
        'price': 'price.raw',
        'state': 'state.raw',
        'publication_date': 'publication_date',
    }
    # Specify default ordering
    ordering = ('id', 'title',)

Usage example ¶

Considering samples above, you should be able to perform the search, sorting and filtering actions described below.

Sample queries ¶

Search ¶

Query param name reserved for search is search. Make sure your models and documents do not have it as a field or attribute.

Multiple search terms are joined with OR.

Let’s assume we have a number of Book items with fields title, description and summary.

Search in all fields

Search in all fields (title, description and summary) for word “education”.

http://127.0.0.1:8080/search/books/?search=education

Search a single term on specific field

In order to search in specific field (title) for term “education”, add the field name separated with | to the search term.

http://127.0.0.1:8080/search/books/?search=title|education

Search for multiple terms

In order to search for multiple terms “education”, “technology” add multiple search query params.

http://127.0.0.1:8080/search/books/?search=education&search=technology

Search for multiple terms on specific fields

In order to search for multiple terms “education”, “technology” in specific fields add multiple search query params and field names separated with | to each of the search terms.

http://127.0.0.1:8080/search/books/?search=title|education&search=summary|technology

Filtering ¶

Let’s assume we have a number of Book documents with the tags (education, politics, economy, biology, climate, environment, internet, technology).

Multiple filter terms are joined with AND.

Filter documents by field

Filter documents by field (state) “published”.

http://127.0.0.1:8080/search/books/?state=published

Filter documents by multiple fields

Filter documents by field (states) “published” and “in_progress”.

http://127.0.0.1:8080/search/books/?state__in=published|in_progress

Filter document by a single field

Filter documents by (field tag) “education”.

http://127.0.0.1:8080/search/books/?tag=education

Filter documents by multiple fields

Filter documents by multiple fields (field tags) “education” and “economy” with use of functional in query filter.

http://127.0.0.1:8080/search/books/?tags__in=education|economy

You can achieve the same effect by specifying multiple fields (tags) “education” and “economy”. Note, that in this case multiple filter terms are joined with OR.

http://127.0.0.1:8080/search/books/?tags=education&tags=economy

If you want the same as above, but joined with AND, add __term to each lookup.

http://127.0.0.1:8080/search/books/?tags__term=education&tags__term=economy

Filter documents by a word part of a single field

Filter documents by a part word part in single field (tags). Word part should match both “technology” and “biology”.

http://127.0.0.1:8080/search/books/?tags__wildcard=*logy

Ordering ¶

The - prefix means ordering should be descending.

Order documents by field (ascending)

Order documents by field price (ascending).

http://127.0.0.1:8080/search/books/?search=title|lorem&ordering=price

Order documents by field (descending)

Order documents by field price (descending).

http://127.0.0.1:8080/search/books/?search=title|lorem&ordering=-price

Order documents by multiple fields

If you want to order by multiple fields, use multiple ordering query params. In the example below, documents would be ordered first by field publication_date (descending), then by field price (ascending).

http://127.0.0.1:8080/search/books/?search=title|lorem&ordering=-publication_date&ordering=price

Ids filter ¶

Filters documents that only have the provided ids.

http://127.0.0.1:8000/api/articles/?ids=68|64|58

Or, alternatively:

http://127.0.0.1:8000/api/articles/?ids=68&ids=64&ids=58

Faceted search ¶

In order to add faceted search support, we would have to extend our view set in the following way:

search_indexes/viewsets.py

# ...

from django_elasticsearch_dsl_drf.filter_backends import (
    # ...
    FacetedSearchFilterBackend,
)

# ...

from elasticsearch_dsl import (
    DateHistogramFacet,
    RangeFacet,
    TermsFacet,
)

# ...

class BookDocumentView(BaseDocumentViewSet):
    """The BookDocument view."""

    # ...

    filter_backends = [
        # ...
        FacetedSearchFilterBackend,
    ]

    # ...

    faceted_search_fields = {
        'state': 'state.raw',  # By default, TermsFacet is used
        'publisher': {
            'field': 'publisher.raw',
            'facet': TermsFacet,  # But we can define it explicitly
            'enabled': True,
        },
        'publication_date': {
            'field': 'publication_date',
            'facet': DateHistogramFacet,
            'options': {
                'interval': 'year',
            }
        },
        'pages_count': {
            'field': 'pages',
            'facet': RangeFacet,
            'options': {
                'ranges': [
                    ("<10", (None, 10)),
                    ("11-20", (11, 20)),
                    ("20-50", (20, 50)),
                    (">50", (50, None)),
                ]
            }
        },
    }

    # ...

Note, that none of the facets is enabled by default, unless you explicitly specify it to be enabled. That means, that you will have to add a query string facet={facet_field_name} for each of the facets you want to see in results.

In the example below, we show results with faceted state and pages_count facets.

http://127.0.0.1:8000/search/books/?facet=state&facet=pages_count

Geo-spatial features ¶

For testing the boundaries the following online services might be helpful:

Filtering ¶

Geo-distance filtering

Filter documents by radius of 100000km from the given location.

http://localhost:8000/search/publishers/?location__geo_distance=100000km|12.04|-63.93

Geo-polygon filtering

Filter documents that are located in the given polygon.

http://localhost:8000/search/publishers/?location__geo_polygon=40,-70|30,-80|20,-90

Geo-bounding-box filtering

Filter documents that are located in the given bounding box.

http://localhost:8000/search/publishers/?location__geo_bounding_box=44.87,40.07|43.87,41.11

Ordering ¶

Geo-distance ordering

http://localhost:8000/search/publishers/?ordering=location|48.85|2.30|km|plane

Suggestions ¶

The suggest feature suggests similar looking terms based on a provided text by using a suggester.

Note

The SuggesterFilterBackend filter backend can be used in the suggest custom view action/route only. Usages outside of the are suggest action/route are restricted.

There are three options available here: term, phrase and completion.

Note

Suggestion functionality is exclusive. Once you have queried the SuggesterFilterBackend, the latter will transform your current search query into suggestion search query (which is very different). Therefore, always add it as the very last filter backend.

Completion suggesters ¶

Document definition ¶

To make use of suggestions, you should properly indexed your documents using fields.CompletionField.

search_indexes/documents/publisher.py

from django.conf import settings

from django_elasticsearch_dsl import DocType, Index, fields

from books.models import Publisher

# Name of the Elasticsearch index
INDEX = Index(settings.ELASTICSEARCH_INDEX_NAMES[__name__])

# See Elasticsearch Indices API reference for available settings
INDEX.settings(
    number_of_shards=1,
    number_of_replicas=1
)


@INDEX.doc_type
class PublisherDocument(DocType):
    """Publisher Elasticsearch document."""

    id = fields.IntegerField(attr='id')

    name = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
            'suggest': fields.CompletionField(),
        }
    )

    info = fields.StringField()

    address = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword')
        }
    )

    city = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
            'suggest': fields.CompletionField(),
        }
    )

    state_province = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
            'suggest': fields.CompletionField(),
        }
    )

    country = fields.StringField(
        fields={
            'raw': fields.StringField(analyzer='keyword'),
            'suggest': fields.CompletionField(),
        }
    )

    website = fields.StringField()

    # Location
    location = fields.GeoPointField(attr='location_field_indexing')

    class Meta(object):
        """Meta options."""

        model = Publisher  # The model associate with this DocType

After that the name.suggest, city.suggest, state_province.suggest and country.suggest fields would be available for suggestions feature.

Serializer definition ¶

This is how publisher serializer would look like.

search_indexes/serializers.py

import json

from django_elasticsearch_dsl_drf.serializers import DocumentSerializer

class PublisherDocumentSerializer(DocumentSerializer):
    """Serializer for Publisher document."""

    location = serializers.SerializerMethodField()

    class Meta(object):
        """Meta options."""

        # Note, that since we're using a dynamic serializer,
        # we only have to declare fields that we want to be shown. If
        # somehow, dynamic serializer doesn't work for you, either extend
        # or declare your serializer explicitly.
        fields = (
            'id',
            'name',
            'info',
            'address',
            'city',
            'state_province',
            'country',
            'website',
        )

    def get_location(self, obj):
    """Represent location value."""
    try:
        return obj.location.to_dict()
    except:
        return {}

ViewSet definition ¶

In order to add suggestions support, we would have to extend our view set in the following way:

search_indexes/viewsets.py

# ...

from django_elasticsearch_dsl_drf.constants import SUGGESTER_COMPLETION
from django_elasticsearch_dsl_drf.filter_backends import (
    # ...
    SuggesterFilterBackend,
)

# ...

class PublisherDocumentViewSet(BaseDocumentViewSet):
    """The PublisherDocument view."""

    document = PublisherDocument

    # ...

    filter_backends = [
        # ...
        SuggesterFilterBackend,
    ]

    # ...

    # Suggester fields
    suggester_fields = {
        'name_suggest': {
            'field': 'name.suggest',
            'suggesters': [
                SUGGESTER_COMPLETION,
            ],
        },
        'city_suggest': {
            'field': 'city.suggest',
            'suggesters': [
                SUGGESTER_COMPLETION,
            ],
        },
        'state_province_suggest': {
            'field': 'state_province.suggest',
            'suggesters': [
                SUGGESTER_COMPLETION,
            ],
        },
        'country_suggest': {
            'field': 'country.suggest',
            'suggesters': [
                SUGGESTER_COMPLETION,
            ],
        },
    }

    # Geo-spatial filtering fields
    geo_spatial_filter_fields = {
        'location': {
            'lookups': [
                LOOKUP_FILTER_GEO_DISTANCE,
            ],
        },
    }

In the example below, we show suggestion results (auto-completion) for country field.

Sample requests/responses ¶

Once you have extended your view set with SuggesterFilterBackend functionality, you can make use of the suggest custom action of your view set.

Request

GET http://127.0.0.1:8000/search/publishers/suggest/?country_suggest__completion=Ar

Response

{
    "_shards": {
        "failed": 0,
        "successful": 1,
        "total": 1
    },
    "country_suggest__completion": [
        {
            "options": [
                {
                    "score": 1.0,
                    "text": "Armenia"
                },
                {
                    "score": 1.0,
                    "text": "Argentina"
                }
            ],
            "offset": 0,
            "length": 2,
            "text": "Ar"
        }
    ]
}

You can also have multiple suggesters per request.

Request

GET http://127.0.0.1:8000/search/publishers/suggest/?name_suggest__completion=B&country_suggest__completion=Ar

Response

{
    "_shards": {
        "successful": 1,
        "total": 1,
        "failed": 0
    },
    "country_suggest__completion": [
        {
            "text": "Ar",
            "options": [
                {
                    "score": 1.0,
                    "text": "Armenia"
                },
                {
                    "score": 1.0,
                    "text": "Argentina"
                }
            ],
            "offset": 0,
            "length": 2
        }
    ],
    "name_suggest__completion": [
        {
            "text": "B",
            "options": [
                {
                    "score": 1.0,
                    "text": "Book Works"
                },
                {
                    "score": 1.0,
                    "text": "Brumleve LLC"
                },
                {
                    "score": 1.0,
                    "text": "Booktrope"
                },
                {
                    "score": 1.0,
                    "text": "Borman, Post and Wendt"
                },
                {
                    "score": 1.0,
                    "text": "Book League of America"
                }
            ],
            "offset": 0,
            "length": 1
        }
    ]
}

Suggestions on Array/List field ¶

Suggestions on Array/List fields (typical use case - tags, where Tag model would be a many-to-many relation to a Book model) work almost the same.

Before checking the Sample requests/responses, do have in mind the following:

Book (see the Sample models)
BookSerializer (see the Sample serializer)
BookDocumentView (see the Sample view)

Sample requests/responses ¶

Once you have extended your view set with SuggesterFilterBackend functionality, you can make use of the suggest custom action of your view set.

Request

GET http://127.0.0.1:8000/search/books/suggest/?tag_suggest__completion=bio

Response

{
    "_shards": {
        "failed": 0,
        "successful": 1,
        "total": 1
    },
    "country_suggest__completion": [
        {
            "options": [
                {
                    "score": 1.0,
                    "text": "Biography"
                },
                {
                    "score": 1.0,
                    "text": "Biology"
                }
            ],
            "offset": 0,
            "length": 2,
            "text": "bio"
        }
    ]
}

Term and Phrase suggestions ¶

While for the completion suggesters to work the CompletionField shall be used, the term and phrase suggesters work on common text fields.

Document definition ¶

search_indexes/documents/book.py

from django.conf import settings

from django_elasticsearch_dsl import DocType, Index, fields

from books.models import Book

# Name of the Elasticsearch index
INDEX = Index(settings.ELASTICSEARCH_INDEX_NAMES[__name__])

# See Elasticsearch Indices API reference for available settings
INDEX.settings(
    number_of_shards=1,
    number_of_replicas=1
)

@INDEX.doc_type
class BookDocument(DocType):
    """Book Elasticsearch document."""
    # ID
    id = fields.IntegerField(attr='id')

    title = StringField(
        analyzer=html_strip,
        fields={
            'raw': KeywordField(),
            'suggest': fields.CompletionField(),
        }
    )

    description = StringField(
        analyzer=html_strip,
        fields={
            'raw': KeywordField(),
        }
    )

    summary = StringField(
        analyzer=html_strip,
        fields={
            'raw': KeywordField()
        }
    )

    # Publisher
    publisher = StringField(
        attr='publisher_indexing',
        analyzer=html_strip,
        fields={
            'raw': KeywordField(),
            'suggest': fields.CompletionField(),
        }
    )

    # Publication date
    publication_date = fields.DateField()

    # State
    state = StringField(
        analyzer=html_strip,
        fields={
            'raw': KeywordField(),
        }
    )

    # ISBN
    isbn = StringField(
        analyzer=html_strip,
        fields={
            'raw': KeywordField(),
        }
    )

    # Price
    price = fields.FloatField()

    # Pages
    pages = fields.IntegerField()

    # Stock count
    stock_count = fields.IntegerField()

    # Tags
    tags = StringField(
        attr='tags_indexing',
        analyzer=html_strip,
        fields={
            'raw': KeywordField(multi=True),
            'suggest': fields.CompletionField(multi=True),
        },
        multi=True
    )

    null_field = fields.StringField(attr='null_field_indexing')

    class Meta(object):
        """Meta options."""

        model = Book  # The model associate with this DocType

ViewSet definition ¶

search_indexes/viewsets.py

from django_elasticsearch_dsl_drf.constants import (
    LOOKUP_FILTER_PREFIX,
    LOOKUP_FILTER_RANGE,
    LOOKUP_FILTER_TERMS,
    LOOKUP_FILTER_WILDCARD,
    LOOKUP_QUERY_EXCLUDE,
    LOOKUP_QUERY_GT,
    LOOKUP_QUERY_GTE,
    LOOKUP_QUERY_IN,
    LOOKUP_QUERY_IN,
    LOOKUP_QUERY_ISNULL,
    LOOKUP_QUERY_LT,
    LOOKUP_QUERY_LTE,
    SUGGESTER_PHRASE,
    SUGGESTER_TERM,
)
from django_elasticsearch_dsl_drf.filter_backends import (
    # ...
    SuggesterFilterBackend,
)

class BookDocumentViewSet(BaseDocumentViewSet):
    """The BookDocument view."""

    document = BookDocument
    # serializer_class = BookDocumentSerializer
    serializer_class = BookDocumentSimpleSerializer
    lookup_field = 'id'
    filter_backends = [
        FilteringFilterBackend,
        OrderingFilterBackend,
        DefaultOrderingFilterBackend,
        SearchFilterBackend,
        SuggesterFilterBackend,
    ]
    # Define search fields
    search_fields = (
        'title',
        'description',
        'summary',
    )
    # Define filter fields
    filter_fields = {
        'id': {
            'field': 'id',
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
                LOOKUP_FILTER_TERMS,
            ],
        },
        'title': 'title.raw',
        'publisher': 'publisher.raw',
        'publication_date': 'publication_date',
        'state': 'state.raw',
        'isbn': 'isbn.raw',
        'price': {
            'field': 'price.raw',
            'lookups': [
                LOOKUP_FILTER_RANGE,
            ],
        },
        'pages': {
            'field': 'pages',
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
            ],
        },
        'stock_count': {
            # 'field': 'stock_count',
            'lookups': [
                LOOKUP_FILTER_RANGE,
                LOOKUP_QUERY_GT,
                LOOKUP_QUERY_GTE,
                LOOKUP_QUERY_LT,
                LOOKUP_QUERY_LTE,
            ],
        },
        'tags': {
            'field': 'tags',
            'lookups': [
                LOOKUP_FILTER_TERMS,
                LOOKUP_FILTER_PREFIX,
                LOOKUP_FILTER_WILDCARD,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_EXCLUDE,
                LOOKUP_QUERY_ISNULL,
            ],
        },
        'tags.raw': {
            'field': 'tags.raw',
            'lookups': [
                LOOKUP_FILTER_TERMS,
                LOOKUP_FILTER_PREFIX,
                LOOKUP_FILTER_WILDCARD,
                LOOKUP_QUERY_IN,
                LOOKUP_QUERY_EXCLUDE,
            ],
        },
        # This has been added to test `exists` filter.
        'non_existent_field': 'non_existent_field',
        # This has been added to test `isnull` filter.
        'null_field': 'null_field',
    }
    # Define ordering fields
    ordering_fields = {
        'id': 'id',
        'title': 'title.raw',
        'price': 'price.raw',
        'state': 'state.raw',
        'publication_date': 'publication_date',
    }
    # Specify default ordering
    ordering = ('id', 'title', 'price',)

    # Suggester fields
    suggester_fields = {
        'title_suggest': 'title.suggest',
        'publisher_suggest': 'publisher.suggest',
        'tag_suggest': 'tags.suggest',
        'summary_suggest': 'summary',
    }

Sample requests/responses ¶

Once you have extended your view set with SuggesterFilterBackend functionality, you can make use of the suggest custom action of your view set.

Let’s considering, that one of our books has the following text in the summary:

Twas brillig, and the slithy toves
Did gyre and gimble in the wabe.
All mimsy were the borogoves
And the mome raths outgrabe.

"Beware the Jabberwock, my son!
The jaws that bite, the claws that catch!
Beware the Jubjub bird, and shun
The frumious Bandersnatch!"

He took his vorpal sword in his hand,
Long time the manxome foe he sought --
So rested he by the Tumtum tree,
And stood awhile in thought.

Term ¶

Request

GET http://127.0.0.1:8000/search/books/suggest/?summary_suggest__term=tovse

Response

{
    "_shards": {
        "failed": 0,
        "total": 1,
        "successful": 1
    },
    "summary_suggest__term": [
        {
            "text": "tovs",
            "offset": 0,
            "options": [
                {
                    "text": "tove",
                    "score": 0.75,
                    "freq": 1
                },
                {
                    "text": "took",
                    "score": 0.5,
                    "freq": 1
                },
                {
                    "text": "twas",
                    "score": 0.5,
                    "freq": 1
                }
            ],
            "length": 5
        }
    ]
}

Phrase ¶

Request

GET http://127.0.0.1:8000/search/books/suggest/?summary_suggest__phrase=slith%20tovs

Response

{
    "summary_suggest__phrase": [
        {
            "text": "slith tovs",
            "offset": 0,
            "options": [
                {
                    "text": "slithi tov",
                    "score": 0.00083028956
                }
            ],
            "length": 10
        }
    ],
    "_shards": {
        "failed": 0,
        "total": 1,
        "successful": 1
    }
}

Pagination ¶

Page number pagination ¶

By default, the PageNumberPagination class is used on all view sets which inherit from BaseDocumentViewSet.

Example:

http://127.0.0.1:8000/search/books/?page=4
http://127.0.0.1:8000/search/books/?page=4&page_size=100

Limit/offset pagination ¶

In order to use a different pagination_class, for instance the LimitOffsetPagination, specify it explicitly in the view.

search_indexes/viewsets.py

# ...

from django_elasticsearch_dsl_drf.pagination import LimitOffsetPagination

# ...

class BookDocumentView(BaseDocumentViewSet):
    """The BookDocument view."""

    # ...

    pagination_class = LimitOffsetPagination

    # ...

Example:

http://127.0.0.1:8000/search/books/?limit=100
http://127.0.0.1:8000/search/books/?offset=400&limit=100

Various handy helpers ¶

Contents:

Table of Contents

Various handy helpers
- More like this

More like this ¶

To get more-like-this results on a random registered model, do as follows:

from django_elasticsearch_dsl_drf.helpers import more_like_this
from books.models import Book
book = Book.objects.first()
similar_books = more_like_this(
    book,
    ['title', 'description', 'summary']
)

Customize results as follows:

from django_elasticsearch_dsl_drf.helpers import more_like_this
from elasticsearch_dsl.query import Q
from books.models import Book
book = Book.objects.first()
query = Q('bool', must_not=Q('term', **{'state.raw': 'cancelled'}))
similar_books = more_like_this(
    book,
    query=query,
    fields=['title', 'description', 'summary'],
    min_term_freq=2,
    min_doc_freq=1,
)

Release history and notes¶

Sequence based identifiers are used for versioning (schema follows below):

major.minor[.revision]

It’s always safe to upgrade within the same minor version (for example, from 0.3 to 0.3.4).
Minor version changes might be backwards incompatible. Read the release notes carefully before upgrading (for example, when upgrading from 0.3.4 to 0.4).
All backwards incompatible changes are mentioned in this document.

0.4.4¶

2017-10-02

Documentation improvements (Elasticsearch suggestions).
More tests (term and phrase suggestions).
Code style fixes.

0.4.3¶

2017-09-28

Documentation fixes.
Fixes in tests.
Improved factories.

0.4.2¶

2017-09-28

Added geo_bounding_box query support to the geo-spatial features.

0.4.1¶

2017-09-26

Fixes in docs.

0.4¶

2017-09-26

Note

This release contains changes that might be backwards incompatible for your project. Make sure to add the DefaultOrderingFilterBackend everywhere you have used the OrderingFilterBackend, right after the latter.

GeoSpatialFilteringFilterBackend filtering backend, supporting geo_distance and geo_polygon geo-spatial queries.
GeoSpatialOrderingFilterBackend ordering backend, supporting ordering of results for geo_distance filter.
OrderingFilterBackend no longer provides defaults when no ordering is given. In order to take care of the defaults include the DefaultOrderingFilterBackend in the list of filter_backends (after all other ordering backends).

0.3.12¶

2017-09-21

Added geo_distance filter. Note, that although functionally the filter would not change its’ behaviour, it is likely to be moved to a separate backend (geo_spatial). For now use as is.
Minor fixes.

0.3.11¶

2017-09-21

Added query argument to more_like_this helper.

0.3.10¶

2017-09-20

Minor fixes.
Simplified Elasticsearch version check.

0.3.9¶

2017-09-12

Python 2.x compatibility fix.

0.3.8¶

2017-09-12

Fixes tests on some environments.

0.3.7¶

2017-09-07

Docs fixes.

0.3.6¶

2017-09-07

Fixed suggestions test for Elasticsearch 5.x.
Added compat module for painless testing of Elastic 2.x to Elastic 5.x transition.

0.3.5¶

2017-08-24

Minor fixes in the ordering backend.
Improved tests and coverage.

0.3.4¶

2017-08-23

Minor fixes in the ordering backend.

0.3.3¶

2017-07-13

Minor fixes and improvements.

0.3.2¶

2017-07-12

Minor fixes and improvements.

0.3.1¶

2017-07-12

Minor Python2 fixes.
Minor documentation fixes.

0.3¶

2017-07-11

Add suggestions support (term, phrase and completion).

0.2.6¶

2017-07-11

Minor fixes.
Fixes in documentation.

0.2.5¶

2017-07-11

Fixes in documentation.

0.2.4¶

2017-07-11

Fixes in documentation.

0.2.3¶

2017-07-11

Fixes in documentation.

0.2.2¶

2017-07-11

Fixes in documentation.

0.2.1¶

2017-07-11

Fixes in documentation.

0.2¶

2017-07-11

Initial faceted search support.
Pagination support.

0.1.8¶

2017-06-26

Python2 fixes.
Documentation and example project improvements.

0.1.7¶

2017-06-25

Dynamic serializer for Documents.
Major improvements in documentation.

0.1.6¶

2017-06-23

Implemented gt`, ``gte, lt and lte functional query lookups.
Implemented ids native filter lookup.

0.1.5¶

2017-06-22

Implemented endswith and contains functional filters.
Added tests for wildcard, exists, exclude and isnull filters. Improved range filter tests.
Improve more_like_this helper test.
Improve ordering tests.
Two additional arguments added to the more_like_this helper: min_doc_freq and max_doc_freq.
Minor documentation improvements.

0.1.4¶

2017-06-22

Added tests for in, term and terms filters.
Minor documentation fixes.

0.1.3¶

2017-06-21

Added tests for more_like_this helper, range and prefix filters.
Minor documentation improvements.

0.1.2¶

2017-06-20

Minor fixes in tests.

0.1.1¶

2017-06-20

Fixes in more_like_this helper.
Tiny documentation improvements.

0.1¶

2017-06-19

Initial beta release.