Blog

Table of Contents

Introduction: Why Elasticsearch Dominates Modern Search Architecture

Imagine you’re building the next Netflix, and your users need to search through millions of movies and TV shows. They want to find content by title, actor, director, genre, or even vague descriptions like “that movie with the car chase in Paris.” They expect autocomplete suggestions as they type, personalized recommendations based on viewing history, and all of this needs to happen in milliseconds. Traditional databases would crumble under these requirements, but Elasticsearch thrives in this environment.

Elasticsearch has revolutionized how we think about search and analytics in modern applications. Born from the Apache Lucene search library, Elasticsearch adds a distributed, RESTful layer that transforms a powerful but complex search engine into something developers can actually use at scale. Today, it powers search functionality for companies like Wikipedia, GitHub, Stack Overflow, and countless others processing billions of searches daily.

Why This Topic Matters for Your Career

As a senior developer, understanding Elasticsearch isn’t just about adding another tool to your toolkit—it’s about understanding a fundamental shift in how we handle unstructured data, real-time analytics, and user experience at scale. The demand for Elasticsearch expertise has skyrocketed, with companies actively seeking engineers who can architect, optimize, and troubleshoot Elasticsearch clusters handling terabytes of data.

In technical interviews, Elasticsearch questions serve as a litmus test for several critical skills:

• System Design Thinking: Can you design a search system that scales horizontally?
• Performance Optimization: Do you understand indexing strategies and query optimization?
• Distributed Systems Knowledge: Can you explain concepts like sharding, replication, and consistency?
• Real-World Problem Solving: Have you dealt with production issues like split-brain scenarios or memory pressure?

What Makes Elasticsearch Interview Questions Challenging

Unlike questions about basic data structures or algorithms, Elasticsearch interviews test your ability to think at multiple levels simultaneously. You need to understand:

• The underlying Lucene engine and how inverted indexes work
• Distributed systems concepts like consensus, partition tolerance, and eventual consistency
• JVM tuning and memory management
• Query optimization and relevance scoring
• Operational concerns like monitoring, backup, and disaster recovery

This guide will take you through all these aspects, with the depth and practical examples you need to ace any Elasticsearch interview, whether you’re interviewing for a senior backend engineer, solutions architect, or DevOps role.

Understanding Elasticsearch: Core Concepts Every Developer Must Know {#core-concepts}

Before we dive into the complex architectural details, let’s establish a solid foundation by understanding what Elasticsearch is and the core concepts that make it tick. Think of this section as learning the vocabulary before we start constructing sentences and paragraphs.

What is Elasticsearch, Really?

At its heart, Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. But that definition, while accurate, doesn’t capture why it’s revolutionary. Let me explain it the way I would to a junior developer over coffee:

Imagine you’re building a library catalog system. In a traditional relational database, finding all books that mention “artificial intelligence” in their content would require a full table scan—reading every single row and checking if the text contains your search term. This is painfully slow, especially with millions of books.

Elasticsearch flips this model on its head. Instead of storing data and then searching through it, Elasticsearch analyzes your data as it comes in and creates an “inverted index”—essentially a map of which documents contain which terms. When you search for “artificial intelligence,” Elasticsearch already knows exactly which documents contain these words and can return results almost instantly.

The Inverted Index: The Secret Sauce

The inverted index is the fundamental data structure that powers Elasticsearch’s blazing-fast search capabilities. Let me illustrate with a simple example:

Suppose we have three documents:

• Doc1: “The quick brown fox”
• Doc2: “The quick brown dog”
• Doc3: “The lazy brown dog”

A traditional forward index would look like:

Doc1 → ["The", "quick", "brown", "fox"]
Doc2 → ["The", "quick", "brown", "dog"]
Doc3 → ["The", "lazy", "brown", "dog"]

Doc1 → ["The", "quick", "brown", "fox"]
Doc2 → ["The", "quick", "brown", "dog"]
Doc3 → ["The", "lazy", "brown", "dog"]

But Elasticsearch creates an inverted index:

"The" → [Doc1, Doc2, Doc3]
"quick" → [Doc1, Doc2]
"brown" → [Doc1, Doc2, Doc3]
"fox" → [Doc1]
"dog" → [Doc2, Doc3]
"lazy" → [Doc3]

"The" → [Doc1, Doc2, Doc3]
"quick" → [Doc1, Doc2]
"brown" → [Doc1, Doc2, Doc3]
"fox" → [Doc1]
"dog" → [Doc2, Doc3]
"lazy" → [Doc3]

Now, when you search for “quick dog”, Elasticsearch can instantly find that Doc2 contains both terms, without scanning through all documents.

flowchart TD
    A[Search Query: quick dog] --> B[Tokenization]
    B --> C[Token 1: quick]
    B --> D[Token 2: dog]
    C --> E[Lookup in Inverted Index]
    D --> F[Lookup in Inverted Index]
    E --> G[Found in: Doc1, Doc2]
    F --> H[Found in: Doc2, Doc3]
    G --> I[Intersection]
    H --> I
    I --> J[Result: Doc2]

Core Terminology: Speaking Elasticsearch Fluently

Understanding Elasticsearch terminology is crucial for both using it effectively and discussing it in interviews. Let’s go through each term with real-world analogies:

1. Index An index in Elasticsearch is like a database in the relational world. It’s a collection of documents that share similar characteristics. For example, you might have a “products” index for an e-commerce site or a “logs-2024-01” index for January 2024 logs.

2. Document A document is the basic unit of information that can be indexed. It’s represented as JSON and is analogous to a row in a relational database. Each document has a unique ID within its index.

json

{
  "_index": "products",
  "_id": "SKU-12345",
  "_source": {
    "name": "Wireless Bluetooth Headphones",
    "price": 79.99,
    "category": "Electronics",
    "brand": "TechSound",
    "features": ["Noise Cancelling", "30-hour battery", "Fast charging"],
    "description": "Premium wireless headphones with active noise cancellation..."
  }
}

{
  "_index": "products",
  "_id": "SKU-12345",
  "_source": {
    "name": "Wireless Bluetooth Headphones",
    "price": 79.99,
    "category": "Electronics",
    "brand": "TechSound",
    "features": ["Noise Cancelling", "30-hour battery", "Fast charging"],
    "description": "Premium wireless headphones with active noise cancellation..."
  }
}

3. Type (Deprecated but Important for Legacy Systems) In older versions of Elasticsearch (pre-6.0), types were used to represent different categories of documents within an index. This concept has been removed because it led to confusion and inefficiencies. If you encounter legacy systems, you might still see types, but modern Elasticsearch uses a single type per index.

4. Mapping Mapping defines how documents and their fields are stored and indexed. It’s similar to a schema in relational databases but more flexible. Mappings determine:

• Which fields are searchable
• Which fields are stored but not indexed
• The data type of each field
• How text fields should be analyzed

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard"
      },
      "price": {
        "type": "float"
      },
      "category": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard"
      },
      "price": {
        "type": "float"
      },
      "category": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}

5. Field Types: The Building Blocks

Understanding field types is crucial for designing efficient Elasticsearch schemas. Here are the main categories:

• Text Fields: Analyzed for full-text search. The text “Quick Brown Fox” becomes tokens [“quick”, “brown”, “fox”]
• Keyword Fields: Exact value matching. Used for filtering, sorting, and aggregations
• Numeric Fields: Integer, long, float, double, used for range queries and aggregations
• Date Fields: Timestamps with various formats
• Boolean Fields: True/false values
• Object Fields: For nested JSON structures
• Nested Fields: For arrays of objects that need to maintain relationships

Interview Tip: When discussing field types, always explain the difference between text and keyword. Junior developers often confuse these, but seniors should know that text fields are analyzed and best for search, while keyword fields are for exact matching, sorting, and aggregations.

6. Shard A shard is a self-contained instance of Lucene that holds a subset of your index’s data. Think of sharding like distributing a large encyclopedia across multiple volumes—each volume (shard) can be stored on a different shelf (node) for faster access.

7. Replica Replicas are copies of shards that provide redundancy and improve search performance. If your primary shard fails, a replica can take over, ensuring high availability.

graph TD
    A[Index: products] --> B[Primary Shard 0]
    A --> C[Primary Shard 1]
    A --> D[Primary Shard 2]
    B --> E[Replica 0-1]
    B --> F[Replica 0-2]
    C --> G[Replica 1-1]
    C --> H[Replica 1-2]
    D --> I[Replica 2-1]
    D --> J[Replica 2-2]
    
    style B fill:#f9f,stroke:#333,stroke-width:2px
    style C fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#f9f,stroke:#333,stroke-width:2px

8. Node A node is a single server that’s part of your Elasticsearch cluster. Nodes can have different roles:

• Master Node: Manages cluster-wide settings and operations
• Data Node: Stores data and executes data-related operations
• Ingest Node: Preprocesses documents before indexing
• Coordinating Node: Routes requests and aggregates results

9. Cluster A cluster is a collection of nodes that work together, sharing data and workload. Clusters are identified by a unique name, and nodes join clusters by specifying this name.

How Elasticsearch Differs from Traditional Databases

This is a favorite interview topic because it tests whether you understand the fundamental trade-offs in system design. Let’s break down the key differences:

1. Schema Flexibility

• RDBMS: Strict schema definition required upfront. Changing schema often requires migration.
• Elasticsearch: Dynamic mapping allows new fields to be added automatically. Schema-on-read approach.

2. Query Language

• RDBMS: SQL with JOIN operations for relationships
• Elasticsearch: JSON-based Query DSL optimized for full-text search and aggregations

3. ACID Compliance

• RDBMS: Full ACID compliance with transactions
• Elasticsearch: Eventually consistent, document-level ACID only

4. Scaling Approach

• RDBMS: Typically vertical scaling (bigger servers)
• Elasticsearch: Horizontal scaling by design (add more nodes)

5. Use Cases

• RDBMS: Transactional systems, financial records, strict consistency requirements
• Elasticsearch: Full-text search, log analytics, real-time analytics, fuzzy matching

Common Mistake: Many candidates position Elasticsearch as a replacement for traditional databases. This is wrong! Elasticsearch complements traditional databases. In most architectures, the primary data store remains a traditional database, with Elasticsearch serving as a specialized search and analytics layer.

The Document-Oriented Approach

Unlike relational databases that spread data across normalized tables, Elasticsearch stores denormalized documents. This fundamental difference has profound implications:

// Relational approach (multiple tables and JOINs)
SELECT p.name, p.price, c.name as category, b.name as brand
FROM products p
JOIN categories c ON p.category_id = c.id
JOIN brands b ON p.brand_id = b.id
WHERE p.name LIKE '%headphones%';

<em>// Elasticsearch approach (single denormalized document)</em>
GET /products/_search
{
  "query": {
    "match": {
      "name": "headphones"
    }
  }
}

// Relational approach (multiple tables and JOINs)
SELECT p.name, p.price, c.name as category, b.name as brand
FROM products p
JOIN categories c ON p.category_id = c.id
JOIN brands b ON p.brand_id = b.id
WHERE p.name LIKE '%headphones%';

<em>// Elasticsearch approach (single denormalized document)</em>
GET /products/_search
{
  "query": {
    "match": {
      "name": "headphones"
    }
  }
}

The denormalized approach means:

• Faster queries (no JOINs needed)
• Better performance for read-heavy workloads
• Increased storage requirements
• Complex update operations when denormalized data changes

Elasticsearch Architecture Deep Dive: How It Really Works

Now that we understand the basic concepts, let’s peek under the hood and examine how Elasticsearch achieves its impressive performance and scalability. This section is crucial for senior-level interviews because it demonstrates not just what Elasticsearch does, but how and why it works the way it does.

The Distributed Nature of Elasticsearch

Elasticsearch was built from the ground up as a distributed system. This isn’t an afterthought or a feature added later—it’s fundamental to its architecture. When you start a single Elasticsearch node, you’re actually starting a fully functional cluster with one member. This design philosophy has several implications:

1. No Single Point of Failure: With proper configuration, your cluster can survive node failures
2. Linear Scalability: Need more capacity? Add more nodes
3. Automatic Work Distribution: The cluster automatically balances data and query load

Let’s visualize a typical Elasticsearch cluster architecture:

graph TD
    subgraph Elasticsearch Cluster
        subgraph Master-Eligible Nodes
            M1[Master Node 1<br/>Active Master]
            M2[Master Node 2<br/>Standby]
            M3[Master Node 3<br/>Standby]
        end
        
        subgraph Data Nodes
            D1[Data Node 1<br/>Shards: P0, R1, R2]
            D2[Data Node 2<br/>Shards: P1, R2, R0]
            D3[Data Node 3<br/>Shards: P2, R0, R1]
        end
        
        subgraph Coordinating Nodes
            C1[Coord Node 1]
            C2[Coord Node 2]
        end
    end
    
    LB[Load Balancer] --> C1
    LB --> C2
    
    C1 --> D1
    C1 --> D2
    C1 --> D3
    
    C2 --> D1
    C2 --> D2
    C2 --> D3
    
    M1 -.-> D1
    M1 -.-> D2
    M1 -.-> D3
    
    Client[Application] --> LB

Master Node Election and Cluster State

One of the most critical aspects of Elasticsearch’s distributed architecture is how it maintains consistency across the cluster. This is managed through the master node election process and cluster state management.

The Master Node’s Responsibilities:

• Creating or deleting indices
• Tracking which nodes are part of the cluster
• Allocating shards to nodes
• Managing cluster-wide settings

The master election process uses an adapted version of the Raft consensus algorithm. Here’s how it works:

1. Quorum Requirement: To elect a master, you need (N/2) + 1 master-eligible nodes to agree, where N is the total number of master-eligible nodes.
2. Election Process: When the current master fails, master-eligible nodes start an election:

   Node1: "I haven't heard from the master. I nominate myself!"
   Node2: "I also nominate myself!"
   Node3: "I'll vote for Node1 since it has a higher node ID"

   Node1: "I haven't heard from the master. I nominate myself!"
   Node2: "I also nominate myself!"
   Node3: "I'll vote for Node1 since it has a higher node ID"

3. Split Brain Prevention: The quorum requirement prevents split-brain scenarios where two parts of a cluster elect different masters.

Pro Insight: Always configure an odd number of master-eligible nodes (3, 5, 7) to maintain quorum even with node failures. Three master nodes can tolerate one failure, five can tolerate two failures.

The Indexing Pipeline: From Document to Searchable Data

Understanding how Elasticsearch indexes documents is crucial for optimizing performance and diagnosing issues. Let’s trace the journey of a document from API call to searchable data:

sequenceDiagram
    participant Client
    participant CoordNode as Coordinating Node
    participant MasterNode as Master Node
    participant PrimaryShard as Primary Shard
    participant ReplicaShard as Replica Shard
    participant Translog as Transaction Log
    participant Lucene as Lucene Index
    
    Client->>CoordNode: Index Document Request
    CoordNode->>CoordNode: Determine target shard<br/>(using routing)
    CoordNode->>PrimaryShard: Forward document
    PrimaryShard->>Translog: Write to transaction log
    PrimaryShard->>Lucene: Add to in-memory buffer
    PrimaryShard->>ReplicaShard: Replicate document
    ReplicaShard->>Translog: Write to transaction log
    ReplicaShard->>Lucene: Add to in-memory buffer
    ReplicaShard-->>PrimaryShard: Acknowledge
    PrimaryShard-->>CoordNode: Success
    CoordNode-->>Client: Index successful
    
    Note over Lucene: Refresh (every 1s by default)
    Lucene->>Lucene: Create new segment<br/>Document becomes searchable
    
    Note over Lucene: Flush (every 30min or when translog is full)
    Lucene->>Lucene: Write segments to disk
    Translog->>Translog: Clear transaction log

Let’s break down each stage:

1. Document Routing When a document arrives at a coordinating node, Elasticsearch must determine which shard should store it. This is done using a routing formula:

shard_num = hash(_routing) % num_primary_shards

shard_num = hash(_routing) % num_primary_shards

By default, _routing is the document ID, but you can specify custom routing:

json

PUT /products/_doc/12345?routing=electronics
{
  "name": "Laptop",
  "category": "electronics"
}

PUT /products/_doc/12345?routing=electronics
{
  "name": "Laptop",
  "category": "electronics"
}

Interview Tip: Custom routing can significantly improve query performance by ensuring related documents are on the same shard, but be careful—it can lead to uneven shard sizes if not managed properly.

2. Primary Indexing Once the target shard is identified, the document is sent to the primary shard, where several operations occur:

• Transaction Log Write: The document is first written to the transaction log (translog) for durability
• In-Memory Buffering: The document is added to an in-memory buffer
• Analysis: Text fields are analyzed according to their configured analyzers

3. Replication After the primary shard indexes the document, it’s replicated to all replica shards. Elasticsearch supports two replication types:

• Synchronous Replication (default): The primary waits for all replicas to acknowledge before responding to the client
• Asynchronous Replication: The primary responds immediately after indexing locally

4. Refresh and Searchability Documents aren’t immediately searchable after indexing. They become searchable after a “refresh” operation, which happens every second by default. This is a trade-off between indexing performance and search freshness.

java

<em>// Force immediate refresh (expensive operation)</em>
IndexResponse response = client.index(request, RequestOptions.DEFAULT);
client.indices().refresh(new RefreshRequest("products"), RequestOptions.DEFAULT);

<em>// Better approach: let Elasticsearch handle refreshing</em>
IndexRequest request = new IndexRequest("products")
    .id("1")
    .source("name", "Product")
    .setRefreshPolicy(RefreshPolicy.WAIT_UNTIL); <em>// Wait until next refresh</em>

<em>// Force immediate refresh (expensive operation)</em>
IndexResponse response = client.index(request, RequestOptions.DEFAULT);
client.indices().refresh(new RefreshRequest("products"), RequestOptions.DEFAULT);

<em>// Better approach: let Elasticsearch handle refreshing</em>
IndexRequest request = new IndexRequest("products")
    .id("1")
    .source("name", "Product")
    .setRefreshPolicy(RefreshPolicy.WAIT_UNTIL); <em>// Wait until next refresh</em>

Lucene Segments: The Foundation of Search Performance

Under the hood, each Elasticsearch shard is a Lucene index, and each Lucene index consists of multiple segments. Understanding segments is crucial for optimizing Elasticsearch performance:

What is a Segment? A segment is an immutable, self-contained index structure. When documents are indexed, they’re initially stored in an in-memory buffer. During a refresh, this buffer is written as a new segment.

flowchart TD
    subgraph "Shard (Lucene Index)"
        A[In-Memory Buffer<br/>New Documents] -->|Refresh| B[Segment 1<br/>Immutable]
        A -->|Refresh| C[Segment 2<br/>Immutable]
        A -->|Refresh| D[Segment 3<br/>Immutable]
        B -->|Merge| E[Merged Segment<br/>Larger, Optimized]
        C -->|Merge| E
        F[Segment 4<br/>Deleted Docs Marked]
    end
    
    style A fill:#ff9999
    style B fill:#99ccff
    style C fill:#99ccff
    style D fill:#99ccff
    style E fill:#99ff99

Key Characteristics of Segments:

1. Immutability: Once written, segments are never modified
2. Append-Only: New documents create new segments
3. Deletion Handling: Deletions are marked in a separate file (.del file)
4. Search Process: Queries must check all segments and merge results

Segment Merging Having many small segments hurts search performance, so Elasticsearch continuously merges segments in the background:

java

// Manual force merge (use carefully in production)
ForceMergeRequest request = new ForceMergeRequest("products");
request.maxNumSegments(1); <em>// Merge down to 1 segment per shard</em>
client.indices().forcemerge(request, RequestOptions.DEFAULT);

// Manual force merge (use carefully in production)
ForceMergeRequest request = new ForceMergeRequest("products");
request.maxNumSegments(1); <em>// Merge down to 1 segment per shard</em>
client.indices().forcemerge(request, RequestOptions.DEFAULT);

Common Mistake: Aggressive force merging on active indices. This is resource-intensive and can harm indexing performance. Reserve force merging for read-only indices.

Query Execution: The Two-Phase Process

When you search in Elasticsearch, the query execution follows a sophisticated two-phase process designed to minimize network traffic while maximizing relevance:

sequenceDiagram
    participant Client
    participant Coord as Coordinating Node
    participant Shard1 as Shard 1
    participant Shard2 as Shard 2
    participant Shard3 as Shard 3
    
    Note over Client,Shard3: Phase 1: Query Phase
    Client->>Coord: Search Request<br/>size=10
    Coord->>Shard1: Query (return top 10)
    Coord->>Shard2: Query (return top 10)
    Coord->>Shard3: Query (return top 10)
    
    Shard1-->>Coord: Doc IDs + Scores<br/>[{id:1,score:0.9}, {id:4,score:0.7}...]
    Shard2-->>Coord: Doc IDs + Scores<br/>[{id:2,score:0.95}, {id:5,score:0.6}...]
    Shard3-->>Coord: Doc IDs + Scores<br/>[{id:3,score:0.85}, {id:6,score:0.5}...]
    
    Coord->>Coord: Merge & Sort by Score<br/>Select top 10 globally
    
    Note over Client,Shard3: Phase 2: Fetch Phase
    Coord->>Shard2: Fetch full document (id:2)
    Coord->>Shard1: Fetch full document (id:1)
    Coord->>Shard3: Fetch full document (id:3)
    
    Shard2-->>Coord: Full document data
    Shard1-->>Coord: Full document data
    Shard3-->>Coord: Full document data
    
    Coord->>Client: Final results with full documents

Phase 1: Query Phase

• The coordinating node sends the query to all relevant shards
• Each shard executes the query locally and returns document IDs and relevance scores
• The coordinating node merges results and identifies the top N documents globally

Phase 2: Fetch Phase

• The coordinating node requests the full document content only for the top N documents
• Shards return the complete document data
• The coordinating node assembles the final response

This two-phase approach is brilliant because:

1. Minimized Network Traffic: Only top results are fetched in full
2. Distributed Scoring: Each shard calculates relevance independently
3. Efficient Sorting: Global sorting happens on lightweight ID/score pairs

Cluster State Management and Consistency

The cluster state is the metadata that defines the cluster’s current configuration. It includes:

• Index mappings and settings
• Shard allocation tables
• Node membership information
• Cluster-wide settings

This state must be consistent across all nodes, which Elasticsearch achieves through:

1. Master-Only Updates Only the elected master node can update the cluster state. This prevents conflicts and ensures a single source of truth.

2. Two-Phase Commit Protocol When updating cluster state:

Master -> All Nodes: "Prepare to commit state version 42"
All Nodes -> Master: "Ready to commit"
Master -> All Nodes: "Commit state version 42"
All Nodes: Apply new state

Master -> All Nodes: "Prepare to commit state version 42"
All Nodes -> Master: "Ready to commit"
Master -> All Nodes: "Commit state version 42"
All Nodes: Apply new state

3. Versioning Each cluster state has a version number. Nodes can quickly determine if they have the latest state:

json

{
  "version": 42,
  "state_uuid": "9KJH3423JK4H23K4J",
  "master_node": "node-1",
  "blocks": {},
  "nodes": {
    "node-1": { "name": "es-node-1", "roles": ["master", "data"] },
    "node-2": { "name": "es-node-2", "roles": ["data"] }
  }
}

{
  "version": 42,
  "state_uuid": "9KJH3423JK4H23K4J",
  "master_node": "node-1",
  "blocks": {},
  "nodes": {
    "node-1": { "name": "es-node-1", "roles": ["master", "data"] },
    "node-2": { "name": "es-node-2", "roles": ["data"] }
  }
}

Data Modeling in Elasticsearch: Mappings, Types, and Index Design {#data-modeling}

Data modeling in Elasticsearch is both an art and a science. Unlike relational databases where you normalize data across tables, Elasticsearch requires a different mindset—one that prioritizes query performance and denormalization. This section will equip you with the knowledge to design efficient Elasticsearch schemas that scale.

Understanding Mappings: Your Schema Definition

Mappings in Elasticsearch define how documents and their fields are indexed and stored. While Elasticsearch can automatically detect and add new field types (dynamic mapping), production systems typically use explicit mappings for better control and performance.

Let’s start with a comprehensive example of a product catalog mapping:

json

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 2,
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "snowball"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "product_id": {
        "type": "keyword"
      },
      "name": {
        "type": "text",
        "analyzer": "custom_analyzer",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "standard"
      },
      "price": {
        "type": "scaled_float",
        "scaling_factor": 100
      },
      "in_stock": {
        "type": "boolean"
      },
      "categories": {
        "type": "keyword"
      },
      "attributes": {
        "type": "object",
        "properties": {
          "color": {
            "type": "keyword"
          },
          "size": {
            "type": "keyword"
          },
          "material": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
        }
      },
      "created_at": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      },
      "reviews": {
        "type": "nested",
        "properties": {
          "user_id": {
            "type": "keyword"
          },
          "rating": {
            "type": "byte"
          },
          "comment": {
            "type": "text"
          },
          "created_at": {
            "type": "date"
          }
        }
      }
    }
  }
}

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 2,
    "analysis": {
      "analyzer": {
        "custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "snowball"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "product_id": {
        "type": "keyword"
      },
      "name": {
        "type": "text",
        "analyzer": "custom_analyzer",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "standard"
      },
      "price": {
        "type": "scaled_float",
        "scaling_factor": 100
      },
      "in_stock": {
        "type": "boolean"
      },
      "categories": {
        "type": "keyword"
      },
      "attributes": {
        "type": "object",
        "properties": {
          "color": {
            "type": "keyword"
          },
          "size": {
            "type": "keyword"
          },
          "material": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
        }
      },
      "created_at": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      },
      "reviews": {
        "type": "nested",
        "properties": {
          "user_id": {
            "type": "keyword"
          },
          "rating": {
            "type": "byte"
          },
          "comment": {
            "type": "text"
          },
          "created_at": {
            "type": "date"
          }
        }
      }
    }
  }
}

Let’s break down the key decisions in this mapping:

1. Field Type Selection

• product_id: Keyword type for exact matching
• name: Text type with custom analyzer for search, plus a keyword sub-field for sorting/aggregations
• price: Scaled float to save storage (stores 19.99 as 1999)
• reviews: Nested type to maintain relationship between review fields

2. Multi-Fields Pattern The name field demonstrates a powerful pattern:

json

"name": {
  "type": "text",
  "analyzer": "custom_analyzer",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

"name": {
  "type": "text",
  "analyzer": "custom_analyzer",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

This allows you to:

• Search on name (analyzed text)
• Sort/aggregate on name.keyword (exact value)
• Query: GET /products/_search?q=name:wireless&sort=name.keyword:asc

3. Dynamic Templates For flexible schemas, dynamic templates let you define rules for automatically mapped fields:

json

"dynamic_templates": [
  {
    "strings_as_keywords": {
      "match_mapping_type": "string",
      "match": "*_id",
      "mapping": {
        "type": "keyword"
      }
    }
  },
  {
    "longs_as_integers": {
      "match_mapping_type": "long",
      "match": "*_count",
      "mapping": {
        "type": "integer"
      }
    }
  }
]

"dynamic_templates": [
  {
    "strings_as_keywords": {
      "match_mapping_type": "string",
      "match": "*_id",
      "mapping": {
        "type": "keyword"
      }
    }
  },
  {
    "longs_as_integers": {
      "match_mapping_type": "long",
      "match": "*_count",
      "mapping": {
        "type": "integer"
      }
    }
  }
]

Text Analysis: The Heart of Search Quality

Text analysis is what transforms “The quick BROWN foxes!” into searchable tokens. Understanding analyzers is crucial for building effective search experiences.

mermaid

flowchart LR
    A[Original Text:<br/>"The quick BROWN foxes!"] --> B[Character Filters]
    B --> C[Tokenizer]
    C --> D[Token Filters]
    D --> E[Final Tokens:<br/>[quick, brown, fox]]
    
    B --> B1[HTML Strip<br/>Mapping<br/>Pattern Replace]
    C --> C1[Standard<br/>Whitespace<br/>Pattern<br/>Keyword]
    D --> D1[Lowercase<br/>Stop Words<br/>Stemming<br/>Synonyms]

Anatomy of an Analyzer:

1. Character Filters (Optional): Preprocess the text
   • HTML Strip: Remove HTML tags
   • Mapping: Replace characters (e.g., “+” → “plus”)
   • Pattern Replace: Regex-based replacements
2. Tokenizer (Required): Break text into tokens
   • Standard: Grammar-based tokenization
   • Whitespace: Split on whitespace only
   • Pattern: Split based on regex
   • Keyword: Don’t tokenize (entire text as one token)
3. Token Filters (Optional): Modify tokens
   • Lowercase: Convert to lowercase
   • Stop: Remove common words (the, is, at)
   • Stemmer: Reduce words to root form (running → run)
   • Synonym: Add synonym tokens

Let’s create a custom analyzer for product search:

json

PUT /products
{
  "settings": {
    "analysis": {
      "char_filter": {
        "special_chars": {
          "type": "mapping",
          "mappings": [
            "+ => plus",
            "& => and",
            "@ => at"
          ]
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "product_synonyms": {
          "type": "synonym",
          "synonyms": [
            "laptop,notebook,computer",
            "phone,mobile,smartphone",
            "tv,television,telly"
          ]
        },
        "edge_ngram_filter": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 20
        }
      },
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "char_filter": ["special_chars"],
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "english_stop",
            "product_synonyms",
            "snowball"
          ]
        },
        "autocomplete_analyzer": {
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "edge_ngram_filter"
          ]
        }
      }
    }
  }
}

PUT /products
{
  "settings": {
    "analysis": {
      "char_filter": {
        "special_chars": {
          "type": "mapping",
          "mappings": [
            "+ => plus",
            "& => and",
            "@ => at"
          ]
        }
      },
      "filter": {
        "english_stop": {
          "type": "stop",
          "stopwords": "_english_"
        },
        "product_synonyms": {
          "type": "synonym",
          "synonyms": [
            "laptop,notebook,computer",
            "phone,mobile,smartphone",
            "tv,television,telly"
          ]
        },
        "edge_ngram_filter": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 20
        }
      },
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "char_filter": ["special_chars"],
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "english_stop",
            "product_synonyms",
            "snowball"
          ]
        },
        "autocomplete_analyzer": {
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "edge_ngram_filter"
          ]
        }
      }
    }
  }
}

Pro Insight: Test your analyzers before deploying! Use the Analyze API:

POST /products/_analyze
{
  "analyzer": "product_analyzer",
  "text": "I'm looking for a laptop + mouse"
}

POST /products/_analyze
{
  "analyzer": "product_analyzer",
  "text": "I'm looking for a laptop + mouse"
}

Nested vs Object vs Flattened Types

One of the most common data modeling challenges in Elasticsearch is handling complex, hierarchical data. Let’s explore the three main approaches:

1. Object Type (Default)

{
  "user": {
    "name": "John Doe",
    "addresses": [
      {
        "type": "home",
        "city": "New York"
      },
      {
        "type": "work",
        "city": "Boston"
      }
    ]
  }
}

{
  "user": {
    "name": "John Doe",
    "addresses": [
      {
        "type": "home",
        "city": "New York"
      },
      {
        "type": "work",
        "city": "Boston"
      }
    ]
  }
}

Internally flattened to:

{
  "user.name": "John Doe",
  "user.addresses.type": ["home", "work"],
  "user.addresses.city": ["New York", "Boston"]
}

{
  "user.name": "John Doe",
  "user.addresses.type": ["home", "work"],
  "user.addresses.city": ["New York", "Boston"]
}

Problem: You lose the relationship between type and city. A query for “home address in Boston” would incorrectly match this document.

2. Nested Type

"addresses": {
  "type": "nested",
  "properties": {
    "type": { "type": "keyword" },
    "city": { "type": "keyword" }
  }
}

"addresses": {
  "type": "nested",
  "properties": {
    "type": { "type": "keyword" },
    "city": { "type": "keyword" }
  }
}

Nested documents maintain relationships but come with costs:

• Each nested document is indexed separately
• Special nested queries required
• Limited to 10,000 nested documents per parent by default
• Performance impact on indexing and querying

3. Flattened Type

"labels": {
  "type": "flattened"
}

"labels": {
  "type": "flattened"
}

Entire object indexed as keywords:

• Great for arbitrary key-value pairs
• No mapping explosion
• Limited query capabilities (only exact matching)

When to Use Each:

• Object: Simple objects where field relationships don’t matter
• Nested: When you need to query on related fields within arrays of objects
• Flattened: Dynamic schemas with many unique fields (e.g., user-defined tags)

Index Design Patterns and Best Practices

Designing indices isn’t just about mappings—it’s about organizing your data for optimal performance, maintenance, and cost. Let’s explore battle-tested patterns:

1. Time-Based Indices For time-series data like logs or metrics:

logs-2024-01-01
logs-2024-01-02
logs-2024-01-03

logs-2024-01-01
logs-2024-01-02
logs-2024-01-03

Benefits:

• Easy deletion of old data (drop entire index)
• Better cache utilization (recent data stays hot)
• Simpler backup strategies
• Can optimize older indices (force merge, close, etc.)

Implementation with Index Templates:

PUT /_index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "refresh_interval": "5s"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "message": { "type": "text" }
      }
    }
  }
}

PUT /_index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "refresh_interval": "5s"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "message": { "type": "text" }
      }
    }
  }
}

2. Index Aliases Aliases provide a layer of abstraction over physical indices:

POST /_aliases
{
  "actions": [
    { "add": { "index": "products_v1", "alias": "products" } },
    { "add": { "index": "products_v1", "alias": "products_write" } }
  ]
}

POST /_aliases
{
  "actions": [
    { "add": { "index": "products_v1", "alias": "products" } },
    { "add": { "index": "products_v1", "alias": "products_write" } }
  ]
}

Use cases:

• Zero-downtime reindexing: Switch aliases after reindexing
• Multi-tenant applications: Filter aliases per customer
• A/B testing: Route traffic to different index versions

3. Rollover Pattern Automatically create new indices based on conditions:

PUT /logs-000001
{
  "aliases": {
    "logs_write": {}
  }
}

PUT /logs_write/_rollover
{
  "conditions": {
    "max_age": "7d",
    "max_size": "50GB",
    "max_docs": 10000000
  }
}

PUT /logs-000001
{
  "aliases": {
    "logs_write": {}
  }
}

PUT /logs_write/_rollover
{
  "conditions": {
    "max_age": "7d",
    "max_size": "50GB",
    "max_docs": 10000000
  }
}

4. Index Lifecycle Management (ILM) Automate index management through different phases:

flowchart LR
    A[Hot Phase<br/>Active writes] --> B[Warm Phase<br/>Read-only, fewer replicas]
    B --> C[Cold Phase<br/>Frozen, searchable snapshots]
    C --> D[Delete Phase<br/>Remove index]
    
    style A fill:#ff9999
    style B fill:#ffcc99
    style C fill:#99ccff
    style D fill:#cccccc

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Parent-Child vs Denormalization

One of the most critical decisions in Elasticsearch data modeling is how to handle relationships. Unlike relational databases, Elasticsearch doesn’t have JOINs, so you need different strategies:

1. Denormalization (Recommended) Store all related data in a single document:

{
  "order_id": "ORD-12345",
  "customer": {
    "id": "CUST-789",
    "name": "John Doe",
    "email": "john@example.com"
  },
  "items": [
    {
      "product_id": "PROD-111",
      "name": "Laptop",
      "price": 999.99,
      "quantity": 1
    }
  ],
  "total": 999.99
}

{
  "order_id": "ORD-12345",
  "customer": {
    "id": "CUST-789",
    "name": "John Doe",
    "email": "john@example.com"
  },
  "items": [
    {
      "product_id": "PROD-111",
      "name": "Laptop",
      "price": 999.99,
      "quantity": 1
    }
  ],
  "total": 999.99
}

Pros:

• Fastest query performance
• Simple queries
• No relationship overhead

Cons:

• Data duplication
• Updates require reindexing multiple documents
• Larger index size

2. Parent-Child Relationships Use when you need to update child documents frequently:

PUT /orders_and_items
{
  "mappings": {
    "properties": {
      "join_field": {
        "type": "join",
        "relations": {
          "order": "item"
        }
      }
    }
  }
}

<em>// Parent document</em>
PUT /orders_and_items/_doc/order-1
{
  "order_id": "ORD-12345",
  "customer_name": "John Doe",
  "join_field": "order"
}

<em>// Child document</em>
PUT /orders_and_items/_doc/item-1?routing=order-1
{
  "product_name": "Laptop",
  "price": 999.99,
  "join_field": {
    "name": "item",
    "parent": "order-1"
  }
}

PUT /orders_and_items
{
  "mappings": {
    "properties": {
      "join_field": {
        "type": "join",
        "relations": {
          "order": "item"
        }
      }
    }
  }
}

<em>// Parent document</em>
PUT /orders_and_items/_doc/order-1
{
  "order_id": "ORD-12345",
  "customer_name": "John Doe",
  "join_field": "order"
}

<em>// Child document</em>
PUT /orders_and_items/_doc/item-1?routing=order-1
{
  "product_name": "Laptop",
  "price": 999.99,
  "join_field": {
    "name": "item",
    "parent": "order-1"
  }
}

Interview Question: “When would you use parent-child relationships over denormalization?”

Junior Answer: “When we have related data that needs to be queried together.”

Senior Answer: “Parent-child relationships are suitable when child documents are updated frequently and independently of parents, when the number of children per parent varies significantly (avoiding document size limits), or when we need to aggregate on child documents across multiple parents. However, they come with significant performance costs—queries are 5-10x slower than denormalized data because Elasticsearch must perform join operations at query time. They also require careful routing to ensure parents and children are on the same shard. In most cases, denormalization with periodic updates is preferable unless update frequency makes it impractical.”

Managing Mapping Explosions

A mapping explosion occurs when dynamic mapping creates too many fields, leading to memory pressure and performance degradation. This is common with dynamic user-generated content:

json

{
  "user_attributes": {
    "favorite_color": "blue",
    "lucky_number": 7,
    "pet_name": "Fluffy",
    <em>// Potentially thousands of unique fields...</em>
  }
}

{
  "user_attributes": {
    "favorite_color": "blue",
    "lucky_number": 7,
    "pet_name": "Fluffy",
    <em>// Potentially thousands of unique fields...</em>
  }
}

Solutions:

1. Limit Total Fields

"settings": {
  "index.mapping.total_fields.limit": 1000
}

"settings": {
  "index.mapping.total_fields.limit": 1000
}

2. Disable Dynamic Mapping

"mappings": {
  "dynamic": "strict",  <em>// Reject unmapped fields</em>
  <em>// or "dynamic": false // Ignore unmapped fields</em>
}

"mappings": {
  "dynamic": "strict",  <em>// Reject unmapped fields</em>
  <em>// or "dynamic": false // Ignore unmapped fields</em>
}

3. Use Flattened Type

"user_attributes": {
  "type": "flattened"
}

"user_attributes": {
  "type": "flattened"
}

4. Dynamic Templates with Limits

"dynamic_templates": [
  {
    "strings_as_keywords": {
      "match_mapping_type": "string",
      "match": "attr_*",
      "mapping": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
]

"dynamic_templates": [
  {
    "strings_as_keywords": {
      "match_mapping_type": "string",
      "match": "attr_*",
      "mapping": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
]

Indexing Strategies: Performance Optimization from Day One

Indexing performance can make or break your Elasticsearch deployment. The difference between a well-optimized indexing pipeline and a naive implementation can be 10x or more in throughput. This section covers advanced indexing strategies that separate senior engineers from juniors.

Bulk Indexing: The Foundation of High-Performance Ingestion

Single-document indexing is like making individual HTTP requests for each pixel of an image—technically possible but painfully inefficient. Bulk indexing is the cornerstone of performance:

<em>// Bad: Individual indexing</em>
for (Product product : products) {
    IndexRequest request = new IndexRequest("products")
        .id(product.getId())
        .source(convertToJson(product));
    client.index(request, RequestOptions.DEFAULT);
}

<em>// Good: Bulk indexing</em>
BulkRequest bulkRequest = new BulkRequest();
for (Product product : products) {
    bulkRequest.add(new IndexRequest("products")
        .id(product.getId())
        .source(convertToJson(product)));
}
BulkResponse bulkResponse = client.bulk(bulkRequest, RequestOptions.DEFAULT);

<em>// Bad: Individual indexing</em>
for (Product product : products) {
    IndexRequest request = new IndexRequest("products")
        .id(product.getId())
        .source(convertToJson(product));
    client.index(request, RequestOptions.DEFAULT);
}

<em>// Good: Bulk indexing</em>
BulkRequest bulkRequest = new BulkRequest();
for (Product product : products) {
    bulkRequest.add(new IndexRequest("products")
        .id(product.getId())
        .source(convertToJson(product)));
}
BulkResponse bulkResponse = client.bulk(bulkRequest, RequestOptions.DEFAULT);

Optimal Bulk Size Calculation: The ideal bulk size depends on your document size and cluster resources. Here’s a systematic approach:

def find_optimal_bulk_size(es_client, test_data):
    bulk_sizes = [100, 500, 1000, 5000, 10000]
    results = {}
    
    for size in bulk_sizes:
        start_time = time.time()
        
        for i in range(0, len(test_data), size):
            batch = test_data[i:i+size]
            bulk_body = []
            
            for doc in batch:
                bulk_body.append({
                    "index": {
                        "_index": "test_index",
                        "_id": doc['id']
                    }
                })
                bulk_body.append(doc)
            
            es_client.bulk(body=bulk_body)
        
        elapsed = time.time() - start_time
        docs_per_second = len(test_data) / elapsed
        results[size] = docs_per_second
        
        print(f"Bulk size {size}: {docs_per_second:.2f} docs/second")
    
    return results

def find_optimal_bulk_size(es_client, test_data):
    bulk_sizes = [100, 500, 1000, 5000, 10000]
    results = {}
    
    for size in bulk_sizes:
        start_time = time.time()
        
        for i in range(0, len(test_data), size):
            batch = test_data[i:i+size]
            bulk_body = []
            
            for doc in batch:
                bulk_body.append({
                    "index": {
                        "_index": "test_index",
                        "_id": doc['id']
                    }
                })
                bulk_body.append(doc)
            
            es_client.bulk(body=bulk_body)
        
        elapsed = time.time() - start_time
        docs_per_second = len(test_data) / elapsed
        results[size] = docs_per_second
        
        print(f"Bulk size {size}: {docs_per_second:.2f} docs/second")
    
    return results

Key Considerations:

• Start with 1,000-5,000 documents per bulk request
• Keep bulk request size under 100MB
• Monitor rejection rates: thread_pool.bulk.rejected
• Adjust based on document size (fewer large documents, more small ones)

Thread Pool Tuning and Queue Management

Elasticsearch uses thread pools to handle different operations. Understanding and tuning these is crucial for optimal performance:

flowchart TD
    A[Incoming Requests] --> B{Request Type}
    B -->|Search| C[Search Thread Pool<br/>Size: 3/2 * CPU + 1<br/>Queue: 1000]
    B -->|Index| D[Write Thread Pool<br/>Size: CPU cores<br/>Queue: 10000]
    B -->|Bulk| E[Bulk Thread Pool<br/>Size: CPU cores<br/>Queue: 200]
    B -->|Get| F[Get Thread Pool<br/>Size: CPU cores<br/>Queue: 1000]
    
    C --> G[Accepted or Rejected]
    D --> G
    E --> G
    F --> G

Monitoring Thread Pool Health:

GET /_cat/thread_pool/bulk,write,search?v&h=node_name,name,active,rejected,completed

node_name name   active rejected completed
node-1    bulk   5      1203     984932
node-1    write  2      0        432432
node-1    search 8      23       234234

GET /_cat/thread_pool/bulk,write,search?v&h=node_name,name,active,rejected,completed

node_name name   active rejected completed
node-1    bulk   5      1203     984932
node-1    write  2      0        432432
node-1    search 8      23       234234

Tuning Strategies:

1. Increase Queue Size (Temporary Fix)

thread_pool:
  bulk:
    queue_size: 500  <em># Default: 200</em>

thread_pool:
  bulk:
    queue_size: 500  <em># Default: 200</em>

2. Add More Nodes (Proper Scaling) If you’re consistently seeing rejections, add data nodes to distribute load.
3. Client-Side Retry Logic

public void indexWithRetry(BulkRequest request, int maxRetries) {
    int retries = 0;
    while (retries < maxRetries) {
        try {
            BulkResponse response = client.bulk(request, RequestOptions.DEFAULT);
            if (!response.hasFailures()) {
                return;
            }
            <em>// Handle partial failures</em>
            request = handleBulkFailures(response, request);
            if (request.numberOfActions() == 0) {
                return;
            }
        } catch (Exception e) {
            if (++retries >= maxRetries) {
                throw new RuntimeException("Max retries exceeded", e);
            }
            try {
                Thread.sleep((long) Math.pow(2, retries) * 1000);
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
            }
        }
    }
}

public void indexWithRetry(BulkRequest request, int maxRetries) {
    int retries = 0;
    while (retries < maxRetries) {
        try {
            BulkResponse response = client.bulk(request, RequestOptions.DEFAULT);
            if (!response.hasFailures()) {
                return;
            }
            <em>// Handle partial failures</em>
            request = handleBulkFailures(response, request);
            if (request.numberOfActions() == 0) {
                return;
            }
        } catch (Exception e) {
            if (++retries >= maxRetries) {
                throw new RuntimeException("Max retries exceeded", e);
            }
            try {
                Thread.sleep((long) Math.pow(2, retries) * 1000);
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
            }
        }
    }
}

Refresh Interval Optimization

The refresh interval determines how quickly indexed documents become searchable. It’s a critical tuning parameter:

PUT /high_volume_index/_settings
{
  "index": {
    "refresh_interval": "30s"  <em>// Default: 1s</em>
  }
}

<em>// Disable refresh during bulk loading</em>
PUT /high_volume_index/_settings
{
  "index": {
    "refresh_interval": "-1"
  }
}

<em>// Re-enable after bulk load</em>
PUT /high_volume_index/_settings
{
  "index": {
    "refresh_interval": "1s"
  }
}

PUT /high_volume_index/_settings
{
  "index": {
    "refresh_interval": "30s"  <em>// Default: 1s</em>
  }
}

<em>// Disable refresh during bulk loading</em>
PUT /high_volume_index/_settings
{
  "index": {
    "refresh_interval": "-1"
  }
}

<em>// Re-enable after bulk load</em>
PUT /high_volume_index/_settings
{
  "index": {
    "refresh_interval": "1s"
  }
}

Performance Impact:

• 1s refresh: Good for near real-time search, higher CPU/IO usage
• 30s refresh: 30x fewer refresh operations, better indexing throughput
• -1 (disabled): Maximum indexing performance, no search visibility

Pro Insight: For log analytics, users rarely need sub-second search latency. Setting refresh_interval to 30s or 60s can dramatically improve indexing throughput without impacting user experience.

Document Routing Strategies

Custom routing can significantly impact both indexing and query performance:

<em>// Route by user ID for a multi-tenant application</em>
IndexRequest request = new IndexRequest("user_data")
    .id(documentId)
    .routing(userId)
    .source(documentJson);

<em>// Query with routing</em>
SearchRequest searchRequest = new SearchRequest("user_data");
searchRequest.routing(userId);
searchRequest.source(new SearchSourceBuilder()
    .query(QueryBuilders.termQuery("user_id", userId)));

<em>// Route by user ID for a multi-tenant application</em>
IndexRequest request = new IndexRequest("user_data")
    .id(documentId)
    .routing(userId)
    .source(documentJson);

<em>// Query with routing</em>
SearchRequest searchRequest = new SearchRequest("user_data");
searchRequest.routing(userId);
searchRequest.source(new SearchSourceBuilder()
    .query(QueryBuilders.termQuery("user_id", userId)));

Benefits of Custom Routing:

1. Improved Query Performance: Queries hit only relevant shards
2. Better Cache Utilization: Related documents on same shard
3. Simplified Multi-Tenancy: Natural data isolation

Risks to Consider:

1. Hot Shards: Popular routing values can overload shards
2. Rebalancing Issues: Difficult to redistribute skewed shards
3. Routing Required: All operations must include routing value

Ingest Pipelines: Transform Data Before Indexing

Ingest pipelines allow you to preprocess documents before indexing, moving computation from query-time to index-time:

PUT /_ingest/pipeline/product_pipeline
{
  "description": "Process product documents",
  "processors": [
    {
      "lowercase": {
        "field": "category"
      }
    },
    {
      "convert": {
        "field": "price",
        "type": "float"
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.price < 20) {
            ctx.price_range = 'budget';
          } else if (ctx.price < 100) {
            ctx.price_range = 'mid-range';
          } else {
            ctx.price_range = 'premium';
          }
        """
      }
    },
    {
      "date": {
        "field": "timestamp",
        "target_field": "@timestamp",
        "formats": ["ISO8601", "yyyy-MM-dd HH:mm:ss"]
      }
    },
    {
      "geoip": {
        "field": "customer_ip",
        "target_field": "customer_location"
      }
    }
  ]
}

<em>// Use the pipeline</em>
PUT /products/_doc/1?pipeline=product_pipeline
{
  "name": "Wireless Mouse",
  "category": "ELECTRONICS",
  "price": "29.99",
  "timestamp": "2024-01-15 10:30:00",
  "customer_ip": "8.8.8.8"
}

PUT /_ingest/pipeline/product_pipeline
{
  "description": "Process product documents",
  "processors": [
    {
      "lowercase": {
        "field": "category"
      }
    },
    {
      "convert": {
        "field": "price",
        "type": "float"
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": """
          if (ctx.price < 20) {
            ctx.price_range = 'budget';
          } else if (ctx.price < 100) {
            ctx.price_range = 'mid-range';
          } else {
            ctx.price_range = 'premium';
          }
        """
      }
    },
    {
      "date": {
        "field": "timestamp",
        "target_field": "@timestamp",
        "formats": ["ISO8601", "yyyy-MM-dd HH:mm:ss"]
      }
    },
    {
      "geoip": {
        "field": "customer_ip",
        "target_field": "customer_location"
      }
    }
  ]
}

<em>// Use the pipeline</em>
PUT /products/_doc/1?pipeline=product_pipeline
{
  "name": "Wireless Mouse",
  "category": "ELECTRONICS",
  "price": "29.99",
  "timestamp": "2024-01-15 10:30:00",
  "customer_ip": "8.8.8.8"
}

Common Pipeline Processors:

• Grok: Parse unstructured log data
• Date: Parse date strings into proper date objects
• GeoIP: Enrich IP addresses with location data
• Script: Custom transformations with Painless
• Enrich: Join with data from other indices

Handling Document Updates: The Versioning Challenge

Elasticsearch documents are immutable at the segment level. Updates actually create new versions, making update patterns crucial for performance:

mermaid

sequenceDiagram
    participant Client
    participant ES as Elasticsearch
    participant Segment as Lucene Segment
    
    Client->>ES: Update document ID: 123
    ES->>ES: Retrieve current document
    ES->>ES: Apply changes
    ES->>Segment: Mark old version as deleted
    ES->>Segment: Index new version
    ES-->>Client: Update successful
    
    Note over Segment: Old version remains until<br/>segment merge

Update Strategies:

1. Full Document Updates

UpdateRequest request = new UpdateRequest("products", "123")
    .doc(XContentType.JSON, 
         "price", 29.99,
         "last_updated", Instant.now());

UpdateRequest request = new UpdateRequest("products", "123")
    .doc(XContentType.JSON, 
         "price", 29.99,
         "last_updated", Instant.now());

2. Partial Updates with Scripts

POST /products/_update/123
{
  "script": {
    "source": "ctx._source.view_count += params.increment",
    "params": {
      "increment": 1
    }
  }
}

POST /products/_update/123
{
  "script": {
    "source": "ctx._source.view_count += params.increment",
    "params": {
      "increment": 1
    }
  }
}

3. Update By Query for Bulk Updates

POST /products/_update_by_query
{
  "script": {
    "source": "ctx._source.price *= 0.9"
  },
  "query": {
    "term": {
      "category": "electronics"
    }
  }
}

POST /products/_update_by_query
{
  "script": {
    "source": "ctx._source.price *= 0.9"
  },
  "query": {
    "term": {
      "category": "electronics"
    }
  }
}

Common Mistake: Using frequent updates on large documents. Each update rewrites the entire document. For frequently changing fields, consider storing them separately or using a time-series approach.

Optimizing for Time-Series Data

Time-series data (logs, metrics, events) has unique characteristics that require specialized indexing strategies:

# Optimized time-series index template
PUT /_index_template/timeseries_template
{
  "index_patterns": ["metrics-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 0,  <em># Add replicas after rollover</em>
      "refresh_interval": "30s",
      "translog.durability": "async",
      "translog.sync_interval": "30s",
      "translog.flush_threshold_size": "1gb",
      "codec": "best_compression",  <em># Trade CPU for storage</em>
      "sort.field": "@timestamp",   <em># Optimize for time-range queries</em>
      "sort.order": "desc"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "metrics": {
          "type": "object",
          "dynamic": true
        }
      }
    }
  }
}

# Optimized time-series index template
PUT /_index_template/timeseries_template
{
  "index_patterns": ["metrics-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 0,  <em># Add replicas after rollover</em>
      "refresh_interval": "30s",
      "translog.durability": "async",
      "translog.sync_interval": "30s",
      "translog.flush_threshold_size": "1gb",
      "codec": "best_compression",  <em># Trade CPU for storage</em>
      "sort.field": "@timestamp",   <em># Optimize for time-range queries</em>
      "sort.order": "desc"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "metrics": {
          "type": "object",
          "dynamic": true
        }
      }
    }
  }
}

Key Optimizations:

1. Time-based indices: Natural data lifecycle management
2. Sorted indices: Faster time-range queries
3. Compression: Reduce storage costs for historical data
4. Async translog: Trade durability for performance
5. No initial replicas: Faster initial indexing

Query DSL Mastery: From Basic to Advanced Search Patterns

The Query DSL (Domain Specific Language) is Elasticsearch’s powerful query language. Mastering it is essential for building sophisticated search experiences. This section progresses from fundamental concepts to advanced techniques used in production systems.

Understanding Query vs Filter Context

One of the most fundamental concepts in Elasticsearch is the distinction between query and filter context:

flowchart TD
    A[Search Request] --> B{Context Type}
    B -->|Query Context| C[Calculate Relevance Score<br/>Affects Ranking<br/>Not Cached]
    B -->|Filter Context| D[Yes/No Match<br/>No Scoring<br/>Cached]
    
    C --> E[Examples:<br/>- match<br/>- multi_match<br/>- query_string]
    D --> F[Examples:<br/>- term<br/>- range<br/>- exists<br/>- bool.filter]
    
    style C fill:#ff9999
    style D fill:#99ff99

Query Context: Answers “How well does this document match?”

{
  "query": {
    "match": {
      "description": {
        "query": "wireless headphones",
        "boost": 2.0
      }
    }
  }
}

{
  "query": {
    "match": {
      "description": {
        "query": "wireless headphones",
        "boost": 2.0
      }
    }
  }
}

Filter Context: Answers “Does this document match?”

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "category": "electronics"
          }
        },
        {
          "range": {
            "price": {
              "gte": 50,
              "lte": 200
            }
          }
        }
      ]
    }
  }
}

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "category": "electronics"
          }
        },
        {
          "range": {
            "price": {
              "gte": 50,
              "lte": 200
            }
          }
        }
      ]
    }
  }
}

Interview Tip: Always use filter context for binary decisions (yes/no matches) to leverage caching and improve performance. Reserve query context for relevance-based ranking.

Full-Text Queries: The Search Experience Foundation

Full-text queries are analyzed and designed for searching text fields:

1. Match Query The workhorse of full-text search:

GET /products/_search
{
  "query": {
    "match": {
      "description": {
        "query": "bluetooth speakers",
        "operator": "and",  <em>// All terms must match (default: "or")</em>
        "fuzziness": "AUTO",  <em>// Handle typos</em>
        "prefix_length": 2,  <em>// First 2 chars must match exactly</em>
        "max_expansions": 50  <em>// Limit fuzzy variations</em>
      }
    }
  }
}

GET /products/_search
{
  "query": {
    "match": {
      "description": {
        "query": "bluetooth speakers",
        "operator": "and",  <em>// All terms must match (default: "or")</em>
        "fuzziness": "AUTO",  <em>// Handle typos</em>
        "prefix_length": 2,  <em>// First 2 chars must match exactly</em>
        "max_expansions": 50  <em>// Limit fuzzy variations</em>
      }
    }
  }
}

2. Multi-Match Query Search across multiple fields with different weights:

{
  "query": {
    "multi_match": {
      "query": "apple macbook",
      "fields": ["name^3", "brand^2", "description"],
      "type": "best_fields",  <em>// Other types: most_fields, cross_fields, phrase</em>
      "tie_breaker": 0.3
    }
  }
}

{
  "query": {
    "multi_match": {
      "query": "apple macbook",
      "fields": ["name^3", "brand^2", "description"],
      "type": "best_fields",  <em>// Other types: most_fields, cross_fields, phrase</em>
      "tie_breaker": 0.3
    }
  }
}

Multi-Match Types Explained:

• best_fields: Uses best matching field’s score (default)
• most_fields: Combines scores from all matching fields
• cross_fields: Treats fields as one big field
• phrase: Runs phrase_match on each field

3. Match Phrase Query For exact phrase matching with slop:

{
  "query": {
    "match_phrase": {
      "description": {
        "query": "quick brown fox",
        "slop": 2  <em>// Allow up to 2 position moves</em>
      }
    }
  }
}

{
  "query": {
    "match_phrase": {
      "description": {
        "query": "quick brown fox",
        "slop": 2  <em>// Allow up to 2 position moves</em>
      }
    }
  }
}

Term-Level Queries: Exact Matching

Term-level queries work on exact values without analysis:

1. Term Query

{
  "query": {
    "term": {
      "status": "active"  <em>// Exact match, not analyzed</em>
    }
  }
}

{
  "query": {
    "term": {
      "status": "active"  <em>// Exact match, not analyzed</em>
    }
  }
}

2. Terms Query Match any of multiple values:

{
  "query": {
    "terms": {
      "category": ["electronics", "computers", "accessories"]
    }
  }
}

{
  "query": {
    "terms": {
      "category": ["electronics", "computers", "accessories"]
    }
  }
}

3. Range Query

{
  "query": {
    "range": {
      "price": {
        "gte": 10,
        "lte": 100,
        "boost": 2.0
      }
    }
  }
}

<em>// Date ranges with date math</em>
{
  "query": {
    "range": {
      "created_at": {
        "gte": "now-7d",
        "lte": "now"
      }
    }
  }
}

{
  "query": {
    "range": {
      "price": {
        "gte": 10,
        "lte": 100,
        "boost": 2.0
      }
    }
  }
}

<em>// Date ranges with date math</em>
{
  "query": {
    "range": {
      "created_at": {
        "gte": "now-7d",
        "lte": "now"
      }
    }
  }
}

Compound Queries: Building Complex Logic

The bool query is the Swiss Army knife of Elasticsearch queries:

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "description": "wireless"
          }
        }
      ],
      "should": [
        {
          "term": {
            "brand": "Apple"
          }
        },
        {
          "term": {
            "brand": "Sony"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "price": {
              "gte": 50,
              "lte": 500
            }
          }
        },
        {
          "term": {
            "in_stock": true
          }
        }
      ],
      "must_not": [
        {
          "term": {
            "category": "refurbished"
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "description": "wireless"
          }
        }
      ],
      "should": [
        {
          "term": {
            "brand": "Apple"
          }
        },
        {
          "term": {
            "brand": "Sony"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "price": {
              "gte": 50,
              "lte": 500
            }
          }
        },
        {
          "term": {
            "in_stock": true
          }
        }
      ],
      "must_not": [
        {
          "term": {
            "category": "refurbished"
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

Bool Query Clauses:

• must: Must match, contributes to score
• should: Should match, contributes to score
• filter: Must match, doesn’t contribute to score
• must_not: Must not match, doesn’t contribute to score

Advanced Query Patterns

1. Function Score Query Modify relevance scores based on custom logic:

{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "name": "laptop"
        }
      },
      "functions": [
        {
          "filter": {
            "term": {
              "brand": "premium"
            }
          },
          "weight": 2
        },
        {
          "gauss": {
            "price": {
              "origin": "100",
              "scale": "50",
              "decay": 0.5
            }
          }
        },
        {
          "field_value_factor": {
            "field": "sales_rank",
            "modifier": "log1p",
            "missing": 1
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "name": "laptop"
        }
      },
      "functions": [
        {
          "filter": {
            "term": {
              "brand": "premium"
            }
          },
          "weight": 2
        },
        {
          "gauss": {
            "price": {
              "origin": "100",
              "scale": "50",
              "decay": 0.5
            }
          }
        },
        {
          "field_value_factor": {
            "field": "sales_rank",
            "modifier": "log1p",
            "missing": 1
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

2. Dis Max Query Use the best matching query without combining scores:

{
  "query": {
    "dis_max": {
      "queries": [
        {
          "match": {
            "title": "java programming"
          }
        },
        {
          "match": {
            "description": "java programming"
          }
        }
      ],
      "tie_breaker": 0.3
    }
  }
}

{
  "query": {
    "dis_max": {
      "queries": [
        {
          "match": {
            "title": "java programming"
          }
        },
        {
          "match": {
            "description": "java programming"
          }
        }
      ],
      "tie_breaker": 0.3
    }
  }
}

3. Percolate Query Reverse search – find queries that match a document:

<em>// First, index queries</em>
PUT /query-index/_doc/1
{
  "query": {
    "match": {
      "message": "error"
    }
  },
  "alert_email": "ops@company.com"
}

<em>// Then, percolate documents</em>
GET /query-index/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "message": "System error: Out of memory"
      }
    }
  }
}

<em>// First, index queries</em>
PUT /query-index/_doc/1
{
  "query": {
    "match": {
      "message": "error"
    }
  },
  "alert_email": "ops@company.com"
}

<em>// Then, percolate documents</em>
GET /query-index/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "message": "System error: Out of memory"
      }
    }
  }
}

Search Templates: Parameterized Queries

Search templates separate query logic from parameters, improving maintainability:

<em>// Create template</em>
PUT _scripts/product-search-template
{
  "script": {
    "lang": "mustache",
    "source": {
      "query": {
        "bool": {
          "must": [
            {{#search_term}}
            {
              "match": {
                "description": "{{search_term}}"
              }
            }
            {{/search_term}}
          ],
          "filter": [
            {{#price_min}}
            {
              "range": {
                "price": {
                  "gte": {{price_min}}
                }
              }
            },
            {{/price_min}}
            {{#categories}}
            {
              "terms": {
                "category": {{#toJson}}categories{{/toJson}}
              }
            }
            {{/categories}}
          ]
        }
      }
    }
  }
}

<em>// Use template</em>
GET /products/_search/template
{
  "id": "product-search-template",
  "params": {
    "search_term": "laptop",
    "price_min": 500,
    "categories": ["computers", "electronics"]
  }
}

<em>// Create template</em>
PUT _scripts/product-search-template
{
  "script": {
    "lang": "mustache",
    "source": {
      "query": {
        "bool": {
          "must": [
            {{#search_term}}
            {
              "match": {
                "description": "{{search_term}}"
              }
            }
            {{/search_term}}
          ],
          "filter": [
            {{#price_min}}
            {
              "range": {
                "price": {
                  "gte": {{price_min}}
                }
              }
            },
            {{/price_min}}
            {{#categories}}
            {
              "terms": {
                "category": {{#toJson}}categories{{/toJson}}
              }
            }
            {{/categories}}
          ]
        }
      }
    }
  }
}

<em>// Use template</em>
GET /products/_search/template
{
  "id": "product-search-template",
  "params": {
    "search_term": "laptop",
    "price_min": 500,
    "categories": ["computers", "electronics"]
  }
}

Highlighting: Showing Search Context

Highlighting helps users understand why documents matched their search:

GET /articles/_search
{
  "query": {
    "match": {
      "content": "elasticsearch performance"
    }
  },
  "highlight": {
    "fields": {
      "content": {
        "type": "unified",  <em>// unified, plain, or fvh</em>
        "fragment_size": 150,
        "number_of_fragments": 3,
        "pre_tags": ["<mark>"],
        "post_tags": ["</mark>"]
      },
      "title": {
        "number_of_fragments": 0  <em>// Highlight entire field</em>
      }
    },
    "require_field_match": false,
"boundary_scanner": "sentence",
    "boundary_scanner_locale": "en-US"
  }
}

// Response includes highlighted fragments
{
  "hits": {
    "hits": [
      {
        "_source": { ... },
        "highlight": {
          "content": [
            "Learn how to optimize <mark>Elasticsearch</mark> for better <mark>performance</mark>.",
            "Common <mark>performance</mark> bottlenecks in <mark>Elasticsearch</mark> clusters."
          ]
        }
      }
    ]
  }
}

GET /articles/_search
{
  "query": {
    "match": {
      "content": "elasticsearch performance"
    }
  },
  "highlight": {
    "fields": {
      "content": {
        "type": "unified",  <em>// unified, plain, or fvh</em>
        "fragment_size": 150,
        "number_of_fragments": 3,
        "pre_tags": ["<mark>"],
        "post_tags": ["</mark>"]
      },
      "title": {
        "number_of_fragments": 0  <em>// Highlight entire field</em>
      }
    },
    "require_field_match": false,
"boundary_scanner": "sentence",
    "boundary_scanner_locale": "en-US"
  }
}

// Response includes highlighted fragments
{
  "hits": {
    "hits": [
      {
        "_source": { ... },
        "highlight": {
          "content": [
            "Learn how to optimize <mark>Elasticsearch</mark> for better <mark>performance</mark>.",
            "Common <mark>performance</mark> bottlenecks in <mark>Elasticsearch</mark> clusters."
          ]
        }
      }
    ]
  }
}