#elasticsearch

[reconshell]

Postgres to Elasticsearch sync

PGSync is a middleware for syncing data from Postgres to Elasticsearch effortlessly. It allows you to keep Postgres as your

[spoon-elastic]

Elasticsearch 7.9.0 released

Elasticsearch 7.9.0 is here! Introducing data streams to simplify ingest, a new wildcard data type for smarter search — and EQL, a powerful, battle-tested language for threat hunting and more.

New elastic sesrh release. Discover brand new functionalities, new kibana presentation, datastream and so on

[accidentallyquadratic]

Elasticsearch IndicesQuery

Elasticsearch is a distributed search engine. One can store “documents” within “indices”, which are collections of documents. One common pattern for storing time based data is to use one index per day. E.g.: tweets-2017-01-01, tweets-2017-02-01 and so on. This makes having many indices decently common, and it also makes searching across many indices common in a single search.

Elasticsearch has a query type called IndicesQuery, which lets you (among other things) optimise your query path by knowing which index to search in. For example, say that you have one index pattern for tweets (tweets-YYYY-MM-DD) and one for Reddit posts (reddit-YYYY-MM-DD). Say that someone searches for “hashtag: kittens OR subreddit: awww”. You know that you don’t have to execute the hashtag query in the Reddit indices and that you do not have to execute the subreddit query in the tweet indices. This is done by specifying to only execute the hashtag in indices that match tweets-*.

When executing the query, Elasticsearch will consider each shard (of an index) separately. Usually one has 2-5 shards per index. For each shard, it will check if the index name matches the pattern given by the IndicesQuery. This is where we find our performance bug.

The algorithm in the 1.X and 2.X versions of ES work by first expanding a pattern into the list of all matching indices – tweets-* becomes ["tweets-2017-01-01", "tweets-2017-01-02", …]. Then, for each index being considered, it checks for membership in that list:

protected boolean matchesIndices(String currentIndex, String... indices) { final String[] concreteIndices = indexNameExpressionResolver.concreteIndices(clusterService.state(), IndicesOptions.lenientExpandOpen(), indices); for (String index : concreteIndices) { if (Regex.simpleMatch(index, currentIndex)) { return true; } } return false; }

(Here, indices is a list of patterns, and the concreteIndices() method applies the wildcard internally to the list of all indices)

This means that the time complexity per shard is O(n), where n is the number of indices in your cluster. Since we consider at least one shard per index, the overall search complexity then goes to O(n^2). Furthermore, this expansion (and Elasticsearch’s Regex.simpleMatch pattern matcher) generate a lot of garbage, stressing the garbage collector and making the process even slower.

(For those of you arriving here via Russ’s blog, you’ll be chagrined to note that ES implements an exponential-time backtracking glob matcher, although that fact isn’t implicated in the bug in question.)

We used this query in our not-too-shabby production cluster. At the time, it had a total of 1152 cores and we searched over roughly 150 TB of data in about 8500 indices. We discovered that we spent almost half of our CPU time in the Regex.simpleMatch method. We patched the algorithm to instead directly check if the current index matches any of the specified indices, making it O(m), where m is the number of index patterns specified in the query (we usually had 2-3). On top of the saved CPU, this also had the benefit of making us spent about 1/3 as much time in Garbage Collection pauses, due to the fewer allocations of Strings.

This seems to have been fixed in the ‘Great Query Rewrite’ for Elasticsearch 5. I do not know if this was explicitly targeted as a performance fix, if it was by accident or if someone just thought 'oh this might be slow’.

This post was contributed by Anton Hägerstrand.

[servermo]

Deploy Typesense on Bare Metal: The Elasticsearch Alternative

Ditch JVM bloat. Deploy the C++ in-memory Typesense engine on ServerMO Bare Metal for lightning-fast RAG vector search and e-commerce faceti

Ditch Elasticsearch JVM Bloat: Deploy C++ Typesense on Bare Metal

Elasticsearch was the undisputed king of search infrastructure for years. But in the era of AI, vector search, and LLMs, its heavy Java (JVM) architecture, complex memory tuning, and garbage collection pauses have become major operational liabilities.

Enter Typesense: Written from scratch in C++, it eliminates JVM overhead entirely. By mapping indices directly into RAM, Typesense delivers blistering sub-50ms query latency for RAG applications, e-commerce faceted filtering, and instant search.

However, deploying an in-memory database comes with zero margin for error. Here is the SRE playbook for deploying Typesense on bare metal without hitting OOM crashes or connection limits:

1. The Vector RAM Equation (OOM Prevention)

Because Typesense operates in RAM, exceeding physical host memory will trigger disk SWAP (destroying latency) or cause the Linux OOM Killer to instantly terminate your database.

When using AI Vector Search, calculate your RAM needs using this formula: 7 Bytes × Dimensions × Total Records

Example: 1 Million records using OpenAI text-embedding-3-small (1,536 dimensions): 7 × 1,536 × 1,000,000 = ~10.75 GB RAM (Always add a 15–20% RAM buffer for the host OS!)

2. Fix Docker Connection Drops (Ulimits)

By default, Docker restricts containers to 1,024 file descriptors. Under high traffic spikes or indexing loads, Typesense will exhaust this immediately.

To fix this in your docker-compose.yml, set the ulimits under your Typesense service configuration:

• Service: typesense

• Image: typesense/typesense:27.1

• Port Mapping: 8108:8108

• Volume: ./data:/data

• Ulimits Fix: Set nofile with soft: 65535 and hard: 65535

• Command: --data-dir=/data --api-key=YOUR_KEY --enable-cors

This simple ulimit tweak stops Typesense from dropping connections during high traffic.

3. The Server-to-Server SSL Chain Trap

If configuring Let's Encrypt SSL inside Typesense directly, never map cert.pem.

While web browsers automatically fetch missing intermediate certificates, backend programming languages (Python, PHP, Node.js) do not and will throw a fatal 'SSL peer certificate not OK' error.

The Fix: Always map fullchain.pem in your configuration so backend API clients can verify the entire SSL chain of trust.

4. Kubernetes Boot Loop Fix (Startup Probes)

Loading massive 50GB+ indices into RAM during startup takes several minutes. Standard livenessProbe checks will assume the pod is stuck and kill it mid-load!

Don't delete probes entirely. Instead, use a startupProbe alongside your livenessProbe:

• Startup Probe Config: Path /health on HTTP port, with failureThreshold: 30 and periodSeconds: 10. This gives Typesense up to 5 minutes to boot into RAM before checks begin.

• Liveness Probe Config: Path /health on HTTP port, with initialDelaySeconds: 5 and periodSeconds: 10.

Scale AI Search with Bare Metal Performance

Avoid exorbitant cloud VM pricing for high-RAM instances. Deploy Typesense on ServerMO Dedicated Bare Metal Servers to unlock unshared memory pathways, lightning NVMe speeds, and true zero-overhead search performance.

Read the complete Typesense Bare Metal Deployment Guide on ServerMO!

[servermo]

Install Meilisearch Ubuntu 24.04: Elasticsearch Alternative

Migrate from legacy Java search engines. Deploy the blazing fast Rust powered Meilisearch configure Nginx reverse proxies and fix hidden mem

Ditch Elasticsearch: Install Native Rust Meilisearch on Ubuntu 24.04

Blindly installing Elasticsearch for full-text search is a massive architectural blunder. Built on the heavy Java Virtual Machine (JVM), Elasticsearch requires 4 to 8 GB of RAM just to boot up, forcing you to pay for expensive, RAM-heavy cloud VMs.

Enter Meilisearch: Engineered natively in Rust, it runs as a lightweight binary, starts on under 50 MB of RAM, offers out-of-the-box typo tolerance, and delivers sub-50ms query responses.

Here is the SRE guide to deploying Meilisearch on Ubuntu 24.04 bare metal without hitting performance traps:

Trap 1: The Base64 Master Key Crash

Never use Base64 strings for systemd environment master keys. Special characters like = and / will violently break the environment parser. Always generate pure hexadecimal keys:

# Generate a 32-byte secure alphanumeric master key MEILI_MASTER_KEY=$(openssl rand -hex 32) # Save securely in an isolated env file echo "MEILI_MASTER_KEY=$MEILI_MASTER_KEY" | sudo tee /etc/meilisearch/env >/dev/null sudo chmod 600 /etc/meilisearch/env

Trap 2: The Gzip Memory Explosion

Default Meilisearch settings reject uploads over 100 MB with a 413 Payload Too Large error. But uploading a giant 5 GB compressed .gz file directly into memory triggers the Linux OOM (Out Of Memory) killer, instantly crashing your server.

The Fix: Increase payload limits in systemd, lock indexing memory, and chunk your database using newline-delimited JSON (.ndjson):

# Split your massive database into 100k line chunks split -l 100000 massive_database.ndjson chunk_ # Stream chunks sequentially curl -X POST 'http://127.0.0.1:7700/indexes/products/documents' \ -H 'Content-Type: application/x-ndjson' \ -H "Authorization: Bearer YOUR_MASTER_KEY" \ --data-binary @chunk_aa

Trap 3: Hardening the Systemd Daemon

To handle heavy production traffic without running out of memory or file descriptors, configure /etc/systemd/system/meilisearch.service:

Ini, TOML [Unit] Description=Meilisearch Enterprise Search Engine After=network.target [Service] Type=simple User=meilisearch Group=meilisearch EnvironmentFile=/etc/meilisearch/env ExecStart=/usr/local/bin/meilisearch \ --env production \ --db-path /var/lib/meilisearch/data.ms \ --dump-dir /var/lib/meilisearch/dumps \ --snapshot-dir /var/lib/meilisearch/snapshots \ --schedule-snapshot \ --snapshot-interval-sec 86400 \ --http-payload-size-limit 500000000 \ --max-indexing-memory 2048Mb \ --http-addr 127.0.0.1:7700 Restart=on-failure RestartSec=5 # Unlock connection throughput LimitNOFILE=65536 [Install] WantedBy=multi-user.target

Trap 4: Nginx Payload Mismatch

When setting up an Nginx reverse proxy with Let's Encrypt SSL, ensure you set client_max_body_size 500M; inside your server block! If Nginx's body size limit doesn't match Meilisearch's HTTP payload limit, Nginx will block your document updates.

Bare Metal Search Supremacy

Ditch noisy cloud neighbors and hypervisor disk I/O bottlenecks. Deploy critical search clusters on ServerMO Bare Metal Dedicated Servers paired with ultra-fast NVMe storage and unmetered network bandwidth.

Read the complete Meilisearch Ubuntu 24.04 Installation Guide on ServerMO.com!

[otuny]

OpenSearch Gains Momentum as the Open Source Standard for Search and Analytics

With downloads reaching 1.4 billion, OpenSearch is no longer just a fork of Elasticsearch. We examine the licensing, maintenance, and feature trade-offs driving this massive adoption shift.

Read the full article

[sunshinedigitalservices]

ELK Stack: Real-Time Log Analytics and Observability

In modern digital infrastructures, monitoring and analyzing data in real time is critical — and the ELK Stack has become a powerful solution for observability and operational intelligence.

The ELK Stack — Elasticsearch, Logstash, and Kibana — enables organizations to centralize, search, analyze, and visualize massive streams of logs and machine data efficiently.

From application monitoring and cybersecurity analysis to infrastructure observability and real-time troubleshooting, ELK helps businesses gain actionable insights from complex data environments.

By transforming raw logs into meaningful dashboards and analytics, the ELK Stack empowers teams to improve performance, detect anomalies, and maintain system reliability at scale.

In a data-driven world, visibility and observability are the foundation of resilient systems.

Read more:

[logicandlegacy]

Elasticsearch & Inverted Indices — The Death of SQL ILIKE (2026)

Rethinking Search: From SQL to Elasticsearch

When tasked with adding a search bar to an application, many developers instinctively turn to their trusty SQL database. However, this approach can lead to performance issues and scalability problems. The reason lies in how SQL databases are designed to handle queries.

The Limitations of SQL

SQL databases utilize B-Trees for indexing, which excel at finding specific values, such as IDs or dates. However, when it comes to searching for text patterns, especially with wildcards at the beginning of a string, B-Trees become inefficient. This leads to a full table scan, where the database must read every row, resulting in significant performance degradation.

Introducing Elasticsearch

Elasticsearch is a distributed, NoSQL search engine built on top of Apache Lucene. It's designed specifically for full-text search and can handle massive amounts of data with ease. By pushing JSON documents into Elasticsearch, it creates an inverted index, mapping each word to a list of documents that contain it. This allows for fast and efficient searching, even with complex queries.

Real-World Applications

Elasticsearch is particularly useful in scenarios where text search is critical, such as:

E-commerce catalogs, where users may search for products with typos or variations in spelling

Log aggregation, where developers need to find specific log entries among millions of lines

Autocomplete and search bars, where users expect instant results as they type

Implementing Elasticsearch

In a production environment, it's recommended to use an existing Elasticsearch cluster or a cloud-based service. The official Python library provides a simple way to interact with the cluster, allowing developers to query the data using a domain-specific language.

from elasticsearch import Elasticsearch es = Elasticsearch("https://my-es-cluster.internal:9200", basic_auth=("admin", "secret")) search_body = { "query": { "multi_match": { "query": "python backend architecture", "fields": ["title^3", "description"], "fuzziness": "AUTO" } } } response = es.search(index="technical_blogs", body=search_body) for hit in response["hits"]["hits"]: print(f"Found: {hit['_source']['title']} (Score: {hit['_score']})")

The Power of Inverted Indices

Elasticsearch's inverted index allows it to search billions of documents in milliseconds. By mapping each word to a list of documents, the engine can quickly find the intersection of multiple sets, resulting in fast and accurate search results. This approach is akin to using a glossary to find specific pages in a book, rather than reading the entire book from cover to cover.

The key to this efficiency lies in the way the index is structured. Instead of mapping documents to their words, an inverted index maps words to their documents. This simple flip in perspective enables Elasticsearch to handle complex searches with ease, making it an essential tool for any application that requires robust text search capabilities.

Read the full technical breakdown on my blog