Оптимизация баз данных при высоких нагрузках

Уровень 0 · материалов: 4

В кластер входят документы, посвященные методам повышения производительности и оптимизации хранения данных в высоконагруженных системах.

Общие признаки: производительность баз данных, высокие нагрузки, денормализация, in-memory движки, аппаратные затраты

Группа выше: Масштабирование и высокая доступность баз данных

Материалы

• #16807 — Optimizing High-Volume URL Queues in MySQL — Представитель этого кластера

  Какая боль! Толпы против Веб — 2:0. Эпизод один — высокоскоростная очередь — yoihj

  Смысл: The main idea is that standard relational database patterns (normalization and random selection) fail at extreme scales (tens of millions of records). The text demonstrates that unconventional methods like denormalization and cursor-based iteration are necessary to maintain performance and hardware longevity in high-load systems.

  A developer shares their trial-and-error process of optimizing a 70-million-URL crawler, moving from failing MySQL queries and broken hard drives to efficient denormalization and cursor-based selection.

• #32074 — Database Denormalization: Purpose, Timing, and Implementation

  Денормализация БД. Зачем? Когда? Как? — JuliaTem

  Смысл: The main idea is that denormalization is a legitimate optimization technique used to improve database read performance by intentionally introducing redundancy, provided that the trade-off between speed and data integrity is carefully managed.

  Denormalization is a purposeful violation of database normal forms to optimize performance, which should be applied selectively based on a cost-benefit analysis of query speed versus maintenance effort.

• #28214 — Creating Reindexer: A High-Performance In-Memory NoSQL Database

  Как мы выбирали между Elastic и Tarantool, а сделали свою (самую быструю) in-memory БД. С Join и полнотекстовым поиском — olegator99

  Смысл: The main idea is that for extreme high-load scenarios with complex filtering requirements, general-purpose databases may be too slow or too expensive, justifying the creation of a specialized in-memory engine like Reindexer to achieve massive performance gains and hardware cost savings.

  Faced with prohibitive hardware costs and performance gaps in existing databases, the authors developed Reindexer, an open-source in-memory NoSQL DB that significantly outperforms Elasticsearch and MongoDB in high-load filtering and search tasks.

• #33595 — Understanding In-Memory Databases and Data Persistence

  Что такое СУБД в оперативной памяти и как она эффективно сохраняет данные — vadim_vyacheslavovich

  Смысл: The main idea is that in-memory databases can achieve the speed of RAM while maintaining the reliability of disk storage by using sequential transaction logging and periodic snapshotting, avoiding the performance bottlenecks associated with random disk I/O.

  In-memory databases combine high speed and data persistence by writing transactions sequentially to a disk log and taking periodic snapshots to optimize recovery.