Уровень 0 · материалов: 7
В кластер входят документы, посвященные архитектуре, компонентам и применению инструментов экосистемы Hadoop для работы с большими данными.
Общие признаки: инструменты обработки Big Data, компоненты Apache Hadoop, сравнение HBase, Hive и Spark, масштабируемость данных
Группа выше: Конвейеры и обработка больших данных
Смысл: The main idea is to introduce Apache Spark as a versatile, high-performance alternative to Hadoop for Big Data processing, showcasing its unified ecosystem of libraries for streaming, SQL, machine learning, and graph analysis.
An overview of Apache Spark, explaining its architecture (RDDs), key libraries (SQL, Streaming, MLlib, GraphX), and practical applications in real-time data analysis.
Смысл: The main idea of the text is to provide a practical and critical guide to the Hadoop ecosystem, explaining how its various components (storage, processing, SQL, and ingestion) fit together to handle Big Data, while steering the reader toward the most efficient and stable tools currently available.
A comprehensive guide to the Hadoop ecosystem covering HDFS, processing engines like Spark, SQL tools, NoSQL options, and data ingestion strategies with a critical perspective on various distributions.
Смысл: The main idea is to introduce Apache Hive as a tool that enables users to query large-scale data stored in HDFS using a familiar SQL-like syntax, thereby simplifying the Big Data processing workflow without sacrificing the horizontal scalability provided by Hadoop.
Apache Hive is a data warehouse software project that provides a SQL-like interface to query data stored in HDFS by translating queries into MapReduce tasks.
Смысл: The text aims to provide a practical entry point for developers to start using Apache Hadoop, demonstrating that Big Data processing can be implemented through either flexible scripting (Python via Streaming) or high-performance native code (Java).
A technical tutorial explaining the Hadoop ecosystem, its installation via Cloudera, and practical implementation of MapReduce using both Python and Java.
Смысл: The main idea is that Hadoop is a specialized tool for massive datasets (multi-terabyte scale) and is inefficient, restrictive, and unnecessarily complex for smaller data volumes that can be handled by SQL, Pandas, or simple scripts.
Stop using Hadoop for small datasets; use Pandas or SQL unless your data exceeds several terabytes.
Смысл: The main idea is to introduce HBase as a scalable solution for Big Data that combines the benefits of random real-time access and high-throughput batch processing, explaining its internal mechanics and optimal use cases.
An educational overview of Apache HBase, covering its Bigtable-inspired architecture, data model, operational mechanisms, and practical application in Big Data ecosystems.
Смысл: The main idea is that in a high-reliability environment (triple replication) on HDD-based hardware, Apache HBase demonstrated superior performance over Apache Cassandra, contradicting some vendor-led benchmarks.
Sberbank's internal benchmarks show HBase significantly outperforming Cassandra in write and read speeds when configured for high data reliability on SATA HDDs.