Уровень 0 · материалов: 16
Документы должны быть посвящены техническим методам и инструментам автоматического извлечения данных с веб-страниц (скрапингу), но не разработке виджетов на основе официальных API.
Общие признаки: извлечение данных из HTML, использование Python, Node.js, библиотеки BeautifulSoup, Scrapy, Selenium, Puppeteer, использование XPath и DOM-дерева, автоматизация сбора данных с веб-сайтов
Группа выше: Парсинг и веб-скрапинг
Смысл: The main idea of the text is to teach beginners how to extract structured data from websites that lack an API by using Python libraries like Requests, BeautifulSoup, and lxml.
A practical tutorial on extracting movie data from KinoPoisk using Python's Requests, BeautifulSoup, and lxml libraries to demonstrate the web scraping workflow.
Смысл: The main idea of the text is to teach beginners how to scrape structured data from websites using Python's lxml library and the pandas library for data management, illustrated through a specific project of collecting MMA statistics.
A technical tutorial demonstrating how to use Python's lxml and pandas libraries to scrape and organize MMA fight data from a website into CSV files.
Смысл: The main idea is to democratize web scraping by teaching non-programmers, specifically SEO specialists, how to use XPath to extract structured data from HTML for analysis and auditing.
A practical tutorial on using XPath syntax and tools like XPath Helper and Screaming Frog to extract specific data from websites for SEO and analysis.
Смысл: The main idea is to introduce beginners to the Scrapy framework by demonstrating a complete end-to-end workflow for scraping a real-world website, from project setup and data extraction to cleaning and exporting results.
A technical tutorial demonstrating how to build a web crawler using the Scrapy framework to extract educational institution data into a CSV file.
Смысл: The main idea is to introduce beginners to web scraping using Python, providing a simple three-step framework (search, extract, save) and a practical code example to automate data collection from a job board.
A beginner-friendly tutorial on how to build a basic web scraper using Python's BeautifulSoup, Requests, and Pandas libraries.
Смысл: The main idea of the text is to provide a practical introduction to web scraping in Python, teaching users how to choose between simple HTTP requests (via Requests and Beautiful Soup) for static content and browser automation (via Selenium) for dynamic, JavaScript-heavy content.
A beginner-friendly guide to web scraping in Python using Requests and Beautiful Soup for static pages and Selenium for dynamic content.
Смысл: The main idea is to demonstrate how to use Python for web scraping and automation to extract media from a social network when a formal API is unavailable.
A guide and Python script for bulk-downloading all photos from a VK.com chat by simulating POST requests and parsing HTML responses.
Смысл: The main idea is to move from fragile, hard-coded scraping to a robust system using XPATH for data extraction and a normalization layer for data structuring, enabling the creation of a searchable product database from unstructured web content.
A technical guide on using C# and XPATH to build resilient web scrapers for online stores and converting raw scraped data into a structured, searchable database.
Смысл: The main idea is to demonstrate the basic mechanics of web scraping in Python by filtering a news feed based on specific HTML tags and classes to extract only relevant information.
A beginner's tutorial on using Python's BeautifulSoup4 and Requests libraries to build a news parser that filters content based on HTML classes.
Смысл: The main idea is to teach developers how to use the Puppeteer library for Node.js to automate browser tasks and extract data from websites (web scraping) through practical examples.
A step-by-step tutorial on using Node.js and Puppeteer to automate Google Chrome for tasks like taking screenshots and scraping web data.
Смысл: The main idea is to follow a specific order of technical approaches to web scraping—from the most stable (APIs) to the least stable (HTML parsing)—to maximize the robustness and scalability of the data extraction process.
To build robust scrapers, try official APIs, then XHR requests, then embedded JSON, then headless browsers, and only use HTML parsing as a last resort.
Смысл: The main idea is to implement a tool for extracting xPaths from third-party websites by combining a proxy server for content delivery and the window.postMessage API for secure, cross-domain communication.
The author overcomes the browser's Same-Origin Policy to allow xPath selection in iFrames by implementing a Java-based proxy server that injects scripts and communicates via postMessage.
Смысл: The main idea is to demonstrate a complete, step-by-step workflow for creating a web scraper in Node.js, from initial site analysis and handling HTTP requests to recursive crawling and DOM parsing.
A practical guide on building a Node.js web scraper using needle for requests, tress for queue management, and cheerio for HTML parsing.
Смысл: The text explains how to bypass the lack of official Instagram API access by creating a web scraper that extracts embedded JSON data from the Instagram website to retrieve user profiles, posts, and high-quality images.
The author demonstrates how to use web scraping and embedded JSON to replace official Instagram API functionality for account and post data retrieval.
Смысл: The main idea of the text is to teach beginners and programmers how to effectively use XPath for web parsing, demonstrating its superiority over regular expressions for navigating the DOM tree of HTML/XHTML pages.
A technical tutorial explaining XPath syntax, functions, and practical application for extracting data from HTML pages, including a PHP implementation example.
Смысл: The main idea is to demonstrate a simplified method for scraping public Instagram data using the 'instabot' Python library for practical applications like giveaway automation and data analysis, while warning about the risks of account suspension.
A technical tutorial on using Python's 'instabot' library to scrape public Instagram profiles, posts, and user interactions, including code examples for data extraction and image downloading.