Dart와 Flutter를 위한 강력하고 직관적인 웹 스크래핑 패키지로, 개발자가 웹 페이지와 API에서 구조화된 데이터를 추출할 수 있도록 도와줍니다. 파서 및 변환을 체인으로 연결할 수 있는 지원 기능을 통해 HTML 탐색, 콘텐츠 필터링, 데이터를 사용 가능한 형식으로 변환하는 과정을 단순화합니다. 크로스플랫폼 사용을 위해 설계되어 Android, iOS, Web, macOS, Windows, Linux에서 원활하게 작동합니다. 최소한의 반복 코드로 데이터 집약 도구, 가격 트래커, 또는 콘텐츠 수집기 구축에 이상적입니다.
사용 사례
웹 데이터 추출
가격 모니터링
콘텐츠 집약
API 응답 파싱
자동 데이터 수집
RSS 피드 스크래핑
주요 기능
체인된 파서 및 변환
크로스 플랫폼 호환성
HTML 및 API 지원
최소 설정
유연한 데이터 변환
강력한 오류 처리
적합한 대상
데이터 채굴 프로젝트
실시간 정보 수집가
자동화 스크립트
Scraping beginners
경량 솔루션을 필요로 하는 개발자
외부 데이터가 필요한 앱
고려 사항
robots.txt 및 이용 약관을 존중하세요
레이트 제한 처리
법적 준수 보장
대상 사이트 변경 모니터링
공격적인 크롤링을 피하세요
먼저 작은 데이터 세트에서 테스트하세요
원본 README
연결된 소스 저장소의 README 원문 스냅샷입니다. 최신 내용은 저장소에서 확인하세요.
Config-based, reusable web scraper for web and API scraping. Scrape, parse web pages or APIs without writing parsers or scraping logic, using simple key/value based configs.
Features
Config-based, reusable web scraper for web and API scraping.
Scrape, parse web pages or APIs without writing parsers or scraping logic, using simple key/value based configs.
High-level web scraper that combines HTML fetching and data parsing.
WebScraperWebScraper()
// Main scraping methodFuture<Map<String, Object>> scrape({
// The URL to scraperequiredUri url,
// Scraper configuration for the URLScraperConfig? scraperConfig,
// Map of domain names to lists of scraper configurationsScraperConfigMap? scraperConfigMap,
// Enable debug logging, also dumps scraped file into current path /dump folderbool debug = false,
// Pre-fetched HTML document (optional, avoids HTTP request if provided)String? html,
// Custom cookies to include in HTTP requests (Overrides ScraperConfig cookies)Map<String, String>? overrideCookies,
// Custom HTTP headers to include in requests (Overrides ScraperConfig headers)Map<String, String>? overrideHeaders,
// Overrides proxy config in ScraperConfig and HTTP Parser optionsProxyAPIConfig? overrideProxyAPIConfig,
// Custom user agent string (overrides ScraperConfig userAgent)String? overrideUserAgent,
})
ScraperConfig class
Configuration for targeting and scraping specific types of URLs.
ScraperConfigScraperConfig({
// List of URL path patterns that this configuration should handleList<String> pathPatterns = const [],
// List of parsers that define how to extract data from the pagerequiredList<Parser> parsers,
// Whether HTML content needs to be fetched from the URLbool requiresHtml = true,
// URL preprocessing and cleaning configurationUrlCleaner? urlCleaner,
// Proxy API configProxyAPIConfig? proxyAPIConfig;
// Cookies for base requestMap<String, String>? cookies;
// Headers for base requestMap<String, String>? headers;
// Whether to force a fresh HTTP request even if HTML is providedbool forceRefresh = false,
// User agent device type for base requestUserAgentDevice userAgent = UserAgentDevice.mobile,
})
UrlCleaner class
Clean the URL before it's passed to a scraper.
UrlCleanerUrlCleaner({
// Set whitelisted or blacklisted URL parameters.List<String>? whitelistParams,
List<String>? blacklistParams,
// Set custom static parameters to a URL.Map<String, String>? appendParams,
})
Parser class
Easy to use and reusable parser class :)
ParserParser({
// `id` is used for final result.// Child parsers can reference to parent parser using `id`.// You can have multiple parsers with same id and same parent and will execute one by one and stop execution once data is successfully parsed by one parser.requiredString id,
// A child can have multiple parents, it will execute once parent parser is successfully executed.requiredList<String> parents,
// Set the parser types.requiredParserType type,
// List of selectors will execute one by one and stop execution once data is successfully parsed by one selector.List<String> selectors = const [],
// Set parser for private usage. Will be not added to final result.bool isPrivate = false,
// Set multiple to `true` if data is a List.bool multiple = false,
// Some parsers require additional options to work properly.// You can pass these options here.// For example, `ParserType.table` requires `ParserOptions.table`.ParserOptions? parserOptions;
// Data transformation options to apply after extraction.TransformationOptions? transformationOptions,
// Custom cleaner function, clean the data and return data.Object? Function(Data, bool)? cleaner,
// If you plan to create configs from JSON, you can pass cleaner name here.// This cleaner should be registered in `CleanerRegistry` to be used.// If you pass cleaner along with cleanerName, cleanerName function will be ignored.String? cleanerName,
})
Parser Types
Type
Description
Selector
ParserOptions
ParserType.element
Extracts element nodes from HTML using CSS selectors.
CSS selector required.
-
ParserType.attribute
Extracts attribute value from HTML element using CSS selectors.
Use CSS selector to select an element and append attribute name with ::. Ex: div#myid::name where name refers to the attribute name.
-
ParserType.text
Extracts text from HTML element using CSS selectors.
CSS selector required.
-
ParserType.image
Extracts image URL from HTML element.
CSS selector required. After selecting an element it tries to find src attribute.
-
ParserType.url
Extracts URL from an HTML element
CSS selector required. After selecting an element it tries to find href attribute.
-
ParserType.urlParam
From an URL it extracts query parameter.
Add parameter name in selector.
ParserOptions.urlParam
ParserType.table
Extracts data from HTML table.
CSS selector required. Select table using this selector.
ParserOptions.table
ParserType.sibling
Used when target element doesn't have a valid selector but sibling does.
CSS selector is required.
ParserOptions.sibling
ParserType.strBetween
Extracts the string between two strings.
Not required
ParserOptions.stringBetween
ParserType.http
Get data using http request
Not required
ParserOptions.http
ParserType.responseHeaders
Sends a GET/POST request and extracts a value from the response headers (e.g. a cookie from set-cookie).
Parser-specific configuration options that control how individual parsers behave during data extraction.
Use the appropriate named constructor for the parser type you're configuring:
// For HTTP parsersParserOptions.http(options: HttpParserOptions(...))
// For response header parsersParserOptions.responseHeaders(options: ResponseHeaderParserOptions(...))
// For table and JSON table parsersParserOptions.table(options: TableParserOptions(...))
// For sibling parsersParserOptions.sibling(options: SiblingParserOptions(...))
// For static value parsersParserOptions.staticValue(options: StaticValueParserOptions(...))
// For string between parsersParserOptions.stringBetween(options: StringBetweenParserOptions(...))
// For URL parameter parsersParserOptions.urlParam(options: UrlParamParserOptions(...))
Response Headers parser
ParserType.responseHeaders sends a request and returns a value from the response headers — handy for grabbing a cookie out of set-cookie.
Parser(
id: "session",
parents: ["_root"],
type: ParserType.responseHeaders,
parserOptions: ParserOptions.responseHeaders(
options: ResponseHeaderParserOptions(
// Optional: defaults to the URL passed to WebScraper.scrape
url: "https://example.com/login",
// HttpMethod.get (default) or HttpMethod.post
method: HttpMethod.get,
// Response header to read (case-insensitive). Omit to return all headers as a Map.
headerName: "set-cookie",
// Optional: extract a single cookie's value from the header
cookieName: "at",
),
),
)
TransformationOptions class
Comprehensive data transformation system that can be applied to extracted data.
TransformationOptionsTransformationOptions({
// Text to add to the beginningString? prepend,
// Text to add to the endString? append,
// List of values to check for matches, returns booleanList<Object>? match,
// Index to extract from a list (negative for reverse indexing)int? nth,
// Delimiter to split data byString? splitBy,
// Whether to decode URL-encoded stringsbool? urldecode,
// Whether to convert map values to a listbool? mapToList,
// Regular expression extraction optionsRegexTransformationOptions? regexMatch,
// Regular expression replacement optionsRegexReplaceTransformationOptions? regexReplace,
// Text replacement optionsReplaceTransformationOptions? replace,
// Text cropping options (start/end)CropTransformationOptions? crop,
// Extract text between two stringsStringBetweenTransformationOptions? stringBetween,
// Extract sibling elementsSiblingTransformationOptions? sibling,
// Table processing optionsTableTransformationOptions? table,
// Return static value optionsStaticValueTransformationOptions? staticValue,
// Custom order for applying transformationsList<TransformationType>? transformationOrder,
})
Error handling
Scraping throws WebScraperError. When a request (the base fetch, or an http / responseHeaders parser) returns 404, the error carries statusCode: 404, signalling the URL is no longer available:
try {
final result = await webScraper.scrape(url: url, scraperConfig: config);
} onWebScraperErrorcatch (e) {
if (e.statusCode == 404) {
// URL is gone
} else {
print(e.message);
}
}
Browser-like requests
Every request automatically includes a coherent set of browser headers — a randomized User-Agent plus matching Accept, Accept-Language, Sec-Fetch-*, and (for Chromium User-Agents) version-aligned Sec-CH-UA client hints. Choose the device profile with ScraperConfig.userAgent (UserAgentDevice.mobile, desktop, windows, mac, linux, android, ios, random). Any headers or overrideUserAgent you set take precedence.
Note: this mimics header-level fingerprints only. TLS/JA3 and HTTP/2 fingerprints come from the Dart networking stack and can't be changed here — route through a ProxyAPIConfig unblocker for sites that fingerprint at those layers.
Creating configs from JSON
You can now create configs from JSON string using ScraperConfig.fromJson method.
Cleaner Registry for parsers created using JSON
You can register cleaners using CleanerRegistry.register method. This is useful when you want to create configs from JSON and want to use custom cleaner for Parser.