Methodology

Cadence is matched to each source's freshness. X runs every 5-15 minutes; Google trending-now every ~10-30 minutes and comparative/related daily; TikTok Creative Center daily; YouTube every 1-6 hours; ALL retail and review ranking snapshots run once daily (rankings change at most hourly, so daily is sufficient and lowest-risk); editorial/awards refresh daily or per awards release. Every chart and card shows both 'last updated' and a 'source freshness' label of live / hourly / daily / analyst-curated — we never use a single ambiguous 'real-time' label.

Source coverage

Source	Domain	Scope	Cadence	Notes
Google Trends	Country Verifiedcountry	US	Comparative/related daily; trending-now ~10-30 min	Country-scoped keyword interest and related terms (US, CA). Also joinable via BigQuery public datasets.
X	Country Verifiedcountry	US	Conversation layer every 5-15 min	Country-aware keyword/brand mention counts plus recent-post evidence.
Reddit	Country Verifiedcountry	US	Every ~15 min	Community signal over a K-beauty subreddit watchlist via the official Reddit Data API (registered OAuth app, rate-limited) — never scraped. Country attribution only from a subreddit's geography (r/SkincareAddictionCanada -> CA; general subs stay global).
Instagram	Global Socialsocial	Global	Daily	Watchlist hashtag/account evidence only. Global directional — never presented as US- or CA-specific.
TikTok	Global Socialsocial	Global	Daily	Creative Center keyword/hashtag/top-product directional signals (ads-based). Global directional — never presented as country-specific.
Pinterest	Global Socialsocial	Global	Daily	Global keyword/visual trend watchlist. Global directional — never presented as country-specific.
YouTube	Global Socialsocial	Global	Every 1-6 h	Shorts-adjacent discovery and review/tutorial proof.
Amazon US	Retail Confirmedretail	US	Once daily	Best-Sellers + category nodes. Permitted/identified user-agent (ClaudeBot/GPTBot/CCBot are name-blocked). Commerce-intent, not direct sales truth.
Amazon.ca	Retail Confirmedretail	CA	Once daily	Bestsellers + Movers & Shakers + New Releases (beauty). Permitted user-agent; throttled against anti-bot 403/503.
Sephora	Retail Confirmedretail	US	Per editorial/awards release	Access-blocked at the access layer (Akamai 403) regardless of robots.txt. Referenced as an editorial/analyst source and via official awards content — NOT auto-ingested.
Ulta	Retail Confirmedretail	US	Per editorial/awards release	Prestige/mass growth surface treated as an editorial reference.
Walmart.ca	Retail Confirmedretail	CA	Per editorial release	Returns anti-bot 418 at the access layer — editorial reference only, not auto-ingested.
YesStyle	Retail Confirmedretail	Global	Once daily	/en/k-beauty + annual/mid-year Beauty Awards. robots allows base paths (avoid ?q* and /product-grid/).
Jolse	Retail Confirmedretail	Global	Once daily	BEST / NEW / TIME DEAL listings.
StyleKorean	Retail Confirmedretail	Global	Once daily	Curated best-selling picks (low weight; curated, not true rank). Now auto-applies CA customs.
Stylevana	Retail Confirmedretail	Global	Once daily	Region-segmented best sellers.
Wishtrend	Retail Confirmedretail	KR	Once daily	Korea-direct Shopify store popular with NA buyers. /collections/ best-sellers; sort_by/filter params disallowed.
TesterKorea	Retail Confirmedretail	KR	Once daily	Korea-direct e-tailer. Bestseller/category listings; only /Member/, /Payment/, /Search/ disallowed.
RoseRoseShop	Retail Confirmedretail	KR	Once daily	Korea-direct Shopify store. /collections/ best-sellers; sort_by/filter params disallowed.
Soko Glam	Retail Confirmedretail	Global	Once daily	US-based curated K-beauty (Shopify). /collections/bestsellers; avoid ?sort_by/filter.
Sukoshi Mart	Retail Confirmedretail	CA	Once daily	Canada-domestic specialist (Shopify). /collections/best-sellers curated; sort_by disallowed.
Lakinza	Retail Confirmedretail	CA	Once daily	Montreal-area Canada-domestic specialist (Shopify), ships domestic with no customs. Curated /collections/; filter params disallowed.
TheKShop	Retail Confirmedretail	CA	Once daily	Montreal Canada-domestic specialist (Shopify). Curated /collections/; filter params disallowed.
Mikaela Beauty	Retail Confirmedretail	CA	Once daily	Edmonton AB Canada-domestic specialist (Shopify). Curated /collections/; filter params disallowed.
ShopDama	Retail Confirmedretail	CA	Once daily	Toronto Canada-domestic specialist. Exposes an official /api/ucp/mcp agent endpoint — the preferred ingestion path over scraping.
Kbeauty Canada	Retail Confirmedretail	CA	Once daily	Canada-domestic specialist (kbeauty.ca). Curated /collections/; filter params disallowed.
Olive Young Global	Retail Confirmedretail	KR	Once daily	Highest-value demand anchor. /display/page/best-seller (Top Orders / Top in Korea) + Olive Young Awards (~180M purchase records). JS-rendered; robots allows /display with Crawl-delay 5. Ships to US and CA.
Hwahae	Retail Confirmedretail	KR	Per awards release / daily	Review-driven rankings + biannual awards, retailer-independent. Used as authenticity/recommendation corroboration; awards pages are the preferred low-risk target.
Glowpick	Retail Confirmedretail	KR	Per awards release / daily	Review-driven rankings + biannual awards, retailer-independent. Used as authenticity/recommendation corroboration; awards-first.

Scope, domains, and the retail-demand authenticity model

Country Intelligence (Google Trends, X, Reddit) is geo-safe: country is attributed only where the source genuinely carries it (e.g. a country-scoped Trends query or a Canada-specific subreddit).
Global Social Signal (Instagram, TikTok, Pinterest, YouTube) is directional and global. These signals are NEVER presented as US- or Canada-specific unless a payload explicitly contains geography.
Retail Demand sources carry a real storefront geography, so US / CA / KR / Global segmentation here is legitimate. Mass-market retailer position is treated as commerce-intent, not direct sales truth.
Retail authenticity model: authentic demand (Korean-domestic purchase data like Olive Young, then mass-market) and review-driven recommendation (Hwahae/Glowpick, retailer-independent) rank ABOVE ad/merchandising-driven Amazon position alone; specialist export retail is corroboration. Disagreement between social hype and retail demand is shown, not hidden.
Access-blocked channels (Sephora US/CA, Walmart.ca) are editorial/analyst references and official awards content only — they are not auto-ingested.

Data source access & compliance policy

We read public pages responsibly. Compliance is a first-class, enforced constraint — not a footnote.

robots.txt gate: robots.txt is the gate. Before any scheduled fetch we fetch and cache the target site's robots.txt and only ingest paths Allowed for our user-agent. Disallowed paths (e.g. Shopify *sort_by* URLs, Amazon's name-blocked AI-bot rules, Gmarket's full block) are never fetched. We honor Crawl-delay (e.g. Olive Young Global: 5s) and throttle aggressively.
Once-daily cadence: All retail and review ranking snapshots run once per day. Rankings change at most hourly, so a single daily snapshot is sufficient, minimizes load on the source, and lowers risk.
Derived-data-only: Derived-data-only. We store derived rankings, positions, product identity, price, and review COUNTS — never bulk copies of copyrighted review text or full page captures.
Access-blocked sites are editorial: Access-blocked sites are editorial, not ingested. Sephora (US/CA) and Walmart.ca return anti-bot 403/418 at the access layer regardless of robots.txt; these are referenced as analyst/editorial sources and via official awards content, not automated snapshots.
Identified user-agent: Honest, identified user-agent with contact info, restricted to an explicit allowlist of permitted UAs. Where a site name-blocks AI bots (Amazon blocks ClaudeBot/GPTBot/CCBot while allowing * on bestseller paths) we use a permitted, identified user-agent.

robots.txt Allowed/Disallowed verdict is recorded on every ingestion run.
Crawl-delay is honored per source.
Once-daily cadence for all retail/review ranking snapshots.
Derived data only — never bulk review text or full page captures.
Identified, allowlisted user-agent with contact.
Access-blocked sites are editorial references, not automated sources.
Official awards pages (Olive Young Awards, YesStyle Beauty Awards, Glowpick/Hwahae awards) are preferred as low-risk, high-signal anchors.