I build crawlers and data pipelines that collect hard-to-reach web data at scale — protected sites, APIs, mobile apps, and on-chain sources — and deliver it clean, structured, and monitored.
Minner
Data Acquisition Engineer
Based in Shanghai · Available for remote data & automation work
About
I'm Minner, a data acquisition engineer based in Shanghai. For the past seven years I've built web data collection systems — starting from individual crawlers and growing into distributed platforms that keep 1,400+ sources running and produce over 200,000 records a day.
My specialty is the hard end: targets that actively resist collection. I've spent years on JavaScript reverse engineering — deobfuscating protected scripts, reconstructing encrypted request parameters and signatures, emulating browser environments, and working through fingerprinting and challenge systems — plus mobile app protocol analysis.
I also lead a data collection team, so I think about data as a service rather than a script: cost and feasibility assessment before we start, delivery formats that fit your workflow, monitoring, and a plan for when a source inevitably changes. On-chain and market data is one of the verticals I work in.
Services
I collect data other people cannot, and turn it into a feed your team can rely on — from a single stubborn source to a fleet of thousands.
Build collectors that keep running — from a single source to distributed fleets covering thousands of sites, with scheduling, incremental sync, retries, and health monitoring.
Get data from targets that actively resist collection: obfuscated and dynamically generated JavaScript, encrypted request parameters and signatures, browser-fingerprint checks, and mobile app protocols.
I work within legal and ToS boundaries — no personal data, no breaking authentication.
Turn raw collected data into something your team can actually use — cleaning, deduplication, storage, and delivery as bulk exports, query APIs, or daily incremental feeds, with monitoring and alerts.
The Web3 vertical: on-chain data collection and analysis — wallet and supply mapping, DEX/CEX market data, exchange API integration, and monitoring with alerting.
Projects
Anonymised write-ups of real production systems — commercial data services, collection infrastructure, and pipelines running at scale.
Problem
A client needed continuous competitive pricing data from eleven electronic-component distributor sites. Several of them sit at the hardest tier of the public web — commercial bot-protection, request signing, and browser-fingerprint checks — and the contract put a hard SLA on it: any interruption to the feed had to be restored within 72 hours.
Solution
I ran this end to end: feasibility and cost assessment before signing, then delivery. Sources were graded into difficulty tiers so effort went where it mattered. The hard tier needed reverse engineering of the protection layer — reconstructing signed request parameters, emulating the browser environment, and pairing that with proxy and session strategy plus query-space optimisation to keep coverage complete without hammering the target.
Result
Around 800,000 records delivered per day across three formats — one-off bulk export, a real-time query API, and a daily incremental async feed. The service ran for 22 months to the end of the contract term with the SLA met.
What this proves
I can carry a commercial data service against actively-defended targets: scoping and pricing the risk up front, breaking the hard sites, and then keeping a contractual feed alive for years — not just writing a scraper that works once.
What I can build for you
Competitive price and product monitoring feeds, market intelligence datasets, and any recurring data service where the target resists collection and uptime actually matters.
Problem
Once collection grew past a thousand sources, the crawlers themselves stopped being the hard part — managing them did. Deploying, scheduling, spotting a spider that had silently broken, and restarting work across several machines was eating the team's time.
Solution
Designed and built an in-house platform: a web control plane for the fleet, plus a lightweight agent deployed on each collection node that reports machine and spider health back and handles start/stop locally. It covers task distribution, node status monitoring, scheduled runs, batch start/stop, and alerting when a job fails.
Result
Four collection nodes and 1,400+ sources running under one system, producing 200,000+ records per day, with a large drop in the time it took to deploy and recover spiders.
What this proves
I build the infrastructure layer, not only individual crawlers — scheduling, observability, and failure recovery for a fleet that has to keep running unattended.
What I can build for you
Crawler orchestration and scheduling systems, health monitoring and alerting for an existing spider fleet, and internal tooling to make a collection team faster.
Problem
A media-intelligence product needed near-complete coverage of a major short-video platform — video, creator, live-stream and commerce data — refreshed constantly. The platform exposes different data on different surfaces, and no single entry point gives you everything.
Solution
Collected across every available surface of the platform (app, web, mobile share pages, and mini-program) and merged them into one dataset. Built the throughput on a distributed task queue with Elasticsearch for search and retrieval, and Redis for dedup and scheduling state.
Result
Over 1M newly-discovered videos per day and 4M+ daily refreshes at steady state, peaking at 5M new and 12M refreshed per day, alongside 100K+ creator profiles and thousands of live streams updated daily.
What this proves
I can design for volume — queue architecture, storage choice, deduplication and refresh strategy — and keep a multi-million-record-per-day pipeline stable.
What I can build for you
Large-scale social, marketplace or content datasets; multi-surface collection where one endpoint isn't enough; and the storage and refresh design that keeps a big dataset current.
Problem
For any token, the questions that decide risk — how much supply is really locked vs circulating, which wallets make the market, how much volume is wash, and when a dormant mega-wallet starts moving — are slow to answer by hand and easy to get wrong.
Solution
A collection and analysis system for on-chain data: it traces mint → treasury → current balances, classifies every significant wallet by role, and quantifies the real float. A Python collector then runs continuously, writing metrics to a database that a FastAPI backend serves, a React dashboard renders, and a Telegram bot uses for tiered alerts.
Result
Applied to a live mid-cap Solana token: mapped ~95% of supply to two dormant wallets (real float ~1–2%), flagged a large share of reported volume as wash trading, and surfaced market-maker distribution into a multi-fold rally.
What this proves
The same collection-to-delivery pipeline applied to a different data source: RPC and DEX data in, classification and analysis in the middle, dashboard and alerts out — shipped as a Dockerized service with CI/CD.
What I can build for you
On-chain monitoring and alerting, wallet and supply dashboards, exchange API data feeds, and market data pipelines with a dashboard on top.
Why Work With Me
I focus on practical delivery, clear communication, and building tools that actually fit your workflow.
I cover the whole chain — collection, the anti-bot work, cleaning and storage, and the API, feed or dashboard your team actually uses.
Before you commit, I assess how hard your targets really are and produce sample data from them, so you know what you're buying instead of guessing.
Sources change and break — that's normal. I build in monitoring, alerting and recovery, and I've kept commercial feeds alive under contractual SLAs for years.
I provide structured updates, explain trade-offs, and work well with async international teams.
I work within legal and ToS boundaries: publicly accessible data only, no personal data, no breaking authentication, and request rates that don't harm the target.
Process
How I work: assess honestly, prove it with real sample data, then build something that keeps running.
I look at your actual sources and judge how hard each one is, what it will cost to run, and where the risk sits — including telling you if something isn't worth it.
Before a full build, I pull real sample data from your real targets, so you can check the fields, coverage and quality against what you actually need.
Collection, the anti-bot work, cleaning and storage — delivered in the format you'll consume: bulk export, query API, or a daily incremental feed.
Sources change. I add monitoring and alerting, and fix breakages so the feed stays alive instead of quietly going stale.
Tech Stack
The stack behind the tools — practical, well-supported, and chosen for reliable delivery.
Scraping
Reverse Engineering
Scale & Infra
Storage & Delivery
On-chain & AI
Contact
Send me a short description of your project. I can help you break it down, design the system, and build a working prototype.
Not sure about the exact scope yet? Send a rough idea. I can help break it down into a practical MVP.