Shopify scraper: the endpoints, the rules and the limits
Every Shopify store publishes its own rules and JSON endpoints. The commands that read them, what the data leaves out, and where scraping breaks.
GuideNovember 12, 2026 · 12 min read
Almost every tool sold as a Shopify scraper is a wrapper around a handful of JSON endpoints that the platform itself serves to anybody who asks. You can call them with curl in one line, and the interesting part is not how to call them: it is which ones a store's own rules permit, what the data leaves out, and the four places the whole approach falls over. ShopLeads is our product, so we say so, and it appears once at the end where an index beats a scraper; the rest of this page is commands.
The pages ranking for this query are tool landing pages, and they all answer the same narrow question: how do I get a store's products into a CSV. None of them mention that a Shopify store now publishes machine-readable instructions for exactly this at three fixed paths, or that the default robots.txt names the endpoints you may not touch. Everything below was read live on November 9, 2026 from the stores named next to it.
Path
What it returns
Allowed by the default robots.txt?
/robots.txt
The crawl rules, a terms-of-service link and a contact address
Paths read on live storefronts on November 9, 2026; the permission column is from Shopify's default robots.txt as served by those stores and as documented by Shopify (Shopify Help).
Read the store's own rules before you write any code
This is the part that has changed, and the part no ranking page covers. Shopify's current default robots.txt opens with a plain-English statement of what is crawlable and then points automated clients at two more files. Read verbatim from a live store on November 9, 2026, it begins: "Shopify storefront. Public product, collection, page, blog, policy, cart, and localized HTML is crawlable", then lists an agent instructions URL, a UCP discovery URL, an MCP endpoint, a link to Shopify's terms and the address [email protected] (RotoWear robots.txt).
The Allow and Disallow lines matter more than the preamble. Shopify documents the defaults: Allow: / for public product, collection, page, blog and policy pages, with Disallow on /admin, /cart/, /checkout, /checkouts/, /account, /orders and the filtered and sorted collection variants (Shopify Help). The live file adds more, including /cart.js, /recommendations/products, /services and /sf_* (RotoWear robots.txt). Note what is absent from every Disallow line: /products.json, /collections.json and /meta.json. The product JSON is not carved out. The cart and recommendation endpoints are.
/agents.md is the second file, and it reads like a rate-limit policy written for you. The version served on November 9, 2026 tells automated clients to identify agent-initiated requests, to "back off on 429 responses and respect Retry-After when provided", to discover public URLs through /sitemap.xml, to read product pages at /products/{handle} and collections at /collections/all, and not to script checkout forms (RotoWear agents.md). Shopify also publishes it in the sitemap index as sitemap_agentic_discovery.xml (Beardbrand), and documents the broader integration surface for agents on its developer site (Shopify.dev).
Every store robots.txt we read links to Shopify's Terms of Service and gives a contact address for bot operators (Shopify Terms). If you are building anything you intend to run at volume or resell, read those terms rather than a blog post about them, and keep your requests identifiable.
The commands that read a Shopify store
Four requests answer almost every question a Shopify products scraper is bought for. Run them against any store you can open.
curl -s https://domain/meta.json — the store profile. On Beardbrand this returned the store name, Austin in Texas, US, USD, the myshopify handle, 68 published products and 48 published collections.
curl -s "https://domain/products.json?limit=250&page=1" — the first 250 products with their variants and images. Increase page until a page comes back short or empty.
curl -s "https://domain/collections.json?limit=250" — every published collection, which is how you slice a large catalog into categories without guessing.
curl -s https://domain/sitemap.xml — the sitemap index, which points at per-type sitemaps for products, collections, pages, blogs and articles.
Two behaviors are worth knowing before you write the loop. The page size caps at 250: on Beardbrand, limit=500 returned byte-for-byte the same payload as limit=250 (Beardbrand). And pagination is a plain walk, not a cursor. Gymshark's US store reports 10,400 published products in /meta.json; pages 1, 2 and 41 of /products.json?limit=250 each returned 250 products and page 42 returned 150, which is exactly 10,400 (Gymshark). When the arithmetic matches the count in /meta.json, you know you got the whole catalog.
If you would rather not write the loop, the oldest open-source option does just this: a Python package that "fetches Shopify's publicly visible products.json file, allowing you to scrape an entire store inventory in seconds" (ShopifyScraper on GitHub).
What products.json contains, and what it leaves out
Read one record before you trust a tool's feature list. On November 9, 2026 a product object from Beardbrand carried exactly these keys (Beardbrand):
Inventory quantity, sales, revenue, customers, orders, the installed app list, the owner
The Ajax API "can't be used to read any customer or order data" (Shopify.dev)
Keys read from a live product record on November 9, 2026 (Beardbrand /products.json). Field sets can differ between stores and Shopify versions.
The single most misread field is available. It is a boolean, not a quantity: it tells you the variant can be bought, and nothing about how many are left. Tools that report stock counts get them elsewhere and say so when they are honest — the Apify actor's own page says a real count "needs grabRealInventory and a store that publishes its stock numbers. A store that does not publish them gets no count and is not billed for one" (Apify).
The second thing the catalog cannot give you is the market price. On Gymshark, /products.json?limit=1 returned the same variant price with no country parameter, with ?country=CA and with ?country=GB (Gymshark). Shopify's own guidance is to build locale-aware URLs from window.Shopify.routes.root for anything market-specific (Shopify.dev), which a server-side scraper does not have. If you need per-market pricing, request the storefront for that market, not a query parameter.
Where a Shopify scraper breaks
1
The store answers 429 instead of JSON
Rate limits and bot management
Shopify says the Ajax API has "no hard rate limits" but is still subject to standard abuse-prevention measures (Shopify.dev), and in practice that means a storefront behind a proxy can refuse you. On November 9, 2026, allbirds.com/products.json answered HTTP 429 with retry-after: 60 from Cloudflare, while the same store's homepage and /meta.json answered normally (Allbirds). The store was not down and not un-scrapable; the request pattern was simply refused.
Honor Retry-After. The store is telling you the cost of ignoring it.
One domain at a time, serially, with a pause, beats a fan-out that trips the limit on the first page.
A 429 on /products.json with a 200 on /meta.json means rate limiting, not a missing endpoint.
2
The storefront is headless, and may serve nothing
Hydrogen and Oxygen
A store built on Shopify's Hydrogen framework serves a custom front end, and whether the legacy JSON endpoints still answer varies by store. Two headless storefronts, read the same morning: Chubbies referenced Hydrogen and Oxygen in its HTML and still answered /products.json and /meta.json with JSON (Chubbies), while Hiut Denim referenced the same stack and answered 404 on both (Hiut Denim).
A 404 on /products.json is not proof the store is not on Shopify. Check the HTML for cdn.shopify.com first.
For a headless store with no JSON, the sitemap plus product pages is the remaining public route.
Shopify documents the Ajax API as unavailable to custom storefronts, which is the same boundary from the other side.
3
You collected the catalog and still cannot answer the question
The data is about products, not businesses
Most people who scrape a store do not want its products. They want to know whether the business is worth contacting: how big it is, what software it runs, whether it is growing, who decides. None of that is in the catalog. Catalog size and product created_at dates are proxies, and that is where the public JSON stops.
Catalog size plus recent created_at dates tells you whether a store is actively merchandised.
The vendor field can reveal that a "brand" is a reseller carrying other people's products.
Installed apps, sales estimates, plan and headcount all come from somewhere other than /products.json.
4
One domain is not one store
Markets and regional storefronts
Larger brands run separate storefronts per region, each with its own catalog and its own counts. Gymshark's US domain reports its own product count and its own myshopify handle, which is not the same storefront a shopper in another country sees. A scraper pointed at the root domain quietly collects one market and reports it as the business.
Compare the count in /meta.json with the pages you actually walked. A mismatch means you are on a different storefront than you think.
The myshopify handle in /meta.json tells you which storefront you reached.
The tools, and what each one charges
If you would rather buy than build, these were the options ranking for this query, read on their own pages on November 9, 2026. All of them read the endpoints above; the difference is packaging and billing.
Hosted actor; whole catalog, a collection, one product URL or a keyword search; no API key needed
Pay per event: $0.0010 per product saved, falling to $0.0007 at higher spend tiers, $0.0004 per collection; its page states 100,000 products costs $70 at the GOLD tier (its pricing)
Read on each tool's own page on November 9, 2026. Pricing moves; re-read before you buy.
Two honest notes from the Apify actor's own page, because they apply to every tool in the category: it does not merge SKUs that a store presents as one product page, and it skips Shopify's 3D and AR media rather than mislabeling it as images (Apify). Those are catalog-modeling decisions you inherit from whichever tool you pick, and they change your row counts.
When you want an index instead of a scraper
A scraper answers "what is in this store". It is one domain deep by design, and the question most buyers of a Shopify products scraper actually have runs the other way: which stores look like this, and who at them decides. Store indexes exist for that. Store Leads tracks over 40 attributes per domain with weekly updates and says its own contact data is non-personal only, with personal contact data available through a third-party integration you enable yourself (Store Leads).
The practical division of labor: an index to find the accounts, the endpoints above for depth on the ones you already care about. The catalog read is still worth doing on a shortlist, because product titles and tags give you the sentence that makes a first email look addressed.
ShopLeads is the index built for the second half of that, and the part worth naming here is the catalog. Product search runs over product title, type, tags and description across stores, so the question that sent you to a scraper in the first place — who sells this — is a filter rather than a 42-page walk per store with a rate limiter in the middle. Stores arrive already confirmed as real DTC brands by rules and an AI check, already paired with their decision-makers, and already carrying installed apps and technologies by exact name, which is the field no /products.json crawl returns at all. Searching and counting are free and never reveal a contact, and the same filters run over a REST API and an MCP server. Signing up is open: the trial runs 7 days with 500 free credits and a card. The Shopify store directory is the browsable version of the same index; start the free trial or read the pricing page.
Request https://domain/products.json?limit=250&page=1 and walk the page number until a page comes back short. Each record carries the product title, handle, tags, vendor, type, creation date, and every variant with its SKU, price and an availability boolean.
Is it legal to scrape a Shopify store?
The store tells you what it permits in its own robots.txt, which on a default Shopify storefront allows public product, collection, page, blog and policy content and disallows admin, cart, checkout, account and order paths. It also links Shopify's Terms of Service and gives a contact address for bot operators, which is what you should read before running anything at volume.
Can you scrape Shopify without an API key?
Yes for public catalog data. Shopify documents the Ajax API as unauthenticated, with no access token required, but it cannot read customer or order data and product JSON responses are limited to 250 variants.
Why does products.json return 429?
The storefront is rate limiting or bot filtering you. Shopify says there are no hard rate limits but standard abuse-prevention measures still apply, and the response usually carries a Retry-After header telling you how long to wait. Honor it and slow down rather than retrying immediately.
Does products.json include stock levels?
No. Each variant carries an available boolean that says whether it can be bought, not how many units remain. Any tool reporting real stock counts is getting them from somewhere other than this endpoint, and only for stores that publish them.
