JobShifuBot
Last updated: July 17, 2026
What this is. JobShifuBot is the crawler that keeps JobShifu's job listings
fresh. It reads public job-board APIs and job postings so the openings we show are current.
It does not access personal data, and it does not sign in to anything.
Identifying JobShifuBot
Our crawler sends this User-Agent header:
Mozilla/5.0 (compatible; JobShifuBot/1.0; +https://jobshifu.com/bot)
What it accesses
- Public job-board APIs and public job-posting pages only.
- The listing data those endpoints return — job title, description, location, and the
apply link — so we can keep our listings up to date.
- No personal data, no accounts, no forms, and nothing behind a login.
How it behaves
- Conditional requests. We use ETags and conditional requests so an
unchanged board answers with a small
304 Not Modified instead of resending
the full listing.
- Rate limits. Requests are capped per host, so we stay well within a
modest request rate to any single site.
- Circuit breakers. If a host starts returning errors, we back off and
stop crawling it rather than retrying hard.
- Retry-After. We honor the
Retry-After response header and
wait the requested time before trying again.
Contact and opt-out
If you would like JobShifuBot to slow down or stop crawling your site, or if you have any
questions about our crawler, email [email protected].
Tell us the host you want us to change and we will adjust our crawl.