What it does
Fetches public HTML pages, sitemaps, and RSS/Atom feeds to build our search index. It does not submit forms or interact with your site.
Niguro Crawler
Meet NiguroBot
NiguroBot is the automated crawler that powers Niguro Search. It discovers, fetches, and indexes public web pages so they can be found by people searching in Nepali.
Identifying NiguroBot in your logs
Every request from our crawler sends the following user-agent string. The
+https://www.niguro.com/bot
points back to this page.
NiguroBot/1.0 (+https://www.niguro.com/bot)
How NiguroBot behaves
Polite, predictable, and respectful of your settings.
Crawl rate
Requests are spaced out and rate-limited per host. NiguroBot honors Crawl-delay and never hammers a single server.
Respects robots.txt
NiguroBot reads and obeys your robots.txt and noindex/nofollow directives before and during crawling.
Privacy
Only publicly available content is stored. We do not collect personal data through crawling.
Technical details
How NiguroBot discovers and indexes content.
Discovery
NiguroBot discovers pages through sitemaps, RSS/Atom feeds, and links found on already-indexed pages. Submitting a sitemap via the Niguro Console helps us discover your content faster.
What we index
We index HTML content, meta descriptions, Open Graph data, and structured data (schema.org). We do not index password-protected pages, submitted form data, or non-public content. File types such as PDF, images, and video metadata may be extracted when available.
Recrawl frequency
The recrawl schedule depends on the site's popularity, update frequency of its content, and our crawl capacity. Most sites are revisited every few weeks. You can check your site's crawl interval on the Niguro Console after adding and verifying your domain.
Verification for site owners
To see crawl stats, request re-indexing, or manage how your site appears in search results, verify your site ownership on the Niguro Console. Verification is done by adding a meta tag to your site's homepage <head>. The Niguro Console provides the exact tag to insert.
How to control or block NiguroBot
If you prefer not to be crawled, add the following to your site's robots.txt:
User-agent: NiguroBotDisallow: /To allow crawling of everything except a specific folder:
User-agent: NiguroBotAllow: /
Disallow: /private/Changes are picked up on the crawler's next visit. For urgent removal of a page from search, use the contact form.
Frequently asked questions
Common questions about NiguroBot and the search index.
How do I verify my site?
Go to the Niguro Console and add your domain, then add the provided meta tag to your site's homepage <head>. Once verified, you can view crawl stats, adjust settings, and request re-indexing.
How do I request removal of a page?
For quick removal, use our contact form with the page URL. For ongoing control, add noindex meta tag or block in robots.txt — NiguroBot will respect it on the next crawl.
Does NiguroBot support nofollow?
Yes. NiguroBot respects rel="nofollow" on links and meta robots directives including nofollow, noindex, nosnippet, and noarchive.
How can I request re-indexing of my site?
Verify your site on the Niguro Console, then use the crawl button to trigger an immediate re-crawl. Unverified sites are crawled on a best-effort schedule.
What IP addresses does NiguroBot use?
NiguroBot crawls from a set of dedicated IP addresses assigned to our servers. The range may change over time. For the most up-to-date list, contact us and we will provide the current addresses.
Questions about NiguroBot?
We're happy to help with crawl issues, indexing requests, or verification. Reach out and a human from the Niguro team will respond.