How to Index a WordPress Site: Problems, Settings & Fixes

How WordPress Controls Indexing (and Where the Settings Live)

WordPress does not manage indexing on its own. Out of the box it has one blunt instrument: the "Discourage search engines from indexing this site" checkbox in Settings → Reading. Everything else -- per-post noindex, taxonomy noindex, canonical tags, XML sitemaps -- is delegated to an SEO plugin. In 2024 that means Yoast SEO or Rank Math for the vast majority of sites, with a small share using All in One SEO or SEOPress. The layered architecture means indexing is controlled at two levels simultaneously. First, the site-wide WordPress toggle (which outputs a noindex header on every page when checked). Second, the SEO plugin's granular controls (which let you noindex specific post types, taxonomies, archives, or individual URLs). Both layers must be configured correctly, because WordPress’s site-wide toggle overrides the plugin for anything page-level if left on. Where to find these controls: - Site-wide toggle: WordPress Admin → Settings → Reading → "Search Engine Visibility" checkbox - Yoast granular controls: Yoast SEO → Search Appearance → [Content Types / Taxonomies / Archives tabs] - Rank Math granular controls: Rank Math → Titles & Meta → [Post Types / Taxonomies tabs] - Per-post override: the Yoast or Rank Math panel at the bottom of the post/page editor, under "Advanced" Start every WordPress indexing audit at the site-wide toggle. Then work down through the SEO plugin. Then check individual URLs.

The "Discourage Search Engines" Setting: How to Find It and Fix It

The single most common WordPress indexing problem is also the easiest to overlook: the "Discourage search engines from indexing this site" checkbox in Settings → Reading. This checkbox is meant to protect staging environments during development. The problem is that developers routinely check it, launch the site, and forget to uncheck it. Google sees a noindex header on every page and stops adding the site to its index. Worse, if the site was previously indexed, Google gradually removes all the pages it had stored. How to check it: 1. Log in to WordPress Admin. 2. Go to Settings → Reading. 3. Look for the "Search Engine Visibility" section near the bottom. 4. If the checkbox next to "Discourage search engines from indexing this site" is checked, uncheck it and click Save Changes. What the checkbox actually does: when checked, WordPress sends an X-Robots-Tag: noindex HTTP header on every frontend response. It also inserts a meta robots noindex tag into the HTML head. Both signals tell Googlebot not to include those pages in search results. Unchecking the box removes both signals on save -- no cache-clearing required at the WordPress level, though if you run a caching plugin (WP Rocket, W3 Total Cache, LiteSpeed Cache) you should purge the full-site cache afterward so the cached HTML no longer contains the noindex tag. After you uncheck it, verify by opening your homepage in a browser, viewing page source (Ctrl+U), and searching for "noindex". If none appears outside of pages you intentionally noindexed, the setting is fixed.

Yoast & Rank Math Indexing Controls: Taxonomies, Archives, and Attachment Pages

Once the site-wide toggle is clear, the next layer is the SEO plugin's taxonomy and post-type controls. These are more nuanced: some content types should be noindexed, and some should not. What to noindex by default: - Tag archives (/tag/your-tag/): tags rarely have enough unique content to justify indexing. With Yoast, go to Search Appearance → Taxonomies → Tags and set to "No" for "Show Tags in search results." With Rank Math, go to Titles & Meta → Tags and toggle "Noindex." - Media attachment pages (/attachment/image-name/): WordPress auto-generates a page for every uploaded image. These pages contain almost no content. In Yoast, go to Search Appearance → Media and disable indexing. In Rank Math, go to Titles & Meta → Attachment and toggle "Noindex." - Author archives (/author/your-name/) on single-author sites: duplicates the homepage or blog feed. Noindex via Yoast → Search Appearance → Archives → Author Archives, or Rank Math → Titles & Meta → Authors. - Date archives (/2024/01/): thin content. Same tabs as author archives. What to keep indexed: - Category archives (/category/name/): often a good hub page, especially if you write supporting content for that category. Keep indexed, but make sure the category has a unique description. Do not noindex categories just because they feel "thin" -- a well-organized category with 10+ posts is a legitimate index target. - Pages and posts: obviously keep indexed. - Custom post types: depends on the content. Products (WooCommerce), courses (LearnDash), listings (Listify) -- all should remain indexed if they carry real content. Per-post overrides: if you need to noindex a single post without touching the global setting, open the post in the editor, scroll to the Yoast or Rank Math panel, click "Advanced," and set "Allow search engines to show this post in search results" to "No" (Yoast) or toggle "Robots Meta noindex" (Rank Math). This overrides the global setting only for that post.

The SEO Plugin Sitemap: Make It List Only Indexable URLs

Both Yoast and Rank Math generate an XML sitemap automatically. The sitemap should be a clean list of every URL you want Google to crawl and consider for indexing. A polluted sitemap -- one that lists noindexed pages, redirect targets, or 404s -- erodes the sitemap's value and wastes crawl budget. Where your sitemap lives: by default, /sitemap.xml (Rank Math) or /sitemap_index.xml (Yoast), which indexes sub-sitemaps for each post type and taxonomy. How to clean it up: 1. Any post type or taxonomy you set to noindex in the plugin settings should also be excluded from the sitemap. Both Yoast and Rank Math handle this automatically: if you set a content type to noindex, it drops out of the sitemap. Verify by opening your sitemap in a browser and confirming tag pages, attachment pages, and date archives are absent. 2. Check for 301-redirected URLs in the sitemap. If you restructured your URL hierarchy and set up redirects, the old URLs should not remain in the sitemap. Update the sitemap by fixing the underlying page or removing the post. 3. Paginated archives (/category/name/page/2/) are fine to include if they have unique content, but they should carry a self-referencing canonical (not canonical to page 1). The default Yoast/Rank Math behavior handles this correctly. 4. After any major change (adding/removing post types from the sitemap), open Google Search Console → Sitemaps, re-enter your sitemap URL, and click Submit. This tells Google to re-fetch the updated list. Note on sitemap pinging: Google retired its ping endpoint for sitemaps in late 2023. Submitting via GSC is the correct Google-specific method. For Bing (and Bing-powered IndexNow), the ping URL https://www.bing.com/ping?sitemap=[URL] still works, and Rank Math supports automatic IndexNow pinging on publish. IndexNow is not a Google mechanism.

Canonicals in WordPress: Self-Referencing Done Right

A canonical tag tells search engines which version of a page is the "master" copy. WordPress with an SEO plugin outputs canonical tags automatically, but several common configurations cause canonicals to point somewhere they should not, which can suppress indexing of the intended URL. Self-referencing canonicals (the correct default): every indexable page should have a canonical that points to itself. For example, https://example.com/article/ should have rel="canonical" href="https://example.com/article/". This is what Yoast and Rank Math generate by default. Common canonical mistakes in WordPress: 1. Pagination canonicaling to page 1: older SEO advice said to canonical /page/2/ to /page/1/. This is no longer recommended. Each paginated page should have its own self-referencing canonical. 2. HTTPS/HTTP mismatch: if WordPress is configured with http:// in Settings → General but your server serves over HTTPS, the canonical will read http:// while the actual URL is https://. Google will see a mismatch. Fix by updating Settings → General → WordPress Address (URL) and Site Address (URL) to use https://. 3. www vs. non-www: same principle. Pick one variant and make sure the WordPress address, canonical outputs, and sitemap URLs all match. Set up a 301 redirect for the other variant. 4. Taxonomy canonicals pointing to the homepage: this happens when a plugin conflict or a custom theme function overrides the canonical output. Check by viewing source on a category or tag page and confirming the canonical is the category URL, not the homepage. To audit canonicals across many pages: crawl your site with Screaming Frog (the free version handles up to 500 URLs), filter the Canonicals tab, and look for any canonical that does not match the Address column.

robots.txt on WordPress: What It Should and Should Not Block

robots.txt controls which URLs crawlers are allowed to fetch. It does not prevent indexing. This is a frequently misunderstood distinction: if a URL is blocked in robots.txt, Googlebot will not crawl it, but Google can still index it based on signals like external links -- it just will not be able to read the page content. To keep a URL out of the index, you need a noindex tag on the page itself, not a Disallow in robots.txt. Default WordPress robots.txt: WordPress generates a virtual /robots.txt if no physical file exists at the root. The default looks like: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php This is generally correct. It blocks crawlers from the admin panel (which would be wasted crawl budget) while allowing AJAX requests needed for some front-end features. What you should block: - /wp-admin/ (already blocked by default) - Staging subfolders, if any are accessible on the same domain - Certain plugin-generated URLs that are functional but not content (some theme preview URLs, for example) What you should NOT block: - /wp-content/uploads/ -- blocking this prevents Google from crawling your images, which removes them from Image Search. - CSS and JavaScript files -- Google needs to render your pages fully. If JS is blocked, Google may not see lazy-loaded content. - /wp-json/ -- can be blocked if you have no public REST API consumers, but it rarely causes indexing problems either way. Where to edit robots.txt in WordPress: Yoast SEO → Tools → File Editor (if a physical robots.txt does not exist, Yoast edits the virtual one). Rank Math → General Settings → Edit robots.txt. Alternatively, create a physical /robots.txt file at your server's document root -- this overrides WordPress's virtual version.

Getting New WordPress Posts Indexed: Discovery + Submission Workflow

Publishing a post in WordPress does not trigger automatic Google indexing. Google finds new content through crawling, which happens on its own schedule based on your site's crawl frequency. To accelerate discovery, you need to combine passive signals (internal links, sitemaps) with active submission. Step-by-step workflow for new post indexing: 1. Publish the post and confirm it is live with a noindex-free response. Open the post URL in a browser, view source, and confirm there is no noindex meta tag. 2. Add an internal link from an existing, already-indexed page. A topically related post or a "You might also like" section works. Internal links are the most reliable passive discovery signal because Googlebot follows them during its regular crawl of your established pages. 3. Update your sitemap in GSC. Go to Google Search Console → Sitemaps. If your sitemap is already submitted and auto-generated by your plugin, it will include the new post automatically. Click Submit to nudge Google to re-fetch it. 4. Use the URL Inspection tool in GSC. Open the tool, paste your new post URL, and click "Request Indexing." This puts the URL into a crawl priority queue. It does not guarantee indexing -- Google still evaluates the page -- but it typically cuts days off discovery time. 5. Optionally use IndexNow (Rank Math Pro or a standalone plugin). IndexNow pushes the URL to Bing and other participating engines the moment you publish. It has no effect on Google. 6. Share the post wherever it will earn real links or clicks: social profiles, newsletters, relevant forum threads (if genuinely helpful). External links pointing to a new URL are a strong discovery accelerator. Expected timeline: for a site with regular crawl activity and strong internal linking, new posts typically appear in Google's index within days, not weeks. For newer sites or those with slower crawl schedules, it can take 7-14 days. There is no hard guarantee on timing -- Google decides. If a post has not appeared in GSC after 14 days, re-inspect it, check for noindex/canonical issues, and confirm the page returns a 200 status code.

Checking What's Indexed: site:, GSC, and Realistic Expectations

After fixing your WordPress indexing settings, you need a way to confirm the changes took effect. Three tools cover this at different levels of detail. site: search operator: type site:yourdomain.com into Google. This returns an approximate count of indexed pages and lets you spot obvious problems -- noindexed pages that somehow appear, or pages you want indexed that do not appear. Limitations: the count is approximate (Google has said it is not reliable), and the results are not sorted by any useful signal. Use it for a quick sanity check, not a precise audit. Google Search Console (Coverage / Indexing report): this is the authoritative source. Go to GSC → Indexing → Pages. The report shows: - Indexed pages (good) - Pages with issues: "Crawled -- currently not indexed," "Discovered -- currently not indexed," "Excluded by noindex," "Blocked by robots.txt," "Redirect," and more. Each category is clickable. Click "Excluded by noindex" to see exactly which URLs have a noindex tag. Click "Blocked by robots.txt" to see what Googlebot cannot fetch. Click "Crawled -- currently not indexed" to see pages Google visited but chose not to index -- this is the most frustrating bucket and usually means thin or duplicate content. URL Inspection tool: paste any single URL for a detailed report on that specific page: last crawl date, canonical as seen by Google, whether it is indexed, and any detected issues. The "Live Test" option fetches the page fresh from Google's crawler. Bulk indexing check with FastIndexing: if you need to audit a large list of URLs (a site migration, a content audit, or post-launch check across hundreds of pages), FastIndexing's bulk status check tool lets you paste a URL list and see Google indexing status at scale -- far faster than checking URLs individually in GSC. Realistic expectations: based on our own tests, approximately 60-75% of correctly configured URLs that go through an active submission workflow appear in Google's index within 14 days. The remaining fraction takes longer or requires content quality improvements before Google chooses to index them. Google makes the final indexing decision; submission tools improve the odds and the timeline, not the outcome.

From the Field: What 15+ Years of SEO Reveals About WordPress Indexing

Dmytro Puhach, Founder · 15+ years in SEO -- I have audited WordPress sites at every scale, from solo-author blogs to enterprise WooCommerce catalogs with 50,000+ SKUs. A few patterns show up again and again. The staging checkbox problem is more common than it should be. I still encounter sites that have been live for months, sometimes years, with the "Discourage search engines" checkbox checked. The site owner assumes something is wrong with their content or their authority. The fix takes 30 seconds. Always check this first. Taxonomy noindex is undersold. Most WordPress tutorials focus on post-level settings. But a large WordPress blog with 200 tag pages, each one a thin list of 3-5 posts, creates significant duplicate-content noise. Noindexing tags (while keeping categories) typically produces a measurable improvement in how Google allocates crawl budget across the remaining, better-quality pages. Robots.txt is misused as a noindex substitute. I regularly see sites with long Disallow lists intended to "hide" thin pages. It does not work that way. Google can still index a blocked URL via external links. Use noindex for content you want excluded from the index, and robots.txt only to block URLs where you do not want the content read. The indexing timeline expectation gap causes real frustration. Site owners publish a post on Monday and check Google on Tuesday. It is not there. They assume something is broken. For most healthy sites, within days -- not weeks -- is the right frame. For new domains with low crawl frequency, give it 14 days before escalating to a full diagnosis. Active submission still matters. The URL Inspection request-indexing button in GSC, combined with a solid internal link from an already-indexed page, is the most reliable way to shorten the gap between publication and indexing. It is not glamorous, but it works consistently.

When to Go Beyond Manual Submission

Manual submission via GSC works well for sites publishing a handful of posts per week. It does not scale. If you are running a WooCommerce catalog, a programmatic content operation, or a site migration that affects thousands of URLs at once, individual GSC submissions become a bottleneck. This is where a multi-channel indexing service adds practical value. FastIndexing submits your URLs across multiple discovery channels simultaneously -- the Google Indexing API (official, for eligible content types), Google Search Console ping equivalents, Bing IndexNow, and additional signals -- and then monitors which URLs actually get indexed, so you have a feedback loop rather than a black box. For WordPress specifically, FastIndexing integrates via a simple URL list (paste or CSV upload) or via the REST API if you want to trigger submissions automatically on publish. Pricing starts from €0,13 per URL, down to €0,11 per URL with volume. You submit the URLs; Google decides what to index. The goal is to make sure Google sees your URLs promptly and with the right signals, so the decision happens sooner rather than later.

Related terms

FAQ

Read next