The site control page that handles forbidden access to web pages has been upgraded. Forbidden Access controls whether robots.txt is obeyed by the Link Alarm robot or bypassed.

Previously the Forbidden Access was a simple on/off setting and only affected external pages; outside of the pages within the Checking Limits settings. Pages included in the Checking Limits were still subject to robots.txt and this caused some frustration for our users.

Now there is far greater control in bypassing robots.txt. Bypass can now be set to not bypass (the default), bypass only on internal links, only on external links, or both. The greater granularity of control will help checking sites where portions of the site are protected by robots.txt while still obeying the rules for external sites.

There are other ways to avoid robots.txt problems on your own site. You can explicitly allow Link Alarm access to your site by adding the following at the top of your robots.txt:

User-agent: LinkAlarm
Allow: /

Remember to be considerate when bypassing robots.txt with sites other than your own. Site owners who have a robots.txt file usually have a reason to do so and bypassing those controls can be seen as rude. So unless it's essential, respect robots.txt on sites you do not own. It's good netiquette.



Comments are closed.

Register for a Free Account

Join thousands of web managers using LinkAlarm on their sites. Register for a free account now.

Your LinkAlarm account includes everything you need to check 100 pages anywhere on the web. To check more pages just credit your account to cover your requirements. The price is only one cent ($US 0.01) per page checked.

Link Alarm Is Up For Sale

I'm sorry to report that we'll be closing the LinkAlarm service in early 2022. After 23 years of operation, 11 under current ownership, it's time for this owner move to other opportunities. It's been great with you and wish you every success in the future. You're welcome to continue using your account until we shut down, probably January 2022.

But wait! Maybe you could be the new owner/operator of Link Alarm. We are open to selling Link Alarm assets and virtual machine that runs the checking service. If you depend heavily on Link Alarm perhaps you would would want to purchase an instance for in-house use.

For an entrepreneurial individual there are opportunities to build upon the solid base that Link Alarm provides. Dress-up the user interface and explore other options like selling Link Alarm as a licensed enterprise service, an open source product or a mobile application. Everything is available including the intellectual property, domains and business name, We are entertaining offers. Reach out to me at

Copyright ©1998-2019 Link Alarm