Robots.tx explained

Hi.

Welcome to this week's web design video blog.

Today Nick and I are going to explain the purpose of the robots.txt file and also share the common rules that you might want to use to communicate with search engine robots like Googlebot.

So the primary purpose of the robots.txt file is to restrict access to your website by search engine robots or BOTS. The file is quite literally a simple dot txt text file that can be opened and created in almost any notepad HTML editor or word processor. To make a start, name your file robots.txt and add it to the root layer of your website. This is quite important as all of the main reputable search engine spiders will automatically look for this file to take instruction before crawling your website. So here's how the file should look to start with. On the very first line: add "User-agent". This first command essentially addresses the instructions to all search BOTS. Once you've adjust a specific or in our case with the asterisks all search BOTS you come on to the allow and disallow commands that you can use to specify your restrictions. To simply ban the bots from the entire website directory including the home page, you'll add the following code: disallow with a capital of D : space and a forward slash. This first four slash represents the root layer of your website.

In most cases you won't want to stripped your entire website just specific folders or files. To do this you specify each restriction on its own line proceed with the disallow command. In example here you can see the necessary code to strict access to a folder called admin along with everything inside it.

If you're looking to restrict individual pages or files the format is very similar online for we're not restricting their entire secure folder just one HTML file within it. You should bear in mind that these directives are case sensitive. You need to ensure that you specify on your robots.txt file the exact match against the file or folder name of your website. So those are the basics and next we come onto slightly more advanced pattern matching techniques.

These can be quite handy if you're looking to block files or directories in bulk without having lines of commands on your robots file. These bulk commands are known as pattern matching and the most common one that you might want to use would be to restrict the access to all dynamically generated URLs that contain the question mark for example. So if you check out line 5 all you need to do to catch all of these type is the forward slash Asterix and then a question mark symbol. you can also use pattern matching to block access to all directories that begin with admin for example: if you check out line 6 you'll see how we've again use an Asterix to match all directories let's start with admin.

So again if for example you had the following folders on your root directory admin - panel admin - files and admin - secure then this one line on your robots file would block access to all three of these folders as they all start with admin. The follow pattern matching command is the ability to identify and restrict access to all files that end with a specific extension. On line 7 you'll see how the single short command will instruct the search engine BOTS and spiders not to crawl and ideally cache any pages that contain the dot PHP extension. So after your initial forward slash, use the asterisks followed by a full stop and then the extension. To signify an extension instead of a string you can clued the command with the dollar sign. The dollar tells the bots that the file to be restricted must end with this extension.

So that's it using those six different techniques you should be able to put together a well optimized robots.txt file that Flags content to search engines that you don't wish to be crawled or cached.

It is important that we point out however that website hackers often frequent robots.txt files as they can indicate where security vulnerability may lie that they might want to throw themselves at. Always be sure to password protect and test the security of your dynamic pages particularly if you're advertising their location on a robots.txt file.

Thank you for watching this week's web design video blog

Hi.

Welcome to this week's web design video blog.

Today Nick and I are going to explain the purpose of the robots.txt file and also share the common rules that you might want to use to communicate with search engine robots like Googlebot.

So the primary purpose of the robots.txt file is to restrict access to your website by search engine robots or BOTS. The file is quite literally a simple dot txt text file that can be opened and created in almost any notepad HTML editor or word processor. To make a start, name your file robots.txt and add it to the root layer of your website. This is quite important as all of the main reputable search engine spiders will automatically look for this file to take instruction before crawling your website. So here's how the file should look to start with. On the very first line: add "User-agent". This first command essentially addresses the instructions to all search BOTS. Once you've adjust a specific or in our case with the asterisks all search BOTS you come on to the allow and disallow commands that you can use to specify your restrictions. To simply ban the bots from the entire website directory including the home page, you'll add the following code: disallow with a capital of D : space and a forward slash. This first four slash represents the root layer of your website.

In most cases you won't want to stripped your entire website just specific folders or files. To do this you specify each restriction on its own line proceed with the disallow command. In example here you can see the necessary code to strict access to a folder called admin along with everything inside it.

If you're looking to restrict individual pages or files the format is very similar online for we're not restricting their entire secure folder just one HTML file within it. You should bear in mind that these directives are case sensitive. You need to ensure that you specify on your robots.txt file the exact match against the file or folder name of your website. So those are the basics and next we come onto slightly more advanced pattern matching techniques.

These can be quite handy if you're looking to block files or directories in bulk without having lines of commands on your robots file. These bulk commands are known as pattern matching and the most common one that you might want to use would be to restrict the access to all dynamically generated URLs that contain the question mark for example. So if you check out line 5 all you need to do to catch all of these type is the forward slash Asterix and then a question mark symbol. you can also use pattern matching to block access to all directories that begin with admin for example: if you check out line 6 you'll see how we've again use an Asterix to match all directories let's start with admin.

So again if for example you had the following folders on your root directory admin - panel admin - files and admin - secure then this one line on your robots file would block access to all three of these folders as they all start with admin. The follow pattern matching command is the ability to identify and restrict access to all files that end with a specific extension. On line 7 you'll see how the single short command will instruct the search engine BOTS and spiders not to crawl and ideally cache any pages that contain the dot PHP extension. So after your initial forward slash, use the asterisks followed by a full stop and then the extension. To signify an extension instead of a string you can clued the command with the dollar sign. The dollar tells the bots that the file to be restricted must end with this extension.

So that's it using those six different techniques you should be able to put together a well optimized robots.txt file that Flags content to search engines that you don't wish to be crawled or cached.

It is important that we point out however that website hackers often frequent robots.txt files as they can indicate where security vulnerability may lie that they might want to throw themselves at. Always be sure to password protect and test the security of your dynamic pages particularly if you're advertising their location on a robots.txt file.

Thank you for watching this week's web design video blog