|
||||||||
|
||||||||
|
||||||||
|
|
|
User Tools
2D Graphics
- Fireworks (272)
- Flash (4177)
- Gimp (257)
- Illustrator (1558)
- Paint Shop Pro (54)
- Photoshop (17363)
3D Graphics
- 3DS MAX (2468)
- AutoCAD (90)
- Blender 3d (293)
- Bryce 3d (29)
- Cinema 4D (144)
- Lightwave 3D (38)
- Maya (579)
Web Design & Development
- Adobe Flex (65)
- ASP (376)
- ASP.NET (489)
- ColdFusion (16)
- CSS (580)
- Dreamweaver (381)
- HTML (413)
- Java (773)
- Javascript (972)
- MM Fusion (1)
- MS FrontPage (20)
- Perl and CGI (128)
- PHP (1563)
- Python (69)
- Ruby on Rails (106)
- SEO (192)
- Web Design (322)
- XML (164)
Business Applications
- MS Excel (449)
- MS PowerPoint (170)
- MS Word (158)
- QuarkXPress (8)
Business Development
- Accounting (45)
- eCommerce (75)
- Legal (7)
- Marketing (242)
Databases
Desktop Programming
- C and Cpp (510)
- Delphi (21)
- Visual Basic (336)
- Visual C (29)
Video Editing
- Final Cut Studio (23)
- Premiere (64)
Miscellaneous
- P2P Filesharing (8)
- Web Hosting (160)
Newsletter
User Login
Stats
- Tutorials: 39,023
- Categories: 572
HTML Tutorials : Understanding a Robots.txt File
| The robots.txt file allows you to control the behavour of web crawlers and spiders that visit your site. Most web crawlers are harmless and simply collect data for various reasons like search engine listings, internet archiving, validating links, security scanning, etc. It's always a good idea to create a robots.txt to tell the crawlers where they can go and where they can not. |  |
| |
| • Comment On Tutorial |
• View Tutorial
• Report Broken Link |
|
|
