Create bookmark
Webbots, spiders, and screen scrapers
a guide to developing internet agents with PHP/CURL
Notes
Please login to add notes
- Acknowledgments
- Tables of Contents
-
+
Introduction
- PART I: Fundamental Concepts and Techniques
-
+
1: What’s in It for You?
-
+
2: Ideas for Webbot Projects
-
+
3: Downloading Web Pages
-
+
4: Parsing Techniques
-
+
5: Automating Form Submission
-
+
6: Managing Large Amounts of Data
- PART II: Projects
- + 7: Price-Monitoring Webbots
- + 8: Image-Capturing Webbots
-
+
9: Link-Verification Webbots
- + 10: Anonymous Browsing Webbots
-
+
11: Search-Ranking Webbots
- + 12: Aggregation Webbots
- + 13: FTP Webbots
- + 14: NNTP News Webbots
- + 15: Webbots That Read Email
-
+
16: Webbots That Send Email
- + 17: Converting a Website into a Function
- PART III: Advanced Technical Considerations
-
+
18: Spiders
-
+
19: Procurement Webbots and Snipers
- + 20: Webbots and Cryptography
-
+
21: Authentication
- + 22: Advanced Cookie Management
-
+
23: Scheduling Webbots and Spiders
- PART IV: Larger Considerations
-
+
24: Designing Stealthy Webbots and Spiders
- + 25: Writing Fault-Tolerant Webbots
-
+
26: Designing Webbot-Friendly Websites
-
+
27: Killing Spiders
- + 28: Keeping Webbots out of Trouble
-
+
A: PHP/CURL Reference
- Creating a Minimal PHP/CURL Session
- Initiating PHP/CURL Sessions
-
+
Setting PHP/CURL Options
- CURLOPT_URL
- CURLOPT_RETURNTRANSFER
- CURLOPT_REFERER
- CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS
- CURLOPT_USERAGENT
- CURLOPT_NOBODY and CURLOPT_HEADER
- CURLOPT_TIMEOUT
- CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR
- CURLOPT_HTTPHEADER
- CURLOPT_SSL_VERIFYPEER
- CURLOPT_USERPWD and CURLOPT_UNRESTRICTED_AUTH
- CURLOPT_POST and CURLOPT_POSTFIELDS
- CURLOPT_VERBOSE
- CURLOPT_PORT
- + Executing the PHP/CURL Command
- Closing PHP/CURL Sessions
- + B: Status Codes
- C: SMS Email Addresses
- Index
The Internet is bigger and better than what a mere browser allows. Webbots, Spiders, and Screen Scrapers is for programmers and businesspeople who want to take full advantage of the vast resources available on the Web. There's no reason to let browsers limit your online experience—especially when you can easily automate online tasks to suit your individual needs.
Learn how to write webbots and spiders that do all this and more:
* Programmatically download entire websites
* Effectively parse data from web pages
* Manage cookies
* Decode encrypted files
* Automate form submissions
* Send and receive email
* Send SMS alerts to your cell phone
* Unlock password-protected websites
* Automatically bid in online auctions
* Exchange data with FTP and NNTP servers
Sample projects using standard code libraries reinforce these new skills. You'll learn how to create your own webbots and spiders that track online prices, aggregate different data sources into a single web page, and archive the online data you just can't live without. You'll learn inside information from an experienced webbot developer on how and when to write stealthy webbots that mimic human behavior, tips for developing fault-tolerant designs, and various methods for launching and scheduling webbots. You'll also get advice on how to write webbots and spiders that respect website owner property rights, plus techniques for shielding websites from unwanted robots.
As a bonus, visit the author's website to test your webbots on sample target pages, and to download the scripts and code libraries used in the book.
Some tasks are just too tedious—or too important!— to leave to humans. Once you've automated your online life, you'll never let a browser limit the way you use the Internet again.
Test the closed alpha on paperc.com
Book Details
Authors
Categories
Computers > Programming > General
Publishers
Publication year : 2007
License: All rights reserved ©
Times read: 1,032

