Perl & LWP

by Sean M. Burke

"Fetching web pages, Parsing HTML, Writing Spiders, & More"

• Introduction to the 2007 online edition of Perl and LWP
• Foreword, by Gisle Aas
• Search!
• Preface
• Structure of This Book
• Order of Chapters
• Important Standards Documents
• Conventions Used in This Book
• Comments & Questions
• Acknowledgments
• Introduction to Web Automation
• History of LWP
• Installing LWP
• Words of Caution
• LWP in Action
• Web Basics
• An HTTP Transaction
• LWP::Simple
• Fetching Documents Without LWP::Simple
• Example: AltaVista
• HTTP POST
• Example: Babelfish
• The LWP Class Model
• Programming with LWP Classes
• Inside the do_GET and do_POST Functions
• User Agents
• HTTP::Response Objects
• LWP Classes: Behind the Scenes
• URLs
• Relative URLs
• Converting Absolute URLs to Relative
• Converting Relative URLs to Absolute
• Forms
• LWP and GET Requests
• Automating Form Analysis
• Idiosyncrasies of HTML Forms
• POST Example: License Plates
• POST Example: ABEBooks.com
• File Uploads
• Limits on Forms
• Simple HTML Processing with Regular Expressions
• Regular Expression Techniques
• Troubleshooting
• When Regular Expressions Aren't Enough
• Example: Extracting Links from a Bookmark File
• Example: Extracting Links from Arbitrary HTML
• Example: Extracting Temperatures from Weather Underground
• HTML Processing with Tokens
• Basic HTML::TokeParser Use
• Individual Tokens
• Token Sequences
• More HTML::TokeParser Methods
• Using Extracted Text
• Tokenizing Walkthrough
• Getting the Data
• Inspecting the HTML
• First Code
• Narrowing In
• Rewrite for Features
• Alternatives
• HTML Processing with Trees
• HTML::TreeBuilder
• Processing
• Example: BBC News
• Example: Fresh Air
• Modifying HTML with Trees
• Deleting Images
• Detaching and Reattaching
• Attaching in Another Tree
• Creating New Elements
• Cookies, Authentication, and Advanced Requests
• Adding Extra Request Header Lines
• Authentication
• An HTTP Authentication Example:The Unicode Mailing Archive
• Spiders
• A User Agent for Robots
• Example: A Link-Checking Spider
• Ideas for Further Expansion
• LWP Modules
• HTTP Status Codes
• 200s: Successful
• 300s: Redirection
• 400s: Client Errors
• 500s: Server Errors
• Common MIME Types
• Language Tags
• Common Content Encodings
• ASCII Table
• User's View of Object-Oriented Modules
• Modules and Their Functional Interfaces
• Modules with Object-Oriented Interfaces
• What Can You Do with Objects?
• What's in an Object?
• What Is an Object Value?
• So Why Do Some Modules Use Objects?
• The Gory Details
• Colophon
• Copyright

https://lwp.interglacial.com/
ISBN-10: 0596001789
ISBN-13: 978-0596001780
June 2002: First edition, O'Reilly
March 2007+: Online edition 1.1

Get the print edition: O'Reilly · Isbn.nu · Powell's · Amazon · Amazon-UK