Comments on: Easy Screen Scraping in PHP with the Simple HTML DOM Library

By: Chea leb

Chea leb — Sun, 31 Jan 2010 10:01:08 +0000

/**
* An example of using remote scraping sessions.
*
* @author Brent Wenerstrom1
*/

using System;
using System.Collections;
using Screen scraper;

public class Data Set Remote Scraping Session Example
{
/**
* The entry point.
*/
public static void Main( string[]args)
{
try
{
// Create a remote Session to communicate with the server.
Remote Scraping Session remote Session = new Remote Scraping Session( “Slashdot” );

// Scraper.
remote Session.Scrape();

// Get the data.
// If the returned value is null an exception will be thrown later.
Data Set data Set = ( Data Set )remote Session.Get Variable( “DATA SET” );

// Temporarily holds data records.
Data Record template Data Record = null;

Console.Write Line( “============================================================================” );

// Iterate through the data records.

for( I Enumerator iterator = data Set.All Data Records.Get Enumerator(); iterator.Move Next(); ) {
{
template Data Record = ( Data Record )iterator.Current;

// Enumerate through the data record, outputting each of the fields.
for( I Dictionary Enumerator en = template Data Record.Get Enumerator(); en.Move Next(); )
{
Console.Write Line( en.Key + “=” + en.Value );
}
Console.Write Line( “============================================================================” );
}

}
// Very important! Be sure to disconnect from the server.
remote Session.Disconnect();
}
catch( Exception e )
{
Console.Write Line( e );
}
}
}

By: streetparade

streetparade — Mon, 17 Aug 2009 19:33:59 +0000

Is there also a function to get the largest text from html body ?

By: Akshay

Akshay — Tue, 04 Aug 2009 19:23:50 +0000

I have created a specific WordPress plugin using phpQuery and cURL to place web scraps inside your wordpress posts, pages or sidebar. Check http://webdlabs.com/projects/wp-web-scraper/ for a demo or download the plugin from http://wordpress.org/extend/plugins/wp-web-scrapper

By: phpfan

phpfan — Sat, 02 May 2009 21:03:23 +0000

is there anyway to get the html source of DOM which is generated from ajax?
{ajax function goes here
document.getElementById(‘div_id’).innerHTML= something;
}

html here

div id=”div_id”(HERE IS WHAT I WANT TO GET)

php goes here
need to do something here to get (HERE IS WHAT I WANT TO GET)….

By: Bruce

Bruce — Fri, 01 May 2009 20:04:59 +0000

Although a little late, hopefully Anar might make it back and see this post. If you’re having problems with being blocked because of the UserAgent, you could use curl if your server has it to pull the HTML. curl offers params that allow you to set the UserAgent, set just about any header you want, pass cookies back and forth, follow redirects, etc.

This class is fantastic. It helped me out instantly on my test and from the first pass, it looks like it’s very easy to use.

By: Brian

Brian — Sat, 18 Apr 2009 04:55:31 +0000

What’s the advantage of using this library over the DOM built into PHP?

By: Everett

Everett — Tue, 07 Apr 2009 13:49:58 +0000

I can read PHP and modify it from working with CMS like Joomla. But I cannot make this run. Can anyone point me to a tutorial on how to get this to run (get installed) on my web host? Thanks! (running PHP 5.X & MySQL)

By: TEST

TEST — Wed, 25 Mar 2009 18:17:35 +0000

Can it be treated like some sort of re-usable component?

Thanks for this great tip

By: TEST

TEST — Wed, 25 Mar 2009 18:16:07 +0000

nice tutorial, Thanks

By: Anar

Anar — Sat, 21 Mar 2009 09:25:11 +0000

Very useful article, but I get problem.
The webpage that I’m trying to parse has something in their script to avoid the parsing. So if I open it via browser (i.e. Firefox) it shows me the page, but when I’m trying to get HTML with PHP script, I get HTML that says something like (“your IP is bla-bla-bla and UserAgent bla-bla-bla blocked”)
Does anybody know how to simulate normal browser request with PHP? How they can detect PHP request?
Thanks in advance,