Google blocking certain user-agents 
Monday, December 3, 2007, 12:17 PM - Tech
If you're trying to write a tool that retrieves information from Google, you want to make sure to explicitly set the UserAgent string to something like what a regular browser would set.

For example, perl's LWP::UserAgent sets the agent string per default to "libwww-perl/#.#". Google apparently doesn't like that and will not return results to you. Setting it to something like "Mozilla/5.0 (X11; U; NetBSD i386; en-US; rv:1.8.1.3)" would work.

Interestingly, they appear to whitelist agents, rather than blacklisting them. That is, a random string as a user agent will not work. This means that they are effectively saying "we will only serve content to agents that we know (and approve) of", banning any client they simply might not know of.

One would assume that they did put this whitelist into place to prevent certain abuses, but then again, it would be trivial for anybody developing tools that lend themselves to abuse to simply reset the agent string to something they would accept, so the win seems negligible to me, while the cost of banning unknown agents seems high.
add comment   |  0 trackbacks   |  permalink   |   ( 3 / 492 )

A Surfer's Log 
Sunday, November 25, 2007, 10:11 PM - Life

Date: 2007-11-25
Where: Pacifica/Linda Mar
Time: 11:15am - 12:45pm
Weather: Cloudy
Wind: Some winds from N.
Crowds: Emptyish (for Pacifica)
Tide: High tide just turned to outgoing tide. Some current.
Waves: Choppy, but good. 3-5', mostly.

Got a 5', some 3'. Ate a 5'. Water was quite a bit colder than the last time.

add comment   |  0 trackbacks   |  permalink   |   ( 3 / 458 )

A Surfer's Log 
Saturday, November 24, 2007, 01:20 AM - Life

Date: 2007-11-23
Where: Pacifica/Linda Mar
Time: 11:30am - 1:00pm
Weather: Sunny
Wind: Some winds from N.
Crowds: Empty (for Pacifica)
Tide: Low low tide, starting to change to incoming tide
Waves: Choppy, but good. 3-5', mostly.

Water's starting to get... "brisk".

add comment   |  0 trackbacks   |  permalink   |   ( 3 / 427 )

A Surfer's Log 
Saturday, November 17, 2007, 12:40 PM - Life

Date: 2007-11-11 -- 2007-11-17
Where: Canoe's Waikiki
Time: mornings, afternoons
Weather: Sunny
Wind: None
Crowds: Crowded.
Waves: 1-3' mostly. Occasional 4'. Slow.

Rented an 8'2" from Surf Bruddahs, which apparently still was too short.
Would have needed at least a 9' to get more out of it.

The water's nice and warm, though. I forgot how nice it
is not to have to wear a wetsuit and booties.


add comment   |  0 trackbacks   |  permalink   |   ( 3 / 420 )

A Surfer's Log 
Sunday, October 28, 2007, 12:37 PM - Life

Date: 2007-10-28
Where: Pacifica/Linda Mar
Time: 9:30am - 11:00am
Weather: Sunny
Wind: none
Crowds: Emptyish (for Pacifica)
Tide: Low tide, starting to change to incoming tide
Waves: Flat. Hardly anything worth paddling into.
A few 2-3'.

add comment   |  0 trackbacks   |  permalink   |   ( 3 / 565 )


Back Next