PHP – Paul Gregg

December 2, 2010June 2, 2020

Stage 2: http:BL with Apache2 mod_perl

After my earlier post Referrer and Comment spammers are a PITA I came up with two mod_perl plugins to Apache and an “apache level” firewall.

The reason for the apache-level firewall is two-fold. There is no direct way for the Apache user to manipulate an iptables chain (as it doesn’t run as root), and second; I was not happy with suid root access or other forms of message passing to a daemon which would manipulate the firewall for me.

Architecture is thus, in httpd.conf place the following two lines:

PerlPreConnectionHandler PGREGG::httpBLBlock
PerlLogHandler PGREGG::httpBLLog

The first tells apache to run the handler in my httpBLBlock.pm module when a connection is received (before the request has been sent by the client). In this handler, I am simply looking for a filename matching that IP in a directory that is writable by the apache user. The contents of the file are a SCORE:httpBL_answer:[LIST]. Based on this, the module checks the mtime of the filename is in the last SCORE days, then the firewall is in effect. If so, we simply tell apache to drop the connection. If the file has expired, we delete the file.

The second line is more interesting, and what creates the firewall filenames. In order to not impede the general speed of request handling, processing is performed in the Logging section of the Apache process. Our module is called by apache after the response has been sent, but before the access_log entry has been written. In our module we perform the http:BL API call and compute the above SCORE based upon the Threat* level and Age* of the API response. (* both Threat and Age are octets in the DNS lookup). We merely discount the Threat down to zero based on the Age (0-255) where an entry 255 days old reduces the SCORE to zero.
If the SCORE is larger than our trigger level (3) then we create the firewall filename, log the entry in our own httpbl.log and return Apache2::Const::FORBIDDEN. This causes Apache to not log the entry in the normal access_log. Otherwise, if all is ok, we return Apache2::Const::OK and Apache logs the hit as normal.

I have a bit of code tidy up, restructure the config/firewall directory and pull some common code out to a shared module before I can release to the world.

An interesting side effect to publishing the last story out through Planet PHP and other news sources along with the Project Honey Pot image is that when browsers viewed those sources, they all asked for the image off my server. In several cases, these were known spammer, Comment spammer, and other abusers. My server then created the firewall entry blocking them before they were able to follow the links back to my server.

I have been reading up more on Apache Bucket Brigades in an attempt to allow the firewall filter to be placed immediately after the request has been received and allow a custom response to the browser. This may help an otherwise unsuspecting user if their machine had been trojaned. I don’t mind admitting I’m thoroughly confused right now 🙂

November 28, 2010June 2, 2020

Referrer and Comment spammers are a PITA.

This shouldn’t be news to anyone – but Referrer and Comment spammers are a real pain in the a*se. Polluting my web logs and making any meaningful log analysis problematic.

So, I now have an itch to scratch and I’m going to do something about it. I would encourage you, the reader, to do something about it too.

Firstly, get yourself over to Project Honey Pot and read up on the project. If you can, set up a Honey Pot or two yourself. Also be sure to read about the http:BL – this works along similar lines to the DNS blacklists used for Email spammers.

Next, I’m going to write a general Apache mod_perl module which will provide integration (lookup) to the http:BL and allow the user to “action”* the abusers. Minimally, it will prevent the normal apache log files from being polluted by diverting the log entries to a httpbl logfile.

* “action” – To provide flexibility, I’m thinking of running an external script with the IP of the abuser. The script can then perform any action you wish. The one I’m going for is an iptables firewall block.

Comments and suggestions welcome.

Project Honey Pot has implementations for several languages, including PHP and Perl (the languages that mean most to me). There may be an implementation for your Web application so you might not be interested in what I’m doing at all 🙂

Project Honey Pot Logo

April 27, 2009February 21, 2013

PHP algorithms: Determining if an IP is within a specific range.

I spend a lot of time lurking in the #PHP channel (efnet and freenode, please – no flamewars) and this topic is a commonly asked one that usually gets a simplified answer in the form of using strpos(), or at best an ip2long() in a greater than and less than answer.

Unfortunately although people usually understand that an IP address is simply an unsigned 32 bit integer, and is easily determined, usually with $_SERVER[‘REMOTE_ADDR’], where the real challenge is – is in specifying the range within which they wish to check that IP address. IP ranges are usually specified in three common ways (in increasing complexity):

Wildcard: 192.168.10.*

Start-End range: 10.1.0.0-10.1.255.255

CIDR*: 172.16.1.0/24

* Classless Inter-Domain Routing

The Wildcard method, or “classy”, allows you to work at Class A (10.*.*.*), Class B (172.16.*.*) or Class C (192.168.10.*) levels of granularity which is how we used to do things in the old days (before the Web decided to make the Internet popular). But, increasingly, this just isn’t granular enough for practical purposes.

Thus was born CIDR (yes, I’m skipping talking about Start-End ranges for now). CIDR brought about the concept that we really didn’t need to break networks on 8, 16, 24 bit boundaries and we could be more granular by allowing the use of any number (from 2-30) to specify a range of networks. Details on why you can’t use “31” is beyond the scope of this article.

CIDR renamed the former Class A, B and C networks as /8, /16 and /24 respectively and reflects the left-most significant bits of the 32-bit IP address. Thus was born the ability to specify very specific IP ranges in the form a.b.c.d/xx. However, part of the problem with this is that although it concisely describes the network start and end, most normal mortal humans couldn’t decipher it. CIDR addressing can also be specified in the form of a longer netmask, e.g. a.b.c.d/255.255.255.224

Thus, the simplified form of Start IP – End IP was put in place for mere mortals and is typically used by those without a networking background. It also features heavily in consumer broadband routers and notably in Microsoft Windows DHCP server.

So having explained how a range, and by inference, that a netmask is, how can we use this knowledge to help us in determining if an IP is within a range?

What this article will attempt to do is guide you though the construction of algorithms to make the checking of IPs simpler.

Logically, Method 1 (the Wildcard), can be easily converted to Method 2 (Start-End range) by using setting Start and End to the Wildcard string and replacing the “*” character with 0 for the Start and 255 for the End, thus for example, “192.168.10.*” becomes “192.168.10.0-192.168.10.255” which should (I hope) be obvious to everyone.

We can then proceed to evaluate both Method1 and Method2 in the same way. In this we’re simply going to use the PHP built in function ip2long() on all 3 values and perform a mathematical check for Start <= IP <= End.

list($lower, $upper) = explode('-', $range, 2);
$lower_dec = ip2long($lower);
$upper_dec = ip2long($upper);
$ip_dec = ip2long($ip);
return ( ($ip_dec>=$lower_dec) && ($ip_dec<=$upper_dec) );

We have, however, a complicating factor here – PHP does not do unsigned integers (32 bit) – which would be necessary for this math to work properly. We can negate this by switing to floating point data types. PHP stores floating types as 64 bit and so will have no problem with IPv4 address space (note – this obviously isn’t granular enough for 128bit IPv6 addressing). Therefore the simplest way to solve the Start <= IP <= End problem with IPs and floating point numers is the following piece of code:

$lower_dec = (float)sprintf("%u",ip2long($lower));
$upper_dec = (float)sprintf("%u",ip2long($upper));
$ip_dec = (float)sprintf("%u",ip2long($ip));
return ( ($ip_dec>=$lower_dec) && ($ip_dec<=$upper_dec) );

Next we have the challenge of handing the CIDR netmasks. What we could do is to take a CIDR format IPaddress/netmask and calculate the Start and End IPs of that block and proceed as before – but that would be no fun – and would mean I haven’t really taught anything through this article.

The method we’re going to use here is how all the world’s Internet routers determine if a destination IP is in a specific CIDR address space. And we’re going to get down and dirty with bitmasks and logical bitwise operators.

So using a real world example, my webserver IP 80.76.201.37 and the netblock within which it resides is 80.76.201.32/27, how does this all work?

Well the /27 indicates that the first 27 bits of the IP address are the same network and IP address in that network (range) will have those same identical first 27 bits. Bits 28-32 are variable and allow 5 bits of variation. If you know your binary, then this means 32 possible IPs. (However with routing, you can’t use the bottom and top IP from any range as these are special and mean the network and broadcast addresses respectively. [This is also why a /31 isn’t much use (except for PPP links) as you can’t use the 2 addresses that space gives you]).

So thinking logically, bitwise, if I take my IP address and the CIDR spec, then all I have to do is check that the first 27 bits all match and I’m good. Correct. So how would we do this in PHP? Sound’s simple, lets just use PHP’s bitwise logical AND operator: &

Again, correct.

In order to do this we need to convert 27 into what 27 really means – a 32 bit number of 27 ones and 5 zeros in binary (which is what 255.255.255.224 really looks like).

In pseudo-code you could then do if (IP & BITMASK) == (RANGE & BITMASK) then all is good and you know that the IP is within the range.

Visualising this using our real IP address (using the very handy unix tool ipcalc):

Address: 80.76.201.37 01010000.01001100.11001001.00100101
Netmask:   255.255.255.224 11111111.11111111.11111111.11100000
Wildcard:  0.0.0.31        00000000.00000000.00000000.00011111
Network:   80.76.201.32/27 01010000.01001100.11001001.00100000
HostMin:   80.76.201.33    01010000.01001100.11001001.00100001
HostMax:   80.76.201.62    01010000.01001100.11001001.00111110
Broadcast: 80.76.201.63    01010000.01001100.11001001.00111111
Hosts/Net: 30

You can see this in the Wildcard line of 0.0.0.31, and the Network ORed with Wildcard results in the Broadcast address: 80.76.201.63.

Knowing this, then the IP address ANDed with the Network address will result in the same value as the Range ANDed with the Network address and so can be used as a comparison for an IP residing within that broadcast range.

How can we work out this Network address in PHP, again we have two strategies, one is to so a simple substr() and take the left most significant bits of the range and then simply pad out to the right with 0s. Or we can do some math with “NOT of 2 to the power of (32-range) – 1”. Thus for our value /27 this gives us the decimal value 31, NOTed results in (65536-31) (representational in the bit form – PHP will see it as a negative integer, but we don’t need to worry about that).

I’m sure by now, your screaming for some code (and if you stuck around this long, you really deserve it).

Code to manipulate a range/netmask into a broadcast address, using math, assuming:

$ip = "80.76.201.37";
$range = "80.76.201.32";
$netmask = 27;

We can convert the IPs to long integers using ip2long (denoted by variable_dec – dec being short for decimal):

$range_dec = ip2long($range);
$ip_dec = ip2long($ip);

This gives us the basis of our math, we now just need to work out the broadcast address.

Strategy 1 using str_pad to create a string by padding with 1s and 0s.

$netmask_dec = bindec( str_pad('', $netmask, '1')
                     . str_pad('', 32-$netmask, '0') );

We can achieve the same result though mathematics by NOTing the wildcard value. This is our Strategy 2:

$wildcard_dec = pow(2, (32-$netmask)) - 1;
$netmask_dec = ~ $wildcard_dec;

Once we know the netmask address (in decimal) as we have here, we can know that, if by ANDing this with the original IP to check results against the Range ANDed with the Netmask, then the IP is within the range defined by the range/mask.

This can be checked easily with:

return (($ip_dec & $netmask_dec) == ($range_dec & $netmask_dec));

I have pulled all of this logic together in a easily included file to provide a single function called ip_in_range($ip, $range) in which $ip is the IP address you want to validate and $range is a any of the above formats, Wildcard, Start-End addressing or CIDR. The function will return a simple TRUE or FALSE if the IP is in that range.

The source code to the all-in function is available here:

http://pgregg.com/projects/php/ip_in_range/ip_in_range.phps

With an example run (and source code):

http://pgregg.com/projects/php/ip_in_range/test.php

I hope this article has been educational, please feel free to leave comments or feedback.
Update: There have been questions about PHP’s signed integers and my use of bit operations in the code. It is important to recognise that when dealing with signed or unsigned 32 bit integers purely as bit patterns for masking with a netmask or broadcast address pattern – the fact that a number (128.0.0.0 or above) really is negative, doesn’t have any impact on the validity of the result. The only impact to not having signed 32 bit integers is in the Start-End range check (example 2 above: 10.1.0.0-10.1.255.255) where a range spanning the switch from positive to negative would be catastrophic to the check. We can safely work around that problem by using floating point numbers as we are only doing <= and >= comparisons and not attempting any bitwise operators (which don’t work on floats).

April 23, 2009

tail -# file, in PHP

I read Kevin’s Read Line from File article today and thought I would add some sample code I wrote a while back to address a similar task in PHP – How to perform the equivalent of “tail -# filename” in PHP.

A typical task in some applications such as a shoutbox or a simple log file viewer is how to extract the last ‘n’ lines from a (very) large file – efficiently. There are several approaches one can take to solving the problem (as always in programming there are many ways to skin the proverbial cat) – taking the easy way out and shelling out to run the tail command; or reading the file line by line keeping the last ‘n’ lines in memory with a queue.

However, here I will demonstrate a fairly tuned method of seeking to the end of the file and stepping back to read sufficient lines to the end of the file. If insufficient lines are returned, it incrementally looks back further in the file until it either can look no further, or sufficient lines are returned.

Assumptions need to be made in order to tune the algorithm for the competing challenges of :

Reading enough lines in
as few reads as possible

My approach to this is to provide an approximation to the average size of a line: $linelength
Multiply by ‘n’: $linecount
and we have the byte $offset from the end of the file that we will seek to to begin reading.

A check we need to do right here is that we haven’t offset to before the beginning of the file, and so we have to override $offset to match the file size.

Also as I’m being over-cautious, I’m going to tell it to offset $linecount + 1 – The main reason for this is by seeking to a specific byte location in the file, we would have to be very lucky to land on the first character of a new line – therefore we must perform a fgets() and throw away that result.

Typically, I want it to be able to read ‘n’ lines forward from the offset given, however if that proves insufficient, I’m going to grow the offset by 10% and try again. I also want to make it so that the algorithm is better able to tune itself if we grossly underestimate what $linelength should be. In order to do this, we’re going to track the string length of each line we do get back and adjust the offset accordingly.

In our example, lets try reading the last 10 lines from Apache’s access_log

So let’s look at the code so far, nothing interesting, we’re just prepping for the interesting stuff:

$linecount = 10; // Number of lines we want to read $linelength = 160; // Apache's logs are typically ~200+ chars // I've set this to < 200 to show the dynamic nature of the algorithm // offset correction. $file = '/usr/local/apache2/logs/access_log.demo'; $fsize = filesize($file);

// check if file is smaller than possible max lines
$offset = ($linecount+1) * $linelength;
if ($offset > $fsize) $offset = $fsize;

Next up we’re going to open the file and using our method of seeking to the end of the file, less our offset, here is the meat of our routine:

$fp = fopen($file, 'r'); if ($fp === false) exit;

$lines = array(); // array to store the lines we read.

$readloop = true;
while($readloop) {
// we will finish reading when we have read $linecount lines, or the file
// just doesn’t have $linecount lines

// seek to $offset bytes from the end of the file
fseek($fp, 0 – $offset, SEEK_END);

// discard the first line as it won't be a complete line // unless we're right at the start of the file if ($offset != $fsize) fgets($fp);

// tally of the number of bytes in each line we read
$linesize = 0;

// read from here till the end of the file and remember each line
while($line = fgets($fp)) {
array_push($lines, $line);
$linesize += strlen($line); // total up the char count

// if we’ve been able to get more lines than we need
// lose the first entry in the queue
// Logically we should decrement $linesize too, but if we
// hit the magic number of lines, we are never going to use it
if (count($lines) > $linecount) array_shift($lines);
}

// We have now read all the lines from $offset until the end of the file
if (count($lines) == $linecount) {
// perfect – have enough lines, can exit the loop
$readloop = false;
} elseif ($offset >= $fsize) {
// file is too small – nothing more we can do, we must exit the loop
$readloop = false;
} elseif (count($lines) < $linecount) {
// try again with a bigger offset
$offset = intval($offset * 1.1); // increase offset 10%
// but also work out what the offset could be if based on the lines we saw
$offset2 = intval($linesize/count($lines) * ($linecount+1));
// and if it is larger, then use that one instead (self-tuning)
if ($offset2 > $offset) $offset = $offset2;
// Also remember we can’t seek back past the start of the file
if ($offset > $fsize) $offset = $fsize;
echo ‘Trying with a bigger offset: ‘, $offset, “n”;
// and reset $lines = array(); } }

// Let’s have a look at the lines we read.
print_r($lines);

At first glance it might seem line overkill for the task, however stepping through the code you can see the expected while loop with fgets() to read each line. The only thing we are doing at this stage is shifting the first line of the $lines array if we happen to read too many lines, and also tallying up how many characters we managed to read for each line.

If we exit the while/fgets loop with the correct number of lines, then all is well, we can exit the main retry loop and we have the result in $lines.

Where the code gets interesting is what we do if we don’t achieve the required number of lines. The simple fix is to step back by a further 10% by increasing the offset and trying again, but remember we also counted up the number of bytes we read for each line we did get, so we can very simply obtain an average real-file line size by dividing this with the number of lines in our $lines array. This enables us to override the previous offset value to something larger, if indeed we were wildly off in our estimates.

By adjusting our offset value and letting the loop repeat, the routine will try again and repeat until it succeeds or fails gracefully by the file not having sufficient lines, in which case it’ll return what it could get.

For the complete, working, source code please visit:

http://pgregg.com/projects/php/code/tail-10.phps

Sample execution:

http://pgregg.com/projects/php/code/tail-10.php

March 19, 2009

TinyURL PHP “flaw” ?

The Register is running a story today TinyURL, your configs are showing which points out that TinyURL has a /php.php page displaying the contents of phpinfo().

The article then goes on to make some scary sounding claims from security consultant Rafal Los “Why would you want to run a web service as ‘Administrator’ because if
I figure out a way to jack that service, I completely, 100% own that
machine.” and “More importantly… why is this server running as ROOT:WHEEL?!“

Sorry Rafal – but you appear to have no idea how web servers work, or all that much about (web) security.

All unix based webservers start as root if they want to bind to the restricted (and default) port 80, after which they switch to the configured UID for request handling. So, right there, goes all Rafal’s claims about pwning the machine.

Check your own server, the _SERVER and _ENV values will reflect the
starting shell/environment, which just happens to be root. In
other words, there is nothing wrong with the settings. Having said that, they do have register_globals turned on, which isn’t ideal – but it isn’t a gaping hole if the underlying php code is safely coded.

Also to TinyURL’s credit, they are running Suhosin patch to harden their server. They’re also running the latest production PHP (which is more than I can say). Granted, they probably don’t want to be exposing phpinfo() – but this all just an overblown storm in a teacup.

February 3, 2009

PHP on LinkedIn.com

Since LinkedIn opened up its Groups system, there has been a huge growth in the number of groups related to PHP. Some with charters, some without; some with a specific community background and others with a specific regional focus. I am posting this to bring attention to some of them.

In order of popularity (member count) some general groups (non-regional) are:

Some of these are useful if you are looking for a job (the recruiters tend to play nice and stay on-topic), others ban job posts and stick to discussions.
There are literally hundreds of groups related to PHP in some shape or fashion – pure PHP, LAMP, PHP&Mysql, Frameworks, and many regional *PUG type groups.

August 7, 2008

To silent fanfare, Microsoft released SQL Server 2005 Driver for PHP

On July 24, Microsoft released version 1.0 of their native SQL Server 2005 Driver for PHP.

http://www.microsoft.com/downloads/details.aspx?FamilyId=61 … 597C21A2E2A&displaylang=en

Some months back I downloaded a beta version of this after having problems working with international characters (UTF-8) with PDO and MSSQL and impressively the SQL Server 2005 Driver for PHP worked very well.

Congratulations to Microsoft for continuing with this and their recent contribution to ADODB. I’m looking for better PDO support now

December 19, 2007

Want to try out the next major version of Delphi for PHP ?

CodeGear is looking for additional beta testers for the next major version of Delphi for PHP (http://www.codegear.com/products/delphi/php).
Interested PHP developers can apply to become a field tester at http://www.surveymonkey.com/s.aspx?sm=AsT7NAxv4zC5WYKIaK9pRw_3d_3d.

September 28, 2007

FastCGI for IIS (PHP related)

I haven’t seen this announced on the PHP blogs (planets) yet and since it may be of interest to those running PHP (with IIS as a CGI) I’ll repost the details here.

Joe Stagner’s Blog (Microsoft employee) has posted that the IIS team have released the FastCGI extension for IIS 5.1/6.0 as a free download from http://www.iis.net/.

Rather than reword it all, here are the salient bits in Joe’s words:

HUGE KUDOS to the IIS team for their hard work and innovation (technical and political) for making FastCGI a reality.)

If your a developer that needs to use a CGI based platform (Like PHP) and work on Windows � then this is a godsend.

They guys went into over-drive to get this ready before the upcoming Zend-Con.

Here are the official particulars.

Since early 2006, Microsoft and Zend have been working together on a technical collaboration with the PHP community to significantly enhance the reliability and performance of PHP on Windows Server 2003 and Windows Server 2008. As part of this collaboration, the IIS product group has been working on a new component for IIS6 and IIS7 called FastCGI Extension which will enable IIS to much more effectively host PHP applications.

May 4, 2007

Delphi for PHP (aka D4PHP)

Yesterday I installed Delphi for PHP (D4PHP), a new development environment from CodeGear, a Borland company.

D4PHP is CodeGear’s first attempt at bringing visual rapid application development to PHP. To be honest my first impression was fear. Here I am, 10+ years into PHP and used to developing raw PHP in vi and Zend Studio, with a couple of forays into Eclipse, staring at a PHP RAD tool without the first clue where to begin.

At first glance, it looks like a familiar IDE environment and my first thought was to File / Open, open a .php and use it as an editor. But no, if I’m going to use it I should use it the way it was meant to be.

"Hello world" was pretty straight forward, but going beyond that left me pretty bewildered, not sure what was going on underneath, nor how it was supposed to plug together. VCLs (Visual Component Library) exist to "simplify" the building of applications, so I thought I would follow the build-a-component guide and turn my preg_find recursive directory iterator/filter/sorter into a VCL. I think I succeeded (it "compiles" ok) but I don’t know how to test it!

Make no mistake – D4PHP is a mindshift for traditional PHP developers like myself.

Step back from the POST handlers and sequential logic we are used to and prepare to look at application components that seem to have a life of their own. I have to confess that being a simple procedural kind of guy that thinks OO is mostly hype, D4PHP is a huge concept for me to grapple with. But I think I might just enjoy it.