April 2009 – Paul Gregg

April 27, 2009February 21, 2013

PHP algorithms: Determining if an IP is within a specific range.

I spend a lot of time lurking in the #PHP channel (efnet and freenode, please – no flamewars) and this topic is a commonly asked one that usually gets a simplified answer in the form of using strpos(), or at best an ip2long() in a greater than and less than answer.

Unfortunately although people usually understand that an IP address is simply an unsigned 32 bit integer, and is easily determined, usually with $_SERVER[‘REMOTE_ADDR’], where the real challenge is – is in specifying the range within which they wish to check that IP address. IP ranges are usually specified in three common ways (in increasing complexity):

Wildcard: 192.168.10.*

Start-End range: 10.1.0.0-10.1.255.255

CIDR*: 172.16.1.0/24

* Classless Inter-Domain Routing

The Wildcard method, or “classy”, allows you to work at Class A (10.*.*.*), Class B (172.16.*.*) or Class C (192.168.10.*) levels of granularity which is how we used to do things in the old days (before the Web decided to make the Internet popular). But, increasingly, this just isn’t granular enough for practical purposes.

Thus was born CIDR (yes, I’m skipping talking about Start-End ranges for now). CIDR brought about the concept that we really didn’t need to break networks on 8, 16, 24 bit boundaries and we could be more granular by allowing the use of any number (from 2-30) to specify a range of networks. Details on why you can’t use “31” is beyond the scope of this article.

CIDR renamed the former Class A, B and C networks as /8, /16 and /24 respectively and reflects the left-most significant bits of the 32-bit IP address. Thus was born the ability to specify very specific IP ranges in the form a.b.c.d/xx. However, part of the problem with this is that although it concisely describes the network start and end, most normal mortal humans couldn’t decipher it. CIDR addressing can also be specified in the form of a longer netmask, e.g. a.b.c.d/255.255.255.224

Thus, the simplified form of Start IP – End IP was put in place for mere mortals and is typically used by those without a networking background. It also features heavily in consumer broadband routers and notably in Microsoft Windows DHCP server.

So having explained how a range, and by inference, that a netmask is, how can we use this knowledge to help us in determining if an IP is within a range?

What this article will attempt to do is guide you though the construction of algorithms to make the checking of IPs simpler.

Logically, Method 1 (the Wildcard), can be easily converted to Method 2 (Start-End range) by using setting Start and End to the Wildcard string and replacing the “*” character with 0 for the Start and 255 for the End, thus for example, “192.168.10.*” becomes “192.168.10.0-192.168.10.255” which should (I hope) be obvious to everyone.

We can then proceed to evaluate both Method1 and Method2 in the same way. In this we’re simply going to use the PHP built in function ip2long() on all 3 values and perform a mathematical check for Start <= IP <= End.

list($lower, $upper) = explode('-', $range, 2);
$lower_dec = ip2long($lower);
$upper_dec = ip2long($upper);
$ip_dec = ip2long($ip);
return ( ($ip_dec>=$lower_dec) && ($ip_dec<=$upper_dec) );

We have, however, a complicating factor here – PHP does not do unsigned integers (32 bit) – which would be necessary for this math to work properly. We can negate this by switing to floating point data types. PHP stores floating types as 64 bit and so will have no problem with IPv4 address space (note – this obviously isn’t granular enough for 128bit IPv6 addressing). Therefore the simplest way to solve the Start <= IP <= End problem with IPs and floating point numers is the following piece of code:

$lower_dec = (float)sprintf("%u",ip2long($lower));
$upper_dec = (float)sprintf("%u",ip2long($upper));
$ip_dec = (float)sprintf("%u",ip2long($ip));
return ( ($ip_dec>=$lower_dec) && ($ip_dec<=$upper_dec) );

Next we have the challenge of handing the CIDR netmasks. What we could do is to take a CIDR format IPaddress/netmask and calculate the Start and End IPs of that block and proceed as before – but that would be no fun – and would mean I haven’t really taught anything through this article.

The method we’re going to use here is how all the world’s Internet routers determine if a destination IP is in a specific CIDR address space. And we’re going to get down and dirty with bitmasks and logical bitwise operators.

So using a real world example, my webserver IP 80.76.201.37 and the netblock within which it resides is 80.76.201.32/27, how does this all work?

Well the /27 indicates that the first 27 bits of the IP address are the same network and IP address in that network (range) will have those same identical first 27 bits. Bits 28-32 are variable and allow 5 bits of variation. If you know your binary, then this means 32 possible IPs. (However with routing, you can’t use the bottom and top IP from any range as these are special and mean the network and broadcast addresses respectively. [This is also why a /31 isn’t much use (except for PPP links) as you can’t use the 2 addresses that space gives you]).

So thinking logically, bitwise, if I take my IP address and the CIDR spec, then all I have to do is check that the first 27 bits all match and I’m good. Correct. So how would we do this in PHP? Sound’s simple, lets just use PHP’s bitwise logical AND operator: &

Again, correct.

In order to do this we need to convert 27 into what 27 really means – a 32 bit number of 27 ones and 5 zeros in binary (which is what 255.255.255.224 really looks like).

In pseudo-code you could then do if (IP & BITMASK) == (RANGE & BITMASK) then all is good and you know that the IP is within the range.

Visualising this using our real IP address (using the very handy unix tool ipcalc):

Address: 80.76.201.37 01010000.01001100.11001001.00100101
Netmask:   255.255.255.224 11111111.11111111.11111111.11100000
Wildcard:  0.0.0.31        00000000.00000000.00000000.00011111
Network:   80.76.201.32/27 01010000.01001100.11001001.00100000
HostMin:   80.76.201.33    01010000.01001100.11001001.00100001
HostMax:   80.76.201.62    01010000.01001100.11001001.00111110
Broadcast: 80.76.201.63    01010000.01001100.11001001.00111111
Hosts/Net: 30

You can see this in the Wildcard line of 0.0.0.31, and the Network ORed with Wildcard results in the Broadcast address: 80.76.201.63.

Knowing this, then the IP address ANDed with the Network address will result in the same value as the Range ANDed with the Network address and so can be used as a comparison for an IP residing within that broadcast range.

How can we work out this Network address in PHP, again we have two strategies, one is to so a simple substr() and take the left most significant bits of the range and then simply pad out to the right with 0s. Or we can do some math with “NOT of 2 to the power of (32-range) – 1”. Thus for our value /27 this gives us the decimal value 31, NOTed results in (65536-31) (representational in the bit form – PHP will see it as a negative integer, but we don’t need to worry about that).

I’m sure by now, your screaming for some code (and if you stuck around this long, you really deserve it).

Code to manipulate a range/netmask into a broadcast address, using math, assuming:

$ip = "80.76.201.37";
$range = "80.76.201.32";
$netmask = 27;

We can convert the IPs to long integers using ip2long (denoted by variable_dec – dec being short for decimal):

$range_dec = ip2long($range);
$ip_dec = ip2long($ip);

This gives us the basis of our math, we now just need to work out the broadcast address.

Strategy 1 using str_pad to create a string by padding with 1s and 0s.

$netmask_dec = bindec( str_pad('', $netmask, '1')
                     . str_pad('', 32-$netmask, '0') );

We can achieve the same result though mathematics by NOTing the wildcard value. This is our Strategy 2:

$wildcard_dec = pow(2, (32-$netmask)) - 1;
$netmask_dec = ~ $wildcard_dec;

Once we know the netmask address (in decimal) as we have here, we can know that, if by ANDing this with the original IP to check results against the Range ANDed with the Netmask, then the IP is within the range defined by the range/mask.

This can be checked easily with:

return (($ip_dec & $netmask_dec) == ($range_dec & $netmask_dec));

I have pulled all of this logic together in a easily included file to provide a single function called ip_in_range($ip, $range) in which $ip is the IP address you want to validate and $range is a any of the above formats, Wildcard, Start-End addressing or CIDR. The function will return a simple TRUE or FALSE if the IP is in that range.

The source code to the all-in function is available here:

http://pgregg.com/projects/php/ip_in_range/ip_in_range.phps

With an example run (and source code):

http://pgregg.com/projects/php/ip_in_range/test.php

I hope this article has been educational, please feel free to leave comments or feedback.
Update: There have been questions about PHP’s signed integers and my use of bit operations in the code. It is important to recognise that when dealing with signed or unsigned 32 bit integers purely as bit patterns for masking with a netmask or broadcast address pattern – the fact that a number (128.0.0.0 or above) really is negative, doesn’t have any impact on the validity of the result. The only impact to not having signed 32 bit integers is in the Start-End range check (example 2 above: 10.1.0.0-10.1.255.255) where a range spanning the switch from positive to negative would be catastrophic to the check. We can safely work around that problem by using floating point numbers as we are only doing <= and >= comparisons and not attempting any bitwise operators (which don’t work on floats).

April 26, 2009

Caught red-handed.

Only two hours after he was supposed to be asleep…

Only light source was the Nintendo DS.

April 23, 2009

tail -# file, in PHP

I read Kevin’s Read Line from File article today and thought I would add some sample code I wrote a while back to address a similar task in PHP – How to perform the equivalent of “tail -# filename” in PHP.

A typical task in some applications such as a shoutbox or a simple log file viewer is how to extract the last ‘n’ lines from a (very) large file – efficiently. There are several approaches one can take to solving the problem (as always in programming there are many ways to skin the proverbial cat) – taking the easy way out and shelling out to run the tail command; or reading the file line by line keeping the last ‘n’ lines in memory with a queue.

However, here I will demonstrate a fairly tuned method of seeking to the end of the file and stepping back to read sufficient lines to the end of the file. If insufficient lines are returned, it incrementally looks back further in the file until it either can look no further, or sufficient lines are returned.

Assumptions need to be made in order to tune the algorithm for the competing challenges of :

Reading enough lines in
as few reads as possible

My approach to this is to provide an approximation to the average size of a line: $linelength
Multiply by ‘n’: $linecount
and we have the byte $offset from the end of the file that we will seek to to begin reading.

A check we need to do right here is that we haven’t offset to before the beginning of the file, and so we have to override $offset to match the file size.

Also as I’m being over-cautious, I’m going to tell it to offset $linecount + 1 – The main reason for this is by seeking to a specific byte location in the file, we would have to be very lucky to land on the first character of a new line – therefore we must perform a fgets() and throw away that result.

Typically, I want it to be able to read ‘n’ lines forward from the offset given, however if that proves insufficient, I’m going to grow the offset by 10% and try again. I also want to make it so that the algorithm is better able to tune itself if we grossly underestimate what $linelength should be. In order to do this, we’re going to track the string length of each line we do get back and adjust the offset accordingly.

In our example, lets try reading the last 10 lines from Apache’s access_log

So let’s look at the code so far, nothing interesting, we’re just prepping for the interesting stuff:

$linecount = 10; // Number of lines we want to read $linelength = 160; // Apache's logs are typically ~200+ chars // I've set this to < 200 to show the dynamic nature of the algorithm // offset correction. $file = '/usr/local/apache2/logs/access_log.demo'; $fsize = filesize($file);

// check if file is smaller than possible max lines
$offset = ($linecount+1) * $linelength;
if ($offset > $fsize) $offset = $fsize;

Next up we’re going to open the file and using our method of seeking to the end of the file, less our offset, here is the meat of our routine:

$fp = fopen($file, 'r'); if ($fp === false) exit;

$lines = array(); // array to store the lines we read.

$readloop = true;
while($readloop) {
// we will finish reading when we have read $linecount lines, or the file
// just doesn’t have $linecount lines

// seek to $offset bytes from the end of the file
fseek($fp, 0 – $offset, SEEK_END);

// discard the first line as it won't be a complete line // unless we're right at the start of the file if ($offset != $fsize) fgets($fp);

// tally of the number of bytes in each line we read
$linesize = 0;

// read from here till the end of the file and remember each line
while($line = fgets($fp)) {
array_push($lines, $line);
$linesize += strlen($line); // total up the char count

// if we’ve been able to get more lines than we need
// lose the first entry in the queue
// Logically we should decrement $linesize too, but if we
// hit the magic number of lines, we are never going to use it
if (count($lines) > $linecount) array_shift($lines);
}

// We have now read all the lines from $offset until the end of the file
if (count($lines) == $linecount) {
// perfect – have enough lines, can exit the loop
$readloop = false;
} elseif ($offset >= $fsize) {
// file is too small – nothing more we can do, we must exit the loop
$readloop = false;
} elseif (count($lines) < $linecount) {
// try again with a bigger offset
$offset = intval($offset * 1.1); // increase offset 10%
// but also work out what the offset could be if based on the lines we saw
$offset2 = intval($linesize/count($lines) * ($linecount+1));
// and if it is larger, then use that one instead (self-tuning)
if ($offset2 > $offset) $offset = $offset2;
// Also remember we can’t seek back past the start of the file
if ($offset > $fsize) $offset = $fsize;
echo ‘Trying with a bigger offset: ‘, $offset, “n”;
// and reset $lines = array(); } }

// Let’s have a look at the lines we read.
print_r($lines);

At first glance it might seem line overkill for the task, however stepping through the code you can see the expected while loop with fgets() to read each line. The only thing we are doing at this stage is shifting the first line of the $lines array if we happen to read too many lines, and also tallying up how many characters we managed to read for each line.

If we exit the while/fgets loop with the correct number of lines, then all is well, we can exit the main retry loop and we have the result in $lines.

Where the code gets interesting is what we do if we don’t achieve the required number of lines. The simple fix is to step back by a further 10% by increasing the offset and trying again, but remember we also counted up the number of bytes we read for each line we did get, so we can very simply obtain an average real-file line size by dividing this with the number of lines in our $lines array. This enables us to override the previous offset value to something larger, if indeed we were wildly off in our estimates.

By adjusting our offset value and letting the loop repeat, the routine will try again and repeat until it succeeds or fails gracefully by the file not having sufficient lines, in which case it’ll return what it could get.

For the complete, working, source code please visit:

http://pgregg.com/projects/php/code/tail-10.phps

Sample execution:

http://pgregg.com/projects/php/code/tail-10.php