I received an email recently (an anonymous email!) which questioned the effectiveness of those FOD detection systems which have a detection time greater than the mean time between aircraft movements. The question in the email was this, “Did you neglect the timing requirement [detection time] when you did your system design???[sic]“ The suggestion was that detection time should be the fundamental design consideration when building a FOD detection system. Personally I believe that reducing the risk from FOD should be the key design consideration. The idea that detection time is the key is based on the following argument:

If a system takes 6 minutes to detect an item of FOD, and the next aircraft is due in 4 minutes, then the FOD detection system is completely ineffective at reducing risk.

It’s the sort of argument that people in marketing dream of, not only does it appear to make a lot of sense, but FOD detection time is easy to quantify, and therefore it’s easy to compare across the various systems. Unfortunately, it does not stand up to any form of rigorous analysis. So, let’s take at look at this in more detail, the first thing we need to do is to define the risk from FOD.

What is risk?

It’s often very difficult to quantify risk in any meaningful way, but with FOD it’s actually quite easy. The risk posed by FOD is proportional to the time the FOD object spends on the runway surface (or other airfield surfaces). Of course, the risk from an item of FOD will also depend on exactly where the FOD is found, and what the object is, but, with a large enough data set these two factors should become less important, and then you’re left with the risk from FOD simply being proportional to the time the object spends on the airfield surface.

For example, a single item of FOD on the runway for 1 hour represents the same risk as two items of FOD on the runway for 30 minutes each. It’s even possible to give units, and I’m going to call the units of FOD risk “FOD minutes” (as it’s simply the number of FOD items multiplied by the time on the surface)

An example

Let’s consider a very simple example of a single item of FOD that is on the runway for 3 hrs, and there is no FOD detection system installed. (I’ve assumed the runway is manually checked every 6 hrs, so the mean time FOD will spend on the runway is 3hrs)

• The risk posed by this one item is 1 x 180 mins = 180 FOD minutes

Now let’s add a FOD detection system and see what happens, let’s assume that the FOD detection system has a detection time of 2 minutes.

• The risk posed by this one item is 1 x 2 mins = 2 FOD minutes

But, it gets even better, if there is an aircraft movement every 4 minutes then for any detection time less than 4 min the FOD risk will actually be zero! Unfortunately this hypothetical case is so simple it’s essentially meaningless, this examples makes 3 assumptions:

1. All FOD items originate from the last aircraft that used the runway
2. The risk from FOD is proportional to the detection time
3. The FOD system detects all FOD

Lets deal with each one of these in turn:

1. Assuming that all FOD originates from the previous aircraft is unrealistic, it assumes that there are no other sources of FOD, such as items from ground vehicles, wildlife, litter, broken concrete or tarmac, stones, tools, luggage etc.
2. The risk from FOD is proportional to the time the FOD spends on the surface, this is the detection time plus the retrieval time. The FOD does not magically disappear after it has been detected.
3. No FOD system is 100% effective. Each will have a probability of detection. So there will still be some FOD on the runway which will be found during manual checks.

A better example

Let’s create a more realistic example that includes the probability of detection, retrieval time, and assumes that there are sources of FOD other than the previous aircraft. First we have to generate a hypothetical airport, this airport has the following parameters:

1. 300 Items of FOD are found at the airport each year.
2. It takes, on average, 5 mins to retrieve an item of FOD.
3. There are 6 hrs between manual FOD checks.
4. The average time between aircraft movements is 4 mins.
5. 50% of all FOD found came from the previous aircraft.

Now let us also introduce two FOD detection systems that differ only in their probability of detection and their detection time.

• System A: Detection time = 1 min. Probability of detection = 90%
• System B: Detection time = 7 mins. Probability of detection = 95%

So which system is most effective at reducing the risk due to FOD? Well, the probability of detection differs by just 5%, but the detection time for system B is 600% greater than for system A. Not only is it 600% greater, but it’s also greater than the the mean time between aircraft movements.

So, the answer is obvious, isn’t it? Actually, each system reduces the risk from FOD by almost exactly the same amount, down to around 10% of the original risk when no FOD detection system was installed (System A reduces the risk to 13%, and system B to 11.3%). More information is included in the image below.

But what if both systems had the same probability of detection and only differed in their detection time? In this case, where both systems have a 95% probability of detection, system A has now reduced the risk from FOD to just 8.2%, and system B remains unchanged at 11.3%. So yes, detection time is a factor. But, a massive reduction in detection time results in a very small reduction in risk.

So, why don’t all the FOD detection systems just reduce their detection time to less than the mean time between aircraft movements?

The fact is that you don’t get something for nothing, and this is true for detection systems. And this is where it starts to get interesting. Each of the systems can probably reduce their detection times, if they wished, but, it would come at a cost, and that cost could be a reduction in probability of detection . And we’ve just seen an example where a system that had a smaller detection time (by 600%), had that advantage wiped out by a system that had a better probability of detection, by just 5%!

This is best explained with an example. Consider a detection system (FOD or not, it doesn’t matter) based on a visible camera system. The system makes detections based on information it receives, and that information is in the form of light entering the camera. Less light, less information, reduced probability of detection. So, what happens when it gets dark? One of two things can happen:

1. The camera collects less light, and the probability of detection reduces
2. The camera stares for longer, i.e. it collects light for longer, and the probability of detection remains constant.

There is a FOD detection system on the market that is based on a visible camera system, and the detection time does indeed increase as it gets dark. Why did they choose to follow option 2 above? why not just follow option 1 and allow the probability of detection to decrease. It’s simple, they realise that probability of detection is far more important that detection time.

So, does this mean that systems with a longer detection time are better?

No, I wish it were that simple, but it’s impossible to compare the trade-off between probability of detection and detection time across systems that are based on different technologies. It’s quite possible that a system based on one form of technology will have a better probability of detection, and a shorter detection time.

Does a reduction in detection time always result in a decrease in probability of detection?

No, it’s possible to reduce detection time while keeping probability of detection constant. When you reduce detection time you’re essentially degrading the quality of the data used to determine the detection, you can still maintain the probability of detection by increasing the system’s sensitivity, but again, you don’t get something for nothing. Increasing the sensitivity will lead to an increase in false alarms, and we’re not talking about a few percent. An increase in the false alarm rate is in itself not a problem, if it means maintaining the probability of detection then you’re still reducing the risk on the runway, which of course is the goal. But, if the false alarm rate is so high that the operator starts to ignore the alarms, that’s when the risk on the runway starts to increase again.

So which system suffers from these trade offs between detection time, probability of detection and false alarm rate?

It doesn’t matter if the FOD detection system uses cameras, radar or lidar, the trade-offs still exist. Actually it doesn’t matter if it’s a FOD detection system or not, it’s fundamental to all detection systems. Radar, cameras (visible, infrared), lidar, human beings, sniffer dogs, or squirrels looking for nuts on the forest floor, they are all detection systems and they all suffer these issues. Lets consider the current FOD detection system used in every airport today, the human being, he drives down the runway at a particular speed trying to detect FOD. Let’s reduce his detection time, it’s easy, ask him to drive twice as fast, unfortunately he now has 1/2 as long to look at each object. So what’s the trade-off in this situation? Typically he will find less FOD (the probability of detection will fall), but what if he has been given strict instructions to find the same amount of FOD (i.e. maintain the probability of detection),  well, he can do that, but  the amount of false alarms will rise. It’s a silly example, but it proves the point.

So why do some vendors still insist that their system is better just because they have a shorter detection time?

Unfortunately, it’s our fault. We like comparing numbers, it’s easy, system A has a shorter detection time than system B, therefore I’m going to buy system A. The best analogy I can make is with digital cameras. The number of megapixels a camera has is still used by many people as the factor that most influences which camera they buy. It’s easy to compare this parameter across various cameras. But, much like detection systems, there is a trade off, if the digital camera sensor size remains constant, then as the number of pixels is increased, each pixel has to be made smaller, and unfortunately this tends to increase the noise in the system, and this can result in reduced image quality. And comparing the image quality from different cameras is much harder than comparing the number of megapixels they have. It’s very similar to FOD detection systems, comparing detection time is easy, but comparing the probability of detection, or false alarm rate is not.

To sum up, I’m not saying that detection time is not important, I’m just saying that it’s not as important as it would first appear (and there are far more important parameters to consider). Always remember that the goal of a FOD detection system is to reduce the risk from FOD, it’s not to have the shortest detection time, or the lowest number of false alarms. And to the person (who didn’t  leave their name, position, company, or a usable email address) who asked “Did you neglect the timing requirement[detection time] when you did your system design???[sic]”, I have one question for you, “did you only consider the detection time when you designed/bought your system?” if the answer is Yes, then all I can say to you is “Oops”

I disagree with you, how do I comment? Is an anonymous email the best method?

Ummm….let me think about this….no, an anonymous email is not the preferred method. I’m aware that not everyone is familiar with a blog style website, so here are some basic instructions for anyone who wishes to comment on this article.

At the bottom of this article (and every article on this website) is a section entitled Leave a Reply, this is where comments should be left. It’s public, it allows others to see the comments, which I believe is useful, especially if they have similar comments. If someone has an issue with a particular assumption I’ve made, such as “50% of FOD originates from the previous aircraft”, please remember that this is not supposed to represent an average value, or a value for a particular airfield. If you have some real data for the airport values used above then please use the spreadsheet (link at the end of this article), do the calculation for your particular airport, include your system’s detection time and probability of detection and include your results in the Leave a Reply section.

If you discover an error in the calculation itself then let me know in the Leave a Reply section, I’m human, mistakes happen. The spreadsheet is available for anyone to download, it’s completely open, there are no hidden or protected fields. Please feel free to check the calculations yourself (build on the original calculation if you wish, add other airfield surfaces, change the risk for each surface, add a probability of detection for the manual checks etc). I have nothing to hide, and I’m a fan of transparency. If you are really uncomfortable with your comments being public then feel free to use the Contact Form, but my personal preference is that we keep any discussions public.

The Spreadsheet

Since starting to write this article the spreadsheet has grown and become more involved than was originally planned. It now includes the ability to input data for 3 FOD detection systems, and also allows the user to select which action is taken when FOD is detected, i.e. to close the runway and retrieve, or to continue operations and retrieve. The spreadsheet is provided, as is.

Download the Spreadsheet (FOD risk v2)

This entry was posted on Tuesday, January 25th, 2011 at 11:15 am and is filed under Editorial, Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.