Avalon Airport
Dick Smith's solicitor's letter to CASA dated 25 June 2008 re flawed Avalon Study
By Dick Smith
Jun 25, 2008, 17:00

As you know, we act for Mr Dick Smith.  We refer to earlier correspondence with you raising Mr Smith’s concerns regarding passenger safety at Avalon Airport.  In particular, as you are aware, Mr Smith considers that there is an overwhelming case that Avalon Airport be classified Class D airspace and that the existing air traffic control tower should be manned for all commercial flights.

OAR has now published its Aeronautical Study of Avalon dated May 2008.  Based on that Study OAR has determined that Class C airspace, with associated ATC services, be activated around Avalon Airport at appropriate times to accommodate passenger transport operations.  Mr Smith considers that to be an entirely unsatisfactory outcome which has been arrived at on the basis of a series of fundamental errors which indicate that CASA has not met relevant legal requirements. 

The purpose of this letter is to draw to your attention Mr Smith’s concerns regarding:

(a)        fundamental deficiencies and flaws in OAR’s Study, primarily relating to the methodology used and the failure to adopt and apply the proven FAA methodology;

(b)        CASA’s consequential failure to fulfil its relevant legal obligations in respect of airspace at Avalon; and

(c)        the critical need for OAR to conduct a supplementary study of Avalon which addresses and remedies the defects and omissions in the May 2008 Study.

We should also point out that this letter reflects independent expert technical advice obtained by Mr Smith from independent specialists who, at his request have reviewed the Study.

Background

A limited aeronautical study of Avalon Airport was conducted by AirServices Australia in September 2006.  That Study identified some long term measures for airspace risk control and reduction.  One option was the provision of an air traffic control service from Melbourne Centre but the Study also recommended that ongoing monitoring of Avalon should occur in order to determine if an Aerodrome Control Service should be introduced.

There was widespread concern and dissatisfaction with the adequacy of AirService’s Study.  That culminated in the then Minister for Transport and Regional Services requesting in August or September 2007 that CASA’s Office of Airspace Regulation (“OAR”) undertake an assessment of whether the measures put in place by AirServices were adequate to maintain the appropriate high level of safety at Avalon, especially in light of the continued expansion of passenger and other aircraft traffic at Avalon.

At that time, AirServices provided a Class C air traffic control service on request at Avalon, under an agreement with the airport operator, Linfox Airports Pty Ltd (“Linfox”).  When Class C air space is not activated, Avalon reverted to Class G air space and no air traffic control services were available.

In October 2007, OAR provided interim advice to the then Minister as to various measures which could be implemented to provide an improved level of safety for aircraft operations in and around Avalon Airport.  OAR acknowledged, however, that its interim report had been completed within a timeframe which it was said did not allow it to apply a comprehensive package of oversight measures, including formal cost-benefit and risk analyses.  In those circumstances, CASA undertook that it would complete a full regulatory review of Avalon Airport within the first half of 2008.  It appears that CASA considers that its May Report discharges that undertaking.  For the reasons given below, that view cannot be sustained.

Legal context

Our earlier letter summarised the important and relevant legal provisions concerning CASA’s statutory responsibilities vis-à-vis Avalon.  That legal context need not be repeated in full, but the key points may be summarised as follows :

·                     under section 9A of the Civil Aviation Act 1988 (“the Act”), CASA is required to regard the safety of air navigation as the most important consideration in exercising its powers and performing its functions; and

·                     section 11A obliges CASA to exercise its powers and perform its functions in a manner which is consistent with the Australian Airspace Policy Statement.

The current Policy Statement which CASA is obliged to act consistently with, is the Statement dated 28 June 2007 (“the 2007 Policy Statement”).  Until such time as the 2007 Policy Statement is amended or replaced by the new Federal Government, it remains binding on CASA as a matter of law.

It is also important to note that the 2007 Policy Statement obliges CASA to establish a work program which includes the Government’s priorities for airspace reform to progress NAS implementation as outlined in that Statement.  The Policy Statement identifies various specific steps which must be followed by CASA before it makes any changes to airspace of a major nature.   Those steps, which are designed to ensure international best practice is followed by CASA, include :

·                     carrying out a risk management analysis consistent with the CASA Risk Management System and Risk Management Framework;

·                     assessing the potential costs and benefits of any proposed change;

·                     inclusive consultation with stakeholders; and

·                     ensuring consistency with Government policy, as expressed in this Policy Statement.

The Policy Statement makes clear that, consistently with CASA’s statutory duty under s.9 of the Act, the safety of passenger transport operations is the most important consideration in determining a framework for the declaration, classification and designation of Australian airspace.  The particular requirements affecting aircraft operating in the vicinity of aerodromes in Class G serviced by a significant number of passenger transport aircraft are stipulated.

In the context of the requirement that CASA implement the Common Risk Management Framework, the Policy Statement stipulates in paragraph 5.8 that CASA “must utilise a risk management based assessment methodology to decide the appropriate classification, designation and allocation of Australian administered airspace”.  Moreover, CASA is required to maintain “a risk management system which is premised on the concept of As Low As Reasonably Practicable (“ALARP”) and conforms to AS/NZS4360”. 

The Policy Statement also contains important material bearing upon CASA’s obligation to act transparently and to engage in wide consultation (see paragraph 5.9).

Finally, Part 6 of the 2007 Policy Statement confirms the Government’s commitment to the NAS reform objectives, including the allocation of air traffic management services on the basis of risk.  The NAS has been adopted as the model for reform of the Australian airspace system since 2002, a point which is reinforced in the current Policy Statement.

Significant deficiencies and omissions in OAR’s Aeronautical Study of Avalon

Based on expert technical advice obtained by Mr Smith, it is evident that the May Study is seriously deficient in at least the following respects :

·                     OAR has failed to carry out a risk management analysis consistent with the Risk Management System and Common Risk Management Framework;

·                     the cost-benefit analysis contains so many inadequacies that, on any reasonable assessment, it falls far short of constituting a valid analysis as required by the 2007 Policy Statement;

·                     OAR has failed to implement its obligations concerning transparency and consultation; and

·                     in these and other respects OAR has failed to comply with the obligation to act consistently with the 2007 Policy Statement.

Each of those four matters is now developed.

1.         Common Risk Management Framework and Methodology

It is evident that three different methodologies were employed by OAR in producing the Study, namely :

(a)        generative interviews with stakeholders;

(b)        Airspace Risk Model (“ARM”) and FN-curves developed by CASA; and

(c)        a simple semi-quantitative “Threat Barrier Model” developed by CASA’s external consultants R2A.

CASA’s multifaceted approach involved a sequence of those three methodologies and compared different airspace models.  R2A’s methodology was based on a New Zealand precedent involving a due diligence model based upon input provided by a range of stakeholders.

The multifaceted methodology used for the Study is seriously deficient in the following respects.

First, it is notable that the methodology is substantially different from that which would be used by the FAA in the United States, based upon detailed legislation governing the establishment of air traffic control services.  The FAA has produced a document which provides detailed information on the conduct of cost-benefit analysis, including particularly in respect of the benefit of tower as opposed to non-tower operations (see FAA document dated August 1990 and entitled FAA-APO-90-7).  The FAA’s establishment and discontinuance criteria has been used in Australia for more than a decade and it is troubling that the criteria was abandoned for the purposes of the May 2008 Study and preference was given by OAR to a far less scientific methodology.  CASA’s methodology is essentially based upon unvalidated and unverified subjective beliefs by selected individual stakeholders, as opposed to a methodology such as that of the FAA which relies upon objective historical incident data.  Any model estimating hazards must be both verified and validated if it is to provide a reliable and informed basis for decision-making in an aeronautical context.  One way of achieving this would be to have an expert who is familiar with the FAA methodology apply Avalon data to the FAA model with a view to then making a recommendation on the establishment or otherwise of a Class D Tower at Avalon.  The FAA methodology is strongly to be preferred given its relative simplicity and lesser vulnerability to error when compared with CASA methodology which involves complex estimations that cannot be properly verified.

Secondly, the ARM and FN-curves developed by CASA relates to acceptable risk criteria in relation to the risk of midair conflicts within regional airport terminal area.  The ARM collision risk model developed by CASA in 1996 is built on a non-radar controlled terminal area model, that is, the FAA methodology above which assesses the costs and safety benefits of providing a Class D airspace visual ATC tower service. This core methodology has been statistically validated against the vast United States safety experience in operating thousands of aerodromes with and without ATC tower services over decades. This should form the basis of a cost benefit analysis which is the decision criterion for the FAA and has been for Australia for more than a decade. As a supplemental criterion, CASA developed FN-curves, and used the risk outcome of the ARM as an input to this.

Thirdly, it is extraordinary that the Threat Barrier Model was used by R2A notwithstanding that a similar model was the subject of trenchant criticisms by CASA’s own consultant, Professor Terry O’Neill, who was engaged by CASA in 2004 to review the Study conducted by AirServices Australia on airspace classification.  Professor O’Neill’s criticisms are reflected in the following conclusions in his Report to CASA :

            “1.        Problems with the way in which AA analysis were conducted and the lack of adequate validation raises considerable doubts about the accuracy of the conclusions presented.  The analyses conducted are insufficient to support a determination that the 2b changes have in fact lead to a change in safety or to determine the extent of any such changes;

            2.         Based on the available evidence, this Report makes no judgements about the relative safety of Class E and C airspace; and

            3.         Obtaining more accurate estimates of the nature and size of changes resulting from the 2b changes would not, by themselves, be adequate to determine the most appropriate course of action to take next”.

As you know, Professor O’Neill was particularly critical of AirServices’ methodology because of its lack of validation.  At page 4 of his Report to CASA, Professor O’Neill said :

            “Validation is generally considered an indispensable part of model development and it would unwise to rely on outputs that had not been subjected to extensive testing and confirmation”.

Moreover, Profession O’Neill was highly critical of a methodology which is based on interviews providing the subjective views of stakeholders.  Professor O’Neill commented as follows on such a methodology :

            “These subjective judgements can be very inaccurate because reality can be counterintuitive in actual practice.  If the key input assumptions and relationships are highly subjective and inaccurate, then the modelling results will also be highly subjective and inaccurate”.

Professor O’Neill’s criticisms of AirServices’ model primarily related to the fact that such a methodology is based on interviews and opinions as opposed to known statistical data.  Interviews are notoriously unreliable in determining the likelihood of low frequency events such as aircraft accidents.  Informed and reliable safety decisions should be supported by actual safety data based on real accident rates, not by well meaning people trying to guess the likelihood of very rare events such as aircraft collisions. 

It is notable that the May 2008 Study itself on page 24 acknowledges that the risk figures produced by the Threat Barrier Model are not scientific, yet those figures were used in the cost-benefit analysis in preference to the figures generated by the accepted ARM (as described on pages 19 and 20 of the Study).

The core basis of the ARM is an internationally developed objective approach for assessing the cost-benefits of an ATC tower service and is strongly to be preferred to the less reliable Threat Barrier Model.  As the Study itself reveals on page 20, the ARM demonstrates that a tower is needed to bring the risk to the ALARP range using CASA’s own risk criteria.

In substance, Professor O’Neill’s criticisms, which apply equally to the Threat Barrier Model, relate to its lack of validation and its dependency on unreliable experiences in order to determine values.  That is to be contrasted with the FAA approach, as described above, which is based on real world experience and is tested to statistical data which is reliable and is tested to a 95% plus confidence level. 

It is remarkable that, having itself commissioned Professor O’Neill’s expert views only a few years ago, CASA has now acted upon a methodology which Professor O’Neill so trenchantly criticised.  In particular, the OAR methodology is seriously wanting because:

·                     it is based on hypothetical models which don’t show evidence of external validation or verification;

·                     critical values are selected on the basis of experiential interviews;

·                     the estimates of costs for establishing the various models considered in the Study are at best uncertain;

·                     costs for operating in Class E airspace over a Class D tower seem to be greatly over-estimated (but it is impossible to reach a considered conclusion because relevant data and information has been withheld on the basis of spurious claims of confidentiality);

·                     critical values used in the ARM Model appear to have been “guesstimated” and deviations from the true values may result in the incorrect selection being made against the stated selection criteria of best NPV; and

·                     if the hazard calculations are flawed, as appears to be the case, then necessarily the estimates of costs and NPVs are unreliable.

The May 2008 Study acknowledges in part the inadequacies of the Threat Barrier Model.  Page 24 of the Study records that the Model is unable to estimate the safety difference between Models 1 and 2 (Class E over Class G and Class E to the surface).  Nor can it estimate the safety difference between Models 3 and 4 (Class E over a Class D tower and Class E over Class C tower).  Accordingly, the “Threat Barrier Model” simply clumps the models together according to the overlying airspace.  Such an analysis is inappropriate because a proper analysis would be weighted towards the type of airspace closer to the runway, as opposed to the overlying airspace, because most accidents occur within five nautical miles of an aerodrome.

Moreover, the Model is based on a crude assumption about the frequency of a limited range of “loss of control” or TCAS Resolution Advisory incidents to TCAS-equipped aircraft.  In each of those incidents the pilots responded correctly to the TCAS Resolution Advisory and avoided an accident, yet the Model considers the pilots’ actions to be past the point of “loss of control”.  This does not make sense.  A correct pilot response to a TCAS Resolution Advisory is not a loss of control.  The Model assumes the pilot does not respond to the TCAS which is clearly incorrect because the quoted incidents indicate that they did respond correctly.

Significantly, the Study also states that the crude assumed figure of 1.25 (LOC) incidents from the Threat Barrier Model was the key input to the cost benefit analysis.  Necessarily, therefore the cost benefit analysis is invalid because such an analysis must be based upon the true and total safety benefits provided by an ATC service, including preventing accidents close to the aerodrome and on the ground, rather than an artificial construct of the type used by OAR which grossly overstates the benefit of a class of airspace located far from the aerodrome.

Given the multiple problems with the Threat Barrier Model, it is submitted that such a modelling approach should only be used where no hard data or scientifically-based model exists, which is not the case here.

2.         Inadequate Cost-Benefit Analysis

The Study’s serious shortcomings in this respect may be summarised as follows.

First, in contrast with previous ATC tower service cost and safety studies, OAR has withheld from public scrutiny important information and data which, if available, would enable the cost benefit analysis to be effectively audited.  It is claimed in paragraph 4.2.2 of the Study that detailed data used in the risk modelling process was not made available because it was “commercial-in-confidence”.  It is difficult to see any rational basis for any such claim in respect of data relating to AirServices Australia.  AirServices Australia is a public authority, not a private corporation.  In effect, it is the monopoly provider of ATC services.  It is difficult to see any basis upon which AirServices Australia would have a valid claim as to commercial confidentiality in respect of the relevant material.  Furthermore, it is difficult to justify the withholding of such information in the name of commercial confidence in circumstances where CASA is obliged to give highest priority to public safety.  Such considerations oblige CASA to publish the full detail in the Study of all quantified traffic, load factors and all cost figures, sources and assumptions for public scrutiny, along with all the assumptions and values associated with the variables in its modelling, as has been the case traditionally in earlier cost-benefit studies.

Secondly, the Study fails to identify many unique safety benefits associated with tower ATC as opposed to remote ATC.  For example, tower ATC can in some cases visually detect and advise pilots so as to prevent more accidents due to :

·                     aircraft collisions where one or more aircraft are on the ground;

·                     aircraft collisions with vehicles or other equipment on the runway;

·                     wheels-up landings;

·                     landing not aligned with the runway or undershoots or overshoots;

·                     landings on the wrong runway relative to the wind; and

·                     accidents due to other hazards such as potential bird strikes or potential meteorological hazards.

Thirdly, the Study singularly fails to make good the proposition that a manned ATC tower service is difficult to justify on a cost basis.  It is acknowledged on page 12 of the Study that such a service “would solve many problems”, but OAR rapidly dismisses that option on the apparent basis that it is seen to be difficult to justify on a cost basis.  How that conclusion is reached is difficult to understand in circumstances where all the manned tower options set out on page 26 of the Study show a positive NPV which justifies them on a cost basis.  Similarly, in paragraph 5.1, the Study notes that an active ATC tower service would mitigate the risks to ALARP, yet the Study fails to recommend implementation of such a service.  That is surprising, particularly where there is no evidence presented in the Study that any of the other options, including controlling the surrounding airspace from Melbourne Centre would reduce the risk to ALARP.

Fourthly, the limited figures published in the Study are of little utility in enabling a proper audit of the cost-benefit analysis.  In particular, the cost benefit analysis is deficient in that :

·                     it includes on page 26 AirServices Australia depreciation figures, but a valid cost benefit analysis should be based on a discounted cash flow and cash flow doesn’t include any accounting provision such as depreciation;

·                     no supporting data has been published in respect of the claimed capital and operational expenses set out on page 26 and in Annexure E.  Access to such detailed data is essential to enable public examination and testing of the claimed expenses.  As noted above, given AirServices Australia’s status as a publicly-owned entity which effectively exercises monopoly powers, there is no warrant for withholding that information on the basis of commercial confidentiality;

·                     the cost-benefit analysis estimates NPV over only five years, which biases the result against any option involving early capital expenditure; and

·                     notably, Explanatory Notes 1 to 6 are missing at the bottom of Annexure E, thereby again depriving the interested reader of important information to access the validity of OAR’s analysis.

3.         Lack of transparency and inadequate consultation

Notwithstanding CASA’s obligation to act transparently in accordance with the Government’s Policy Statement, it is evident that vital information has been withheld from the published final Study on the basis of spurious claims of commercial confidentiality in respect of AirServices Australia’s data, as discussed above. 

OAR’s failure to give proper consideration to the viable option of manning the tower for all commercial passenger flights is a further example of its failure to act transparently.  Another example is the stark omission of Explanatory Notes 1 to 6 in Annexure E.

The Policy Statement also makes clear CASA’s duty to consult extensively in the performance of its statutory functions.  Although OAR and R2A consulted with various selected stakeholders, it is evident that many other interested organisations and pilots were not.  In particular, the two major organisations which represent other 12,000 general aviation pilots (the Aircraft Owners and Pilots Associations and Recreational Aviation Australia) were not consulted at all, as Mr Smith has confirmed with both those bodies.

4.         Acting inconsistently with the Government’s 2007 Policy Statement

As is evident from our analysis above, it is evident that CASA has failed in several respects to honour and fulfil its obligation to act consistently with Government policy as expressed in the 2007 Policy Statement.  In particular :

(a)        the current Policy Statement makes abundantly clear that any reclassification of airspace must follow the United States NAS system.  This has not occurred.  Avalon does not have a local radar approach facility and therefore, under NAS, it should be Class D airspace;

(b)        moreover, for reasons given above, OAR has failed to carry out a proper risk management analysis as required by the Policy Statement and the cost-benefit analysis purportedly conducted by R2A has so many inadequacies that it fails to constitute a valid cost-benefit analysis as required by the Policy Statement;

(c)        furthermore, for reasons given above, OAR has failed to implement its obligations concerning transparency and consultation, as required by the 2007 Policy Statement.

Given the number of serious deficiencies and flaws in the May 2008 Study, it provides an insufficient basis for OAR’s determination to maintain in effect the status quo, instead of moving to Class D airspace.

In circumstances where the May 2008 Study provides an unreliable and unsafe basis for OAR’s determination to provide Class C airspace at Avalon Airport:

(a)                CASA is requested to advise as a matter of urgency what steps its proposes to take to deal with the multiple problems and deficiencies created by the May Study;

(b)                in particular, CASA is asked to confirm that it will take appropriate steps to ensure that OAR conducts as a matter of priority a Supplementary Study in relation to Avalon Airport, taking into account the matters raised herein, including the proposal that an expert familiar with the FAA methodology be engaged to apply Avalon data to the FAA model with a view to recommending whether or not a Class D Tower should be operated at Avalon; and

(c)                CASA is asked to confirm whether it is prepared in the interim to direct that airspace around Avalon Airport be reclassified Class D.

In the event that unsatisfactory replies are received in respect of any of these matters, Mr. Smith reserves all rights to commence legal proceedings in relation to the May 2008 Study and OAR’s determination based on that Study.

Yours faithfully 

Mark O’Brien
Partner