Regulator Performance Framework self assessment report 2019-20

We undertook our fifth annual self-assessment of its performance against the Regulator Performance Framework in August 2020
21 December 2020

Executive summary

Self-assessment

The Australian Maritime Safety Authority (AMSA) undertook its fifth annual self-assessment of its performance against the Regulator Performance Framework (RPF) in August 2020.

The self-assessment was informed by:

  • the draft 2019–20 AMSA annual performance statements
  • the annual results of AMSA’s RPF customer survey; and
  • the professional knowledge and experience of AMSA’s Executive team.

The 2019–20 self-assessment results broadly indicate that AMSA management:

  • believes it has made tangible progress over the past year on the opportunities for improvement identified from previous assessments (2016 to 2018).
  • is aware that there is room for improvement across the range of RPF KPIs, particularly in relation to the domestic commercial vessel (DCV) industry.

The sheer volume of AMSA-customer interactions (174,937: including phone, online and face-to-face interactions), as demonstrated by consistently high call numbers to our call centre AMSA Connect, provides AMSA with significant opportunity to positively impact our industry.

Just over 89 percent (89.3%) of customers were satisfied with the service delivered by AMSA Connect (see measure R.1.1: National System Customer Service), well above the call centre industry average of 70%, but not quite reaching our stretch target of 90%.

AMSA continues to be fortunate that the daily interactions of its staff with stakeholders and its long established reputation positively contributes, arguably contributing a ‘buffer’ of goodwill. However, AMSA management is very aware that such goodwill will not continue indefinitely in the absence of continued effort by management to improve our regulatory performance.

1.1 Self-assessment validation

In previous years, the AMSA self-assessment has been validated by the AMSA Advisory Committee (AAC) – an Australian peak maritime representative body. Due to the disbandment of the AAC in 2020, we were unable to secure a validation body in time for this year’s survey.

Background

1.2 Purpose

Commencing in July 2015, the purpose of the Regulator Performance Framework (RPF) is to encourage regulators to undertake their functions with the minimum impact necessary to achieve regulatory objectives, and to effect positive ongoing and lasting cultural change.

The RPF consists of six outcomes-based key performance indicators (KPIs) which set the Government’s overarching expectations of performance:

  1. regulators do not unnecessarily impede the efficient operation of regulated entities
  2. communication with regulated entities is clear, targeted and effective
  3. actions undertaken by regulators are proportionate to the risk being managed
  4. compliance and monitoring approaches are streamlined and coordinated
  5. regulators are open and transparent in their dealings with regulated entities; and
  6. regulators actively contribute to the continuous improvement of regulatory frameworks.

More information on the RPF is available on the Department of Prime Minister and Cabinet website.

1.3 Requirement

Regulators must self-assess their performance against the RPF annually. The results of the self-assessment must be:

  • validated by an approved external stakeholder body—see comments above regarding validation
  • certified by AMSA’s accountable authority—the AMSA Board; and
  • provided to AMSA’s portfolio Minister and published no later than 31 December each year.

Method

1.4 Evidence

Where possible AMSA leveraged existing processes for data collection and analysis. The primary sources of evidence used for the self-assessment and validation were:

  • the draft 2019–20 AMSA annual performance statements—available as part of AMSA’s 2019–20 annual report.
  • the annual results of AMSA’s RPF customer online and phone surveys

AMSA self-assessment

Over the period 27 July – 7 August 2020 the AMSA Executive completed a self-assessment survey of AMSA’s performance against the RPF. In addition to exercising their professional judgement, the self-assessment was informed by documents listed at section 1.4.

The self-assessment survey consisted of six key statements/questions aligned to the RPF key performance indicators:

  • Q1: AMSA helps vessel owners and seafarers safely operate or work on a vessel without getting in the way
  • Q2: Our communications with those we regulate are clear and useful
  • Q3: Given the risks involved in the industries AMSA regulates, the level of regulation is about right
  • Q4: AMSA’s compliance and monitoring arrangements are well organised and efficient
  • Q5: AMSA explains its regulatory decisions well, and
  • Q6: AMSA is always trying to improve maritime regulations to create a safer and more efficient industry.

The response options were:

1234567
strongly disagreedisagreesomewhat disagreesomewhat agreeagreestrongly agreen/a

This year’s survey also included questions on AMSA’s progress against the opportunities for improvement identified in 2016–17 and 2017–18 self-assessments:

  • 2016–17: how much progress has been made towards:
    • improving engagement and communication with a wide and diverse group of stakeholders – notably the DCV industry?
  • 2017–18: how much progress has been made towards:
    • looking for more efficient ways of doing business with the DCV industry?
    • providing standardised technical/legislative advice to the DCV industry, and minimising the influence of individual interpretation by regulatory officers?

The response options for these questions were:

  • no or little progress
  • some progress; and
  • significant progress.

If the response was ‘no or little progress’, respondents were asked to explain. If the response was ‘some’ or ‘significant’ progress, respondents were asked for examples.

Respondents were also asked to identify three opportunities for AMSA’s delivery of regulation in the future (see 1.8 for results).

Results

1.5 Self-assessment against RPF KPIs

The summary results of the 2019–20 self-assessment against each RPF KPI question are shown in Figure One (solid purple bars), including the results of the RPF customer survey (patterned purple bars). For comparative purposes Figure One also shows the:

  • 2016–17 (solid blue bars), 2017–18 (solid orange bars) and 2018–19 (solid green bars) self-assessment results
  • 2016–17, 2017–18, and 2018-19 annual results of the RPF customer survey (patterned blue, orange and green bars respectively).

Question 1: helps safe operation or work on vessels without getting in the way

Question 2: communications are clear and useful

Question 3: level of regulation is about right

Question 4: compliance and monitoring arrangements are well organised and efficient

Question 5: explains regulatory decisions well

Question 6: trying to improve maritime regulation

The results and comments for each self-assessment KPI question are at Attachment 1.

1.6 Progress against 2016–17 opportunities.

Progress has been made. Table 2 details the evidence provided by the AMSA Executive in support of their assessment.

Table 2: Progress assessment – evidence

Improving engagement and communication with a wide and diverse group of stakeholders—notably the DCV industry
Response: (n=3) some progress and (n=1) significant progress
“More targeted communication and engagement efforts (via industry representative committees and select education / info programs) has been achieved, however segmentation of the regulated community has not been substantially matured during this period and this will bring further improvements to AMSA's effectiveness in these areas”.
“Increased regular communication from AMSA through various channels. New and more structured regional safety committees. Various SMS workshops. Better information available on website”.
“We are listening to industry concerns in Queensland where we have issued a number of exemptions for safety equipment carriage requirements which being extended across a number of jurisdictions”.
“Our digital systems are still not contemporary. Improving our accessibility to industry would be incredibly helpful. Online chat and an App to provide greater mobility in services to our DCV community”.

1.7 Progress against 2017–18 opportunities

Progress has been made. Table 3 details the evidence provided by the AMSA Executive in support of their assessment.

Table 3: Progress assessment—evidence

Looking for more efficient ways of doing business with the DCV industryProviding standardised technical/legislative advice to the DCV industry, and minimising the influence of individual interpretation by regulatory officers
Response: (n=4) some progressResponse: (n=3) some progress and (n=1) significant progress
“Some improvements in transactional systems have been achieved ensuring key regulatory measures are in place. Access to customer assistance/advice via the telephone and via in-fields experts has been well developed”.“Training and collaboration of staff, sharing and access to information internally”.
“Refining of exemptions for less, or alternate regulatory touch. As of right arrangements rather than an application and fees”.“Although not close to that area, it does appear we continue to make progress”.
“We need to improve the data and resolve some of the legislative issues which hamstring our efforts. As indicated earlier, appetite plays a role in this as well as grandfathering under the National System. Rock and a hard place!”.“No comment”.

1.8 2019–20 Opportunities

Respondents were asked to consider how AMSA might deliver regulation in the future. The opportunities identified were:

  • Amendments to the National Law
  • Resolve some of the legislative impediments (noting the challenges)
  • Digitisation of services
  • Review administrative processes
  • Promote stronger engagement from industry
  • Continue to focus on customer satisfaction and engagement

Analysis

1.9 Self-assessment against RPF KPIs

Overall, there has been a slight increase since the 2018–19 self-assessment, with an average 4.96 ‘agree’ (4.43 in 2018–19).

There have been small movements across all the individual RPF KPIs, but given the small sample size of the AMSA Executive (n=4), these movements are immaterial.

1.10 Self-assessment against RPF customer survey

There has been a slight decline in the RPF customer survey results. It should be noted, however, that the number of survey respondents was much lower in 2019–20 than previous years. 

The most significant gap is for the KPI ‘AMSA explains its regulatory decisions well’ with the Executive scoring this question at 5.25 and the stakeholder participants scoring this question at 3.9.

Stakeholder respondents scored question 4: ‘Compliance and monitoring arrangements are well organised and efficient’ the lowest, only scoring 3.4, however this is a slight improvement from the 2018–19 results (3.22).

1.11 Impact of domestic commercial vessel participants

While the sample size is small, the RPF customer survey results provide a useful insight. The survey is available both online and as an opt-in phone survey for those who contact AMSA Connect. It is separate to AMSA’s complaints handling process, however, where specific issues are raised the feedback is passed to the operational area to address.

The DCV industry accounted for almost half of the RPF customer survey participants (note: participants who responded via the telephone survey were not provided the opportunity to select their industry sector. As these were all directed via a contact with AMSA Connect, it is assumed that these respondents were from the DCV industry).

The survey results indicate that DCV, as a single nationally regulated industry group, continues to be ambivalent towards AMSA. The DCV industry is now AMSA’s largest stakeholder group, and will therefore play a central role in gauging AMSA’s long-term success.

Those who identified as being from the DCV industry demonstrated the lowest levels of satisfaction with AMSA’s performance.

Conclusion

The 2019–20 self-assessment results broadly indicate that AMSA management:

  • believes it has made tangible progress over the past year
  • is very aware that there is room for improvement across the range of RPF KPIs, particularly in relation to the DCV industry.

AMSA continues to be fortunate that the daily interactions of its staff with stakeholders and its strong broader reputation contributes to a positive overall perception and a buffer of goodwill. However, AMSA management is very aware that ongoing goodwill is not guaranteed.

Attachment 1: AMSA 2019-20 RPF self-assessment detailed responses

#RPF indicatorAMSA survey questionAdditional comments
RPF KPI 1Regulators do not unnecessarily impede the efficient operation of regulated entities.AMSA helps vessel owners and seafarers safely operate or work on a vessel without getting in the way.“Emphasis is given to effective safety regulation but with strong attention to not unnecessarily ‘encumbering’ the regulated entity.”
RPF KPI 2Communication with regulated entities is clear, targeted and effective.Our communications with those we regulate are clear and useful.“Focus is given to clarity and relevant of information when developing communication material for our regulated community.” “The breadth of operators and nature of operations means that this is quite a challenge, we won’t always get it right but we continue to review and improve.” “I think we target our communications to address the issue at hand — would be useful to be able to translate how successful that communication was into observable industry changes.”
RPF KPI 3

Actions taken by regulators are proportionate to the risk being managed.

 

Given the risks involved in the industries AMSA regulates, the level of regulation is about right.“A national regulator implementing national standards will always be difficult when it comes to smaller ‘niche’ sectors. We need to continue to build on a risk-based flexible approach recognising current limitations of the National Law.”
RPF KPI 4Compliance and monitoring approaches are streamlined and coordinated.AMSA’s compliance and monitoring arrangements are well organised and efficient“This will continue to improve in the future as we mine the data that becomes available.” “Better data will provide better focus and then results. We need to sort our data issues out!”
RPF KPI 5Regulators are open and transparent in their dealings with regulated entities.AMSA explains its regulatory decisions well 
RPF 6Regulators actively contribute to the continuous improvement of regulatory frameworks.AMSA is always trying to improve maritime regulations to create a safer and more efficient industry“I think AMSA definitely focuses on improving regulations however the degree of success can suffer from the lack of support or timing.”
rpf
Last updated: 22 December 2020