INTOSAI IT Audit IT Methods Awareness

INTOSAI IT AuditIT Methods Awareness Outline

Scope

Overview

It Methods

Methods Description

Methods Usage

Audit Reporting

Scope

It Methods Described For:

Project Selection, Control, Evaluation

Systems Development

Systems Acquisition

Enterprise Architecture Development

Security Assessment

Overview

Methods Listed Here Are Generally Accepted in The Community

Methods Assess or Prescribe “What” Must Be Done Not “How” to Accomplish Activity

Methods Provide a Framework to Audit It Activity

It Methods Methods Description Module 1 Project Selection, Control, Evaluation Project Selection, Control, Evaluation

Wisely Managed Investments in It Can Improve Organizational Performance

Internet and Local Area Networks Enable Data Sharing and Research

Data Warehouse Permits Organizations to Discover Unknown Fiscal or Physical Resources

Project Selection, Control, Evaluation

However, Along With the Potential to Improve Organizations, It Projects Can Become Risky, Costly, Unproductive Mistakes

In Response, Gao Developed Guidance, That Provides a Method for Evaluating and Assessing How Well an Agency Is Selecting and Managing Its It Resources

Project Selection, Control, Evaluation

The Select/control/evaluate Model Has Become a Central Tenet of the It Investment Management Approach

Project Selection, Control, Evaluation

During the Selection Phase the Organization

Selects Those It Projects That Will Best Support Its Mission Needs and

Identifies and Analyzes Each Project’s Risks and Returns Before Committing Significant Funds to a Project.

Project Selection, Control, Evaluation

During the Control Phase the

Organization Ensures That, As Projects Develop, the Project Is Continuing to Meet Mission Needs at Expected Levels of Cost and Risk

If the Project Is Not Meeting Expectations Steps Are Taken to Address the Deficiencies

Project Selection, Control, Evaluation

Lastly, During the Evaluation Phase,

Actual Versus Expected Results Are Compared to

Assess the Project’s Impact on Mission Performance,

Identify Any Changes or Modifications to the Project That May Be Needed, and

Revise the Investment Management Process Based on Lessons Learned

Project Selection, Control, Evaluation

Gao’s Information Technology Investment Model (Itim) Model Is Comprised of Five Stages of Maturity

Each Stage Builds Upon the Lower Stages and Enhances the Organization’s Ability to Manage Its It Investment Stages

Project Selection, Control, Evaluation Five Stages Of Investment Maturity Project Selection, Control, Evaluation Progressing Through the ITIM Stages of Maturity Project Selection, Control, Evaluation

Itim Is a Tool for Assessing the Maturity of an Organization

An Itim Assessment Can Be Conducted for an Entire Organization or For One of Its Lower Divisions

Itim Is Applicable to Organizations of Different Sizes

Project Selection, Control, Evaluation

Itim Allows Auditors to Assesses the Maturity of Organizations to Manage Investments

Itim Provides a Maturity Stage or “Level” for an Organization

Each Maturity Stage or “Level” Has Required Practices or Activities

Project Selection, Control, Evaluation ITIM Required Processes Project Selection, Control, Evaluation

Applying the Model Requires Assessing

Critical Processes, Such As the Processes Used to Create an It Investment Portfolio

Core Elements, (Purpose, Organizational Commitment, Prerequisites, Activities, and Evidence of Performance)

Questions / Discussion

Questions

Comments

Discussion

Etc.

Methods Description Module 2 Systems Development Systems Development

Systems Development Includes Activities Such As

Project Management,

Requirements Management,

Configuration Management

Software Development, Testing, Etc.

Systems Development

Many Organizations Rely on Software-intensive Systems to Perform Their Missions

Software Quality Is Governed by the Quality of the Processes Used To Develop the Software

(Provide Reference)

Systems Development

The Software Engineering Institute Has Developed a Number of Models That Facilitate Assessing the Maturity of Organizations Developing Software

The Models Are Called Capability Maturity Models (Cmm)

Systems Development

What Is the Cmm?

An Ordered Collection of Practices for the Acquisition, Development or Maintenance of Systems

Ordered by “Key Process Area”

Practices Determined by the Community Through Broad Peer Reviews

Defines the Stages Through Which Organizations Evolve As They Improve Their Acquisition Process

Identifies Key Priorities, Goals and Activities on the Road to Improving an Organization's Capability to Do Its Job

Systems Development

The Cmm Provides a Framework for

Identifying an Organization’s Process Strengths and Weaknesses

Assisting an Organization Develop a Structured Plan for Process Improvement

Systems Development

Who Uses the Sw-cmm?

Organizations That Develop or Maintain Products That Contain Software

Organizations Who Want to Improve Their Software Development Processes

Audit Organizations Who Want to Assess the Maturity Of Organizations Developing or Maintaining Software Products

Systems Development

The Cmm Is Structured Into

Five Maturity Levels

Each Level Has Key Process Areas (Kpa)

Each Kpa Has Goals

Goals Require Certain Activities Be Performed

Management Provides Support and Verifies That Activities Are Being Performed

Systems Development Systems Development

The Five Levels Are

1. Initial: The Software Process Is Characterized As Ad Hoc and Few Processes Are Defined

2. Repeatable: Basic Project Management Processes Are Established; Improvement Activities Are Begun

3. Defined: Software Processes Are Documented and Standardized; All Projects Use an Approved, Tailored Version of the Organization’s Standard Software Processes

Systems Development

The Five Levels Are (Contd.)

4. Managed/quantitative: Detailed Measures of the Software Processes, Products, and Services Are Collected; the Software Processes and Products Are Quantitatively Measured and Controlled

5. Optimizing: Continuous Process Improvement Is Enabled by Quantitative Feedback From the Process and From Piloting Innovative Ideas and Technologies

Systems Development Software CMM Levels and KPAs Systems Development

Cmm Common Features

Commitment To Perform

Ability To Perform

Activities

Measurement & Analysis

Verification

Systems Development

Commitment To Perform

Describes What an Organization Must Do to ‘Set the Stage’ for Process Improvement / Implementation

Involves Establishing Policy

Assigning Responsibility

Systems Development

Ability To Perform

Describes the Preconditions That Must Be Present to Facilitate Process Improvement / Implementation

Assignment of Duties to Groups

Providing Trained or Experienced Personnel

Ensuring Adequacy of Resources

Systems Development

Activities

Describe the Activities, Roles, and Procedures That Are Necessary to Implement the Key Process Area

Requires Formal and Informal Planning Documents

Requires Formally Documented Procedures

Requires (Depending on Kpa) Coordination With Other Affected Groups, Tracking Contractor Performance, Etc.

Systems Development

Measurement & Analysis

Describes the Practices That Must Be Accomplished to Enable the Group to Track the Status of the Kpa

Effort & Funds Expended by the Project Team in Conducting Its Activities

Tracking Their Schedule and Progress (for Developing Formal Plans, Requirements, Etc.)

Systems Development

Verification

Describes the Practices That Must Be Performed to Ensure That Project and Senior Management Oversee the Activities of the Group

Includes Periodic or As Needed

Project Level Reviews

Senior Management Level Reviews

Systems Development Example From Model Questions / Discussion

Questions

Comments

Discussion

Etc.

Methods Description Module 3 Systems Acquisition Systems Acquisition

Systems Acquisition Includes Activities Such As

Project Management,

Requirements Management,

Solicitation, Contractor Tracking

Evaluation, Risk Management, Etc.

Systems Acquisition

Many Organizations Rely on Software-intensive Systems to Perform Their Missions

Organizations Have Been Increasingly Contracting Out for Software or Engineering Services

Systems Acquisition

The Software Engineering Institute Has Developed a Number of Models That Facilitate Assessing the Maturity of Organizations That Acquire Software or Systems

The Models Are Called Capability Maturity Models (Cmm)

Systems Acquisition

Just As For Software Development There Is the Sw-cmm (or Just Cmm)

For Assessing or Improving Acquisition Related Activities, The Sei Has Developed the Software Acquisition Capability Maturity Model (Sa-cmm)

Systems Acquisition

Who Uses The Sa-cmm?

Organizations That Acquire or Support Acquisition of Products That Contain Software, Including Software Support and Maintenance

Organizations That Are Responsible for Acquisition Life Cycle From Requirements Development Through System Delivery and Support

Audit Institutions That Want To Assess How Effectively Software or Services Are Being Acquired

Systems Acquisition

The Sa-cmm Is Also Structured Into

Five Maturity Levels

Each Level Has Key Process Areas (Kpa)

Each Kpa Has Goals

Goals Require Certain Activities Be Performed

Management Provides Support and Verifies That Activities Are Being Performed

Systems Acquisition Systems Acquisition

The Five Levels Are

1. Initial: The Software Process Is Characterized As Ad Hoc and Few Processes Are Defined

2. Repeatable: Basic Project Management Processes Are Established; Improvement Activities Are Begun

3. Defined: Software Processes Are Documented and Standardized; All Projects Use an Approved, Tailored Version of the Organization’s Standard Software Processes

Systems Acquisition

The Five Levels Are (Contd.)

4. Managed/quantitative: Detailed Measures of the Software Processes, Products, and Services Are Collected; the Software Processes and Products Are Quantitatively Measured and Controlled

5. Optimizing: Continuous Process Improvement Is Enabled by Quantitative Feedback From the Process and From Piloting Innovative Ideas and Technologies

Systems Acquisition Systems Acquisition Example From Model Questions / Discussion

Questions

Comments

Discussion

Etc.

Methods Description Module 4 Enterprise Architecture Development Enterprise Architecture Development

An Enterprise Architecture (Ea) Establishes the Agency-wide Roadmap to Achieve an Agency’s Mission

Eas Are “Blueprints” for Systematically and Completely Defining an Organization’s Current (Baseline) or Desired (Target) Environment

Enterprise Architecture Development

Eas Are Essential for Evolving Information Systems and Developing New Systems That Optimize Mission Value

For Eas to Be Useful and Provide Business Value, Their Development, Maintenance, and Implementation Should Be Managed Effectively

Enterprise Architecture Development

An EaIs A Strategic Information Asset, Which Documents the Mission And,

The Information , Technology, and the Processes Required to Perform the Mission

An Ea Includes a Baseline Architecture, Target Architecture, and a Sequencing Plan

Enterprise Architecture Development

Eas Typically Include

Business or Operational Architecture

Work Processes and Locations

Information or Data Architecture

Data or Information Needed to Perform Business

Technical or Systems Architecture

Technology Standards, It Systems Description

Enterprise Architecture Development EA Process Sections Refer to: A Practical Guide To Federal Enterprise Architecture Enterprise Architecture Development

Obtain Executive Buy-in and Support

Ensure Agency Head Buy-in and Support

Issue an Executive Enterprise Architecture Policy

Obtain Support From Senior Executives and Business Units

Establish Management Structure and Control

Establish a Technical Review Committee

Establish a Capital Investment Council

Establish an Ea Executive Steering Committee

Appoint Chief Architect

Define an Architecture Process and Approach

Define the Intended Use of the Architecture

Define the Scope of the Architecture

Determine the Depth of the Architecture

Enterprise Architecture Development

Develop the Baseline Enterprise Architecture

Collect Information

Generate Products and Populate Ea Repository

Develop the Target Enterprise Architecture

Collect Information

Generate Products and Populate Ea Repository

Develop the Sequencing Plan

Identify Gaps

Define and Differentiate Legacy, Migration, and New Systems

Planning the Migration

Enterprise Architecture Development

Use the Enterprise Architecture

Integrate the Ea With Cpic and Slc Processes

Train Personnel

Establish Enforcement Processes and Procedures

Maintain the Enterprise Architecture As the Enterprise Evolves

Reassess the Enterprise Architecture Periodically

Manage Products to Reflect Reality

Questions / Discussion

Questions

Comments

Discussion

Etc.

Methods Description Module 5 Security Assessment Security Assessment

Information and the Systems That Process It Are Among the Most Valuable Assets of Any Organization

Adequate Security of These Assets Is a Fundamental Management Responsibility

Security Assessment

Agency Must

Ensure That Systems and Applications Provide Appropriate Confidentiality, Integrity, and Availability

Protect Information Commensurate With the Level of Risk and Magnitude of Harm Resulting From Loss, Misuse, Unauthorized Access, or Modification

Security Assessment

Agencies Must

Plan for Security

Ensure Appropriate Officials Are Assigned Security Responsibility

Authorize (or ”Certify") System Processing Prior to Operations and Periodically As Necessary

Security Assessment

The Federal It Security Assessment Framework Provides a Method for

Determining the Current Status of Their Security Programs

Establishing a Target for Improvements Where Necessary

The Framework May Be Used to Assess the Status of Security Controls

Security Assessment

The Framework Comprises Five Levels to Guide Assessments of Security Programs and Prioritization of Improvement Efforts

Security Assessment

Level 1 Documented Policy

Level 2 Documented Procedures

Level 3 Implemented Procedures and Controls

Level 4 Tested and Reviewed Procedures and Controls

Level 5 Fully Integrated Procedures and Controls

Security Assessment

Level 1 of the Framework Includes

Formally Documented and Disseminated Security Policy

Level 2 of the Framework Includes

Formal, Complete, Documented Procedures for Implementing Policies Established at Level One

Security Assessment

Level 3 of the Framework Includes

Security Procedures and Controls That Are Implemented

Level 4 of the Framework Includes

Routinely Evaluating the Adequacy and Effectiveness of Security Policies, Procedures, and Controls

Level 5 of the Framework Includes

A Comprehensive Security Program That Is an Integral Part of an Agency’s Organizational Culture

Security Assessment Security Assessment Security Assessment Security Assessment Security Assessment Questions / Discussion

Questions

Comments

Discussion

Etc.

Methods Usage Methods Usage

Most Methods

Have Specific Activities to Be Performed

Can Be Applied to Specific Projects

Need a Team of About 3 - 4 Auditors

Requires Training or Understanding of the Method

Methods Usage

Since Methods Have Specific Activities

Questionnaires Can Be Generated

Results Can Be Tabulated

Analysis Can Be Formed Quickly From the Results

Methods Usage

The Sw-cmm and Sa-cmm Methods Require the Audit Lead Be Specifically Trained

Methods Usage Sample Data Collection Instrument Audit Reporting Audit Reporting

Audit Report Can Be Briefing Slides or Full Reports

Briefing Slides Can Contain Both Summary or Detailed Results

Audit Reporting Sample SA-CMM Summary Results Audit Reporting Sample SA-CMM Acquisition Risk Management Detailed Results Questions / Discussion

Questions

Comments

Discussion

Etc.

Contacts

Keith Rhodes

Phone 1 202 512 6412

Emailrhodesk@gao.gov

Madhav Panwar

Phone1 202 512 6228

Email panwarm@gao.gov