CDF Offline Computing Upgrades for Run 2

Pekka K. Sinervo Head, CDF Offline Group CDF/Fermilab/University of Toronto

1. Offline Group and Management

2. Run 2 Computing Environment

3. Software Development

4. Production and Analysis Systems

5. Offline Computing Budget and Schedule

Fermilab Director’s Review January 17, 1995

CDF Offline Group

The CDF Offline group is responsible for

• the CDF Offline Environment on Central Offline Facilities – FNALD VMS Cluster (mixed AXP and VAX)

– CDFSGA UNIX Analysis Server

– Distributed Computing Resources

• Production reconstruction of CDF data

• Development and maintenance of production, analysis and MC software

• Maintenance and distribution of software and databases • Systems to maintain and distribute documentation

• Distribution and management of datasets

Consists of collaborators and programmers drawn from:

• Fermilab Research Division CDF group

• Fermilab Computing Division (CDF group)

• 15 outside collaborating institutions

Organisation and Management

Offline group is managed by a physicist who

• Coordinates activities of groups and institutions • Acts as liaison between CDF ⇔ RD and CD

Group is divided into task forces or subgroups:

1. Software Development Task Force

2. Production Group

3. Datasets Task Force

4. Database Task Force

5. Code Distribution Task Force

6. MC Group

Task force leader(s) are members of Software Coordination and Management Board SCM Board meets every 2-4 weeks

• review progress and set policy

CDF Computing Upgrades

We are assuming that CDF will take

• 1000 pb−1 of data in Run 2

We have spent about $500-700K/year on computing in order to keep up with data analysis needs

• In 1993, bulk was spent on offline production farms

• In 1994, spent $900K on central analysis systems

• In 1995, improvement of access to Run IB datasets – expanding STK SILO to 5 Tb

– improving access of workstations to data

Difficult to predict specific hardware requirements beyond this date

• Can use experience to guide general planning

• Must retain flexibility in order to take advantage of most cost-effective solutions

Offline Upgrade Strategy

Run II Offline upgrade has two components:

1. software environment

2. hardware platforms

Little desire to adiabatically migrate Run IB software environment

• Still use obsolete programming methodology (f77)

• Code management system is VMS-based & not portable

Hardware platforms can be adiabatically upgraded

• Already have large investment in mixed VMS/UNIX system • Needs of Run IB force us to move carefully

Software Environment

CDF is only just beginning to address evolution of software environment

• Have to redesign code management system

• Introduce new programming methodologies

An informal effort has started to incorporate OOP into CDF environment A formal Offline Upgrade group has to be formed to carry on this work

• Problem is lack of human resources

• Exigencies of Run IB occupy most of the qualified personnel • Goal is to create this group over next 3 months

General Hardware Requirements

Have used Run IB data processing model to estimate hardware costs

1. Data coming from detector will be in two streams

(a) Stream A that corresponds to ∼ 10% of total cross section

• processed within hours of data taking

• used for detector monitoring and data validation (b) Stream B takes total cross section

• Processed with 24 hours with “final” reconstruction code

• Available within 2 days – requires calibration constants immediately

– probably generated in Level 3 system

2. Data is available on-line to physicists in ∼ 2 days • Several formats (DST’s, PAD’s, ntuples)

• Divided up based on L3 trigger decision

• All Stream A PAD’s available on-line – Would be SCSI disk today

– Rapid access to data servers from w/s’s

Requirements (cont.)

3. Access to Stream A DST’s and Stream B PAD’s via heirarchical storage system (robotic tape library?)

• Would require approximately 10-20 Tb storage

• Pursuing similar model for Run 1B analysis

4. Central Computing Facilities

• Central facility will be required – Post-production data processing

– MC sample generation and reconstruction

– Software development

• Will need data and software servers – Access to data disks and ATL

– Access to mass storage system

• High band-width network between servers and w/s’s

Computing Environment for Run 2

1. Focus on vendor-independent software

• Define CDF environment in a platform-independent way • Adopt a broader standard (POSIX?)

2. Integrate new languages and code development environments

• What programming languages do we support/foster?

• Move away from YBOS and FORTRAN-77

3. Migrate all central systems from VMS

⇒ Move to machine-independent environment ⇒ perform code-management on multiple platforms

4. Further improve distributed computing support

5. Design production processing system for Run 2

Resource Requirements

Central computing facilities will evolve over next 3 years

• Requirements for Run 1B should be compatible with evolution toward Run II facility

• Just completing major upgrade of central UNIX system

– Will incrementally upgrade CDFSGA ∗ add processors and memory

∗ add additional storage

– Upgrading STK SILO to 2 Tb

– VMS facility probably saw last major upgrade?

∗ Not clear until we gain more experience with CDF-

SGA

Distributed computing support will also evolve

• Incremental enhancements in network bandwidth

• WIll be adding “satellite” file and data servers

Software Development

CDF is committed to making transition to non-VMS central system

• Must establish UNIX-based code development environment

• Identify the methodology we will use

– Programming languages – Object-Oriented?

– Tools used for reconstruction and analysis code development

Will be making transition from VMS to OSF/1 for outside institutions

• Still unclear as to whether we will continue to support other platforms

– SGI/IRIX

– RS6000/AIX

• Will depend on effort required to support OSF/1

Cost and Schedule

Cost of computing systems for Run II difficult to estimate

• Can use experience from Run 1B

• But following is a coarse “guess” – Leave production system acquisition as last step

FY Item Cost
1994 VMS Upgrade  400K
CDFSGA Upgrade  400K
1995 SILO Upgrade  200K
Network Upgrades  400K
Data server  150K
1996 Central Systems Upgrades  700K
SILO Upgrade  300K
1997 2nd UNIX System  400K
New ATL or mass storage system  400K
1998 New production system  800K
Satellite file/software servers  200K
1999 Storage media upgrade  500K

Schedule

Hardware acquisition schedule will be drive by Run IB requirements

• Virtually impossible to predict details with any confidence

Software Development Schedule is most pressing

• Establish Offline Upgrade Group (Apr 95)

• Preliminary design document (Sep 95)

• Finalise design (Dec 95)

• Review and approval complete (Apr 96)

• Begin implementation (May 96)

Programming Resources

Most critical resource is professional and physicist programmers

• Will probably rewrite significant components of reconstruction software

• Need to develop new software framework

Most immediate need is programmer support

• Work in FY 1995 will require 4-5 FTE programmers 1. Design UNIX code management system

2. Develop tools for distributing computing systems

3. Support new software development paradigm for CDF

Inadequate programmer support has set CDF back 3-4 months in Run 1b

• Have lost two experienced programmers in last 2 years

• Existing staff is stretched extremely thin

• No programming staff to support new initiatives

• Mitigated by excellent support from CD Task Force – Marilyn Schweitzer (OCS tapes for UNIX)

– Bob Yeager (STK SILO Interface for UNIX)

– Mark Leininger & Randy Herber (software librarians)

Summary

Run 1B upgrades will evolve offline system into Run II

• Will continue to upgrade central computing systems

• Continue to improve data-handling capabilities

• Improve distributed computing environment

Expect Run 2 “Computing Environment” to take advantage of new software development systems

• New code management system

• New programming languages and tools

• Greater reliance on distributed computing systems

• Unrealistic to predict equipment at this point

Concern that Run 2 offline development will be limited by programming personnel resources