Section 4000.10  General

 

a)         The Local Records Commission (the Commission) for agencies comprising counties of less than 3,000,000 inhabitants shall consist of a county board chairperson/president, a mayor/president of a city, village or incorporated town, a county auditor, a State's Attorney (all of whom shall be appointed by the Governor), the State Archivist and the State Historian. [50 ILCS 205/6] The chairperson/president of the county board shall be the chairman of the Commission.  A member of the Commission may designate a substitute.

 

b)         The Commission shall meet at 10:00 a.m. on the first Tuesday of each month.  If the first Tuesday falls on a holiday, the Commission shall meet on the first Wednesday.

 

c)         All meetings of the Commission shall be open to the public and will be held in the John Daly Conference Room, Margaret Cross Norton Building, Springfield, Illinois unless otherwise stated in the publicly posted notice of the meeting.

 

d)         The Commission shall determine what records no longer have administrative, legal, fiscal, research, or historical value; determine what records should be destroyed or otherwise disposed of; and authorize and approve the destruction or other disposal of records. The State Archivist may deposit records in the State Archives, State Library or State Historical Museum, or with a local historical society, museum or library.

 

e)         No public record, except as otherwise provided by law, shall be disposed of by any officer or agency, unless written approval of the Commission is first obtained.

 

f)         The Commission reserves the rights to review, modify or revoke approved records disposal schedules after due notice is given to the agency and an open meeting on the subject is held.

 

g)         The presiding judge of any court of record or the head of each agency shall provide for compliance with this Part. In the case of a violation of the Local Records Act [50 ILCS 205] or of this Part discovered by the Commission, the Attorney General and the county's State's Attorney will be notified.

 

h)         Non-record materials may be destroyed at any time by the agency in possession of those materials without the prior approval of the Commission.  Whenever there is doubt that certain items are non-record materials, the agency should consider them to be records until their status is determined.

 

(Source:  Amended at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.20  Definitions

 

Administrative Value – Those aspects of records containing facts concerning an agency's administrative decisions that an agency needs for its immediate day-to‑day function.  This value almost always diminishes and is lost over time.

 

Agency – Any court, and all parts, boards, departments, bureaus and commissions of any county, municipal corporation or political subdivision. [50 ILCS 205/5]

 

Analog Records – Records created and maintained on a physical medium.  Examples include, but are not limited to, paper documents, analog motion picture film, analog photographs and analog audio tape.

 

Application for Authority to Dispose of Local Records – Also referred to as a Records Retention Schedule, the document stating the official retention, maintenance and disposition requirements for a record series, or type of record, based on administrative, fiscal, legal or archival values for the scheduled records.  This schedule is of no force unless approved by the Local Records Commission (see Section 7 of the Local Records Act).

 

Authentic Copy – A reproduction of a record that duplicates the content of the original record and that has been certified as authentic by the creating agency so that it may be submitted as legal evidence.

 

Born-Digital Records – Records created in a digital format, as opposed to those created in other media and then converted to digital surrogates.  Examples include, but are not limited to, word processing documents, electronic spreadsheets and digital photographs.

 

Chairman – Chairman of the Local Records Commission.

 

Commission or LRC – The Local Records Commission created by Section 6 of the Local Records Act.

 

Database – A collection of data elements organized in such a way that a computer program can select desired pieces of data.  A database is typically used as an electronic filing system through which users can quickly sort and retrieve data as necessary.

 

Digital Surrogate – A reproduction of content on analog media that has been scanned, photographed, encoded or otherwise converted to a digital file that, when printed, viewed or played, replicates the original content.

 

Digitization Process – The methods, tools and procedures by which a digital surrogate is created for an original record.  Examples include scanning and encoding of audio/video signals into digital data.

 

Electronic Microimaging – Any process in which digital documents or images (scanned or born-digital) are converted to permanent record microfilm.

 

Electronic Record – A record generated, communicated, received or stored by electronic means.  Both born-digital records and digital surrogates of analog records are considered electronic records.  Databases or components of databases may or may not be considered records, depending upon their function and contents.  Electronic records can be contained in various storage media.

 

Fiscal Value – Those aspects of records containing monetary information that accounts for the receipt or expenditure of funds.

 

Geographic Redundancy – The practice of replicating business data at two or more geographically distinct sites in order to protect against catastrophic data loss. Geographic redundancy can be provided through duplicate storage systems in different locations, or through contracting with vendors for remote or "cloud" storage.

 

Illinois State Archives – Department of the Archives and Records, Office of the Secretary of State, established pursuant to the State Records Act [5 ILCS 160].

 

Legal Value – Records that contain evidence of legally enforceable rights or obligations of the State, such as legal decisions and opinions; fiscal documents representing agreements, such as leases, titles and contracts; and records of actions in particular cases, such as claim papers and legal dockets.

 

List – An Application for the Authority to Dispose of Local Records that have accumulated.

 

Local Records Disposal Certificate – The document on which all local government agencies list all records of which they wish to dispose.  Agencies must file the Certificate with the Commission 30 days prior to the destruction of any records present on their approved Application for Authority to Dispose of Local Records.  Agencies may not dispose of records until the Certificate has been approved and returned to them.

 

Metadata – Commonly referred to as "data about data", metadata is structured data that describes, explains, locates or otherwise makes it easier to retrieve, use or manage an information resource.  Metadata is typically organized into distinct categories, such as administrative, descriptive, preservation or structural.

 

Non-Record Material – Types of non-record material include, but are not limited to:

 

Material not filed as evidence of administrative activity or for its informational content.

 

Extra copies of documents preserved only for convenience of reference.

 

Stocks of printed or reproduced documents kept for supply purposes, when file copies have been retained for record purposes.

 

Books, periodicals, newspapers, posters, finding aids and other library and museum materials made or acquired and preserved solely for reference or exhibition purposes.

 

Private materials neither made nor received by a local agency pursuant to State or local law or in connection with the transaction of public business.

 

Perforated, magnetized and photographically coded cards and tapes, provided that documents containing the same information have been filed in the same office and the cards and tapes were not prepared as evidence of administrative decisions or transactions subject to audit.

 

Transitory messages, consisting of material that is created primarily to communicate information of short-term value.  These can include messages sent via email, instant messaging (IM), text messaging (SMS) or paper correspondence.  Examples of transitory messages include, but are not limited to, reminders to employees about scheduled meetings or appointments; most telephone messages (whether in paper, voicemail or other electronic form); announcements of office events such as holiday parties or group lunches; and recipient copies of announcements of agency-sponsored events such as exhibits, lectures, workshops, etc.  Transitory messages are not intended to formalize or perpetuate knowledge and do not set policy, establish guidelines or procedures, certify a transaction or become a receipt.

 

Permanent – To be retained forever.

 

Permanent Record Film – A photographic camera original, or an exact copy of an original film, so composed and treated that the image and support will have maximum keeping quality under archival room storage conditions of 65-70 degrees Fahrenheit and 30-40% humidity.

 

Public Record − Any book, paper, map, photograph or other official documentary material, regardless of physical form or characteristics, made, produced, executed or received by any agency or officer pursuant to law or in connection with the transaction of public business and preserved or appropriate for preservation by such agency or officer, or any successor thereof, as evidence of the organization, function, policies, decisions, procedures, or other activities thereof, or because of the informational data contained therein. [50 ILCS 205/3]

 

Raw Stock – Sensitized photographic material that has not undergone the process of development.

 

Records Retention Schedule or Schedule – Same as Application for the Authority to Dispose of Local Records.

 

Record Series – A group of identical or related documents (either as to form or content) that is arranged under a single filing system or kept together as a unit because they consist of the same form, relate to the same subject, result from the same activity, or have certain common physical characteristics (i.e., maps, blueprints, etc.).  A series may contain both forms and correspondence.

 

Research, Historical or Archival Value – Records that document a specific local program, a unique program, a departure from previous local policy, formation of public policy, the activities of an important government official, or a trend or movement by the citizenry.

 

State Archivist – The Illinois Secretary of State.

 

System Decommissioning – The removal of a system from service, such as when a system used to manage business records is shut down when it is no longer being utilized or is being replaced by a new system.

 

(Source:  Amended at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.22  Incorporations by Reference

 

a)         No incorporation by reference in this Part includes any amendment or edition later than the date specified.

 

b)         The following materials are incorporated in this Part:

 

1)         ANSI/AIIM MS23 (2004) – Recommended Practice – Production, Inspection, and Quality Assurance of First Generation, Silver Microforms of Documents.

 

2)         ANSI/AIIM MS62 (1999) – Recommended Practice for COM (Computer Output Microfilm) Recording System Having an Internal Electronic Forms Generating System – Operational Practices for Inspection & Quality Control.

 

(Source:  Added at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.30  Procedures for Compiling and Submitting Lists and Schedules of Records for Disposal

 

a)         The presiding judge of any court of record or the head of each agency shall submit to the Commission lists or schedules of public records in his or her custody that are not needed in the transaction of current business and that do not have sufficient administrative, legal or fiscal value to warrant their further preservation.

 

b)         New lists or schedules are required whenever the informational content of a record series is changed.

 

c)         An original and one copy of all Applications for the Authority to Dispose of Local Records shall be submitted to the Commission on forms available from the Commission, Margaret Cross Norton Building, Springfield, Illinois, 62756.

 

d)         Nonrecord materials may be destroyed at any time by the agency in possession of the materials without the prior approval of the Commission. However, whenever the head of any agency doubts whether certain papers are nonrecord materials, he or she should presume that they are records.

 

e)         The Archivist shall be the local records advisor and shall appoint such assistants as necessary to assist local governments in carrying out the purposes of the Local Records Act [50 ILCS 205/5], including the preparation of lists and schedules of records.

 

f)         Applications for the Authority to Dispose of Local Records must be received in the office of the Local Records Unit of the Illinois State Archives at least 5 business days before a scheduled Commission meeting to be placed on the agenda of that meeting.  If received after that time, applications will be placed on the agenda of the next Commission meeting.

 

g)         The Commission will consider all applications presented at each meeting.  The Commission may approve, amend, deny or defer approval of an application pending clarification, modification or deletion of information presented on any portion of the application.  Once approved, applications are non-expiring.

 

h)         During the review of each application, the Commission will consider if the description of the record series is complete and easily understandable with regard to how and why the record was created, what purpose it serves, where else the information can be found, and if the proposed retention is appropriate in light of the record's administrative, fiscal, legal, research or historical value.

 

(Source:  Amended at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.40  Procedures for the Physical Destruction or Other Disposition of Records Proposed for Disposal

 

a)         Subject to statutory provisions, agencies may dispose of records authorized for destruction by the Commission.

 

b)         All records for which disclosure is prohibited by law that contain social security, driver's license or State identification numbers, or that identify a person by name and birth date, must be destroyed by a lawful, secure manner that does not allow for the reconstruction or reuse of the original record information.

 

1)         Approved methods of destruction for paper based records for which disclosure is prohibited by law or that identify a person include:  burning; shredding, in which either a crosscut shredder cutting to a maximum width of ⅜ inches or an industrial sized strip cut shredder is used, if it is incorporated with a baler or the shredded paper is further destroyed; pulping using standard wet process pulpers; or pulverizing using a dry destruction process that may include the use of hammer mills, choppers, huggers or disintegrating equipment.

 

2)         Approved methods of destruction for non-paper based records for which disclosure is prohibited by law or that identify a person include:  burning in a pyrolytic furnace or other incinerator or incendiary device; destroying in a dry pulverizing system; shredding; grinding, which is defined as abrading through the surface of an optical disc (compact disc); milling; knurling; disintegration; or degaussing.  Computer software or hardware must be overwritten, erased or wiped/sanitized in a manner that prevents retrieval.

 

3)         The handling and transportation of the records designated for destruction must be done in a reasonably secure manner that is designed to prevent public access to the records.

 

c)         Thirty days prior to disposal or destruction of any records, regardless of physical format or characteristics, the agency shall submit a Local Records Disposal Certificate to the Commission and proceed with disposal only after a copy of that certificate has been reviewed and approved by the Chairman and returned to the agency.  The original copy of this Local Records Disposal Certificate will be kept in the files of the Commission and the duplicate copy approved and returned by the Chairman shall be retained by the agency.

 

d)         In the case of records with scheduled retention of less than one year, a single Local Records Disposal Certificate may be used for more than one disposal event within a given year.  Local Records Disposal Certificates submitted with this intent must include a schedule of proposed records disposal in addition to the normally required information.  Agencies may not proceed with disposal of records until receipt of approval from the Commission, as delineated in subsection (c).

 

e)         If an agency's records have been damaged by water, fire, smoke, insects or vermin, mold or some other natural disaster that poses a health or safety risk to employees, that agency may apply to the Commission for permission to dispose of those records ahead of their scheduled disposal date.  The request must include a Local Records Disposal Certificate accompanied by the agency's explanation of why the records need to be disposed of early.  The Commission may grant the request only after physically reviewing the damaged records.

 

(Source:  Amended at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.50  Standards for the Reproduction of Records by Microphotographic and Electronic Microimaging Processes with a View to the Disposal of the Original Records

 

a)         Records proposed for microfilming or electronic microimaging with a view to disposal of the original records must be on a list or retention schedule approved by the Commission.

 

b)         In submitting lists or schedules of records scheduled for permanent retention for which microfilm copies are to be substituted, the head of each agency shall certify that microfilm copies, made in accordance with standards of the Commission, will be adequate substitutions for the original records.

 

c)         Quality of the Film Used.  The film stock must be silver halide and the processing of the film shall comply with the minimum standards of quality required by the Commission as set forth in Section 4000.60.

 

d)         Preparation of the Records for Filming or Electronic Microimaging

 

1)         All documents in the file shall be microfilmed, unless their size or physical form prevents microfilming, in which case an explanation of their omission shall be microfilmed at the appropriate point on the roll of film and be worded substantially as follows:

 

"(Item Description) was omitted from this roll of film because ____________________________________.  It may be located _____________________________."

 

2)         Any records not filmed shall be maintained by the agency under terms specified on the approved records retention schedule.

 

e)         Integrity of the Original Records

 

1)         The integrity of the original records shall be preserved through a photographic or electronic microimaging process so that the image on film, or exact duplicates of the image, will be adequate substitutes for the original records in that they will serve the purposes for which the records were created or maintained and that the copies will contain all significant record detail needed for probable future reference and will not permit additions, deletions or changes to the reproductions of the original images.

 

2)         Prior to microfilming, the original documents shall be prepared, arranged, classified and indexed to readily permit the subsequent location, examination and reproduction of the photographs.  Any significant characteristics of the records that would not reflect photographically (e.g., that the record is indistinct or that certain figures are of a color not suited to recording on microfilm) shall be indicated by means of an explanatory target inserted to guide the user.  Any notations on the face or reverse side of any document shall be photographed and identified as forming an integral part of the original document.  A significant characteristic is any part of the record necessary for its interpretation, including all words, numbers and illustrations.

 

A)        Each film roll, camera negative, or sheet (including 105 mm continuous fiche film rolls, but not COM) shall be identified by or contain the following targets:

 

i)          A technical target for measuring resolution.

 

ii)         A film density target (8½ x 11 inch bond paper).

 

iii)        A roll number START target in characters that can be read without magnification.

 

iv)        A TITLE target giving name of the office having custody of the records, a brief title of the record series, dates, file arrangement, and the number of the schedule approved by the Commission authorizing the project.

 

v)         Listed between the START file and END file targets must be explanatory targets for omission, deletion, misfiles, retakes, or any example given in this subsection (e)(2).

 

B)        At the end of each roll/sheet of film, after the document images, shall be targets as follow:

 

i)          An END target containing the number of the list or schedule approved by the Commission authorizing the project.

 

ii)         Roll number.

 

iii)        Brief title of the record series.

 

iv)        Beginning and ending file designations.

 

v)         A camera/electronic microimaging operator's certificate as follows:

 

"I hereby certify that I have on this ____ day of ________, 20___, photographed or electronically microimaged the documents appearing on this roll of film, that they are true copies of the documents found in the record file described above, and that the integrity of the above described record file has been maintained on this film by microfilming or electronically microimaging each document in the exact order in which it was found in the file. Reproductions designed to serve as permanent records comply with the regulations and standards of the Local Records Commission."

 

vi)        Signature of camera operator.

 

vii)       A film density target (8½ x 11 inch bond paper).

 

viii)      A technical target for measuring resolution.

 

f)         Security microfilm shall have no breaks, cuts or splices in the body of the film, which shall be the area following the START target and preceding the Camera/Electronic Microimaging Operator's Certificate.  However, a retake of a length of film may be spliced ahead of the START target or after the Camera/Electronic Microimaging Operator's Certificate, providing that the retake be given its own START target and Camera/Electronic Microimaging Operator's Certificate.  This shall be done in such a manner as not to overload a reel or cartridge.  Exceptions to this rule are:

 

1)         If the trailing end of a reel is fogged or unreadable, the camera or electronic microimaging operator shall rephotograph the original documents or obtain the corresponding electronically microimaged documents from a point 12 images in advance of the last readable image prior to the fogged or unreadable area.  The retake will include a Camera/Electronic Microimaging Operator's Certificate and will be spliced to the trailing end of the fogged or unreadable portion of the film.

 

2)         When a court-ordered expungement of specific records is issued and deletions are made from the roll of film, the court expungement order and a Certificate of Deletion, illustrated below, must be photographed or electronically microimaged and the images spliced to the beginning of the film.

 

CERTIFICATE OF DELETION

 

This is to certify the deletion of microfilm images on this roll of
microfilm occurred due to Court Order #							,
dated			, signed by Judge					.
No other images other than those listed in this order were deleted.
Signature of Officer

 

g)         The camera or microimaging system used to microfilm the records shall be one that accurately reproduces the content of the original records with sufficient photographic contrast and resolution to be readable through three generations of reproduction.

 

h)         Each roll of original film or camera negative must be inspected after processing and before duplicate copies are made.  The inspection must be conducted in such a manner as to reveal defects such as improper density, poor resolution, blurred or obscured images, improper document sequence, or improper identification targets.  If a defect prohibits a clear, legible, hard copy print from the files, the original records must be rephotographed.  The following methods are suitable means of inspection:

 

1)         Random sampling of the film, including samples from the beginning, middle and end of the roll or microform.  (It is suggested that this be done on all film as a minimum quality control.)

 

2)         Visually inspecting the film by passing each image through a reader and checking for overlapping, double or folded images, or other types of problems that would impair retrieving any information on the microimages.

 

3)         Performing all of the requirements of subsection (h)(2) plus counting the number of microimages on the film and comparing that against the number of documents that were to be microfilmed.  (If the numbers coincide, the conclusion is made that every document has been microfilmed.)

 

4)         Individually comparing each document with each microimage that was actually created.  (This visual verification provides the highest assurance that every document has been properly filmed.)

 

i)          If more than 1% of the original images needs to be refilmed (approximately 30 images per roll), the entire roll must be refilmed.

 

j)          Updateable Microfiche Systems.  An agency considering using an updateable microfiche system should first contact the Commission to review the proposed application.  This application will be approved if the updateable microfiche meets the following specifications:

 

1)         Each microfiche must have the specified targets at the beginning and end of each fiche as required by subsections (e)(2)(A) and (B).

 

2)         Each time a microfiche is updated, either a Camera Operator's Certificate must be inserted at the end of the added documents or annotated reference to the original Camera Operator's Certificate must appear on each added image.

 

3)         Only records bearing retention periods of 10 years or less may be placed on updateable microforms.

 

4)         If a court-ordered expungement is necessary, a Certificate of Deletion must appear at the place of the deleted image.

 

k)         Prior to the destruction of records microfilmed under the authority of approved records schedules, the agency shall file a Local Records Disposal Certificate with the Commission that lists all records that are to be destroyed and certifies compliance with this Section.

 

l)          Each film carton shall be identified by a label or exterior marking indicating:

 

1)         Roll number.

 

2)         Name of office.

 

3)         Title of the record series.

 

4)         Names of the file units at the start of the roll, at space targets, and at the end of the roll.

 

5)         The number of the application authorizing the microfilming of the record/record series.

 

m)        Inspection.  Security or master films of permanent record microforms, and records microfilmed to dispose of the original record, shall be inspected every 2 years during their scheduled life.  The inspection shall be made using a 1% randomly selected sample in the following categories:  70% − microforms not previously tested; 20% − microforms tested in the last inspection; and 10% − control group.  The control group shall represent samples of microforms from the oldest microforms filmed through the most current.

 

(Source:  Amended at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.60  Minimum Standards of Quality for Permanent Record Photographic Microcopying Film

 

a)         These standards are concerned with both raw stock for permanent record films and with the processed films ready for storage.  They are not restricted to microfilm but apply equally to motion picture films, roll films, and sheet films. They reflect incorporations listed in Section 4000.22.

 

b)         All such film stock shall be of approved permanent type polyester based film that includes an anti-halation dye system that meets the minimum specifications of ANSI/AIIM MS23.                                                                

 

c)         Each frame of microfilm shall be exposed and processed so that every line and character on the document appears on the microfilm with sufficient clarity to permit reproducibility through three successive generations of reproduction. With regard to operational procedures, inspection and quality control of silver gelatin microfilm, ANSI/AIIM MS23 shall apply.

 

d)         The background photographic densities must be appropriate to the type of documents being filmed. Appropriate background densities are as follows:

 

Classification	Description of Documents	Background Density
Group 1	High-quality, high-contrast printed books and periodicals; black type face; fine-line originals; black opaque pencil writing; and documents with small, high-contrast print....................................................	1.00 to 1.30
Group 2	Pencil and ink drawings; faded and very small print (for example, footnotes at the bottom of a printed page); scenic checks; documents with printed pictorial images; and newspapers.......................................	0.90 to 1.10
Group 3	Low-contrast manuscripts and drawings; graph paper with pale, fine-colored lines; letters typed with a worn ribbon; poorly printed, faint documents.........................................	0.80 to 1.00 (1:24 reduction or less)
Group 4	Very low-contrast (worst case) documents can require extremely low background density....................	0.75 to 0.85 (1:24 reduction or less)
Group 5	COM.................................................	1.50 to 2.00

           

(Source:  Amended at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.70  Digital Reproduction

 

a)         Analog records may not be destroyed in favor of digital surrogates unless the digital surrogates are produced in compliance with this Section and unless done pursuant to a retention schedule approved by the Commission.  Agencies must certify compliance by filing a Local Records Disposal Certificate with the Commission prior to the destruction of any analog records for which digital surrogates are to be substituted.

 

b)         In submitting schedules of analog records scheduled for permanent retention for which digital surrogates are to be substituted, each agency shall certify that the copies will be made in accordance with the regulations of the Commission and will be authentic copies of the analog records.

 

c)         Analog records scheduled for permanent retention may not be destroyed in favor of digital surrogates unless those surrogates are maintained in compliance with Section 4000.80.  If the requirements of Section 4000.80 are not met, records must be additionally maintained either in original format or in a microfilm format that complies with Sections 4000.50 and 4000.60.

 

d)         File Integrity.  The integrity and authenticity of the analog records shall be preserved through the digitization process so that the images or surrogates will be authentic copies of the analog records.  They must serve the purposes for which the original records were created or maintained and the copies must contain all significant record detail needed for probable future reference.

 

e)         Digital surrogates of analog records must be created and stored in file formats approved by the Commission (see Appendix A).  These formats include, but are not limited to, PDF, PDF/A and TIFF.

 

f)         Access.  The digital surrogates shall be prepared, arranged, classified and indexed to readily permit subsequent location, examination and reproduction of individual records.  Hardware, software and documentation must be maintained to allow ready access to each file.

 

g)         External Vendors.  Agencies may contract with external vendors to perform any of the tasks involved with the digitization of records. The vendors must comply with all State laws and rules governing the digitization process.  The contracting agency will remain responsible for the proper management of records in the temporary custody of the vendor.

 

h)         Technical Standards for Creation of Digital Surrogates

 

1)         Quality Control.  Prior to production, an agency shall assemble a sample set of source documents or records equivalent in characteristics to the source documents for the purposes of evaluating scanner results.  Scanner quality must be evaluated in accordance with current industry best practices at the time of production, such as, but not limited to, Riley & Whitsel's "Practical Quality Control Procedures for Digital Imaging Projects" and the Federal Agencies Digitization Guidelines Initiative's Technical Guidelines for Digitizing Cultural Heritage Materials:  Creation of Raster Image Master Files.  If agencies are uncertain as to whether they are following appropriate best practices, they should consult with the Illinois State Archives.

 

2)         Quality Assurance.  Before production, an agency shall develop written quality assurance procedures based upon the results of the pre-production quality sample.  Before the original documents are destroyed, quality assurance must be conducted in accordance with current industry best practices at the time of production, such as, but not limited to, Riley & Whitsel's "Practical Quality Control Procedures for Digital Imaging Projects" and the Federal Agencies Digitization Guidelines Initiative's Technical Guidelines for Digitizing Cultural Heritage Materials:  Creation of Raster Image Master Files.  If agencies are uncertain as to whether they are following appropriate best practices, they should consult with the Illinois State Archives.

 

3)         Scanning Resolution.  Scanning resolution must be adequate to ensure that no information is lost.  A scanning resolution with a minimum of 200 dots per inch is required for recording documents that contain no type font smaller than 6 point.  A minimum scanning resolution of 300 dots per inch is required for engineering drawings, maps and other documents with a type font smaller than 6 point or with background detail.  The selected scanning resolution must be validated with tests on actual source documents.

 

(Source:  Added at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.80  Management of Electronic Records

 

a)         Born-digital Records.  Born-digital records shall be subject to the same record schedules as those records originally created in other media.

 

b)         Databases.  Databases or components of databases may or may not be considered records, depending upon their function and contents.  An agency's Records Retention Schedule, as approved by the Commission, will be used to make such a determination.  

 

c)         Permanent Records.  Records scheduled for permanent retention must be stored in file formats approved by the Commission (see Appendix A) at the time the records are permanently removed from the active system, at the time of active system decommissioning, or at the request of the Commission.

 

d)         Storage Media.  Electronic records may be stored on a hard disk, magnetic tape, networks using a combination of these, or other media approved by the Commission (see Appendix B).  Stored records must be regularly migrated to new media in accordance with current industry best practices, such as, but not limited to, ANSI/ARMA 16-2007, The Digital Preservation Coalition's Digital Preservation Handbook and the University of Illinois' "Best Practices for Media Selection and Migration".  If agencies are uncertain as to whether they are following appropriate best practices, they should consult with the Illinois State Archives.

 

e)         Access.  Electronic records must be maintained in such a way that each record is individually accessible for the length of the scheduled retention.

 

f)         Backup Copies.  A minimum of two total copies of all electronic records must be preserved for the length of scheduled retention.  Copies must be stored according to current industry best practices for geographic redundancy, such as, but not limited to, NIST Special Publication 800-34 Rev. 1 – Contingency Planning Guidelines for Federal Information Systems and the New York State Archives' "Record Advisory:  Electronic Records Disaster Preparedness and Recovery".  If agencies are uncertain as to whether they are following appropriate best practices, they should consult with the Illinois State Archives.

 

g)         External Vendors

 

1)         Agencies may contract with external vendors for the storage or management of electronic records. Vendors must comply with this Section.  Contracting agencies will remain responsible for the proper management of records in the custody of vendors.

 

2)         Contracts for the storage of electronic records by external vendors must allow for the return of all electronic data files and indexing information to the agency at the expiration of the contract or, in the case of vendor failure, in a format complying with the requirements of subsections (c) and (e).

 

h)         Identification.  Each electronic record must have a unique identifier to allow for ongoing management of that record.  If electronic records are stored on discrete storage media, each physical unit must have a unique identifier.

 

i)          System Requirements for the Management of Permanent Records

 

1)         Electronic Records Scheduled for Permanent Retention.  These records must be stored and managed in accordance with subsections (d) through (j).  If those requirements are not or cannot be met, then an additional microfilm or print copy must be created for permanent preservation. Microfilm copies must be created in accordance with Sections 4000.50 and 4000.60.  Certain record types that are unsuitable for print or microfilm reproduction, such as audio or video files, are exempt from this requirement.

 

2)         Classification.  Systems used to store and access electronic records must allow records to maintain their relationships with one another.

 

3)         Security.  Systems used to store and access electronic records must not permit unauthorized additions, deletions or changes to the records. Access to the system must be limited and strictly controlled.

 

4)         Access.  Systems used to store and access electronic records must allow for the retrieval of individual records and their associated metadata in a timely manner.

 

5)         Metadata.  Systems used to store and access electronic records must capture relevant structural, descriptive and administrative metadata at the time a record enters the system.  The system must generate additional metadata whenever a record is moved within the system or migrated to another format or storage medium.

 

6)         Format Migration.  Systems used to store and access electronic records must allow for the migration of stored records, and their associated metadata, notes and attachments, from one file format to another.

 

7)         System Maintenance.  Each agency shall ensure that hardware, software and documentation (including maintenance documentation) used to store and access electronic records are retained for the entire life of that system.

 

8)         System Changes.  If hardware, software and/or documentation used to store and access electronic records is replaced, or if the electronic records are migrated to a new system, the agency must ensure that the replacement hardware, software and/or documentation meets all requirements mandated in the approved records schedule and in this Section.

 

j)          Legacy Systems.  Agencies must make efforts to bring existing systems used for the storage of electronic records into compliance with this Section.  If systems are unable to accomplish some of the required functions, agencies must attempt to achieve the same results through separate processes.  Vendor contracts for the storage or management of government electronic records must be updated for compliance when possible.

 

(Source:  Added at 39 Ill. Reg. 2652, effective February 9, 2015)

Section 4000.APPENDIX A   Sustainable File Formats for Electronic Records − A Guide for Government Agencies

 

Electronic records are produced and kept in a wide variety of file formats, often dictated by the type of software used to create and access a record. Accessibility and user convenience are also common factors that determine the use of one format over another. When dealing with electronic records that have retention requirements past their initial use, however, one must also take into consideration the sustainability of the format used.

 

Sustainability in this context refers to continued accessibility over time.  For example, will a given electronic record be available for users in ten years?  What about twenty?  Fifty?  While no file format can guarantee perpetual accessibility, certain formats have distinct advantages over others in this regard. These formats are often referred to as "sustainable" formats.  Sustainable formats often include the below features:

 

1)         Published Documentation and Open Disclosure:  Specifications for the format are published and accessible to the public.  This means that anyone who wants to create tools to work with the format can do so with no restrictions of copyright. Formats that share these characteristics are commonly called "open-source" or "non-proprietary." Because anyone can create tools to access such formats they have a low chance of becoming inaccessible in the future, even if the formats themselves become obsolete.

 

2)         Widespread Adoption and Use:  The more widely a format is used, the more likely it is to have multiple tools used to access and manipulate it.  This reduces the chance of a format becoming inaccessible due to one software publisher going out of business.  Widespread adoption also serves as an indicator of general format stability, and serves as a safeguard against loss of accessibility.  A wider user base means more stakeholders who have a vested interest in keeping a format going.

 

3)         Self-describing Formats:  These formats contain metadata (data about the data) within their structure that interprets the content, context and structure of the file.  This means that descriptive information (the file name, date of creation, identification of data within the file, for example) can be kept within the file itself, and external documentation is not required.  When discussing long-term preservation this is particularly important, since records often become disassociated from their original software environment and accompanying files. The more self-contained a format is, the better the chances of the data contained within being accessible down the road.

 

4)         Unencrypted Files:  Electronic records with long-term retention should not be encrypted in any way, as this can severely compromise the future accessibility of those records. Encryption methods change dramatically over time, and the specific software tools needed to access current encrypted records may not exist in the future. A good electronic records management system can handle security, restricting access to records as needed, while leaving the records themselves unchanged.

 

The following is a list of formats currently recommended for long-term preservation by the Illinois State Archives.

 

Best Choice:

 

PDF/A (Portable Document Format / Archives):  A variant of PDF that is specifically aimed at long-term preservation, its specifications are published in the standard ISO 19005-1:2005. It sacrifices certain functions, such as the ability to have external hyperlinks or embed audio or video, for the sake of greater reliability. The most notable different between PDF and PDF/A is the latter's ability to embed all necessary fonts within the file itself.  This makes the file totally self-extracting, without any need to access external font information to properly present the formatting of the document. PDF/A also embeds descriptive metadata within the file itself, making it self-describing.  These two factors make PDF/A the preferred format for long-term preservation of textual electronic records, both born-digital and digitized.  Files can be converted to PDF/A by a number of different software tools and plug-ins to existing word-processor software.

 

Other Options:

 

PDF (Portable Document Format):  A format commonly used to present formatted, page-oriented documents. PDFs can contain text, images, graphics, video and audio, as well as hyperlinks to outside documents. Originally created by Adobe Systems as a propriety format, the source code for PDF and its variants have since been made freely available, making it an open-source format.  PDF is widely adopted around the world. Some later versions of PDF can include self-describing metadata. PDFs are acceptable for short to medium-term storage, but are not suitable for long term (20+ years) or permanent preservation. For long-term applications the PDF/A variant is preferred.

 

XML (Extensible Markup Language):  A standard format for structured documents and data on websites, XML is also a preferred format for the preservation of metadata associated with records.  XML is maintained and developed by the World Wide Web Consortium (W3C), but is open-source.  XML enjoys nearly universal adoption, and can be accessed and worked on by scores of freely available software tools.  XML is self-describing, but requires association with an appropriate schema (also freely available) in order to properly render all formatting.

 

HTML (Hypertext Markup Language):  A standard format for structured documents and data on websites currently maintained and developed by the World Wide Web Consortium (W3C).  HTML is open-source, and is universally adopted.  Unlike XML, HTML does not contain descriptive metadata headings.  This limits the machine-readability of HTML, particularly when attempting to perform advanced search functions within files. 

 

Plain Text:  The most basic form of text file, plain text can be rendered by any software that can read text, across any platform.  Plain Text renders only basic characters, spaces and punctuation, however, and does not preserve formatting such as italics or bold letters.  It is therefore typically used only for relatively small amounts of information such as software instructions or short notes. Plain Text is open-source and universally adopted. Common file extensions for Plain Text include .txt and .text.

 

ODF (OpenDocument Format):  An XML-based file format used for spreadsheets, charts, presentations and word processing documents.  ODF was developed by Sun Microsystems, but is an open format, is freely available to anyone and has been published as an ISO standard (ISO/IEC 26300:2006).  Owing to its relatively recent creation (2005) ODF is not as widely adopted as some other formats, but it is supported by almost all current office suites and word processing programs.  File extensions for ODF files vary depending upon the specific type of file, but include .odt (word processing), .ods (spreadsheets) and .odp (presentations).

 

Still Images

 

Best Choice:

 

TIFF (Tagged Image File Format):  TIFF was initially created in the 1980s in an effort to standardize file formats created by commercial scanners.  The format has gone through a number of revisions since then, becoming an international standard for electronic images.  The format is currently owned by Adobe Corporation, but the specifications are open and freely available.  Unlike many image file formats, TIFF is uncompressed.  This means that the files are larger than a compressed format (such as JPEG) but there is no loss of data.  This ensures that the file can be reproduced over time at its full fidelity.  TIFF files can contain "tags" that store descriptive metadata about the file.  TIFF files may have a file extension of .tif (Windows) or .tiff (Macintosh).

 

Other Options:

 

JPEG 2000 (Joint Photographic Experts Group): JPEG-2000 was created by the Joint Photographic Experts Group in 2000 as a next-generation format for electronic images.  The format is part of an international standard: ISO/IEC 15444:2004.  JPEG-2000 files can be compressed in either lossy or lossless fashion, although only the lossless variety is acceptable for long-term preservation.  The format is still relatively new, and thus does not have the same wide-spread use as TIFF.  This makes it a slightly riskier choice for preservation, although usage of the format is growing.  The lossless compression of JPEG 2000 provides some space savings over TIFF, but it may be better suited as a format for access rather than preservation.  The standard file extension for JPEG 2000 is .jp2.

 

PNG (Portable Network Graphics):  A file format initially created with the approval of the World Wide Web Consortium (W3C) as a replacement to GIF (Graphics Interchange Format).  PNG is most often used to present images on the web, and can be accessed with a wide variety of web browser and image display software.  PNG uses a "lossless" compression algorithm which reduces the size of the file without losing any data.  This means that images in PNG format do not suffer from "generation loss," where the quality of an image suffers over time with repeated use.  Specifications for PNG are open and freely available, and the format can contain extensive metadata within its structure.

 

Spreadsheets

 

Due to the complexity of spreadsheet structure it is challenging to perfectly represent data over time.  Different software uses varied means to record formulae and link data, and so advanced functions are not always replicable in more open formats.  The below formats represent the best approach for long-term accessibility, but both may be unable to represent certain formatting or functions of spreadsheets originally created in formats such as Microsoft's XLS.  Agencies may want to save copies of spreadsheets with long-term retention in both the native format and in one of the below.  This redundant method can preserve the maximum functionality of the spreadsheet while still protecting the core data from format obsolescence.

 

CSV (Comma Separated Values):  A simple format which can be used to represent spreadsheet data.  CSV files can be accessed with any spreadsheet software or text editor, but at the cost of potential loss of advanced functionality enjoyed by more proprietary spreadsheet formats. There is therefore a tradeoff with using CSV: universal interoperability is excellent for long-term preservation, but the loss of advanced formulae may compromise the core data of the record.  Basic spreadsheets containing tabular data without advanced functions may be better served by CSV than others.

 

ODF:  (See previous entry for general data on ODF) The spreadsheet format of ODF, .ods, is a good choice for preservation of spreadsheets, as it supports more advanced functionality than CSV.  However, spreadsheets originally created in other formats such as XLS may suffer some functionality loss upon conversion to ODF due to the non-standardized methods by which different software execute formulae.

 

Audio

 

Best Choice:

 

BWF (Broadcast WAVE Format):  A variant of the WAVE format, BWF (sometimes called BWAVE) was developed by the European Broadcasting Union with long-term preservation in mind.  BWF takes the existing WAVE file structure and adds additional metadata support.  The specifications for BWF are open and freely available, and the format is a de facto standard for digital audio for those in the radio, motion picture and television industries.  It is also used extensively by audio archives throughout the world.  The format is self-describing, as it contains its own structural and descriptive metadata.  BWF files are uncompressed, and can be played by any software that is WAVE compatible.  In order to display, add or modify metadata in a BWF file, however, one must use software that specifically supports the format.  Free software is available that can attach BWF metadata to existing WAVE files.  The file extension for BWF is .wav, the same as standard WAVE files.

 

Other Option:

 

WAVE (Waveform Audio File Format):  WAVE is a format created by Microsoft and IBM in the early 1990s.  Though proprietary, the format is fully documented and has been used as the basis for the preservation-oriented variant BWF (see above entry).  WAVE files are uncompressed, so they lose no audio data as with some other audio formats.  The format also enjoys near-universal adoption, as it is compatible with virtually every audio player available, across computer platforms.  Software utilities to convert other formats to WAVE are plentiful and inexpensive (or free).  WAVE has limited metadata capabilities, so is a second choice for long-term preservation behind BWF (see above).  WAVE can still be an acceptable format for non-permanent audio, provided that appropriate external metadata is associated with the WAVE files.

 

Video

 

Whereas best practices typically dictate that only uncompressed formats be used for preservation of electronic content, the area of video preservation becomes more complex.  Uncompressed video can take up huge amounts of space in a storage environment, and thus formats utilizing "lossless" or "near-lossless" compression have become more acceptable in some cases.  Compression of these types utilizes algorithms to reduce the size of a file without irrevocably losing any data.  This can be compared to "lossy" compression, which sacrifices some data to achieve smaller size.  Lossy compression is unacceptable for long-term preservation because it permanently alters the structure of digital content and can lead to gradual reduction in quality over time.

 

MPEG-4 (Motion Picture Experts Group):  MPEG-4 is an open-standard format developed by the Motion Picture Experts Group as a format for encoding video content for dissemination on the web.  There are two main encoding versions, and numerous subcategories, of the format.  Documentation for all varieties of MPEG-4 is extensively published as part of an international standard: ISO/IEC 14496-14:2003.  The compression of a given MPEG-4 video file will depend upon the specific software and coding used in its creation, and can range from lossy to lossless.  For long-term preservation only lossless or near-lossless compression should be used.  MPEG-4 supports the embedding of descriptive metadata to help support future access.  A number of software tools, both free and paid for, are available to convert existing video files to MPEG-4 format.

 

Motion JPEG 2000 (Joint Photographic Experts Group):  Motion JPEG-2000 is a derivative of JPEG 2000 which codes and displays video.  The format is part of an open international standard: ISO/IEC 15444-3:2004.  Motion JPEG-2000 files can be compressed in either lossy or lossless fashion, although only the lossless variety is acceptable for long-term preservation.  The format is still relatively new, so adoption is not yet as widespread as older video formats.  A number of software tools are available that can convert other video formats into Motion JPEG-2000, and it can support a variety of descriptive and structural metadata.  File extensions for the format are .mj2 and .mjp2.

 

(Source:  Old Appendix A repealed at 9 Ill. Reg. 17796, effective November 5, 1985; new Appendix A added at 39 Ill. Reg. 2652, effective February 9, 2015)

 

Section 4000.APPENDIX B   Reliable Storage Media for Electronic Records − A Guide for Government Agencies

 

Modern computer systems use a wide variety of storage media to store and access electronic data. What media is used depends on a number of factors, but cost, speed of access and ease of use are common drivers of selection decisions. Often overlooked are concerns of long-term reliability and sustainability. Electronic records are vulnerable to degradation or loss if not maintained in an appropriate storage environment which takes into consideration media reliability and guards against technological obsolescence.

 

To say that media is reliable is to indicate that it can be trusted to preserve and provide access to data stored on it over time. While no storage medium can guarantee reliability and sustainability, certain media formats have distinct advantages over others in this regard. Understanding a media format's strengths, weaknesses and expected life span allows IT managers to appropriately protect the data stored on that media.

 

No storage media alone can ensure the preservation of electronic records. Selecting appropriate media is one part of a greater preservation strategy which includes using sustainable file formats, actively managing files over time, planning for future technology change and securing adequate resources to support preservation activities.

 

The following factors should play a part in the selection of any storage media for electronic records.

 

Durability: Durability is a factor representing the ability of electronic storage media to withstand wear and environmental conditions. Corruption (data rot) can occur as the electrical charge, magnetic orientation, or physical material degrades, causing unintended changes or loss of data.

 

Assessing Durability: Durability of media is commonly expressed in terms of "mean time between failures," which indicates how long a given drive/tape/disk can be expected to operate before failure.

 

Widespread Adoption and Use: Widespread adoption and use is a factor indicating a wide user base, meaning more stakeholders have a vested interest in keeping the storage media viable and well-supported. Widespread adoption also serves as an indicator of general media stability and generally provides a lower overall lifecycle cost of storage. The more widely a storage medium is used, the more likely it is to have long-term support to maintain it. Widely adopted technologies are typically documented and based on open standards supported by multiple hardware vendors. This reduces the chance of a medium becoming inaccessible due to one vendor going out of business.

 

Assessing Adoption and Use: While there is no universal benchmark that indicates something is "widely used" one should look for examples of a given technology being used by other institutions, government agencies and private corporations. Multiple manufacturers and distribution sources are also good indicators of a widely-used technology.

 

Integrity: Integrity is a factor indicating the ability of electronic storage media to protect against and correct data corruption. The use of parity bits, error correcting codes, checksum algorithms, physical and digital access controls, and other measures help ensure that data is not corrupted. The media format used and how data is stored on it determines which of these measures can be applied. Compressed, de-duplicated, or encrypted data is more susceptible to corruption as non-functional software or an uncorrectable error can make a large amount of data unreadable.

 

Assessing Integrity: All storage media have listed specifications that indicate what types of integrity protection are possible, but one must also consider the source. New technologies frequently come with lofty manufacturer claims which should be considered critically until independently verified through outside testing or use.

 

Redundancy: Redundancy is a factor that indicates the data stored on electronic storage media is being replicated to ensure recovery of data in the event of a data loss incident. A minimum of one additional copy of any data representing electronic records must be maintained to protect against such a loss. At least one copy should be stored in a geographically separate location. Depending upon cost and performance needs, multiple types of storage may be used, such as a hard drive for the primary copy and magnetic tape for the backup copy.

 

Assessing Redundancy: Redundancy can be assessed by determining if you have one additional copy in a geographically separate location. IT policies should ensure that all electronically stored data will be restorable in the case of total loss of the primary storage environment.

 

The following is a list and descriptions of storage media formats currently recommended by the Illinois State Archives for use in storing electronic records.

 

Format	Short term	Long term	Do Not Use
HDD	X	X
SSD (Internal)	X	X
Magnetic Tape	X	X
Cloud Storage	X	X
Optical (All Types)	X
USB Flash			X
Obsolete media			X

 

STORAGE MEDIA

 

LONG-TERM RETENTION

 

The following formats are considered acceptable choices for the retention of records greater than a decade.

 

Magnetic Tape:

 

A durable recording medium which uses a plastic film coated with magnetic material to record information, magnetic tape has been used to record computer data since the 1950s. Early formats of this medium consisted of open reel tapes, but modern varieties all use a cartridge of some sort.

 

The most widely used current version is LTO (Linear Tape-Open), which is based on open standards, as opposed to several proprietary competitors. LTO is currently in its 6th generation, with LTO-6 introduced in 2012. LTO-6 tapes have an uncompressed storage capacity of 2.5 Terabytes (TB). A number of different companies currently manufacture LTO tapes and drives, and LTO technology now accounts for close to 90% of the data tape market. LTO drives have some backwards compatibility, being able to read tape from two generations past and write to tape one generation past (an LTO-6 drive can read LTO-4, 5 and 6, and write to 5 or 6). Older versions of LTO tapes can remain viable for a few decades so long as users possess the appropriate drive, but it is best to migrate to newer versions every two generations to avoid potential loss of access.

 

Other current tape technologies are the Oracle Storagetek T1000X series and IBM TS1140 line. Both offer higher capacities and faster transfer speeds than LTO, but at a higher cost per GB. They are both proprietary formats, with drives and automated libraries available only from Oracle and IBM, respectively. Both companies are long-established and stable, but if either chooses to discontinue their tape technologies users will have no choice but to switch formats entirely. LTO thus remains the safer choice in terms of adoption and support.

 

Pros of magnetic tape:

 

Durability up to 30 years (best practice migration in 8-12 years)

 

High capacity, low cost compared to other storage technologies

 

Widely used, mature technology

 

High transfer rates, low error rates

 

Low energy consumption

 

Cons of magnetic tape:

 

Slow access time (average 50 seconds)

 

Wears out faster with frequent access

 

Hard Disk Drive (HDD):

 

Hard disk drives store data on a stack of rapidly spinning metal disks coated in magnetic material. HDDs have been used for primary storage in computers since the early 1960s, and are used in the vast majority of personal computers and servers today. They can be internally mounted or connected externally. For stability and monitoring it is recommended that only internal HDDs be used for long-term records storage, with external drives being used for file transport or backup duties only. Due to their extensive use HDDs are inexpensive, and are available from a wide variety of manufacturers. They can be prone to unexpected failures, however, so active monitoring, regular media refreshment and appropriate backups must be used to ensure the safety of the records stored within.

 

To help manage the inherent risks associated with HDD technology a RAID (Redundant Array of Independent Disks) setup should be used. RAID uses a battery of drives that are interlinked and automatically duplicate data across the drives, thus protecting content from loss. There are different levels of RAID which correspond to greater or lesser amounts of duplication, but for records preservation RAID 6 or 10 are recommended. Both involve high levels of fault tolerance, meaning one or more drives in the array could fail at once with no irretrievable loss of data.

 

Pros of HDD:

 

Rapid access to content

 

High capacity, low initial cost

 

Widely used, mature technology

 

Easily scalable through networking

 

Cons of HDD:

 

Short life span (average 4-6 years, best practice migration in 3-5 years)

 

High energy consumption

 

Expensive for large-scale applications or for long-term content

 

Higher error rate than tape

 

Solid State Drive (SSD):

 

A flash memory storage device first developed in the mid-1990s with no moving parts that typically uses the same shape, interface, and power source as standard hard drives. Data is stored in static electronic chips rather than on magnetized spinning platters. This results in much shorter time required for drive start-up, read, random access, latency and data transfer as well as reduced energy use, but at a cost up to ten times that of standard hard drives. As the price continues to drop in the coming ten to twenty years, solid state drives are expected to replace standard hard drives as the primary storage medium for laptops, desktops, servers, mobile devices, and external storage.

 

Pros of SSD:

 

Resilient to physical shock

 

Lower failure rate compared to standard hard drives

 

Fast access time (<0.1 ms)

 

Low energy consumption

 

Cons of SSD:

 

High cost compared to other storage technologies

 

Limited lifetime due to limited number of times a storage block can be written

 

Susceptible to data loss due to power outages or long-term unpowered storage

 

Maturing technology with most commercial availability beginning in 2007

 

Cloud Storage:

 

"Cloud storage" refers not to a particular type of media, but a method for managing data using networked storage providers. Cloud hosting companies provide technical infrastructure which often spans across many geographical areas, providing high levels of redundancy and remote access for customers. While not a new concept, commercial cloud storage has only seen widespread adoption by both private and public entities in the last decade. There are many cloud service providers but much of the commercial market is dominated by companies like Amazon, Microsoft and Google. Cloud storage services can range from bare-bones warehousing with minimal security and upkeep to highly customized management of data, with integrity checks, enhanced security and faster access speeds.

 

Cloud storage in general has shown itself to be very reliable regarding the preservation of data. Nonetheless, greater concerns arise surrounding the protection of that data from inappropriate access. Data breaches can and do happen, and network security must be a primary focus for any agency wishing to use cloud storage for their records. Only established providers with proven track records should be used, but they do not necessarily need to be one of the large corporate entities. Many smaller cloud providers actually use one of the large hosts, and simply add their own layers of services on top of the bare storage. Cloud storage providers may not automatically provide long-term preservation services such as fixity checks, audit logging or creation of additional metadata, so agencies must still plan on performing these tasks themselves or specifically contracting cloud providers to do so.

 

Pros of cloud storage:

 

Highest level of duplication and geographic redundancy

 

Easy access from multiple locations

 

Keeps up with technology trends without additional investment

 

Trades unpredictable maintenance costs for known subscription fee

 

Can be cheaper than investing in own technology

 

Cons of cloud storage:

 

Relatively higher risk of security breach

 

Laws may prevent the storage of certain types of sensitive data in the cloud

 

Some providers may not be reliable or may go out of business

 

Less control over data / loss of physical custody

 

SHORT-TERM RETENTION (10 years or less)

 

The following formats are inappropriate for the long-term storage of electronic records, but may be used for short-term storage of records.

 

Optical Media:

 

A thin, circular, plastic disc with a reflective layer upon which data is stored in the form of pits and lands. The reflective layer typically resides on the label side of the disc facing inward although double-sided and dual-layer discs are also available. It can be engraved (read-only), dye-based (write-once), or alloy-based (rewritable). A laser is used to read data from the spinning disc based on changes in the reflection caused by the pits and lands. Several forms of optical media are widely adopted and supported; descriptions of the most common forms are provided below. Optical media is subject to damage due to scratches or breakdown of the recording dye, although proper storage and handling, regular migration to new media, and use of a gold reflective layer can mitigate these risks. Many types of writable optical media also use volatile organic dyes to store information, and can degrade over time. As cloud and network-based storage become common, optical media usage is expected to decline.

 

CD: The Compact Disc was originally developed in the early 1980s, evolving from the older LaserDisc format, and it is still widely supported. A standard CD is 4.7 inches in diameter and can hold up to 80 minutes of audio or 700 MB of data, although smaller and non-round shapes also exist. The most popular CD formats include CD-ROM (read-only), CD-R (write once), and CD-RW (rewritable). Both the drives and recordable media are speed rated, indicated as a multiplier of 1x (1200 Kbps). The writing speed as set by the recording software should not exceed the rated speed of the recording media to prevent data corruption. The CD format includes strong error correction coding to prevent data loss due to scratches, fingerprints, or other environmental contaminants. CDs suffer from low capacity compared to other modern storage technologies.

 

DVD: The Digital Versatile Disc was developed in 1995. The DVD uses the same dimensions as a CD but offers a standard capacity of 4.7 GB or 8.5 GB for dual layer formats. Most DVD players can also read CDs. The most popular DVD formats include DVD-ROM (read-only), DVD-R and DVD+R (write once), and DVD-RW and DVD+RW (rewritable). The plus and minus formats require different recording media and drives to write. Most DVD players can read all DVD and CD formats while some can also write in all formats. Similar to CDs, both the DVD drives and recordable media are speed rated, indicated as a multiplier of 1x (10.5 Mbps). The writing speed as set by the recording software should not exceed the rated speed of the recording media to prevent data corruption. The DVD format also includes strong error correction coding to prevent data loss due to scratches, fingerprints, or other environmental contaminants. DVDs should not be used for long-term data storage, because their reliability over time has not yet been adequately demonstrated.

 

Blu-ray: The Blu-ray Disc was developed in 2006. Blu-ray uses the same dimensions as a CD and DVD but offers a standard capacity of 25 GB or 50 GB for dual layer formats. Most Blu-ray players can also read CDs and DVDs. The most popular Blu-ray formats include BD-ROM (read-only), BD-R (write once), and BD-RE (rewritable). Similar to DVDs, both the Blu-ray drives and recordable media are speed rated, indicated as a multiplier of 1x (36 Mbps). The writing speed as set by the recording software should not exceed the rated speed of the recording media to prevent data corruption. The Blu-ray format also includes strong error correction coding to prevent data loss due to scratches, fingerprints, or other environmental contaminants. Like DVD, Blu-ray discs have not been adequately evaluated for long-term stability.

 

M-Disc (Millenial Disc): A relatively new player on the optical front, M-Disc technology has only been available since 2009, and has yet to see wide-spread adoption. M-Discs use a proprietary "stone-like" material in the storage layer, sandwiched between plastic discs. Currently both DVD and Blu-ray variants are available, with capacities similar to standard varieties of those discs. Special M-Disc drives are required to write data to the discs, but standard DVD or Blu-ray drives are able to read data from them. Millenniata, the company behind the format, claims such discs have a projected lifespan of 1000 years. Independent stress tests have shown that M-discs are indeed more resistant to environmental degradation than traditional varieties, but the life span statement cannot be corroborated. Even if such claims are accurate, optical drives are already losing ground to newer storage technologies, and will likely be rendered completely obsolete within decades. It is possible that M-Disc technology will be much more widely adopted in the future, but at this juncture it is too limited to consider as a long-term storage option.

 

Pros of optical media:

 

Convenient and portable

 

Widely supported formats available

 

Low energy consumption in storage

 

Cons of optical media:

 

Not well-suited for frequent writing or for fast read access from multiple discs

 

Limited capacity per disc compared to other modern storage

 

Widely varying lifetime depending on use and care

 

Aging technology being slowly replaced by flash media and cloud storage

 

DO NOT USE

 

The following formats should not be used for the short-term or long-term storage of electronic records. If records are currently stored on such media they should be migrated onto appropriate media to avoid their permanent loss.

 

USB Flash Drive:

 

A rewritable portable data storage device developed in 2000 with no moving parts that connects to a computer using the Universal Serial Bus interface. Data is stored electrically in chips using power from the USB interface itself. USB flash drives offer many of the same performance benefits and limitations as Solid State Drives but typically have a smaller capacity. USB flash drives come in a variety of shapes and sizes from standard three inch "stick" to novelty shapes and even some barely larger than the USB plug. USB flash drives quickly displaced floppy disks as the preferred means to quickly write and transport data but may themselves be displaced as cloud and network-based storage become common. These portable devices are useful for short-term information sharing, but are far too easily lost or compromised to serve as reliable storage for electronic records.

 

Pros of flash drives:

 

Convenient

 

Durable

 

Widely supported

 

Fast transfer rate (up to 5 Gbps with USB 3.0)

 

Low energy consumption

 

Cons of flash drives:

 

Easily misplaced or stolen

 

Limited write protection

 

Limited lifetime due to limited number of times a storage block can be written

 

Older Magnetic Tape Formats:

 

Due to lack of support for obsolete technology, agencies should avoid any tape formats which have been discontinued. Caution should also be exercised when using older variants of current tape technology such as LTO-1 or LTO-2, as backward compatibility of LTO technology only goes back two generations. Any records currently stored on older formats such as these should be migrated forward to more current versions to ensure continued accessibility.

 

Any Size Floppy, ZIP, JAZ Disk:

 

A wide array of magnetic diskette technologies have now been almost completely abandoned in favor of more current storage options. Even if still technically usable, such disks should not be used for storage of electronic records.

 

ADDITIONAL RESOURCES

 

Best practices for a digital storage infrastructure for the long-term preservation of digital files (Digitizing Contemporary Art)

 

Best Practices for Media Selection and Migration (University of Illinois)

 

Selecting Storage Media for Long-Term Preservation (UK National Archives)

 

FAQ about Optical Storage Media (National Archives)

 

(Source:  Added at 39 Ill. Reg. 2652, effective February 9, 2015)