Storage

Common IDE Categories

Early IDE (ATA-0 & ATA-1)

• Commonly used with 286, 386, and early 486 computers
• IDE for 16bit ISA controllers, connects IDE storage devices directly to the ISA/AT bus
• Some storage devices support multi-sector transfers. If the controller device driver enables this feature, max thoughput can be increased by 35%.
• ISA systems are limited to PIO modes 0 & 1, rarely seeing transfers > 2.2MB/s at 8 Mhz.
• Caching controllers can achive transfer rates up to 4MB/s
• Storage transfer rates increase if ISA transfer clocks are increased.

XT-IDE

• A card that allows IDE devices to work from an 8 bit ISA slot
• Also used to describe storage devices that support a rarely used 8 bit transfer protocol that's slightly different than normal IDE
• Also used as shorthand for "XtIDE Universal BIOS", which is an ISA Option ROM for PCs that is used to overcome IDE BIOS limitations.

EIDE, Fast ATA (ATA-2 & ATA-3 )

• Common between 1993-1997: From 486 socket 2 to early Pentium socket 7
• Drivers and BIOS images released before June 1994 rarely support storage devices > 512MB
• Some VLB controllers allow fixed rate speeds to be configured by jumper, but if the speed is too fast, random data corruption may occur.
• VLB IDE controllers often have hardware or drivers that stop at PIO2 speeds to avoid data corruption, making throughput > 5.5 MB/s difficult to achieve.
• To avoid random data corruption, PIO3 & 4 modes require a working IORDY signal from the storage device to prevent failed transfers.
• PIO3 & 4 modes rarely reach speeds > 10 MB/s because of IORDY overhead.
• Some early IDE storage devices never assert IORDY, so some VLB controllers from this period include a jumper to always assert IORDY so these devices can function.
• The Promise 20630 VLB controller can reach transfer rates up to 12MB/s when multiword DMA transfers (WDMA) are enabled via driver, but it only does DMA into a buffer on the controller, and transfers to main memory still happen via PIO. There is no CPU benefit in multitasking operating systems.
• Most early PCI controllers support WDMA modes, and this allow for background data transfers directly into main memory without a CPU load when using a multitasking operating system.
• ATA3 introduced S.M.A.R.T. — Self-Monitoring, Analysis, and Reporting Technology.

ATA-4, UDMA/33, Ultra ATA/33

• Common between 1998-1999: Pentium II & III, Super Socket 7, Early Athlon
• UDMA2 operated at 33MHz for significant performance increases over previous generations
• 80 connector IDE cables recommended but not required
• UDMA bus master transfer modes significantly improve performance in multitasking operating systems
• Few BIOS directly allow UDMA transfers in DOS without a UMDA driver in config.sys
• Windows 98 requires users to change a setting in Device manager to enable UDMA Bus mastering
• Some Award BIOS from this period require a patch to allow UDMA operation in Win98. https://www.vogons.org/viewtopic.php?p=1272522#p1272522
• Controllers from this period were often paired with BIOS support that support >8.4GB using LBA28 addressing

Mature PATA (ATA-5, ATA-6, ATA-7: Ultra ATA-66, ATA-100 & ATA-133)

• Common between 2000-2010: Later Pentium 3 & Athlon through Core2
• UDMA4 & UDMA5 & UDMA6 require an 80 connector IDE cable. An 80 connector can be simulated by grounding Pin #34 on the storage device. https://www.vogons.org/viewtopic.php?t=77881
• UDMA5 & UDMA6 rarely show significant performance improvement over UDMA4 unless paired with the fastest PATA storage devices and a south bridge that uses >= Intel ICH5, VIA Enhanced-Vlink or AMD HyperTransport.
• Controllers from this period were often paired with BIOS support for storage >128GB using LBA48

More information about PATA ATA is available here: https://en.wikipedia.org/wiki/Parallel_ATA

BIOS Storage Limitations

• July 1994 - 528 MB limit - Original CHS (Cylinders/Heads/Sectors) limit - BIOS before this date rarely support drives over 528MB. As released, these systems were limited to 1024 Cylinders, 16 Heads, 63 Sectors/Track. These systems either don't allow larger values to be entered, or were not stable when larger values are configured.
• May 1996 - 2015 MB limit - INT 13h limit - Some BIOS before this date allow drives up to 2015 MB by updating INT 13h to allow Cylinder values up to 4095. Head limit remained at 16, Sectors/Track at 63. This was not compatible with Microsoft Operating Systems released before Win95 SR2.
• February 1997
  • 4.2 GB limit - ECHS limit (Extended CHS) - Some BIOS before this date allow a 'Large' mode in the BIOS that produces an alternate geometry by doubling the number of heads and halving the number of cylinders shown to DOS until cylinders <= 1024. The limit for this method for drives that report 16 heads is 4032 MB (C=1024, H = 128, S = 63).
  • 7.9 GB limit - Revised ECHS limit. Other BIOS from this period have a 'Large' mode in the BIOS that presents an alternate geometry using multiples of 15 heads, up to 240 heads. This method stops working at 7560 MB (C=1024, H=240, S=63)
  • 8.4 GB limit - Final ECHS limit - BIOS geometry uses head value from the sequence 16, 32, 64, 128, 255 to present an alternate geometry up to 8.4 GB (C=1024, H=255, S=63).
• August 1999 - 33.8 GB limit - BIOS before this date often stored the cylinders as a 16 bit value, so they may have issues when a drive reports cylinders > 65535. Drives from this era often had a jumper to limit cylinders = 65535.
• September 2001 - 128 GB limit - BIOS before this date only used ATA-5, which had LBA28 addressing, where 28 bits were used to identify each LBA sector, limiting drive capacity to 128GB. ATA-6 added an additional 48bit LBA sector field.
• January 2006 - 2 TB limit - MBR Limit - Storage devices configured with a legacy Master Boot Record (MBR) are limited to 2TB per device. Larger capacities require that the storage device is configured with a "GUID Partition Table" (GPT). Few systems with BIOS firmware are able to boot from GPT configurations without using a 3rd party boot loader and a hybrid partitioning scheme. UEFI firmware is usually required to boot from GPT configured storage.

Vendor Specific BIOS Limitations

• Phoenix BIOS 4.03 and 4.04 - The BIOS config screen stops responding when a capacity > 3277 MB is entered.
• Award BIOS Display Bug - Award BIOS between July 1996 and May 1996 work with capacities 2016MB to 8.4GB but fail to display the size correctly in the Setup and POST screens. The capacity is displayed modulo 2016, so 2016MB shows as 0MB, 4031MB shows as 2015MB, 4032MB shows as 0MB, etc.
• Award UDMA Bug - Award BIOS released before March 1999 that support UDMA report eligibility values that prevent DMA from staying enabled in Windows 98. Most BIOS can be patched to correct the issue.

Notes on Storage Limitations

• Hard drives that report a CHS geometry of 16383 Cylinders, 16 Heads and 63 Sectors/Track are indicating that they are larger than ECHS geometry translation allows and that LBA or LBA48 should be used
• Hard drives that report an LBA capacity of 0xfffffff are indicating that they are larger than LBA allows and that LBA48 should be used
• Number of Heads had a limit of 255 in MS-DOS so no BIOS was ever enhanced to allow more than 255 Heads.
• Sectors/Track was limited to 63 in early BIOS and was never addressed because increasing the number of allowable Cylinders & switching to LBA fixed most storage issues.

Overcoming Limitations and Incompatibilities

• Systems with BIOS limitations can be configured to work with larger drives using one of these three techniques:
  • Configure the drive with geometry with Cylinders < 1024, Heads <= 16, and Sectors <= 63 in the BIOS and install drive overlay software that replaces the BIOS routines when the boot sector is loaded
  • Upgrade your BIOS using a new BIOS from the motherboard manufacturer or from a third party such as MR BIOS that works with your motherboard
  • Install an add-in card with an option ROM extension that replaces the BIOS disk routines before BOOT such as XTIDE UniversalBIOS or SIIG Enhanced BIOS
• Drive overlay software is usually the easiest option
• "EZ Drive" overlay software usually works with partition types and drive geometries that are compatible with modern operating systems. This can be helpful if you plan to move storage devices between computers.
• Early LBA drives do not always work correctly with mature LBA controllers. This can be resolved by configuring the drive mode as 'CHS' or 'Normal' in the BIOS.
• Mature ATA6 storage devices may have trouble negotiating a compatible configuration with some early ATA-3 & ATA-4 controllers. This can be resolved by disabling ATA33 or UDMA modes in the BIOS.
• PATA speeds faster than UDMA2 (such as ATA-5, Ultra ATA/66, UDMA3, UDMA/66) require a ground or open circuit on IDE pin #34. Usually this can be done with an 80 wire IDE cable, but can also be accomplished by grounding pin #34.
• Some systems with BIOS before 1992 may have trouble accessing very fast storage devices because the expect reuults to stay in the IO bus longer. Upgrading the BIOS or andding an Option ROM like XTide BIOS usually fixes this.
• Some pins such as 27 & 28 have changed functionality over time. Some early cards connect these pins in ways that later caused unexpected ISA bus issues. https://www.vogons.org/viewtopic.php?t=107640

Common OS Storage Limitations

• Microsoft DOS (including Windows 95/98) require CHS addressing with heads <= 255
• Full support for enhanced INT 13h (Cylinders >= 1024 / disks larger than 8.4GB) arrived with these releases:
  • Windows 95 OEM Service Release 2
  • Windows 2000
  • IBM OS/2 Warp Client version 4.0.14.
  • LILO version 21.4
• Windows 98 require a patch for LBA48 support and is limited to storage devices 128GB without it.
• Linux kernels older than 2.0.39 / 2.2.14 / 2.3.21 require a patch to work with cylinders > 65535.
• Microsoft Operating Systems up through Windows 7 & Server 2008r2 use CHS addressing when booting from device partitioned with a master boot record (MBR)
• Windows XP SP1 x64 is able to mount devices configured with a "Guid Partition Table" (GPT) but cannot boot from them.
• Booting from devices configured with a "Guid Partition Table" (GPT) requires UEFI and:
  • Vista SP1 x64
  • MacOS 10.4
  • Linux 2.6.X

Common File System / Partitioning Limitations

• DOS with FAT12 allowed for 32MB partitions using the largest cluster size
• DOS with FAT16B allowed for 2GB partitions with the default cluster size
• DOS/Win with FAT32 allowed for 2TB partitions with the default cluster size
• Master Boot Record (MBR) limits devices to 2TB with the default sector size and is limited to 4 partitions per device.

Modern Storage for Retro Computers

CF (Compact Flash)

A popular portable storage format developed in 1994 that is compatible with IDE signaling and ATA protocols. Standard has not significantly changed since PATA matured.

• CF devices have an extensive family tree and that comes with a wide spectrum of behavior, especially if you have a device with firmware older than 2006
• Devices labeled "industrial" often work best for retro-computing, supporting the best response times & compatibility.
• "Consumer" devices are often optimized for storing large images files and may not perform as well on random access work loads
• CF continued to grow the ATA specification through in 2010 with the release of the CF 6.0 standard. CF devices with firmware & controllers > 2006 usually offer better performance.
• While some CF devices support speeds faster than 133Mhz/UDMA6, those speeds are not accessible from a PC.

Pros

• Still very common and affordable
• Designed to operate as a PATA SSD in "true IDE mode", so the devices have native ATA support. Connecting these devices to a 40 pin IDE header only requires an inexpensive pass though adapter.
• Portable media that is easy to move between old and new computers
• Rugged form factor that is sturdy and not prone to loss or electrical damage
• "Industrial" compact flash devices made after 2006 usually have very good compatibility, performance, and life expectancies.

Cons

• Compact flash is now a legacy product and new devices are less common than they were 15 years ago.
• Some CF devices report themselves to be "removable media", even when in "true IDE mode", which can cause problems for Windows 2000 & XP installation.
• Few CF devices support ATA block transfer sizes > 1. This can cause lower max transfer rates when compared with other devices that support block sizes > 1.
• Very few CF devices if any support ATA TRIM, making them less desirable for operating systems that support this feature.
• Pin28 issues on old boards. Some pre 1991 IDE controllers cause video corruption with CF's https://www.vogons.org/viewtopic.php?p=1142705

SD to IDE Bridges:

Most SD-->IDE adapters available today are all variations on the Sinitechi device with firmware versions 1.2, 1.3, 1.4, 1.5 on an FC1307 chip. All perform very similarly. Performance can be limited by old or low quality SD media, but generally any <= 128GB SD made since 2010 works fine in these devices.

Pros

• SD media is very common and affordable
• USB SD readers are common, allowing for easy transfers between old and new computers
• Provides good performance on retro-computing workloads compared to legacy spinning disks or CF devices older than 2006.
• MWDMA support is pretty solid.

Cons

• VLB & early PCI EIDE controllers may experience difficulties with PIO modes faster than PIO2 because of the IORDY implementation on the FC1307.
• Communication between the FC1307 and the SD card is limited to 25MB/s, which limits performance on IDE controllers supporting ATA/33 and faster.
• Data loss can occur with sudden power loss because writes are not immediately committed.
• Can operate in an undefined manner when used with SD cards larger than 128GB
• Can enter an "undefined state" if the first partition on the SD cards is not FAT32 or NTFS.
• TRIM is not supported, making these devices less desirable when using operating systems that supports TRIM.
• Slower than SATA SSDs

Disk on Module (Dom):

Pros

• Does not require an adapter or cable, sits right on the IDE block for a less cluttered case
• Most DOM devices offer performance on par with the best SD and CF devices

Cons

• Hardware mod (short pin 34) or Female-Female 40 pin cable necessary to negotiate speeds faster that ATA-4 / UDMA2
• Some DOM devices have issues negotiating ATA protocols with early ATA-2 and ATA-3 controllers
• DOM devices usually have ide latches that firmly connect to the IDE pin-blocks. These latches can stress brittle plastic on old motherboards
• Difficult to connect to a new PC without a Female-Female 40 pin cross over cable

Sata SSD:

Pros

• Best performance, especially when the device has a write buffer
• Usually supports TRIM and wear leveling
• In 2021, NGFF and M2 devices have affordable pricing for builds between 16GB and 64GB
• Usually responds well to an ATA Secure erase command to reset wear leveling

Cons

• You may need one or more adapters if you want to connect your device to a 40 pin IDE controller
• More convenient to move between systems than a DOM, but not as easy as SD and CF
• Sata --> Pata converters usually require Sata 1 protocols and some recent Sata devices lack full Sata 1 support. (e.g. PNY)

Write Amplification:

• Solid state devices can experience "Write amplification" when writing to pages that already contain data. The write is amplified because what would be a simple write on magnetic storage becomes multiple writes in these cases on solid state storage. This causes solid state devices to become slower as they fill up. This issue may shorten expected life span of the device because more flash cell writes are performed than necessary.
• Solid state devices use techniques such as flash address translation, idle garbage collection, wear leveling, over provisioning, and the ATA TRIM command to reduce write amplification and prolong the life of the device.
• Retro enthusiasts sometimes only partitioning a portion of the solid state storage device capacity as a form of over provisioning. For example, if 8GB of storage is needed, a 16GB device with an 8GB partition is used. The unallocated space should help extend the life of the device.
• It isn't clear how any single CF, SD, and DOM devices handles address translation, wear leveling or garbage collection without doing a lot of research on that specific device. These techniques were generally trade secrets and not publicly documented.
• TRIM functionality helps an SSD identify which pages have real data and which pages hold ghosts of deleted files. CF and SD specifications allow for TRIM or "TRIM like" functionality, no CF devices or SD-->PATA bridge have been identified that support it when operating through a PATA interface. Even if they did, few operating systems before 2009 (Windows 7) support the TRIM command, which means even if the device supports TRIM functionality, accessing it from a retro operating system will be difficult.
• For Win98, consider adding "ConservativeSwapfileUsage=1" to your system.ini to reduce unnecessary swapfile writes.
• For WinXP, consider adding extra ram to reduce swap file usage.
• It might be possible to reset the write amplification on a device using an ATA Secure erase command, but not all devices function correctly after this.

Removable Storage Speed Ratings

CF Revisions

• CompactFlash 1.0 (1995) supported PIO2 with capacity up to 137 GB.
• CompactFlash 2.0 (2003) added PIO4 followed by DMA 33 in mid-2004.
• CompactFlash 3.0 (2004) added UDMA 66 (UDMA3)
• CompactFlash 4.0 (2006) added IDE Ultra DMA Mode 6 and UDMA 133
• CompactFlash 5.0 (2010) added LBA 48 for drives larger than 137 GB
• CompactFlash 6.0 (November 2010) added UltraDMA Mode 7 (167 MByte/s), ATA-8/ACS-2 sanitize command, and TRIM support
• CFast (2008) Also called CompactFast, is a SATA based standard with a similar form factor that is not interchangeable with PATA CF devices
• XQD card (2011) Is a PCI-E based standard with a similar form factor than is not compatible with either PATA CF or SATA CFast

SD Speed Ratings

Early SD speed classes only measured minimum write throughput, which does not align with usual computing work loads.

Comparison of SD card Speed Class ratings

Minimum sequential writing speed		Speed Class
	Suggested max. bitrate	Speed Class	UHS Speed Class	Video Speed Class
2 MB/s	15 Mbit/s	Class 2 (C2)	N/A	N/A
4 MB/s	30 Mbit/s	Class 4 (C4)	N/A	N/A
6 MB/s	45 Mbit/s	Class 6 (C6)	N/A	Class 6 (V6)
10 MB/s	75 Mbit/s	Class 10 (C10)	Class 1 (U1)	Class 10 (V10)
30 MB/s	220 Mbit/s		Class 3 (U3)	Class 30 (V30)
60 MB/s	460 Mbit/s			Class 60 (V60)
90 MB/s	700 Mbit/s			Class 90 (V90)

Application Performance Class is defined in SD Specification 5.1 and 6.0 which mandates a minimum IOPS for reading and writing, which more closely aligns with computing work loads.

Comparison of SD card Application Performance Class ratings

Name	Minimum random IOPS		Minimum sustained sequential writing
	Read	Write
Application Performance Class 1 (A1)	1500 IOPS	500 IOPS	10 MB/s
Application Performance Class 2 (A2)	4000 IOPS	2000 IOPS

Storage After IDE/PATA

SATA I

• SATA I controllers worked in "legacy mode".
  • The send and receive commands through x86 I/O space (0x1F0, 0x170) like classic IDE drives.
  • The controller captures the commands from the emulated ports put them into FIS (Frame Information Structures) for the SATA device.
• SATA I storage devices accept ATA commands over FIS and processes them like legacy PATA.
• Some early "SATA" devices were simply PATA devices with a built in SATA-PARA bridge
• SATA devices didn't implement TCQ (Tagged Command Queueing) because TCQ for PATA was not particularly efficient.

SATA II

• SATA II (>= 2004) devices support AHCI Mode (Advanced Host Controller Interface).
• When in ACHI mode they do LBA48 addressing with a slightly updated ATA command set, but the commands are sent directly through FIS to the SATA device, no more legacy PATA port emulation, reducing latency.
• SATA devices from this generation often support NCQ (Native Command Queuing) at a depth of up to 32 commands in flight.
• Some early VIA SATA I controllers only work with SATA II devices that can be configured to force legacy mode.

SATA III

• SATA III offers some refinements of the AHCI command set, but mainly raised the throughput from 3GB/s to 6GB/s which was important for SSD SATA devices.
• Some SATA III devices have trouble falling back to SATA I controllers operating in legacy mode.

NVME

• Common after 2018.
• Uses the same M2 connector as NGFF Sata, but uses different pins.
• NVME devices are native PCIe devices that do no not support ATA or ACHI command sets. That's why don't see cheap NVME - PATA bridges like you do with SATA.
• NVME command set uses linear addressing with up to 64 bit addresses. NVME devices use memory queues implemented in pairs, a submission queue (ASQ) & completion queue (ACQ). Each queue pair is up to 65,535 commands deep, and there can be up to 65,535 queue pairs. Eliminating ATA & ACHI and switching to in-memory queues provides extremely low latency compared to PATA or SATA. Having many queues is beneficial for VM hosts and for processors with many cores.
• Cannot be used as a boot device without UEFI firmware support and Operating System support
• Microsoft provided Windows 7 drivers, and community created Windows XP drivers
• Linux GRand Unified Bootloader (GRUB) has boot support for NVME devices

Optane

• Intel Optane, is a memory technology based on 3D XPoint memory, that offered much lower latency and higher endurance than NAND flash
• Used in some NVME devices, but never achieved a capacity/price ratio to survive in the market

Post PATA Connectors

• SATA was introduced with Sata Data & Power pair connectors that replaced PATA 40 pin ide header and 4 pin molex power
• mSata (aka Mini Sata) was common between 2009 to 2014
  • Devices were most commonly 30mm x 50.95mm
• M.2 Sata (aka NGFF, Next Gen Form Factor) arrived in 2013. Common device sizes are:
  • 2230 - 22 × 30 mm Small laptops, embedded systems
  • 2280 - 22 × 80 mm Most common SSD size
• M.2 has 3 different key patterns:
  • B key, single notch on left, usually SATA
  • M key, single notch on right side, usually NVMe with 4 lanes of PCIe
  • B+M key, two notches, can be SATA or NVMe with 2 lanes of PCIe

Notes not worked into the main document yet

• MBR partitioned storage devices are limited 2TB. Larger devices require GPT (GUID Partition Table). GPT use requires operating system support (Vista SP1, macOS 10.4, Linux version 2.6.x ). Booting from GPT requires UEFI support.
• VIA8237 and VIA8237R SATA controllers are incompatible with SATA-II and SATA-III storage unless the devices can be configured to force SATA 1 operation. The VT8237R Plus is not affected.