shredos vs. diskpart clean all

[cylkee]

I'm new to disk 'shredding' so perhaps this might be a naive question, but is diskpart> clean all the equivalent of hdparm -N to max the sectors (for overwriting any HPA) followed by shredos 'Fill with Zeros'.

My use case is basically for potential resale of some HDDs - I'm not on the run from the NSA, but I am interested in shredding any HPA.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-vista/cc766465(v=ws.10)?redirectedfrom=MSDN

From the above link:

clean [all]

Removes any and all partition or volume formatting from the disk with focus. On master boot record (MBR) disks, only the MBR partitioning information and hidden sector information are overwritten. On GUID partition table (GPT) disks, the GPT partitioning information, including the Protective MBR, is overwritten; there is no hidden sector information.

all

Specifies that each and every sector on the disk is zeroed, which completely deletes all data contained on the disk.

With shredos, I followed the Ventoy ISO route but I've had some trouble with hdparm -N as it keeps throwing back this on whichever disk I've tried. The following response is on a Samsung SP1604N UATA 160GB, in a USB enclosure, already quick cleaned so no partitions or anything:

SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 max sectors = 0/1, HPA is enabled

Also tried hdparm --dco-restore (with the confirm flag) but it responded with something similar and had no effect.

[PartialVolume]

I'm new to disk 'shredding' so perhaps this might be a naive question, but is diskpart> clean all the equivalent of hdparm -N to max the sectors (for overwriting any HPA) followed by shredos 'Fill with Zeros'.

Yes it does look like it's the same however I've not tested or used diskpart clean all

With shredos, I followed the Ventoy ISO route but I've had some trouble with hdparm -N as it keeps throwing back this on whichever disk I've tried. The following response is on a Samsung SP1604N UATA 160GB, in a USB enclosure, already quick cleaned so no partitions or anything:

The chipset being used in the USB enclosure could be the issue. It may not fully support ata pass through https://hardwarerecs.stackexchange.com/questions/6970/what-usb-to-ide-adapter-that-can-send-the-ata-secure-erase-command

As a first step I would remove the drive from the USB enclosure and plug it into a PC motherboard with an IDE connector. There is a good chance the hdparm -N /dev/xxxx command will then work properly. Also when you run shredos or nwipe if you are not seeing the serial number for the drive it may also be an indication that the USB enclosure chipset doesn't support ATA pass through

[PartialVolume]

Some more info here about ATA pass through and USB enclosures and interfaces in particular a list of USB chipsets that support ATA pass through.

[PartialVolume]

and here https://www.smartmontools.org/wiki/USB

[cylkee]

Thanks for your quick replies @PartialVolume. And thanks for the tips about the USB enclosures and support for ATA passthrough - I think this is likely. Not sure if/when I'll get to test with a PC motherboard. I'm rolling with diskpart clean all for now. Would appreciate any other opinions from others here re. the equivalence question :)

[PartialVolume]

I'm rolling with diskpart clean all for now.

Couple of points.

• I would expect diskpart to also be unable to reset the HPA on a USB adapter that doesn't allow ATA pass through. Hopefully it doesn't silently ignore that failure leaving you to assume the HPA is reset. I'm not on a Windows system at the moment so can't test that theory.
• diskpart doesn't appear to have an option to verify the wipe. I don't know if that's because it automatically verifies the disk after the wipe. If in doubt I would either use a hexeditor and take a quick look at the disk and makesure at least the start/end/middle sectors are all zero'ed. And if you want a more thorough verification run shredos and select the Verify Zeros method which will check every single sector.

[cylkee]

Edit: TL;DR - no, not equivalent.

--

Thanks again for your tips.

So I shrugged off my laziness and dug out an old PC. Initially I wondered why the boot loader was crashing at 100%, then I realised that the 64bit ISO I used on my laptop aint gonna work on this 32bit heap of junk! 🤣

I connected the Samsung SP1604N UATA 160GB which has earlier had the diskpart clean all treatment. Happy to say that hdparm -N reported max/max sectors and HPA disabled. And a quick glance at hexedit followed by a shredos Verify Zeros was also successful. Great, I thought...

However, it appears that diskpart clean all does not, in fact, disable or overwrite HPA. I know this because all my other (older) drives, earlier given the diskpart clean all treatment, were reporting unequal sectors using hdparm -N saying HPA is enabled (and a couple were detecting greater sectors than max(!), saying HPA setting seems invalid (buggy kernel device driver?)). So it would appear the Samsung SP1604N UATA 160GB never had HPA enabled. Presumably, diskpart does not know about HPA because it's working at the OS level..?

Unfortunately, on these other drives I'm unable to change the max sectors, for example hdparm -N p257980288 /dev/sda responds:

/dev/sda:
 setting max visible sectors to p257980288 (permanent)
SG_IO: bad/missing sense data. sb[]: 70 00 05 00 00 00 00 0a 04 51 ef 01 21 04 00 00 00 7f 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 max sectors   = 39876480/257980288, HPA is enabled

I also tried a --dco-restore but a similar bad/missing sense data error was given. I tried a power cycle and tried -N again. Also tried quitting nwipe before attempting, for example hdparm -N p257980288 /dev/sda.

Any suggestions on how to get past this? Perhaps something to configure in the BIOS? - the HDDs are just being auto detected at the moment. Access Mode is auto, but other options are CHS, LBA, Large. And then if Manual I can set Cylinder, Head, Precomp, Landing Zone, Sector. In another screen PIO is set to Auto, with options for modes 0 to 4. In fact, here's the manual https://www.manualslib.com/manual/628283/Shuttle-Fn41v3.html?page=64#manual

Or, am I over-paranoid about HPA? The extra space is not a concern. Suggestion is that user data is never normally added to HPA https://superuser.com/questions/1613602/can-the-hpa-and-dca-contain-user-data-from-the-filesystem

Maybe helpful info:

IDE connected HDDs, three at a time, IDE0 master and slave, IDE1 master
Mainboard FN41V3 Shuttle Inc., circa 2004
Phoenix AwardBIOS
AMD CPU
Ventoy 1.0.78
shredos-2021.08.2_21_i586_0.32.023_20220126.iso

I have another older PC in bits (circa 2000) - is it worth trying that?

By the way, I can say that shredos is way more informative during the wipe/verify process. diskpart clean all just gives you a flashing cursor and then a success or error when it's done. And, as you say, no way to check or perform verification.

[PartialVolume]

HPA wasn't introduced until the ATA/ATAPI-4 (aka ATA-4, Ultra ATA/33) standard in 2001. So if the drive or the controller in the PC predate 2001 or are from about that time then possibly HPA won't be accessible. see the table in this wiki page Features introduced with each ATA revision

So assuming that the IDE controller in the PC is at least ATA/ATAPI-4 or later and likewise the disk then in theory it should work.

Also some bios firmware from that time has limitations in regards to size of disk. So could be a bios issue although I've got my doubts, but you never know. See History of BIOS and IDE limits In particular the 33.8GB (1999) and 137GB (2001) limitations.

I have a 80GB maxtor on a Pentium 4 32 bit from that period that behaves the same way in reporting a unlikely disk size and blaming it on a buggy driver. Despite that shredos wipes the 80GB disk without any issues.

BTW, thanks for investigating this, when I come to write the HPA code into nwipe I'm starting to realize just how many edge cases I may have to take care off. So any feedback from others that are using or even trying to use hdparm -N to identify and set a HPA is all useful info to make the nwipe implementation of it more reliable.

[PartialVolume]

@cylkee So investigating a bit further. Above I mentioned that I had a Pentium 4 with a 80GB Maxtor drive and hdparm -N /dev/sda was reporting a suspect HPA value which sort of implied HPA was enabled and complained about maybe a buggy driver?

Well this fixed it for me.

hdparm --dco-setmax xxxxxxxx /dev/sda (or whatever your drive name is)

Replace xxxxxxxx with the total number of sectors expected for your disc. The second number reported by hdparm -N /dev/sdx

Now rerunning hdparm -N /dev/sda reports 160086528/160086528 HPA is disabled. All Good.

[PartialVolume]

However, the hdparm --dco-setmax xxxxxxxx /dev/sda command changes both values reported by hdparm -N.

Also hdparm -Npxxxxx /dev/xxx which is used to set the HPA size does not work. Possibly because it needs a space between -N and pxxxxx, i.e hdparm -N pxxxxx /dev/xxx

I'm currently wiping the drive after having apparently reduced its size with the --dco-setmax command. I'll let you know if it honours the HPA value or whether it ignores it and wipes the maximum number of sectors.

[PartialVolume]

So ShredOS does respect the HPA that is set. However using the hdparm --dco-setmax has highlighted something interesting. It sets both sides of the HPA result return by hdparm -N. This means you can't rely upon the HPA not enabled message. You can still hide sectors. You need to know the actual physical number of blocks for the drive and if hdparm -N does not return that value you need to set it using hdparm --dco-setmax xxxxxxxx /dev/sda.

You can then use hdparm -N pxxxxxx /dev/sda Note the space between -N and pxxxx, without that space the command fails. to reduce the disc size thus enabling HPA if you so wanted. For wiping you would not do that.

So long story short, there are certainly a few things to think about when using hdparm to check and reset the HPA if necessary.

One other thing, When changing the HPA value you obviously need to restart nwipe preferably I would reboot shredos before wiping the disc and recheck the HPA value after the reboot.

[cylkee]

hdparm --dco-setmax xxxxxxxx /dev/sda (or whatever your drive name is)

Replace xxxxxxxx with the total number of sectors expected for your disc. The first number reported by hdparm -N /dev/sdx

Thanks again. So I did this shortly after you first wrote about it. Did a wipe and it came back with an fdatasync: Input/output error, but I had not rebooted shredos, if that even was the cause. Now doing another wipe.

HPA wasn't introduced until the ATA/ATAPI-4 (aka ATA-4, Ultra ATA/33) standard in 2001

Basically, these are the other HDDs:

• Maxtor, DiamondMax Plus 9 6Y120L0, 120GB ATA/133, Aug 2003

  • reported the buggy driver thing, was the one successful with --dco-setmax
  • manual says 240121728 sectors
  • hdparm -N now says 240121728/240121728, HPA is disabled - but you suggest we can't rely on this statement, which might be true given that this figure is the same as the manual, and what we see of manual comparisons with -N below.
  • geometry = 14946/255/63 which by my calculation is 240107490..?

• Maxtor, 5T060H6, 61GB, ATA-5, Apr 2001

  • --dco-setmax had no effect - bad/missing sense data etc.
  • manual says 120103200 sectors
  • hdparm -N says 120103200/254320928, HPA is enabled - attempt to alter with hdparm -N p254320928 fails with bad/missing sense data
  • geometry = 7476/255/63 which by my calculation is 120101940..?

• IBM Deskstar, IC35L040AVER07-0, 41GB, ATA-5, Mar 2001

  • --dco-setmax had no effect - bad/missing sense data etc.
  • manual says 80418240 sectors
  • hdparm -N says 80418240/264967616, HPA is enabled - attempt to alter with hdparm -N p264967616 fails with bad/missing sense data - staggering that the apparent max sectors is almost 3.3 times what is available, although I understand this could be for compatibility reasons at the time, but probably also a bit of marketing, i.e. a line of identical physical kit, with different labels/restrictions.
  • geometry = 5005/255/63 which by my calculation is 80405325..?

• Quantum Fireball LCT 15, 20GB, ATA-4?, mid 2000

  • --dco-setmax had no effect - bad/missing sense data etc.
  • manual says 39876480 sectors
  • HPA is enabled

• Seagate, ST3491A, 428.1MB (yup!), 1994?

  • reported the buggy driver thing, was not successful with --dco-setmax, perhaps unsurprisingly
  • manual says 836070 sectors
  • https://www.ebay.co.uk/itm/224839924293 <- lost for words...

So if the manuals are not revealing the true physical geometry of the HDDs, how can we ever know without taking them apart? And am I misunderstanding the geometry output of hdparm with no flags?

[PartialVolume]

Thanks for replying with with so much detail, much appreciated. I can confirm everything you have said from my own tests on various drives:

1. If changing HPA using sudo hdparm -N pxxxxxxxx /dev/sdx always reboot because otherwise the OS & many commands will use the previous hard disk size that was noted when the OS was booted.
2. sudo hdparm --dco-setmax xxxxxxxxxx /dev/sdx works on some drives not on others
3. On some, maybe older drives, the second value returned by sudo hdparm -N p xxxxxxxx /dev/sdx cannot be relied upon. In which case you need to know from the disk specifications what the max number of sectors are.
4. Calculating the absolute max sectors from drive geometry doesn't match the reported real number of sectors reported by sudo hdparm -N /dev/sdx so you can't rely upon that method to determine real max sectors.
5. On some drives if the second value returned by sudo hdparm -N /dev/sdx is reported by hdparm as suspect then in some cases this can be corrected by the sudo hdparm --dco-setmax xxxxxxxxx /dev/sdx command and then sudo hdparm -N pxxxxxxxx /dev/sdx works.
6. On some older drives sudo hdparm -N /dev/sdx can report both values the same and that HPA is disabled when in fact it is not.

It's an interesting dilemma for a program such as nwipe to accurately determine the true number of sectors on many different types of drives in order to determine whether the HPA is active.

Here's an idea. In trying to determine the real max sectors perhaps the only reliable way is to have some C code change the HPA reported value so it is larger, then try to read the sector using a low level call avoiding any OS libraries so that we don't want to have to reboot each time we change the HPA. If the sector read fails, then we use a binary sort method to determine the next lower HPA setting, i.e half way between the known good sector and failed sector. Repeat this until we find the largest addressable sector. There could be a few issues with this method such as if there is a long delay before the disk reports a read fail. This could make the process slow however using the binary sort method to determine highest addressable sector is the most efficient way in terms of reducing the number of sectors we are reading. The only way to really know whether this would work is if I write a standalone program and run it on as many drives as possible. It's output would show the initial reported values provided by hdparm then the max addressable sector that it's determined itself. If it works reliably compared to only using hdparm then the code could be incorporated into nwipe.

[PartialVolume]

Here's an idea. In trying to determine the real max sectors perhaps the only reliable way is to have some C code change the HPA reported value so it is larger, then try to read the sector using a low level call avoiding any OS libraries so that we don't want to have to reboot each time we change the HPA. If the sector read fails, then we use a binary sort method to determine the next lower HPA setting, i.e half way between the known good sector and failed sector. Repeat this until we find the largest addressable sector. There could be a few issues with this method such as if there is a long delay before the disk reports a read fail. This could make the process slow however using the binary sort method to determine highest addressable sector is the most efficient way in terms of reducing the number of sectors we are reading. The only way to really know whether this would work is if I write a standalone program and run it on as many drives as possible. It's output would show the initial reported values provided by hdparm then the max addressable sector that it's determined itself. If it works reliably compared to only using hdparm then the code could be incorporated into nwipe.

Forget that, it won't work. I just found you can't keep changing the HPA without power cycling the drive each time. Perhaps the first thing to do in regards to adding HPA detection to nwipe is simply identify whether the disc appears to have a HPA and flag it on the drive selection screen in which case you then have the option to manually correct it it you want.