Hardware Installation Notes

Tips On Working With 3Ware RAID Controllers

3Ware RAID controllers are dependable, reliable controllers, capable of decent throughput, which are supported by many versions of Linixi. For IDE drives, the 7000 series controllers are often available in the discount/auction market for very little money, especially the 7006-2. For SATA drives, the 9500 series controllers are likewise available for discount prices, especially the 9500S-LP. These notes are geared towards working with those cards, although they are extensible to other models of the 3Ware controllers.

The steps to physically setting up a RAID array using the 3Ware controllers are covered quite well in their instructions. These are additional notes that address topics that aren't covered by the traditional documentation.

Installing 3DM2

Most versions of Linux come with device drivers built-in, for the 3Ware cards. All that remains to be installed is the 3Ware management program which can be installed to monitor and manage the RAID arrays on a system using either commands or a Web-based UI.

To install this package, get the latest version of it from the 3Ware Web site (you can use the 3DM2 supplied for the 9000 series controllers, with the 7000 series controllers):

http://www.3ware.com/support/download.asp.

To get the proper software, one can pick the 9690SA (or 9690SA-4I) series of controllers and then the combined 3DM2 and CLI package. In the past, the version that we have downloaded is the Linux 32-bit version:

3DM2_CLI-Linux_x86-9.5.0.1.tgz.

For this version, unpack the file as Joe User:

tar -xvzf 3DM2_CLI-Linux_x86-9.5.0.1.tgz

Run the resulting binary as root:

su
setupLinux_x86.bin -console

You'll love answering the quesions using only the number keys. We pick /usr/local/3Ware as the install directory (mainly because we can't remember who AMCC is, apart from the local movie theatre). Some versions of the installer always want to use a TLD of AMCC so they will actually put things in /usr/local/3Ware/AMCC, which makes no nevermind to us. But, if you want one less directory level, you could pick /usr/local/AMCC.

We usually want to use "root" as the recipient of the local email messages but the "Wizard" won't allow it (only addresses of the form you@bug.me). But, after it is done "configuring" your system, you can go to /etc/3dm2/3dm2.conf and simply put in "root".

Currently, we are using the 10.2.1 Linux version:

3DM2_CLI-Linux_10.2.1_9.5.4.tgz

For this version, and despite its being meant for Linux, the clever guys at LSI Corp (the new home for 3Ware) have decided to package the 3DM2 CLI as a zip file. So, you're going to need something (like Windoze Zip) that can unpack the zip file. We unpack it on a Windoze box, copy it to a Linux box, and then make a new tar file, which we can then use as normal. This is completely up to you. You could just unzip the package and copy the files to your Linux system.

Once you have the package unpacked, run the resulting shell script as root:

su
./install.sh -i

Answer 'Y' to the "agree to the license" prompt and then pick '0' to install both the CLI and Web interface. The install script does everything else for you. There are no other questions to answer.

If you wish to tune 3DM2, particularly if you want it to use a Web port other than 888, you can directly edit /etc/3dm2/3dm2.conf. We prefer to use port 7080. After editing /etc/3dm2/3dm2.conf, 3DM2 needs to be recycled (see below).

Once the install is done, 3DM2 should start automagically. In addition, it will start itself whenever the system boots. If you want to start/stop it manually, its in /etc/init.d, called tdm2 (that's intuitive):

/etc/rc.d/init.d/tdm2 start
/etc/rc.d/init.d/tdm2 stop

To view the controller's staus from the Web, you need to use HTTPS, not HTTP. The first time around, you need to accept the bogus certificate from the system running 3DM2 (pay no attention to all of the whining that FireFox or IE does). If you picked port 7080 to run it under, try this:

https://localhost:7080/

After a fresh install, the secret passwords are "3ware". You can change them, if you want.

If you receive a connection reset error when you connect to the 3DM2 Web page, you should check out KB16625 for a fix:

http://kb.lsi.com/KnowledgebaseArticle16625.aspx

Monitoring RAID Array Disks with SMART

If the disks used in a RAID array that is attached to a 3Ware controller are SMART enabled, they can be monitored by the SMART daemon smartd.

To monitor the disks in the array, install smartd in the usual manner (either by selecting it from your operating system's package manager or installing it from the source directly). Once this is done, hack the sample configuration file (/etc/smartd.conf) that comes with smartd as shown here, in this case to monitor 3Ware-controlled SATA disks:


#DEVICESCAN -H -m root  (comment out all DEVICESCAN directives)
     .
     .
     .
#
# Monitor the two SATA disks that are connected to the system's 3Ware
# 9500S-4LP, which uses the 3w-9xxx driver.  This is a very silent check.
# It only reports SMART health status if it fails but, in that case, it
# will send an email to root.
#
# Short self tests are scheduled for 5AM on Saturday for the first disk
# and 5AM on Sunday for the second disk.
#
# The long self test, on occasion, has been known to hang the disk
# array (or at least keep it so busy that it appears hung) so it is not
# run at any time.
#
/dev/twa0 -d 3ware,0 -H -l error -l selftest -t -I 190 -I 194 -I 231 \
     -s S/../../6/05 -m root
/dev/twa0 -d 3ware,1 -H -l error -l selftest -t -I 190 -I 194 -I 231 \
     -s S/../../7/05 -m root
  

If your array uses older, IDE disks, hack the sample configuration file (/etc/smartd.conf) that comes with smartd as shown here:


#DEVICESCAN  (comment out all DEVICESCAN directives)
     .
     .
     .
# Monitor the two IDE disks that are connected to the system's 3Ware
# 7006-2, which uses the 3w-xxxx driver.  This is a very silent check.
# It only reports SMART health status if it fails but, in that case, it
# will send an email to root.
#
# Long self tests are scheduled for 5AM on Saturday for the first disk
# and 5AM on Sunday for the second disk.
/dev/twe0 -d 3ware,0 -H -l error -l selftest -t -I 190 -I 194 -I 231 \
    -s L/../../6/05 -m root
/dev/twe0 -d 3ware,1 -H -l error -l selftest -t -I 190 -I 194 -I 231 \
    -s L/../../7/05 -m root
  

You may have noticed, from inspection of the two config file snippets shown above, that the name of the 3Ware device depends on which device driver is employed. The older 3w-xxxx driver names the array device /dev/twen (where 'n' is the device number). The newer 3w-9xxx driver names the array device /dev/twan (where 'n' is, once again, the device number).

In either case, the entire array appears as a single device. It might also be named /dev/sda to appear transparent to existing software and packages but SMART always uses the twa or twe device names.

To monitor and test individual devices in the array, the "-d 3ware,..." parameter is used to tell SMART that the array is a 3Ware array and to select which device is of interest. The devices in the array are numbered starting from zero, up to n-1 where n is the number of physical disks in the array.

This naming convention can also be used with the smartctl command to list or set the SMART information associated with an specific disk. For example, to list all SMART information for the first IDE drive in a RAID array, use this command:

su
/usr/sbin/smartctl -a -d 3ware,0 /dev/twe0

To list all SMART information for the second SATA drive in a different RAID array, use this command:

su
/usr/sbin/smartctl -a -d 3ware,1 /dev/twa0

Recovering From A Disk Failure

When one disk in a 2-disk array goes bad, the following steps should allow you to recover the array (these steps also work for creating a brand new array from a cloned drive):

  1. Move the good disk to Slot 0. You can leave it in Slot 1, if it was already there but, for some reason, some of the controller cards seem to recover better if the good disk is in Slot 0.
  2. Insert a new, blank disk in Slot 1. If the disk isn't blank and has been used in a 3Ware array before, see the steps below about deleting the array and starting over. Alternately, you can put the disk in a standalone Linux or Unix box and use the following command to zero out the first few tracks to make the RAID controller think it is blank (assuming the disk is in the hdc slot):

    dd if=/dev/zero of=/dev/hdc bs=512 count=1024

  3. Boot the machine with the original and new drives in it.
  4. Press Alt-3 to bring up the 3Ware BIOS utility.
  5. Select the new, blank disk and the existing RAID array (i.e. the original disk or disks).
  6. Select "Rebuild".
  7. You can try accepting the default setting to "Halt on source errors", the first time around.
  8. Press F8 to save the changes and continue with booting (answer the obvious questions, first).
  9. Once the machine boots up, the array should rebuild itself after about 10 minutes have elapsed. It may take several hours. Your only absolute clue that it is done is when the activity light on the controller card (a tiny, chip LED just beside the IDE connectors on the 7006-2, or next to the J3 connector on the 9500S-LP) stops flashing. However, while the array is being rebuilt, the activity lights on any removable drive bays, or any lights connected to the status light header pins will be on almost solidly and, if 3DM is installed, you will see an indicator that the array is being rebuilt as well as a percentage progress indicator on the 3DM Web page.
  10. Or, you could just run the rebuild overnight, while you're napping.
  11. Reboot the machine.
  12. Press Alt-3 to bring up the 3Ware BIOS utility.
  13. Observe the state of the array. If the newly added drive is marked unused and the array marked degraded, the array did not get rebuilt properly.

In the event that an array did not get rebuilt properly or if you are reusing a disk that was previously in an array, you can follow these steps to delete the failed/previous array information from the disk and start over (or see the procedure using dd, above).

  1. Shut down the machine.
  2. Remove the original, good disk.
  3. Boot up the machine with only the disk to be deleted in the array.
  4. Press Alt-3 to bring up the 3Ware BIOS utility.
  5. Select the array (which will now be marked incomplete) and then select Delete.
  6. Shut down the system, reinsert the original, good disk and boot the machine back up.

Should an array fail to rebuild properly, you can try the following steps to get it to do so, despite the occasional I/O error.

  1. Make sure all of the disks that will be in the array are installed in the machine. The original, good disk should be in Slot 0, if possible.
  2. Boot the machine.
  3. Press Alt-3 to bring up the 3Ware BIOS utility.
  4. Select the new, disk and the existing RAID array.
  5. Select "Rebuild".
  6. This time, select the "Ignore source errors" setting.
  7. Press F8 to save the changes and continue with booting (answer the obvious questions, first).
  8. Once the machine boots up, the array should rebuild itself after about 10 minutes have elapsed (see above for how to tell when its done).
  9. Reboot the machine.
  10. Press Alt-3 to bring up the 3Ware BIOS utility.
  11. Once again, observe the state of the array. If the newly added drive is now part of the array, you're in business. If it is not or the array is still marked degraded, the array did not get rebuilt properly. You might want to run diagnostics on the disks you're using and/or replace the controller card.

Cloning A Disk

If you'd like to clone a disk for use in another system or with another controller card, this is the preferred method for doing so (although you will be facing a short period of time, while the array is being rebuilt, when you only have one copy of the disk). You can proceed as outlined above in "Recovering From A Disk Failure" to make a duplicate of a good drive from a working array. Start with the original drive in Slot 0 and the clone drive in Slot 1. Follow the steps for recovering from a degraded array.

Once the array with the new drive has been rebuilt, you can shut down the machine and remove the new drive. You should be able to move the new drive to another controller card and proceed to build a working array from it with a second new drive.

Meanwhile, when you insert the original Slot 1 drive back into the system, the RAID controller will degrade the array and you will have to rebuild it again. You should first delete the prior array information from the original Slot 1 drive, as outlined above, either by removing the Slot 0 drive and then deleting the array or by using dd to write zeros over the first few tracks.

Copying A Disk

If you are having problems with an array and want a spare copy of the good drive (e.g. before you do anything to the failing array) without resorting to the controller-assisted cloning method described above or you just want to make a copy of a good drive without exposing yourself during that short period when there's only one good drive in your array, you can make a copy of a drive as outlined below.

Once you've done this, in the case of a failing drive you can do your blasting on the copied drive and save the old drive, just in case your recovery efforts don't succeed and you need to go back. For just straight copies, you can put the original drive back in the array and it should start up without the need to rebuild.

  1. Put the disk to be copied (after a failure, presumably this is the drive that is still good) in the Secondary/Master slot of a Linux machine that has dd, or better yet sg_dd, installed on it.
  2. Put the disk you're copying to into the Primary/Slave slot of the same machine.
  3. Boot the machine and copy the entire original disk to the new disk. This script (named copydisks) can be used to ensure no mistakes:
    
    #!/bin/bash
    #
    # Copy two disks, from /dev/hdc (master) to /dev/hdb (slave),
    # using raw mode for fast copying.  The "rev" option can be used
    # to copy from /dev/hdb to /dev/hdc.
    #
    # Used to ensure accurate copying and no screw-ups from typos, etc.
    # Be warned that whatever disk is mounted on /dev/hdb is gone so
    # exercise caution.
    #
    SourceHD=/dev/hdc
    DestHD=/dev/hdb
    
    if test x"$1" == xrev; then
        echo Reversed!  Copying from /dev/hdb to /dev/hdc!
        SourceHD=/dev/hdb
        DestHD=/dev/hdc
    fi
    
    Sectors=`/sbin/fdisk -lu $SourceHD 2>&1 | grep total \
        | sed -r "s/^.+?total ([0-9]+) sectors$/\1/"`
    /usr/bin/raw /dev/raw/raw1 $SourceHD
    /usr/bin/raw /dev/raw/raw2 $DestHD
    /sbin/fdisk -lu $SourceHD 2>&1
    /sbin/fdisk -lu $DestHD 2>&1
    
    echo
    echo Will copy $Sectors sectors from $SourceHD to $DestHD
    read -p "Is this OK? [n|y] " PromptResp
    if test x"$PromptResp" != xy; then
        echo Responded \"$PromptResp\", exiting!
        exit 1
    fi
    
    echo
    echo Starting copy:
    echo /usr/local/bin/sg_dd bs=512 bpt=16384 count=$Sectors \
    time=1 if=/dev/raw/raw1 of=/dev/raw/raw2
    /usr/local/bin/sg_dd bs=512 bpt=16384 count=$Sectors time=1 \
        if=/dev/raw/raw1 of=/dev/raw/raw2
      

    To do the copy, run:

    ./copydisks

    Note that the disk image may be too big for sg_dd or dd to copy all in one operation, in which case the copy will end short of the whole disk. If that happens, you can restart the copy simply by running sg_dd or dd, beginning from just before where copying gave up. For example:

    sg_dd skip=1234567 seek=1234567 bs=...

  4. After you've copied the whole disk, you can keep it for back-em-up porpoises.

If you want to put the disk into the existing RAID array, use it to build a totally new array for a duplicate machine, or for some other procedure that requires it be connected to a 3Ware controller, you will first need to alter the disk serial number in the first track of the drive.

When the 3Ware controller builds the array, it puts the serial numbers of the drives in the array into a control block that it writes to the first track of each of the array's drives. The act of copying the drive makes the serial numbers be mismatched (i.e. the copy contains the original drive's serial number but the drive it is copied to has a new, different serial number).

To make the copy acceptable to a 3Ware controller, you need to use your favorite disk editing program to locate the original disk's serial number and update it to the new disk's serial number. At that point, you should be able to proceed, using the new disk, as outlined above in "Recovering From A Disk Failure".

Expanding An Array's Size

Expanding an array's size is complicated principally by the fact that the only thing that will touch the array's drives is the 3Ware controller. You could try putting two controller cards in one system, as there is no physical reason preventing it. However, the crap operating system software (especially Linux) out there may not be too keen on booting up this configuration.

Better to use a separate IDE drive as a temporary copy buffer, as outlined in the following steps (incidentally, these steps asume you're expanding a Linux drive, not Windows):

  1. Get a pair of brand new disks and a controller card and put them in a separate machine (either that or shut down and use the machine with the existing array). You will need a machine with an available PCI slot as well as a separate IDE connection, plus a CD or DVD drive. Remove anything else in the way of disks and/or exotic hardware, since you only want the new disks and the single controller card installed in the machine at this time.
  2. You will need a copy of the install CD for the system that is already on the array disk that will be expanded.
  3. Boot the machine.
  4. Press Alt-3 to bring up the 3Ware BIOS utility.
  5. Select the two new disks.
  6. Choose Build to build the array.
  7. Once you've done this, press F8 to save the configuration and proceed with booting from the OS install CD.
  8. Install the identical OS to the one on the old array (e.g. Linux).
  9. Be sure to partition the disk exactly the same as the original array (except that you can grow the last partition to fill the remainder of the array drives). If you change the sizes of any partitions except the last one, you will be in trouble. Here is a typical layout that is often used:
    
      Device   Boot  Start  End   Blocks  ID  System
    /dev/sda1    *      1    65   522081  83  Linux
    /dev/sda2          66   326  2096482  82  Linux swap
    /dev/sda3         327   ...      ...  83  Linux
      
  10. When the install is done, shut the machine down.
  11. Hook up a third disk to the IDE connection. It helps if all disks are identical in size (especially in the case of the RAID array).
  12. Boot the system with either the OS Install disk in CD/DVD drive or a disk partitioner (such as Acronis Partition Expert). If you use the OS Install Disk, proceed to install another copy of the OS on the third, IDE disk until you get to the part where the disks get partitioned. If you use the partitioner, create matching partitions on the third, IDE disk to those on the array being expanded.
  13. Remove the third, IDE drive from the machine and hook it up to the original system (Primary/Master). Or, if you're only using one system, remove the new array and hook back up the original array.
  14. Insert a stand-alone Linux CD (e.g. Knoppix) in the machine's CD/DVD drive.
  15. Boot from this CD. For Knoppix, the boot command is:

    failsafe 2

  16. After the boot is complete, copy all of the partitions, one by one, from the original array to the IDE drive:

    dd bs-65536 if=/dev/sda1 of=/dev/hda1
    dd bs-65536 if=/dev/sda3 of=/dev/hda3

    Note that we're assuming it was a Linux system with three partitions, one of which (/dev/hda2) was the swap partition.

  17. Shut the system down.
  18. Either swap the old array with the new array or transport the IDE drive back to the system that has the new array.
  19. Boot again from the stand-alone Linux CD (e.g. Knoppix), in the machine's CD/DVD drive. For Knoppix, the boot command is:

    failsafe 2

  20. You can copy all of the partitions (excluding swap and the last one) with DD, since they are identical in size. For example:

    dd bs=65536 if=/dev/hda1 of=/dev/sda1

  21. The last partition can be copied using the machine's file system and regular copy commands. This will preserve the size of the destination file system. Or you can just use dd and rebuild the file system later on. The next steps describe how to do the copy using the machine's file system:

    dd bs=65536 if=/dev/hda1 of=/dev/sda1
    mkdir /mnt/in
    mkdir /mnt/out
    mount /dev/sda3 /mnt/out
    rm -rf /mnt/out/*
    mount /dev/hda3 /mnt/in
    cp -rfp /mnt/in/* /mnt/out

  22. Once the copy completes, remove the boot CD.
  23. Shut the system down and disconnect the IDE drive.
  24. Insert the OS install CD, once again, into the CD/DVD drive.
  25. Boot the OS install disk and follow the steps for rescue.
  26. Allow it to find and mount the existing OS.
  27. Verify the integrity of the file systems on the new array:

    fsck.ext3 /dev/sda1
    fsck.ext3 /dev/sda3

  28. Ignore any messages about how bad it is to recover a mounted file system.
  29. Change to the newly mounted file system:

    chroot /mnt/sysimage

  30. Repair the boot loader (assuming you're using grub):

    /sbin/grub
    > find /grub/stage1

    or, if that fails

    > find /boot/grub/stage1
    > root (hd0,0) (use the correct address from the find command)
    > setup (hd0)
    > quit

  31. You should now be able to boot from the newly expanded RAID array.

Migrating A SATA/IDE Drive To An Array

If you wish to migrate the existing SATA or IDE drive or drives of a Linux system to a RAID array you can use whatever method you choose to copy the drive(s) to the array.

The array must be set up, connected to a controller and installed in a system. It can be the existing system or it might be a new, upgraded system that will eventually run the array.

One way to do the copy is to install the original disk and the controller/array in the same system. The array is built and then one boots up a standalone OS (such as Knoppix), from a CD or USB stick, that can be used to partition the array, copy partiton data from the original drive to the array, expand array partitions, etc.

Another way to do the copy is to install the controller/array in a separate system that has a network connection, build the array and then boot up a standalone OS (such as Knoppix), from a CD or USB stick, that can once again be used to partition the array and run lftp. Once the array is partitioned, lftp is used to mirror the file system over the network, one top-level directory at a time. For example:

su
mount /dev/sda3 /media/sda3
cd /media/sda3
lftp 192.168.1.1 -u root
[secret password]
mirror -a /bin bin
mirror -a /etc etc
mirror -a /home home
     .
     .
     .

You probably only need to create a top-level directory for /boot, if your system mounts a separate partition (e.g. /dev/sda1) on /boot. In that case, you'll need to copy the /boot partition separately but, otherwise, you should copy the /boot sub-directory too. Either do:

mkdir boot

or do:

mirror -a /boot boot

You should definitely not copy the /dev and /proc directories, instead just create their place holders (since the OS builds their content on the fly):

mkdir dev
mkdir proc

We have successfully used this technique to migrate a running version of Linux to a new RAID array on a separate system. The only difficulty occurred when the /var directory was mirrored. Some of the lock files in /var/lock and /var/run could not be copied by FTP. The solution was to mirror the directory on the original system, in a user's home directory, and the copy the mirror. On the original system:

cp --preserve -R /var /home/jblow/var_mirror

On the system with the array, under lftp:

mirror -a /home/jblow/var_mirror var

Note that, for FTP to be able to copy all of the system files, you may have to allow root to log in to FTP on the original system. If the original system is running VSFTPD, you need to hack the files: /etc/vsftpd/ftpusers and /etc/vsftpd/user_list and comment out the line with "root" on it, in each of the files. Then, restart VSFTPD:

/etc/init.d/vsftpd restart

If you need to set up a swap partiton, you can do so like this:

su
mkswap /dev/sda2

Once all of the partitions have been created, you must install the boot loader (Linux uses GRUB) on the disk. As above, you can avoid the trouble of setting up the boot sector with the boot loader by just copying the first few tracks of the source disk directly with dd. However, since we've already set up the partition table, we must do the copy very carefully to avoid wiping it out. First begin by copying the boot loader from the original system to a file:

dd bs=512 count=63 if=/dev/hda of=/home/joeblow/BootLoader.dump

Now, on the system with the RAID array:

su
cd /ramdisk
lftp 192.168.1.1 -u root
[secret password]
cd /home/joeblow
get BootLoader.dump
quit
dd bs=446 count=1 if=BootLoader.dump of=/dev/sda
dd bs=2 skip=255 seek=255 count=1 if=BootLoader.dump of=/dev/sda
dd bs=512 skip=1 seek=1 count=62 if=/BootLoader.dump of=/dev/sda

This copies the first 446 bytes of the first sector (a.k.a. the MBR), which contains the boot loader code. It skips over the next 64 bytes, which contain the partition table. Then, it copies the two-byte magic cookie that defines it as the MBR. Finally, it copies the next 62 sectors (a.k.a. the DOS Compat Space), which contain the GRUB 1.5 code. Once that is done, you should have a bootable disk that is partitioned the way you want it.

You might want to test that the system can boot the disk (admittedly, it might not get far but you should at least see GRUB Stage 1.5 running), before you proceed any further.

You should now go and tune up the grub.conf (or menu.lst) file in the /boot/grub directory, the /etc/fstab file, any networking files that will be affected by a change in the NIC (if you're using a new system), and any other files that may be need to get the sytem running.

And, if your system experiences a kernel panic, that says the root and other file systems cannot be found, when the system is booted, it is probably due to the initrd image not containing the 3Ware device drivers. These drivers are included among the kernel modules by default, for all later versions of the Linux kernel, but the initrd image from the original system probably didn't include them. This can be fixed by going back to the original system and building a new initrd image (this must be done on a running system) that includes the appropriate drivers. For the 7000-series RAID controllers, you should do:

/sbin/mkinitrd -v -f --with=3w-xxxx --with=scsi_mod --with=sd_mod \
  /boot/initrd-`uname -r`.img.new `uname -r`

For the 9500-series RAID controllers, you should do:

/sbin/mkinitrd -v -f --with=3w-9xxx --with=scsi_mod --with=sd_mod \
  /boot/initrd-`uname -r`.img.new `uname -r`

The new initrd image should then be FTPed to the new system with the RAID array and renamed to replace the existing initrd image. Your system should then boot with no problems.

Getting Your Data Off The Array

As was mentioned above, the only thing that will touch the disks in the array is the RAID controller. This can present a problem, should you ever wish to have a non-RAID-managed copy of the data (e.g. you'd like a regular, bootable disk that doesn't need the RAID controller to run).

The procedure for doing so is basically outlined above in "Expanding An Array's Size". Essentially, you cofigure a system with a RAID controller, with the array connected, plus a non-RAID (i.e. IDE or SATA) drive.

You boot up the system with your favorite copy software and proceed to create new partitions on the non-RAID drive. Then, you do a partition-by-partition copy, instead of copying the drive directly on a track-by-track basis.

Once the copying is done, you may need to reinstall the boot loader, if the disk is supposed to be bootable. All the steps are described above.