top of page
efvasbaterrcomp

HowTo fix a mdadm problem auto-read-only or resync PENDING: Best practices and common pitfalls



As for the post of Estevan TH.about HDD sentinel which claims to "repair hard disk drive problems" I pretend that there is no repair for hardware problems such as pending sectors in software. Remapping is not repair.For file system problems you have your chkdsk command on a Windows machine.Any free SMART monitoring software can monitor changes at the hardware level but there is not necessarily a SMART warning before any drive failure.




HowTo fix a mdadm problem auto-read-only or resync PENDING



The problem is that the operating system will randomly select to mount either the source, or the copy of the partition. For example, on the first mount action the source partition might be mounted. On the next mount action the copy of the partition might be mounted. Over time this random nature of partition mounting might make files seem to mysteriously appear or disappear depending upon which partition is mounted. Random mounting of the source or the copy of the partition might also cause severe data corruption or loss.


Nodes that connect to that resource later on will not have seen theTrim/Discard requests, and will therefore start a full resync; depending onkernel version and file system a call to fstrim might give the wanted result,though.


One consequence of this is that special directories such as C:\windows may appear as /WINDOWS or /windows (or other things) depending on the precise details of how they were created. In Windows itself this would not be a problem.


SYNOPSISmdadm [mode] [options] DESCRIPTIONRAID devices are virtual devices created from two or morereal block devices. This allows multiple devices (typically diskdrives or partitions thereof) to be combined into a single device tohold (for example) a single filesystem.Some RAID levels include redundancy and so can survive some degree ofdevice failure.Linux Software RAID devices are implemented through the md (MultipleDevices) device driver.Currently, Linux supportsLINEAR md devices,RAID0 (striping),RAID1 (mirroring),RAID4, RAID5, RAID6, RAID10, MULTIPATH, andFAULTY. MULTIPATH is not a Software RAID mechanism, but does involvemultiple devices:each device is a path to one common physical storage device.FAULTY is also not true RAID, and it only involves one device. Itprovides a layer over a true device that can be used to inject faults.MODESmdadm has several major modes of operation:TagDescriptionAssemble Assemble the components of a previously createdarray into an active array. Components can be explicitly givenor can be searched for.mdadm checks that the componentsdo form a bona fide array, and can, on request, fiddle superblockinformation so as to assemble a faulty array.Build Build an array that doesn’t have per-device superblocks. For thesesorts of arrays,mdadm cannot differentiate between initial creation and subsequent assemblyof an array. It also cannot perform any checks that appropriatecomponents have been requested. Because of this, theBuild mode should only be used together with a complete understanding ofwhat you are doing.Create Create a new array with per-device superblocks.Follow or Monitor Monitor one or more md devices and act on any state changes. This isonly meaningful for raid1, 4, 5, 6, 10 or multipath arrays, asonly these have interesting state. raid0 or linear never havemissing, spare, or failed drives, so there is nothing to monitor.Grow Grow (or shrink) an array, or otherwise reshape it in some way.Currently supported growth options include changing the active sizeof component devices and changing the number of active devices in RAIDlevels 1/4/5/6, as well as adding or removing a write-intent bitmap.Incremental Assembly Add a single device to an appropriate array. If the addition of thedevice makes the array runnable, the array will be started.This provides a convenient interface to ahot-plug system. As each device is detected,mdadm has a chance to include it in some array as appropriate.Manage This is for doing things to specific components of an array such asadding new spares and removing faulty devices.Misc This is an ’everything else’ mode that supports operations on activearrays, operations on component devices such as erasing old superblocks, andinformation gathering operations.Auto-detect This mode does not act on a specific device or array, but rather itrequests the Linux Kernel to activate any auto-detected arrays.OPTIONSOptions for selecting a mode are:TagDescription-A, --assemble Assemble a pre-existing array.-B, --build Build a legacy array without superblocks.-C, --create Create a new array.-F, --follow, --monitor SelectMonitor mode.-G, --grow Change the size or shape of an active array.-I, --incremental Add a single device into an appropriate array, and possibly start the array.--auto-detect Request that the kernel starts any auto-detected arrays. This can onlywork ifmd is compiled into the kernel — not if it is a module.Arrays can be auto-detected by the kernel if all the components are inprimary MS-DOS partitions with partition typeFD. In-kernel autodetect is not recommended for new installations. Usingmdadm to detect and assemble arrays — possibly in aninitrd — is substantially more flexible and should be preferred.If a device is given before any options, or if the first option is--add, --fail, or--remove, then the MANAGE mode is assumed.Anything other than these will cause theMisc mode to be assumed.Options that are not mode-specific are:TagDescription-h, --help Display general help message or, after one of the above options, amode-specific help message.--help-options Display more detailed help about command line parsing and some commonlyused options.-V, --version Print version information for mdadm.-v, --verbose Be more verbose about what is happening. This can be used twice to beextra-verbose.The extra verbosity currently only affects--detail --scan and--examine --scan. -q, --quiet Avoid printing purely informative messages. With this,mdadm will be silent unless there is something really important to report.-b, --brief Be less verbose. This is used with--detail and--examine. Using--brief with--verbose gives an intermediate level of verbosity.-f, --force Be more forceful about certain operations. See the various modes forthe exact meaning of this option in different contexts.-c, --config= Specify the config file. Default is to use/etc/mdadm.conf, or if that is missing then/etc/mdadm/mdadm.conf. If the config file given ispartitions then nothing will be read, butmdadm will act as though the config file contained exactlyDEVICE partitions and will read/proc/partitions to find a list of devices to scan.If the wordnone is given for the config file, thenmdadm will act as though the config file were empty.-s, --scan Scan config file or/proc/mdstat for missing information.In general, this option givesmdadm permission to get any missing information (like component devices,array devices, array identities, and alert destination) from theconfiguration file (see previous option);one exception is MISC mode when using--detail or--stop, in which case--scan says to get a list of array devices from/proc/mdstat. -e , --metadata= Declare the style of superblock (raid metadata) to be used. Thedefault is 0.90 for--create, and to guess for other operations.The default can be overridden by setting themetadata value for theCREATE keyword inmdadm.conf. Options are:TagDescription0, 0.90, defaultUse the original 0.90 format superblock. This format limits arrays to28 component devices and limits component devices of levels 1 andgreater to 2 terabytes.1, 1.0, 1.1, 1.2Use the new version-1 format superblock. This has few restrictions.The different sub-versions store the superblock at different locationson the device, either at the end (for 1.0), at the start (for 1.1) or4K from the start (for 1.2).--homehost= This will override anyHOMEHOST setting in the config file and provides the identity of the host whichshould be considered the home for any arrays.When creating an array, thehomehost will be recorded in the superblock. For version-1 superblocks, it willbe prefixed to the array name. For version-0.90 superblocks, part ofthe SHA1 hash of the hostname will be stored in the later half of theUUID.When reporting information about an array, any array which is taggedfor the given homehost will be reported as such.When using Auto-Assemble, only arrays tagged for the given homehostwill be assembled.For create, build, or grow:TagDescription-n, --raid-devices= Specify the number of active devices in the array. This, plus thenumber of spare devices (see below) must equal the number ofcomponent-devices (including "missing" devices)that are listed on the command line for--create. Setting a value of 1 is probablya mistake and so requires that--force be specified first. A value of 1 will then be allowed for linear,multipath, raid0 and raid1. It is never allowed for raid4 or raid5.This number can only be changed using--grow for RAID1, RAID5 and RAID6 arrays, and only on kernels which providenecessary support.-x, --spare-devices= Specify the number of spare (eXtra) devices in the initial array.The number of component devices listedon the command line must equal the number of raid devices plus thenumber of spare devices.After initial array creation, new devices are added to the array using the--add command. If you add devices in excess of the number needed for the array,they are automatically treated as spare devices. For grow mode, it isnot possible to grow the number of spare devices, instead you need togrow (or shrink) the number of active devices in the array. Spare devicesare handled automatically after initial array creation.-z, --size= Amount (in Kibibytes) of space to use from each drive in RAID level 1/4/5/6.This must be a multiple of the chunk size, and must leave about 128Kbof space at the end of the drive for the RAID superblock.If this is not specified(as it normally is not) the smallest drive (or partition) sets thesize, though if there is a variance among the drives of greater than 1%, a warning isissued.This value can be set with--grow for RAID level 1/4/5/6. If the array was created with a size smallerthan the currently active drives, the extra space can be accessedusing--grow. The size can be given asmax which means to choose the largest size that fits on all current drives.-c, --chunk= Specify chunk size in kibibytes. The default is 64.--rounding= Specify rounding factor for linear array (==chunk size)-l, --level= Set raid level. When used with--create, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4, 4,raid5, 5, raid6, 6, raid10, 10, multipath, mp, faulty. Obviously some of these are synonymous.When used with--build, only linear, stripe, raid0, 0, raid1, multipath, mp, and faulty are valid.Not yet supported with--grow. -p, --layout= This option configures the fine details of data layout for raid5,and raid10 arrays, and controls the failure modes forfaulty. The layout of the raid5 parity block can be one ofleft-asymmetric, left-symmetric, right-asymmetric, right-symmetric, la, ra, ls, rs. The default isleft-symmetric. When setting the failure mode for levelfaulty, the options are:write-transient, wt, read-transient, rt, write-persistent, wp, read-persistent, rp, write-all, read-fixable, rf, clear, flush, none. Each failure mode can be followed by a number, which is used as a periodbetween fault generation. Without a number, the fault is generatedonce on the first relevant request. With a number, the fault will begenerated after that many requests, and will continue to be generatedevery time the period elapses.Multiple failure modes can be current simultaneously by using the--grow option to set subsequent failure modes."clear" or "none" will remove any pending or periodic failure modes,and "flush" will clear any persistent faults.To set the parity with--grow, the level of the array ("faulty")must be specified before the fault mode is specified.Finally, the layout options for RAID10 are one of ’n’, ’o’ or ’f’ followedby a small number. The default is ’n2’. The supported options are:’n’ signals ’near’ copies. Multiple copies of one data block are atsimilar offsets in different devices.’o’ signals ’offset’ copies. Rather than the chunks being duplicatedwithin a stripe, whole stripes are duplicated but are rotated by onedevice so duplicate blocks are on different devices. Thus subsequentcopies of a block are in the next drive, and are one chunk furtherdown.’f’ signals ’far’ copies(multiple copies have very different offsets).See md(4) for more detail about ’near’ and ’far’.The number is the number of copies of each datablock. 2 is normal, 3can be useful. This number can be at most equal to the number ofdevices in the array. It does not need to divide evenly into thatnumber (e.g. it is perfectly legal to have an ’n2’ layout for an arraywith an odd number of devices).--parity= same as--layout (thus explaining the p of-p). -b, --bitmap= Specify a file to store a write-intent bitmap in. The file should notexist unless--force is also given. The same file should be providedwhen assembling the array. The file may not reside on a filesystem that isbuilt on top of the array the bitmap file is for or else a kernel deadlockwill occur. This is not a bug, it’s a feature. If the wordinternal is given, then the bitmap is stored with the metadata on the array,and so is replicated on all devices. If the wordnone is given with--grow mode, then any bitmap that is present is removed.To help catch typing errors, the filename must contain at least oneslash (’/’) if it is a real file (not ’internal’ or ’none’).Note: external bitmaps are only known to work on ext2 and ext3.Storing bitmap files on other filesystems may result in serious problems.Note: The choice of internal versus external bitmap can have a drastic impacton performance.While an internal bitmap is the most convenient as it doesn’t requirea device totally separate from the array on which to store the bitmapfile, it has a larger impact on performance than an external bitmap.This is because we can’t predict which device in the array might fail, sowe store a copy of the bitmap on every device in the array when using aninternal bitmap. This means that prior to allowing a write to a section ofthe array that is currently marked clean in the bitmap, we must issue awrite to change the bit for that section of the array from clean to dirty,and must wait for the bitmap write to complete on all of the array devicesbefore the pending write to the array data area can proceed.Especially if the array is under heavy load, these syncronous writes candrastically impact performance. An external bitmap file is less convenient,but there is only one copy of the bitmap, so there is only one bitmap writethat must complete before the pending write to the array data can proceed.In addition, if your bitmap file device is not heavily loaded, and thearray is, then you will notice a considerable performance benefitfrom the fact that bitmap writes are not competing with array reads/writes.The performance impact of this option can be somewhat mitigated byappropriate selection of a bitmap chunk size (next option).--bitmap-chunk= Set the chunksize of the bitmap. Each bit corresponds to that manyKilobytes of storage.When using a file based bitmap, the default is to use the smallestsize that is at-least 4 and requires no more than 2^21 chunks.When using aninternal bitmap, the chunksize is automatically determined to make best use ofavailable space.Note: This option can drastically effect performance of the array.The more granular the bitmap is, then the morefrequently writes will trigger syncronous bitmap updates and be delayeduntil the bitmap update is complete. The trade off is that a moregranular bitmap means a shorter array resync time after any event causesthe array to go down unclean. Given raw drive speeds can be in excessof 100MB/s on modern SATA/SAS drives, any bitmap chunk up to 262144 (256MB)can generally be synced in a matter of just a few seconds. Smaller chunkscan be synced faster, but you reach a point of diminishing returns that isquickly offset by the increased write performance degradation seen inevery day operation. Considering that the smaller bitmap chunk sizeswill only ever be a benefit on rare occasions (hopefully never), but thatyou will pay for a small bitmap chunk every single day, it is recommendedthat you select the largest bitmap chunk size you feel comforable with.-W, --write-mostly subsequent devices lists in a--build, --create, or--add command will be flagged as ’write-mostly’. This is valid for RAID1only and means that the ’md’ driver will avoid reading from thesedevices if at all possible. This can be useful if mirroring over aslow link.--write-behind= Specify that write-behind mode should be enabled (valid for RAID1only). If an argument is specified, it will set the maximum numberof outstanding writes allowed. The default value is 256.A write-intent bitmap is required in order to use write-behindmode, and write-behind is only attempted on drives marked aswrite-mostly. --assume-clean Tellmdadm that the array pre-existed and is known to be clean. It can be usefulwhen trying to recover from a major failure as you can be sure that nodata will be affected unless you actually write to the array. It canalso be used when creating a RAID1 or RAID10 if you want to avoid theinitial resync, however this practice — while normally safe — is notrecommended. Use this only if you really know what you are doing.--backup-file= This is needed when--grow is used to increase the number ofraid-devices in a RAID5 if there are no spare devices available.See the section below on RAID_DEVICE CHANGES. The file should bestored on a separate device, not on the raid array being reshaped.-N, --name= Set aname for the array. This is currently only effective when creating anarray with a version-1 superblock. The name is a simple textualstring that can be used to identify array components when assembling.-R, --run Insist thatmdadm run the array, even if some of the componentsappear to be active in another array or filesystem. Normallymdadm will ask for confirmation before including such components in anarray. This option causes that question to be suppressed.-f, --force Insist thatmdadm accept the geometry and layout specified without question. Normallymdadm will not allow creation of an array with only one device, and will tryto create a raid5 array with one missing drive (as this makes theinitial resync work faster). With--force, mdadm will not try to be so clever.-a, --auto=no,yes,md,mdp,part,pNN Instruct mdadm to create the device file if needed, possibly allocatingan unused minor number. "md" causes a non-partitionable arrayto be used. "mdp", "part" or "p" causes a partitionable array (2.6 andlater) to be used. "yes" requires the named md device to havea ’standard’ format, and the type and minor number will be determinedfrom this. See DEVICE NAMES below.The argument can also come immediately after"-a". e.g. "-ap".If--auto is not given on the command line or in the config file, thenthe default will be--auto=yes. If--scan is also given, then anyauto= entries in the config file will override the--auto instruction given on the command line.For partitionable arrays,mdadm will create the device file for the whole array and for the first 4partitions. A different number of partitions can be specified at theend of this option (e.g.--auto=p7). If the device name ends with a digit, the partition names add a ’p’,and a number, e.g. "/dev/home1p3". If there is notrailing digit, then the partition names just have a number added,e.g. "/dev/scratch3".If the md device name is in a ’standard’ format as described in DEVICENAMES, then it will be created, if necessary, with the appropriatenumber based on that name. If the device name is not in one of theseformats, then a unused minor number will be allocated. The minornumber will be considered unused if there is no active array for thatnumber, and there is no entry in /dev for that number and with anon-standard name.--symlink=no Normally when--auto causesmdadm to create devices in/dev/md/ it will also create symlinks from/dev/ with names starting withmd ormd_. Use--symlink=no to suppress this, or--symlink=yes to enforce this even if it is suppressingmdadm.conf. For assemble:TagDescription-u, --uuid= uuid of array to assemble. Devices which don’t have this uuid areexcluded-m, --super-minor= Minor number of device that array was created for. Devices whichdon’t have this minor number are excluded. If you create an array as/dev/md1, then all superblocks will contain the minor number 1, even ifthe array is later assembled as /dev/md2.Giving the literal word "dev" for--super-minor will causemdadm to use the minor number of the md device that is being assembled.e.g. when assembling/dev/md0, --super-minor=dev will look for super blocks with a minor number of 0.-N, --name= Specify the name of the array to assemble. This must be the namethat was specified when creating the array. It must either matchthe name stored in the superblock exactly, or it must matchwith the currenthomehost prefixed to the start of the given name.-f, --force Assemble the array even if some superblocks appear out-of-date-R, --run Attempt to start the array even if fewer drives were given than werepresent last time the array was active. Normally if not all theexpected drives are found and--scan is not used, then the array will be assembled but not started.With--run an attempt will be made to start it anyway.--no-degraded This is the reverse of--run in that it inhibits the startup of array unless all expected drivesare present. This is only needed with--scan, and can be used if the physical connections to devices arenot as reliable as you would like.-a, --auto=no,yes,md,mdp,part See this option under Create and Build options.-b, --bitmap= Specify the bitmap file that was given when the array was created. Ifan array has aninternal bitmap, there is no need to specify this when assembling the array.--backup-file= If--backup-file was used to grow the number of raid-devices in a RAID5, and the systemcrashed during the critical section, then the same--backup-file must be presented to--assemble to allow possibly corrupted data to be restored.-U, --update= Update the superblock on each device while assembling the array. Theargument given to this flag can be one ofsparc2.2, summaries, uuid, name, homehost, resync, byteorder, devicesize, orsuper-minor. Thesparc2.2 option will adjust the superblock of an array what was created on a Sparcmachine running a patched 2.2 Linux kernel. This kernel got thealignment of part of the superblock wrong. You can use the--examine --sparc2.2 option tomdadm to see what effect this would have.Thesuper-minor option will update thepreferred minor field on each superblock to match the minor number of the array beingassembled.This can be useful if--examine reports a different "Preferred Minor" to--detail. In some cases this update will be performed automaticallyby the kernel driver. In particular the update happens automaticallyat the first write to an array with redundancy (RAID level 1 orgreater) on a 2.6 (or later) kernel.Theuuid option will change the uuid of the array. If a UUID is given with the--uuid option that UUID will be used as a new UUID and willNOT be used to help identify the devices in the array.If no--uuid is given, a random UUID is chosen.Thename option will change thename of the array as stored in the superblock. This is only supported forversion-1 superblocks.Thehomehost option will change thehomehost as recorded in the superblock. For version-0 superblocks, this is thesame as updating the UUID.For version-1 superblocks, this involves updating the name.Theresync option will cause the array to be markeddirty meaning that any redundancy in the array (e.g. parity for raid5,copies for raid1) may be incorrect. This will cause the raid systemto perform a "resync" pass to make sure that all redundant informationis correct.Thebyteorder option allows arrays to be moved between machines with differentbyte-order.When assembling such an array for the first time after a move, giving--update=byteorder will causemdadm to expect superblocks to have their byteorder reversed, and willcorrect that order before assembling the array. This is only validwith original (Version 0.90) superblocks.Thesummaries option will correct the summaries in the superblock. That is thecounts of total, working, active, failed, and spare devices.Thedevicesize will rarely be of use. It applies to version 1.1 and 1.2 metadataonly (where the metadata is at the start of the device) and is onlyuseful when the component device has changed size (typically becomelarger). The version 1 metadata records the amount of the device thatcan be used to store data, so if a device in a version 1.1 or 1.2array becomes larger, the metadata will still be visible, but theextra space will not. In this case it might be useful to assemble thearray with--update=devicesize. This will causemdadm to determine the maximum usable amount of space on each device andupdate the relevant field in the metadata.--auto-update-homehost This flag is only meaningful with auto-assembly (see discussion below).In that situation, if no suitable arrays are found for this homehost,mdadm will rescan for any arrays at all and will assemble them and update thehomehost to match the current host.For Manage mode:TagDescription-a, --add add listed devices to a live array. When the array is in a degraded stateand you add a device, the device will be added as a spare device andreconstruction on to the spare device will commence. Upon completion ofthe reconstruction, the device will be transitioned to an active device.If you add more devices than the array’s normal capacity of active devices,then they are automatically added as hot spare devices. In order toutilize the spare devices, use the Grow mode of mdadm to increase the numberof active devices in the array.--re-add re-add a device that was recently removed from an array. This only appliesto devices that were part of an array built without a persistent superblock,and for which a write intent bitmap exists. In this isolated case, thekernel will treat this device as a previous member of the array even thoughthere is no superblock to tell it to do so. For all add operations involvingarrays with persistent superblocks, use the --add command above and thekernel will automatically determine whether a full resync or partial resyncis needed based upon the superblock state and the write intent bitmapstate (if it exists).-r, --remove remove listed devices. They must not be active. i.e. they shouldbe failed or spare devices. As well as the name of a device file(e.g./dev/sda1) the wordsfailed anddetached can be given to--remove. The first causes all failed device to be removed. The second causesany device which is no longer connected to the system (i.e an ’open’returnsENXIO) to be removed. This will only succeed for devices that are spares orhave already been marked as failed.-f, --fail mark listed devices as faulty.As well as the name of a device file, the worddetached can be given. This will cause any device that has been detached fromthe system to be marked as failed. It can then be removed.--set-faulty same as--fail. --write-mostly Subsequent devices that are added or re-added will have the ’write-mostly’flag set. This is only valid for RAID1 and means that the ’md’ driverwill avoid reading from these devices if possible.--readwrite Subsequent devices that are added or re-added will have the ’write-mostly’flag cleared.Each of these options require that the first device listed is the arrayto be acted upon, and the remainder are component devices to be added,removed, or marked as faulty. Several different operations can bespecified for different devices, e.g.mdadm /dev/md0 --add /dev/sda1 --fail /dev/sdb1 --remove /dev/sdb1Each operation applies to all devices listed until the nextoperation.If an array is using a write-intent bitmap, then devices which havebeen removed can be re-added in a way that avoids a fullreconstruction but instead just updates the blocks that have changedsince the device was removed. For arrays with persistent metadata(superblocks) this is done automatically. For arrays created with--build mdadm needs to be told that this device was removed recently by using--re-add instead of--add command (see above).Devices can only be removed from an array if they are not in activeuse, i.e. they must be spares or failed devices. To remove an activedevice, it must first be marked asfaulty. For Misc mode:TagDescription-Q, --query Examine a device to see(1) if it is an md device and (2) if it is a component of an mdarray.Information about what is discovered is presented.-D, --detail Print detail of one or more md devices.-Y, --export When used with--detail or--examine, output will be formatted askey=value pairs for easy import into the environment.-E, --examine Print content of md superblock on device(s).--sparc2.2 If an array was created on a 2.2 Linux kernel patched with RAIDsupport, the superblock will have been created incorrectly, or atleast incompatibly with 2.4 and later kernels. Using the--sparc2.2 flag with--examine will fix the superblock before displaying it. If this appears to dothe right thing, then the array can be successfully assembled using--assemble --update=sparc2.2. -X, --examine-bitmap Report information about a bitmap file.The argument is either an external bitmap file or an array componentin case of an internal bitmap.-R, --run start a partially built array.-S, --stop deactivate array, releasing all resources.-o, --readonly mark array as readonly.-w, --readwrite mark array as readwrite.--zero-superblock If the device contains a valid md superblock, the block isoverwritten with zeros. With--force the block where the superblock would be is overwritten even if itdoesn’t appear to be valid.-t, --test When used with--detail, the exit status ofmdadm is set to reflect the status of the device.-W, --wait For each md device given, wait for any resync, recovery, or reshapeactivity to finish before returning.mdadm will return with success if it actually waited for every devicelisted, otherwise it will return failure.For Incremental Assembly mode:TagDescription--rebuild-map, -r Rebuild the map file(/var/run/mdadm/map) thatmdadm uses to help track which arrays are currently being assembled.--run, -R Run any array assembled as soon as a minimal number of devices areavailable, rather than waiting until all expected devices are present.--scan, -s Only meaningful with-R this will scan themap file for arrays that are being incrementally assembled and will try tostart any that are not already started. If any such array is listedinmdadm.conf as requiring an external bitmap, that bitmap will be attached first.For Monitor mode:TagDescription-m, --mail Give a mail address to send alerts to.-p, --program, --alert Give a program to be run whenever an event is detected.-y, --syslog Cause all events to be reported through ’syslog’. The messages havefacility of ’daemon’ and varying priorities.-d, --delay Give a delay in seconds.mdadm polls the md arrays and then waits this many seconds before pollingagain. The default is 60 seconds.-f, --daemonise Tellmdadm to run as a background daemon if it decides to monitor anything. Thiscauses it to fork and run in the child, and to disconnect form theterminal. The process id of the child is written to stdout.This is useful with--scan which will only continue monitoring if a mail address or alert programis found in the config file.-i, --pid-file Whenmdadm is running in daemon mode, write the pid of the daemon process tothe specified file, instead of printing it on standard output.-1, --oneshot Check arrays only once. This will generateNewArray events and more significantlyDegradedArray andSparesMissing events. Running mdadm --monitor --scan -1 from a cron script will ensure regular notification of any degraded arrays.-t, --test Generate aTestMessage alert for every array found at startup. This alert gets mailed andpassed to the alert program. This can be used for testing that alertmessage do get through successfully.ASSEMBLE MODETagDescriptionUsage:mdadm --assemble md-device options-and-component-devices... Usage:mdadm --assemble --scan md-devices-and-options... Usage:mdadm --assemble --scan options... This usage assembles one or more raid arrays from pre-existing components.For each array, mdadm needs to know the md device, the identity of thearray, and a number of component-devices. These can be found in a number of ways.In the first usage example (without the--scan) the first device given is the md device.In the second usage example, all devices listed are treated as mddevices and assembly is attempted.In the third (where no devices are listed) all md devices that arelisted in the configuration file are assembled.If precisely one device is listed, but--scan is not given, thenmdadm acts as though--scan was given and identity information is extracted from the configuration file.The identity can be given with the--uuid option, with the--super-minor option, will be taken from the md-device record in the config file, orwill be taken from the super block of the first component-devicelisted on the command line.Devices can be given on the--assemble command line or in the config file. Only devices which have an mdsuperblock which contains the right identity will be considered forany array.The config file is only used if explicitly named with--config or requested with (a possibly implicit)--scan. In the later case,/etc/mdadm.conf is used.If--scan is not given, then the config file will only be used to find theidentity of md arrays.Normally the array will be started after it is assembled. However if--scan is not given and insufficient drives were listed to start a complete(non-degraded) array, then the array is not started (to guard againstusage errors). To insist that the array be started in this case (asmay work for RAID1, 4, 5, 6, or 10), give the--run flag.If the md device does not exist, then it will be created providing theintent is clear. i.e. the name must be in a standard form, or the--auto option must be given to clarify how and whether the device should becreated.This can be useful for handling partitioned devices (which don’t havea stable device number — it can change after a reboot) and when using"udev" to manage your/dev tree (udev cannot handle md devices because of the unusual deviceinitialisation conventions).If the option to "auto" is "mdp" or "part" or (on the command lineonly) "p", then mdadm will create a partitionable array, using thefirst free one that is not in use and does not already have an entryin /dev (apart from numeric /dev/md* entries).If the option to "auto" is "yes" or "md" or (on the command line)nothing, then mdadm will create a traditional, non-partitionable mdarray.It is expected that the "auto" functionality will be used to createdevice entries with meaningful names such as "/dev/md/home" or"/dev/md/root", rather than names based on the numerical array number.When using option "auto" to create a partitionable array, the devicefiles for the first 4 partitions are also created. If a differentnumber is required it can be simply appended to the auto option.e.g. "auto=part8". Partition names are created by appending a digitstring to the device name, with an intervening "p" if the device nameends with a digit.The--auto option is also available in Build and Create modes. As those modes donot use a config file, the "auto=" config option does not apply tothese modes.Auto AssemblyWhen--assemble is used with--scan and no devices are listed,mdadm will first attempt to assemble all the arrays listed in the configfile.If ahomehost has been specified (either in the config file or on the command line),mdadm will look further for possible arrays and will try to assembleanything that it finds which is tagged as belonging to the givenhomehost. This is the only situation wheremdadm will assemble arrays without being given specific device name oridentity information for the array.Ifmdadm finds a consistent set of devices that look like they should comprisean array, and if the superblock is tagged as belonging to the givenhome host, it will automatically choose a device name and try toassemble the array. If the array uses version-0.90 metadata, then theminor number as recorded in the superblock is used to create a name in/dev/md/ so for example/dev/md/3. If the array uses version-1 metadata, then thename from the superblock is used to similarly create a name in/dev/md (the name will have any ’host’ prefix stripped first).Ifmdadm cannot find any array for the given host at all, and if--auto-update-homehost is given, thenmdadm will search again for any array (not just an array created for thishost) and will assemble each assuming--update=homehost. This will change the host tag in the superblock so that on the next run,these arrays will be found without the second pass. The intention ofthis feature is to support transitioning a set of md arrays to usinghomehost tagging.The reason for requiring arrays to be tagged with the homehost forauto assembly is to guard against problems that can arise when movingdevices from one host to another.BUILD MODETagDescriptionUsage:mdadm --build md-device --chunk=X --level=Y --raid-devices=Z devices This usage is similar to--create. The difference is that it creates an array without a superblock. Withthese arrays there is no difference between initially creating the array andsubsequently assembling the array, except that hopefully there is usefuldata there in the second case.The level may raid0, linear, multipath, or faulty, or one of theirsynonyms. All devices must be listed and the array will be startedonce complete.CREATE MODETagDescriptionUsage:mdadm --create md-device --chunk=X --level=Y --raid-devices=Z devices This usage will initialise a new md array, associate some devices withit, and activate the array.If the--auto option is given (as described in more detail in the section onAssemble mode), then the md device will be created with a suitabledevice number if necessary.As devices are added, they are checked to see if they contain raidsuperblocks or filesystems. They are also checked to see if the variance indevice size exceeds 1%.If any discrepancy is found, the array will not automatically be run, thoughthe presence of a--run can override this caution.To create a "degraded" array in which some devices are missing, simplygive the word "missing"in place of a device name. This will causemdadm to leave the corresponding slot in the array empty.For a RAID4 or RAID5 array at most one slot can be"missing"; for a RAID6 array at most two slots.For a RAID1 array, only one real device needs to be given. All of theothers can be"missing".When creating a RAID5 array,mdadm will automatically create a degraded array with an extra spare drive.This is because building the spare into a degraded array is in general faster than resyncingthe parity on a non-degraded, but not clean, array. This feature canbe overridden with the--force option.When creating an array with version-1 metadata a name for the array isrequired.If this is not given with the--name option,mdadm will choose a name based on the last component of the name of thedevice being created. So if/dev/md3 is being created, then the name3 will be chosen.If/dev/md/home is being created, then the namehome will be used.When creating a partition based array, usingmdadm with version-1.x metadata, the partition type should be set to0xDA (non fs-data). This type selection allows for greater precision sinceusing any other [RAID auto-detect (0xFD) or a GNU/Linux partition (0x83)],might create problems in the event of array recovery through a live cdrom.A new array will normally get a randomly assigned 128bit UUID which isvery likely to be unique. If you have a specific need, you can choosea UUID for the array by giving the--uuid= option. Be warned that creating two arrays with the same UUID is arecipe for disaster. Also, using--uuid= when creating a v0.90 array will silently override any--homehost= setting.The General Management options that are valid with--create are:TagDescription--run insist on running the array even if some devices look like they mightbe in use.--readonly start the array readonly — not supported yet.MANAGE MODETagDescriptionUsage:mdadm device options... devices... This usage will allow individual devices in an array to be failed,removed or added. It is possible to perform multiple operations withon command. For example: mdadm /dev/md0 -f /dev/hda1 -r /dev/hda1 -a /dev/hda1 will firstly mark/dev/hda1 as faulty in/dev/md0 and will then remove it from the array and finally add it backin as a spare. However only one md array can be affected by a singlecommand.MISC MODETagDescriptionUsage:mdadm options ... devices ... MISC mode includes a number of distinct operations thatoperate on distinct devices. The operations are:TagDescription--query The device is examined to see if it is(1) an active md array, or(2) a component of an md array.The information discovered is reported.--detail The device should be an active md device.mdadm will display a detailed description of the array.--brief or--scan will cause the output to be less detailed and the format to besuitable for inclusion in/etc/mdadm.conf. The exit status ofmdadm will normally be 0 unlessmdadm failed to get useful information about the device(s); however, if the--test option is given, then the exit status will be:TagDescription0The array is functioning normally.1The array has at least one failed device.2The array has multiple failed devices such that it is unusable.4There was an error while trying to get information about the device.--examine The device should be a component of an md array.mdadm will read the md superblock of the device and display the contents.If--brief or--scan is given, then multiple devices that are components of the one arrayare grouped together and reported in a single entry suitablefor inclusion in/etc/mdadm.conf. Having--scan without listing any devices will cause all devices listed in theconfig file to be examined.--stop The devices should be active md arrays which will be deactivated, aslong as they are not currently in use.--run This will fully activate a partially assembled md array.--readonly This will mark an active array as read-only, providing that it isnot currently being used.--readwrite This will change areadonly array back to being read/write.--scan For all operations except--examine, --scan will cause the operation to be applied to all arrays listed in/proc/mdstat. For--examine, --scan causes all devices listed in the config file to be examined.MONITOR MODETagDescriptionUsage:mdadm --monitor options... devices... This usage causesmdadm to periodically poll a number of md arrays and to report on any eventsnoticed.mdadm will never exit once it decides that there are arrays to be checked,so it should normally be run in the background.As well as reporting events,mdadm may move a spare drive from one array to another if they are in thesamespare-group and if the destination array has a failed drive but no spares.If any devices are listed on the command line,mdadm will only monitor those devices. Otherwise all arrays listed in theconfiguration file will be monitored. Further, if--scan is given, then any other md devices that appear in/proc/mdstat will also be monitored.The result of monitoring the arrays is the generation of events.These events are passed to a separate program (if specified) and maybe mailed to a given E-mail address.When passing events to a program, the program is run once for each event,and is given 2 or 3 command-line arguments: the first is thename of the event (see below), the second is the name of themd device which is affected, and the third is the name of a relateddevice if relevant (such as a component device that has failed).If--scan is given, then a program or an E-mail address must be specified on thecommand line or in the config file. If neither are available, thenmdadm will not monitor anything.Without--scan, mdadm will continue monitoring as long as something was found to monitor. Ifno program or email is given, then each event is reported tostdout. The different events are:TagDescriptionDeviceDisappeared An md array which previously was configured appears to no longer beconfigured. (syslog priority: Critical)Ifmdadm was told to monitor an array which is RAID0 or Linear, then it willreportDeviceDisappeared with the extra informationWrong-Level. This is because RAID0 and Linear do not support the device-failed,hot-spare and resync operations which are monitored.RebuildStarted An md array started reconstruction. (syslog priority: Warning)RebuildNN WhereNN is 20, 40, 60, or 80, this indicates that rebuild has passed that manypercentage of the total. (syslog priority: Warning)RebuildFinished An md array that was rebuilding, isn’t any more, either because itfinished normally or was aborted. (syslog priority: Warning)Fail An active component device of an array has been marked asfaulty. (syslog priority: Critical)FailSpare A spare component device which was being rebuilt to replace a faultydevice has failed. (syslog priority: Critical)SpareActive A spare component device which was being rebuilt to replace a faultydevice has been successfully rebuilt and has been made active.(syslog priority: Info)NewArray A new md array has been detected in the/proc/mdstat file. (syslog priority: Info)DegradedArray A newly noticed array appears to be degraded. This message is notgenerated whenmdadm notices a drive failure which causes degradation, but only whenmdadm notices that an array is degraded when it first sees the array.(syslog priority: Critical)MoveSpare A spare drive has been moved from one array in aspare-group to another to allow a failed drive to be replaced.(syslog priority: Info)SparesMissing Ifmdadm has been told, via the config file, that an array should have a certainnumber of spare devices, andmdadm detects that it has fewer than this number when it first sees thearray, it will report aSparesMissing message.(syslog priority: Warning)TestMessage An array was found at startup, and the--test flag was given.(syslog priority: Info)OnlyFail, FailSpare, DegradedArray, SparesMissing andTestMessage cause Email to be sent. All events cause the program to be run.The program is run with two or three arguments: the eventname, the array device and possibly a second device.Each event has an associated array device (e.g./dev/md1) and possibly a second device. ForFail, FailSpare, andSpareActive the second device is the relevant component device.ForMoveSpare the second device is the array that the spare was moved from.Formdadm to move spares from one array to another, the different arrays need tobe labeled with the samespare-group in the configuration file. Thespare-group name can be any string; it is only necessary that different sparegroups use different names.Whenmdadm detects that an array in a spare group has fewer activedevices than necessary for the complete array, and has no sparedevices, it will look for another array in the same spare group thathas a full complement of working drive and a spare. It will thenattempt to remove the spare from the second drive and add it to thefirst.If the removal succeeds but the adding fails, then it is added back tothe original array.GROW MODEThe GROW mode is used for changing the size or shape of an activearray.For this to work, the kernel must support the necessary change.Various types of growth are being added during 2.6 development,including restructuring a raid5 array to have more active devices.Currently the only support available is toTagDescriptionochange the "size" attributefor RAID1, RAID5 and RAID6.oincrease the "raid-devices" attribute of RAID1, RAID5, and RAID6.oadd a write-intent bitmap to any array which supports these bitmaps, orremove a write-intent bitmap from such an array.SIZE CHANGESNormally when an array is built the "size" it taken from the smallestof the drives. If all the small drives in an arrays are, one at atime, removed and replaced with larger drives, then you could have anarray of large drives with only a small amount used. In thissituation, changing the "size" with "GROW" mode will allow the extraspace to start being used. If the size is increased in this way, a"resync" process will start to make sure the new parts of the arrayare synchronised.Note that when an array changes size, any filesystem that may bestored in the array will not automatically grow to use the space. Thefilesystem will need to be explicitly told to use the extra space.RAID-DEVICES CHANGESA RAID1 array can work with any number of devices from 1 upwards(though 1 is not very useful). There may be times when you want toincrease or decrease the number of active devices. Note that this isdifferent than hot-add or hot-remove which changes the number ofinactive devices.When reducing the number of devices in a RAID1 array, the slots whichare to be removed from the array must already be vacant. That is, thedevices which were in those slots must be failed and removed.When the number of devices is increased, any hot spares that arepresent will be activated immediately.Increasing the number of active devices in a RAID5 is much morework. Every block in the array will need to be moved to a new location.From 2.6.17, the Linux Kernel is able to do this safely, includingrestarting an interrupted "reshape".When relocating the first few stripes on a raid5, it is not possibleto keep the data on disk completely consistent and crash-proof. Toprovide the required safety, mdadm disables writes to the array whilethis "critical section" is reshaped, and makes a backup of the datathat is in that section. This backup is normally stored in any sparedevices that the array has, however it can also be stored in aseparate file specified with the--backup-file option. If this option is used, and the system does crash during thecritical period, the same file must be passed to--assemble to restore the backup and reassemble the array.BITMAP CHANGESA write-intent bitmap can be added to, or removed from, an activearray. Either internal bitmaps or an external bitmap stored in a filecan be added. In the case of internal bitmaps, there is one copy of thebitmap per device (since you never know what device might fail, you needa copy on every device). The bitmap is stored between the array dataand the superblock, which limits the total number of bits available.For a bitmap in an external file, only one copy is needed, but thisassumes that the bitmap file is not on an array device or else failureof that device would take the only copy of the bitmap with it. Forthis reason, the fact that the kernel will deadlock if you attemptto use a file that resides on the array it is the bitmap for isconsidered a safety feature.INCREMENTAL MODETagDescriptionUsage:mdadm --incremental [--run] [--quiet] component-device Usage:mdadm --incremental --rebuild Usage:mdadm --incremental --run --scan This mode is designed to be used in conjunction with a devicediscovery system. As devices are found in a system, they can bepassed tomdadm --incremental to be conditionally added to an appropriate array.mdadm performs a number of tests to determine if the device is part of anarray, and which array it should be part of. If an appropriate arrayis found, or can be created,mdadm adds the device to the array and conditionally starts the array.Note thatmdadm will only add devices to an array which were previously working(active or spare) parts of that array. It does not currently supportautomatic inclusion of a new drive as a spare in some array.mdadm --incremental requires a bug-fix in all kernels through 2.6.19.Hopefully, this will be fixed in 2.6.20; alternately, apply the patchwhich is included with the mdadm source distribution. Ifmdadm detects that this bug is present, it will abort any attempt to use--incremental. The tests thatmdadm makes are as follow:TagDescription+Is the device permitted bymdadm.conf? That is, is it listed in aDEVICES line in that file. IfDEVICES is absent then the default it to allow any device. Similar ifDEVICES contains the special wordpartitions then any device is allowed. Otherwise the device name given tomdadm must match one of the names or patterns in aDEVICES line.+Does the device have a valid md superblock. If a specific metadataversion is request with--metadata or-e then only that style of metadata is accepted, otherwisemdadm finds any known version of metadata. If nomd metadata is found, the device is rejected.+Does the metadata match an expected array?The metadata can match in two ways. Either there is an array listedinmdadm.conf which identifies the array (either by UUID, by name, by device list,or by minor-number), or the array was created with ahomehost specified and thathomehost matches the one inmdadm.conf or on the command line.Ifmdadm is not able to positively identify the array as belonging to thecurrent host, the device will be rejected.+mdadm keeps a list of arrays that it has partially assembled in/var/run/mdadm/map (or/var/run/mdadm.map if the directory doesn’t exist). If no array exists which matchesthe metadata on the new device,mdadm must choose a device name and unit number. It does this based on anyname given inmdadm.conf or any name information stored in the metadata. If this namesuggests a unit number, that number will be used, otherwise a freeunit number will be chosen. Normallymdadm will prefer to create a partitionable array, however if theCREATE line inmdadm.conf suggests that a non-partitionable array is preferred, that will behonoured.+Once an appropriate array is found or created and the device is added,mdadm must decide if the array is ready to be started. It willnormally compare the number of available (non-spare) devices to thenumber of devices that the metadata suggests need to be active. Ifthere are at least that many, the array will be started. This meansthat if any devices are missing the array will not be restarted.As an alternative,--run may be passed tomdadm in which case the array will be run as soon as there are enoughdevices present for the data to be accessible. For a raid1, thatmeans one device will start the array. For a clean raid5, the arraywill be started as soon as all but one drive is present.Note that neither of these approaches is really ideal. If it canbe known that all device discovery has completed, then mdadm -IRs can be run which will try to start all arrays that are beingincrementally assembled. They are started in "read-auto" mode inwhich they are read-only until the first write request. This meansthat no metadata updates are made and no attempt at resync or recoveryhappens. Further devices that are found before the first write canstill be added safely.EXAMPLES mdadm --query /dev/name-of-device This will find out if a given device is a raid array, or is part ofone, and will provide brief information about the device. mdadm --assemble --scan This will assemble and start all arrays listed in the standard configfile. This command will typically go in a system startup file. mdadm --stop --scan This will shut down all arrays that can be shut down (i.e. are notcurrently in use). This will typically go in a system shutdown script. mdadm --follow --scan --delay=120 If (and only if) there is an Email address or program given in thestandard config file, thenmonitor the status of all arrays listed in that file bypolling them ever 2 minutes. mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hd[ac]1 Create /dev/md0 as a RAID1 array consisting of /dev/hda1 and /dev/hdc1. echo ’DEVICE /dev/hd*[0-9] /dev/sd*[0-9]’ > mdadm.conf mdadm --detail --scan >> mdadm.conf This will create a prototype config file that describes currentlyactive arrays that are known to be made from partitions of IDE or SCSI drives.This file should be reviewed before being used as it maycontain unwanted detail. echo ’DEVICE /dev/hd[a-z] /dev/sd*[a-z]’ > mdadm.conf mdadm --examine --scan --config=mdadm.conf >> mdadm.conf This will find arrays which could be assembled from existing IDE andSCSI whole drives (not partitions), and store the information in theformat of a config file.This file is very likely to contain unwanted detail, particularlythedevices= entries. It should be reviewed and edited before being used as anactual config file. mdadm --examine --brief --scan --config=partitions mdadm -Ebsc partitions Create a list of devices by reading/proc/partitions, scan these for RAID superblocks, and printout a brief listing of allthat were found. mdadm -Ac partitions -m 0 /dev/md0 Scan all partitions and devices listed in/proc/partitions and assemble/dev/md0 out of all such devices with a RAID superblock with a minor number of 0. mdadm --monitor --scan --daemonise > /var/run/mdadm If config file contains a mail address or alert program, run mdadm inthe background in monitor mode monitoring all md devices. Also writepid of mdadm daemon to/var/run/mdadm. mdadm -Iq /dev/somedevice Try to incorporate newly discovered device into some array asappropriate. mdadm --incremental --rebuild --run --scan Rebuild the array map from any current arrays, and then start any thatcan be started. mdadm /dev/md4 --fail detached --remove detached Any devices which are components of /dev/md4 will be marked as faultyand then remove from the array. mdadm --create --help Provide help about the Create mode. mdadm --config --help Provide help about the format of the config file. mdadm --help Provide general help.FILES/proc/mdstatIf you’re using the/proc filesystem,/proc/mdstat lists all active md devices with information about them.mdadm uses this to find arrays when--scan is given in Misc mode, and to monitor array reconstructionon Monitor mode./etc/mdadm.confThe config file lists which devices may be scanned to see ifthey contain MD super block, and gives identifying information(e.g. UUID) about known MD arrays. Seemdadm.conf(5)for more details./var/run/mdadm/mapWhen--incremental mode is used, this file gets a list of arrays currently being created.If/var/run/mdadm does not exist as a directory, then/var/run/mdadm.map is used instead.DEVICE NAMESWhile entries in the /dev directory can have any format you like,mdadm has an understanding of ’standard’ formats which it uses to guide itsbehaviour when creating device files via the--auto option.The standard names for non-partitioned arrays (the only sort of mdarray available in 2.4 and earlier) are either ofTagDescription/dev/mdNN/dev/md/NNwhere NN is a number.The standard names for partitionable arrays (as available from 2.6onwards) are either of/dev/md/dNN/dev/md_dNNPartition numbers should be indicated by added "pMM" to these, thus "/dev/md/d1p2".NOTEmdadm was previously known asmdctl. mdadm is completely separate from theraidtools package, and does not use the/etc/raidtab configuration file at all.SEE ALSORAID, see:TagDescription -raid.osdl.org/ (based upon Jakob \(/Ostergaard’s Software-RAID.HOWTO)The latest version ofmdadm should always be available from Related man pages:mdadm.conf(5),md(4).raidtab(5),raid0run(8),raidstop(8),mkraid(8). Advertisements 2ff7e9595c


1 view0 comments

Recent Posts

See All

Comments


bottom of page