BIGDIRS is a utility to help locate directories that have a large number of entries. (Large directories can trigger system aborts in the Transaction Manager.) BIGDIRS can also purge all the files in a group or directory, generally without trigging an XM abort. BIGDIRS can list files that have a lot of extents (e.g., you could ask for a list of all files with at least 1000 extents). The simplest way of using it to find large directories is: :run bigdirs.pub.allegro; info="search" which will report a list of large directories. The search will take from one to several minutes, depending upon the number of files on your system, your CPU speed, and the load on the system. You can also run bigdirs interactively: :run bigdirs.pub.allegro Usage: :run bigdirs.pub.allegro; info = "[options] [command [options]]" Examples: Look through entire system for large directories: :run bigdirs.pub.allegro; info = "/" Look through the DATA account for large directories: :run bigdirs.pub.allegro; info = "/DATA/" Look through the DATA account for files with >= 1000 extents: :run bigdirs.pub.allegro; info = "/DATA extents 1000" Purge all files in the group OUT.HPSPOOL, in reverse order: :run bigdirs.pub.allegro; info = "PURGEGROUP OUT.HPSPOOL" Interactively explore directories (or purge files): :run bigdirs.pub.allegro Commands: Exit or Quit Terminate the program Help [prefix] LISTFile pathname [FIRST #] [LAST #] [DEPTH #] [noTERSE] [noSHOWBig] [noSIZEs] [Verbose] [Quiet] Lists files (ala 'listf,2'). If DEPTH is specified, the recursion depth is restricted to that number. If FIRST or LAST is specified, the listing is suppressed until we reach FIRST, and stopped after LAST (i.e., the directory search is told to top, not merely to suppress output). If SIZES is specified, BIGDIRS will compute the exact number of sectors and extents for each file. If SHOWBIG is specified, BIGDIRS will usually report the file with the most extents, and the file "deepest" in the directory (both reported at the end of the LISTF). QUIET suppresses the per-file listing, resulting in just the summary. PURGEGroup group[.account] [RM | FAKE] [Verbose] [] Purges files from the group specified, in *REVERSE* order. (See also: PURGEPATH) Without the keyword RM, *NO FILES WILL BE PURGED*, instead you'll see 'purge' commands sent to stdlist (you can catch them and execute them later, if desired). With 'FAKE', NO FILES WILL BE PURGED ('FAKE' is the opposite of 'RM'). 'FAKE' is the default! (to minimize accidental changes) With VERBOSE, each file seen is listed. The PUBSYS option (also settable via 'SET') will allocate our temporary mapped file in the MPEXL_SYSTEM_VOLUME_SET. (And the temp file will be opened with the name maptemp.pub.sys, should you care to file equate it.) The FEQ option allows you to file equate the mapped file, MAPTEMP. NOTE: despite its name, PURGEGROUP does not purge the group when done (if 'RM' is specified) ... it leaves the group in the directory. You can purge the group by doing: (optional) :purgegroup foo.fum; onvs = fie and then: :purgegroup foo.fum PURGEPath some_hfs_path [RM | FAKE] [Verbose] [] Purges files from the directory ("path") specified, but in *REVERSE* order. (See also: PURGEGROUP) Without the keyword RM, *NO FILES WILL BE PURGED*, instead you'll see 'purge' commands sent to stdlist (you can catch them and execute them later, if desired). With 'FAKE', NO FILES WILL BE PURGED. With VERBOSE, each file seen is listed. The PUBSYS option (also settable via 'SET') will allocate our temporary mapped file in the MPEXL_SYSTEM_VOLUME_SET. (And the temp file will be opened with the name maptemp.pub.sys, should you care to file equate it.) The FEQ option allows you to file equate the mapped file, MAPTEMP. At the end, BIGDIRs attempts to 'purgedir' the path. SEARCH [pathname] [options] [search_options] search_options ::= ALL | BIG | DEPTH # | EXTENTS # | ENTries # | Kbs # | Mbs # | SOME | Search the specified path (directory) for large directories, and/or files with many extents. Note: non-SEARCH-specific options are documented further below. See also: LISTFile ALL List every directory (this can be quite verbose). (Modes: QUIET, BIG, SOME, ALL) (Will not list every file, even if 'EXTENTS' specified.) BIG Only list big directories (default). (Modes: QUIET, BIG, SOME, ALL) DEPTH # Maximum recursion depth (0 = unlimited, 1 = no recursion) ENTRIES # (or: NOENTRIES) ENTRIES # tells BIGDIRS to list only directories with at least the specified number of entries. # of 0 means don't check. EXTents # EXTENTS tells BIGDIRS to list *files* with at least the specified number of extents. Files with a large number of extents (e.g., 1000 or more) can cause problems by using many entries in some MPE/iX system tables. If EXTENTS is not specified, files are ignored and only directories are checked. KBS # KBS # tells BIGDIRS to list only directories of at least the specified number of kilobytes. # of 0 means don't check. Note: the MBS option is usually used, but specifying KBS allows you to test BIGDIRS with smaller directories. MAXFiles # MAXFILES tells BIGDIRS to stop any SEARCH or PURGE after checking the specified number of files. Although we have not encountered directory loops, this option was added to detect a possible loop in the directory and stop the search/purge when one is detected. The default value is 300,000, which should be sufficient for nearly every system. A value of 0 disables this check. Note: BIGDIRS has a compiled-in limit of how many filenames it can remember while scanning directories. This number is not related to MAXFILES. MBS # MBS # tells BIGDIRS to list only directories of at least the specified number of megabytes. # of 0 means don't check. SOME Only list big and somewhat big directories. (More verbose than BIG, less verbose than ALL.) (Modes: QUIET, BIG, SOME, ALL) VARs (See the 'options' section below) ---- Defaults for SEARCH: -ENTRIES 20000 -KBS 2048 -VARS -OR -PATH / (A path of "/" means that BIGDIRS will scan the entire system directory.) // Note: the directory "size" (for -mbs/-kbs) is based on the EOF of the directory file. SET [option]* RESET [option]* SET (or RESET) the specified options. After parsing, all the global options are displayed. TESTMAP [ALLOCate] [#] [ALLOCate] [List] TESTMAP is used by Allegro's R&D VERsion Common options: AND A directory will be reported if it has both at least the specified number of entries and at least the specified number of KBs or MBs. (Mostly used by SEARCH) (See also: OR) DEBUG1 Enable debugging for R&D noFAKE FAKE tells BIRDIRS to not actually purge/purgegroup. noFEQ The FEQ option will allocate by opening a file named 'maptemp', which you can file equate. (You'd file equate it to point to desired volume set, in case you're having problem with disk allocation on the logon volume set.) Note: see 'PUBSYS' for important notes about FEQ. NORMal NORMAL selects a medium level of output. OR A directory will be reported either if it has enough entries or if it is large enough (-kbs or -mbs) to be considered interesting. (See also: AND) (Mostly used by SEARCH) PAGING noPREFERKBS noPREFERMBS Tell BIGDIRS how you like seeing disk usage reported. (E.g., in kilobytes or megabytes) PROGRESS [#seconds] NOPROGRESS PROGRESS tells BIGDIRS to report progress while searching for interesting directories and/or files. The optional #seconds tells us how often to report. NOPROGRESS disables this progress reporting. Session default: PROGRESS 15 Job default: PROGRESS 300 (and, it goes to the console) PUBSYS PUBSYS overrides FEQ. The default is neither, which means that the temporary file is allocated on the volume set your login group is home'd to. Note: PUBSYS and FEQ can only be specified via the INFO parameter. Note: PUBSYS and FEQ are ignored if the temporary file is already allocated (e.g., due to a prior 'purgegroup' command). QUIET Generally try to produce less output RM RM *enables* the actual purging of files/directories/groups. If you don't specify 'RM' (either via 'SET RM' or by using the RM option on a PURGEPATH or PURGEGROUP command), then nothing will actually be deleted! RM is a safety check ... to ensure that you really want to delete files! STATISTICS STATISTICS tells BIGDIRS to display some statistics at the end. TIMES TIMES tells BIGDIRS to report the CPU and elapsed time it used. VARs VARS tells BIGDIRS to create some CI variables when it is done: BIGDIRS_NUM_DIRS ... number of directories seen BIGDIRS_NUM_BIG ... number of "big" directories (based on ENTRIES, KBS or MBS) BIGDIRS_BIGGEST_ENTS ... largest number of entries seen in a directory BIGDIRS_BIGGEST_KBS ... size of largest directory seen (Only applies to SEARCH) Verbose Generate more output. ------ --------------------------------------------------------------- How directory inserts/deletes work. On MPE/iX, every directory (account, group, or HFS) is a separate file, with entries in alphabetic order. When a file (or directory) is inserted into a directory, MPE/iX locates the place to insert the file, and does the following: 1. start a transaction (to protect the integrity of the file system in the event of a system abort before all of the updated data gets posted to disk); 2. moves the entire rest of the directory "up" one slot, starting at the insertion point (the "slot" that will have the new entry eventually); (This data is recorded by Transaction Management.) 3. inserts the new entry into the slot just freed up by the move; 4. terminates the transaction. Thus, if you are adding a file at slot 10, and there are 90 files after it, the amount of data moved by step 2 would be 90 times the size of a slot. When the number of entries in a directory becomes large (over 25,000 for a group directory, or about 10,000 for an HFS directory), the amount of data being moved usually exceeds one or more megabytes. When the size of this data approaches about five megabytes, system aborts due to "stalled transaction" become more likely. Deleting entries from a directory is done in a similar manner, except that the "move" of data is "down", overwriting the slot formerly in use by the deleted entry. --- Deleting all files in a directory. In some cases, users have a directory so large that most insertions/deletions into it will trigger a system abort ... and they want to delete the directory (or group). Unfortunately, in such cases the attempt to delete the directory or group will also trigger the system abort. The description above of how directory insertion/deletion works suggests the workaround: get a list of the files (and/or directories) in the directory (or group), put them in reverse alphabetic order, and delete them one at a time. (E.g., :purge Z, :purge FRED, :purge A) This works because deletions at the end of a directory result in very small transactions. --- Note: control-Y may not interrupt a SEARCH, because of how interrupts interact with directory searching. If this happens, you can usually interrupt BIGDIRS by doing: :setjcw jcw 1 :resume --- A few options can be specified via PARM (the bits are additive): parm = 1 verbose parm = 2 times parm = 4 noprogress (i.e., progress is the default) parm = 8 debug1 // bigdirs 20240129