Mass Imputation#

Execution: banff.massimp()
SDE function type: Review, Selection, Treatment
Input status flags: None
Output status flags: IMAS

Description#

Performs donor imputation for a block of variables using a nearest neighbour approach or random selection.

The massimp procedure is intended for use when a large block of variables is missing for a set of respondents, typically when detailed information is collected only for a subsample (or second phase sample) of units. While the donorimp procedure uses both system and user matching fields, mass imputation only considers user matching fields to find a valid record (donor) that is most similar to the one which needs imputation (recipient).

Mass imputation considers a recipient any record for which all the variables to impute (must_impute) are missing on indata, and considers a donors any record for which none of the listed variables are missing. If matching fields (must_match) are provided by the user, the massimp procedure uses them to find the nearest donor using the same distance function as donorimp. If matching fields are not provided, a donor is selected at random.

Unlike donorimp, the massimp procedure does not use edits. Before running the procedure, users should ensure that the pool of potential donors do not include any errors, including outliers or consistency errors.

Users may create by-groups by specifying by variables. These by-groups act as imputation classes. Use the min_donors and percent_donors parameters to ensure an appropriate number or ratio of recipients and donors exist in each imputation class before performing imputation.

For a full mathematical description of the procedure methods, with examples, please see the Functional Description.

Input and output tables#

Descriptions of input and output tables are given below. Banff supports a number of input and output formats; please see the Banff User Guide for more information.

Input Table	Description
indata	Input statistical data. Mandatory.

Output Table	Description
outdata	Output statistical table containing imputed data. Note that outdata will only contain successfully imputed records and affected fields. Users should update indata with the values from outdata before continuing the data editing process.
outstatus	Output status file identifying imputed fields with IMAS status flags, and their values after imputation.
outdonormap	Output table of recipient-donor pairs for successfully imputed records.

For details on the content of output tables, please see the Output Tables document.

Parameters#

Parameter	Python type	Description
unit_id	str	Identify key variable (unit identifier) on indata. Mandatory. Must be unique for each record. Records with a missing value are dropped before processing.
must_impute	str	Variables(s) to be imputed. Mandatory. To be a recipient, all the variables listed on `must_impute` must be missing for an observation. If all these variables are non missing, then this observation is a potential donor. Example: `must_impute="revenue_q1 revenue_q2 revenue_q3 revenue_q4"`
must_match	str	User defined matching field(s). must_match is optional; when not used, random parameter must be specified and a donor will be selected randomly for each recipient. Example: `must_match="revenue headcount"`
random	bool	Random selection of donors. When `random` is used alongside `must_match`, random selection will be applied to recipients with missing values for all `must_match` fields.
min_donors	int	Minimum number of donors required to perform imputation. Default=30.
percent_donors	float	Minimum percentage of donors required to perform imputation. Default=30.
n_limit	int	Maximum number of times a donor can be used.
mrl	float	Multiplier ratio limit. This parameter is multiplied by the ratio of the numbers of recipients to donors, then the result will be the maximum number of times a donor can be used.
seed	int	Specify the root for the random number generator. The seed is used to ensure consistent results from one run to the next. If not specified or specified as a non-positive value, a random number is generated by the procedure.
accept_negative	bool	Treat negative values as valid. Default=False. By default, a positivity edit is added for every variable in the list of edits; this parameter permits users to remove this restriction. If required, users may directly add positivity edits for individual variables.
by	str	Variable(s) used to partition indata into by-groups for independent processing. In massimp, by-groups can also be seen as imputation classes. Example: `by = "province industry"`
presort	bool	Sorts input tables before processing, according to procedure requirements. Default=True.
no_by_stats	bool	Reduces log output by suppressing by-group specific messages. Default=False.

Notes#

Nearest neighbour or random donor#

The parameters must_match and random determine whether the nearest-neighbour algorithm or random selection is used to select donors. The following table shows how specifying these parameters affects mass imputation.

`must_match` specified	`random` specified	Syntax	Imputation
No	No	Incorrect	Results in an error, no imputation is performed.
No	Yes	Correct	Random selection of donors.
Yes	No	Correct	Nearest neighbour selection using `must_match` variables.
Yes	Yes	Correct	Nearest neighbour selection using `must_match` variables, or random selection for recipients with missing values for the `must_match` variables.

If a recipient has missing values for some but not all must_match variables, the distance to the closest donor will be based only on the must_match variables that have valid values. If a recipient has missing values for all must_match variables, then it will be randomly matched to a donor if the random parameter is specified, and it will not be matched to any donor if the random parameter is not used.

Multiple equivalent solutions#

In some cases, for a given recipient, there may be multiple equidistant donors (i.e. having the same distance from the recipient) whose values would allow the recipient to pass the edits. When this occurs, the procedure selects one of these solutions at random.

For development or testing purposes, users may wish to produce consistent results over multiple runs of the procedure, and may do so using the seed parameter. It ensures that the same solutions will be selected from one run to the next, if executed on the same set of inputs. Note that if seed is not specified, the system generates a default seed.

This parameter can also be used to replicate results when random donor selection is performed.

Limiting the repeated use of donors#

Users may limit the repeated use of donors with the interrelated n_limit and mrl parameters. The donor limit is calculated as follows, depending on whether one or both are specified:

`n_limit`	`mrl`	Donor Limit
No	No	Number of times a donor can be used is unlimited.
No	Yes	round(`mrl`*(recipients/donors)).
Yes	No	`n_limit`.
Yes	Yes	round(max(`n_limit`,`mrl`*(recipients/donors))).

When limiting the number of donors with the n_limit parameter, the number of remaining donors may end up being less than min_donors. In such a case, mass imputation will continue and ignore min_donors which was validated at the beginning. The same applies for percent_donors.