Vol. 19, #1 – Do We Know Who We Are? Limitations Of The 2011 National Household Survey (NHS)

November 2013

The release of the much anticipated income data from the 2011 National Household Survey (NHS) in September has placed a chill over the planners and providers of social services, municipal services and economic development strategies. Good information is essential for the development of responsive and efficient social and economic infrastructure. Tragically, many of the concerns about the impact on the integrity of the Statistics Canada data as a result of the Government of Canada replacing the mandatory long form census with a voluntary National Household Survey have proven true (See Community Dispatch Vol. 18, No. 5). This Community Dispatch analyzes the limitations to the recently released data on income, homeownership and shelter cost of Canadians. The information is critical in understanding issues of income security and inequality across our land, province and communities. How do we answer the question: Do we know who we are?

Joey Edwardh

This Community Dispatch focuses on the limitations of the income data from the National Household Survey especially in data quality, data comparability and low income measures.

Data Quality

Due to the change in data collection methodology from a mandatory census to a voluntary survey the non-response rate of the NHS is significantly higher than those of the previous long form census. At the national level, the NHS Global Non-response Rate (GNR)¹ is 26.1% compared to 6.5% with the 2006 long form census. In Ontario, the County of Peterborough has a high non-response rate of over 38%. A high non-response rate implies low data quality.

In addition to an overall lower response rate, certain population groups are less likely to respond to the survey such as high and low income individuals, Aboriginals, newcomers and visible minority groups. At this point, there is no way to determine their respective response rates. In order to maintain a high level of data quality, Statistics Canada suppresses data release in geographic areas with a non-response rate higher than or equal to 25% in previous censuses. However, given the high non-response rate and the associated non-response bias² of the NHS, Statistics Canada raised the data suppression threshold from 25% to 50%. In doing so, data are released for more geographic areas but at lower data quality.

“We’re concluding it [the NHS income data] pretty much is garbage,” said Professor David Hulchanski, University of Toronto.³

The map shows the geographic distribution of the 2011 NHS Global Non-response Rate (GNR) by census tract⁴ for Halton Region. Although there is no census tract with a GNR over 50%, there are many areas (shaded in dark brown) with non-response rates equal to or over 25% (suppression threshold for previous censuses). Almost two-thirds (63%) of the census tracts recorded non-response rates equal to or over 25%. The map can serve as a reference point for data quality when interpreting NHS data by census tract in Halton Region.

Data Comparability

Although the 2011 NHS provides a snapshot of the income composition in Canada including income sources, government transfers, income redistribution and income by family types, the data should not be compared with those from previous censuses and other similar sources.

Previous long form censuses include usual residents in collective dwellings (lodging or rooming houses, hotels, nursing homes, hospitals, staff residences, jails and so on) and persons living abroad whereas the NHS target population excludes them. The census and NHS represent two different populations.

In fact, Statistics Canada warns users to use caution when comparing income estimates from the NHS to other household income surveys, administrative data or earlier censuses.⁵ The methodology of how the information was collected, the concepts used and response patterns can affect the comparability of income information.

One cannot establish reliable trend lines from previous censuses to see if Canadians are better or worse off in terms of their incomes. The data gap created between 2005 and 2010 (income data are collected a year prior to the census) precludes any assessment of the impact of the economic recession in 2008. This also includes potential impact of any government economic action plan.

“…the lack of comparability between the 2006 census and the 2011 NHS will make it very difficult for researchers and analysts to judge to what extent our social programs blunted the impact of the Great Recession on the incomes of many middle-class and working families…” said Andrew Jackson, Packer Professor of Social Justice at York University and senior policy adviser to the Broadbent Institute.⁶

Low Income Measures

In previous censuses, Statistics Canada employs Low Income Cut-offs (LICO) as a measure of low income. It is an income threshold below which families or unattached individuals spend 20% more than average on food, shelter and clothing needs. LICO thresholds are also set at income levels by family size and size of community.

For the 2011 NHS, instead of LICO, Statistics Canada chose the after-tax Low Income Measure (LIM-AT) as a measure of low income. Individuals are defined as having low income if the after-tax income of their households fall below 50% of the median adjusted household after-tax income in 2010.

While LIM is an important measure used broadly internationally, it is important that researchers have access to data flowing from both measures; LIM and LICO.

Low income estimates from the 2011 NHS compared to previous censuses show markedly different trends than those derived from other surveys and administrative data such as the Survey of Labour and Income Dynamics (SLID) or the T1 Family File (T1FF).⁷ As such, Statistics Canada stated that the NHS low-income estimates are not comparable to census-based estimates produced in the past.

The data gap on low income between 2005 and 2010 will hinder work on poverty reduction in Halton’s local areas and neighbourhoods.

Conclusion

“The main finding of the NHS is that, with weaker ability to track change or measure a growing portion of society, we’re losing sight of the Canada we’re becoming.” said Armine Yalnizyan, Senior Economist at the Canadian Centre for Policy Alternatives.⁸

Given the problems associated with data quality and comparability, the usefulness of the income data from the NHS is severely limited. Improper use of the data can lead to erroneous conclusions, faulty comparisons, spurious debate and, ultimately, unwise decisions.

At Community Development Halton, we will continue to assess other data sources to meet the data needs of CDH and our community partners.

In the meantime, CDH asks that the federal government reinstate the mandatory long form census for 2016 and beyond.

5. Statistics Canada, Income Composition in Canada, National Household Survey, 2011, Cat. No. 99-014-X201100

6. Jackson, Andrew. 2013. Even a bad survey cannot blind us to income inequality. http://www.theglobeandmail.com/report-on-business/economy/economy-lab/even-a-bad-survey-cannot-blind-us-to-income-inequality/article14251936/

7. Statistics Canada, Persons Living in low-income neighbourhoods, National Household Survey, 2011, Cat. No. 99-014-X2011003

8. Yalnizyan, Armine. 2013. National Household Survey provides blurred look at housing.
http://www.theglobeandmail.com/report-on-business/economy/economy-lab/national-household-survey-provides-blurred-look-at-housing/article14271791/

  Community Dispatch

Produced by Community Development Halton
860 Harrington Court
Burlington, Ontario L7N 3N4
(905) 632-1975, (905) 878-0955; Fax: (905) 632-0778; E-mail: office@cdhalton.ca