Vol. 18, #5 – 2011 National Household Survey (NHS)

June 2013

In 2010, the federal government replaced the mandatory long form census with a voluntary National Household Survey (NHS). This change in the data collection method has drawn strong opposition and protests from data users, municipalities, for profit and nonprofit organizations, academics, researchers, economists and statisticians. A voluntary survey has serious negative impact on data quality, reliability and comparability. There is no way of knowing how good or bad the data will be.

Community Development Halton (CDH), along with eleven charitable nonprofit agencies, launched a Charter challenge against the federal government’s action to abolish the mandatory long form census. Unfortunately, all appeals, petitions and legal actions to reinstate the long form census were unsuccessful.

On May 8, 2013, Statistics Canada released its first set of data from the 2011 NHS. The release covers immigration, citizenship, place of birth, ethnicity, visible minorities, religion and Aboriginal peoples.

As expected, the voluntary nature of the survey has resulted in low response rate and non-response bias. These limitations have prompted Statistics Canada to issue warnings and cautions to its readers about the reliability and comparability of the NHS data.

Furthermore, unlike previous long form census, the NHS data are currently published only at the national, provincial, Census Metropolitan Area (CMA) and Census Agglomeration (CA) level.¹ No cross-tabulated data have been released at the Census Subdivision² (CSD) level. Statistics Canada is still determining how and when data at the lower geographies (i.e. Census Tract and Dissemination Area) will be released.

Response Rate

Although the NHS was distributed to 1 out of 3 households instead of 1 out of 5 as in previous long form census, the response rate³ is significantly lower than that of the long form census. At the national level, the NHS response rate is 68.6% compared to 93.5% with the 2006 long form census. In Halton Region, the response rates at the local municipal level are higher than both the provincial and national averages.

Source: Statistics Canada, National Household Survey, 2011, final response rate

In addition, the response rate to each question on the returned survey questionnaire also varies. The response rate for questions on demographic characteristics (age, sex, marital status) is highest. The response rate is lower for questions on employment, income and dwelling characteristics.

Non-response bias

The voluntary nature of the survey has resulted in significant non-responses. In the 2011 NHS, more than 30% of those who received the survey did not respond. It has been established that those who did respond to the survey have different characteristics than those who did not respond. For example, Aboriginals, new immigrants and low-income individuals are less likely to respond to the survey.

Here is a simple example to illustrate the impact of non-response bias. In Area A, 50% of the 100 residents voluntarily responded to the survey. The response rate is 50% (50/100 x 100%). One out of two residents responds. However, among the residents, 25 are low-income individuals and did not respond to the survey at all. Their response rate is 0% (0/25 x 100%). The responses are then weighted by the overall response rate so that the data from the sample represent the target population. The final result will show Area A has no low income individuals since no information of that population group was captured in the survey.

To take this example further, five years ago, the same population responded to a mandatory survey with a 90%+ response rate. The low-income population would have been captured in the survey. The data will show about 25 low income persons lived in Area A five years ago and there is no low income person living there now. How do we interpret the data? Does it mean:

• all low-income persons are no longer in poverty?
• Or they have all moved to another part of the city.
• Or the data is unreliable?

In reality, different population groups may respond differently. Their response rates may range from zero to several percentage points. Unfortunately, there is no way to confirm this.

Statistics Canada, in its first release of the NHS data, points out an example of overestimation resulting from non-response bias. The 2011 NHS showed that Canada received the highest number of immigrants from the Philippines between 2006 and 2011. However, the administrative data from Citizenship and Immigration Canada (CIC) indicated that the NHS has overestimated the Filipino immigrants by 10,000 for the same period.

Data suppression

Statistics Canada suppresses the release of NHS data to maintain confidentiality and data quality. In order to prevent the release of information that could be used to identify respondents, census geographic areas whose population is below a certain threshold are not released or published.

Statistics Canada also uses the global non-response rate as a measure of data quality. This indicator combines the complete non-response (household) and partial non-response (specific question) into a single rate. The NHS’s global non-response rate is 26.1% for Canada and 27.1% for Ontario respectively.

The NHS estimates of any given geographic area that have a global non-response rate greater or equal to 50% are not published in the standard census products. As a result, NHS data for about 25% of the 4,567 CSDs are not available. The data release rate at the CSD level of the 2006 long form census was 96.6%.

In Ontario, NHS data are published for 429 CSDs (81.4% of all CSDs).⁴ On the other hand, Saskatchewan has the lowest release rate of 57.4%. In other words, NHS data are not available for more than 40% of the municipalities in Saskatchewan.

How does this impact our work?

Given the high non-response rate and the associated non-response bias of the NHS data, it is very likely that we will not have access to up-to-date and reliable socio-demographic data at the local and neighbourhood level. This situation deteriorates for cross-tabulated data. For example, we may know how many recent immigrants live in a census tract but may not know their income, employment or education situation. This is key information required to plan, develop and deliver community and neighbourhood based services and programs.

Can we compare the 2011 NHS data with previous censuses? The short answer is a qualified “yes” for high-level geography (e.g. national, provincial or regional). However, as mentioned earlier, for lower level of geography (e.g. municipal and neighbourhood – CT/DA), readers will find it difficult to determine if the changes reflect a real change or simply the result of poor quality of data.

The same scenario applies to the comparison of 2011 NHS data among different geographic areas.

The loss of the 2011 long form census also creates a void for Statistics Canada to “adjust” other surveys⁵ when the data are affected by response biases. The data quality of other surveys will suffer.

What can we do?

Short of a long form census replacement, it is difficult to find other data sources that will provide the same level of data quality and geographic coverage.

With the release of the NHS data and their associated deficiencies, the case against a voluntary survey is crystal clear. It also confirms that a larger sample size and more money do not cut it. As a community based nonprofit agency committed to social development for all members of our community, CDH and others will continue its pressure on the federal government and our federal representatives to reinstate the mandatory long form census.

In the meantime, Community Development Halton will continue to assess other data sources to meet its data needs as well as those of CDH partners. Some of these data sets are:

• The administrative data from Citizenship and Immigration Canada (CIC) provide gender-based information on permanent (landed immigrants) and temporary residents by age, level of education, marital status, language ability, occupation skill level and intention to work. The data are available at the municipal level annually.

• The Transportation Tomorrow Survey (TTS) provides transportation related data by traffic zone (similar to Census Tract) for the GTHA (Greater Toronto and Hamilton Area) and immediate surrounding municipalities. The survey corresponds to census year. The 2011 results will be available in the fall of 2013.

• From the income tax forms submitted each year by Canadians, Statistics Canada produces sets of economic and demographic information at various census standard geographies including Census Tract and Postal codes. Low income data based on Low Income Measures (LIM) are available. The information is also available on an annual basis with a time lapse of about 18 to 24 months.

In conclusion, the loss of the mandatory long-form census significantly affects communities in answering the question “who are we?” This, in turn, limits our capacity to plan and deliver appropriate economic, social and health services.

  Community Dispatch

Produced by Community Development Halton
860 Harrington Court
Burlington, Ontario L7N 3N4
(905) 632-1975, (905) 878-0955; Fax: (905) 632-0778; E-mail: office@cdhalton.ca