BJS: Bureau of Justice Statistics

Home  |  About Us  |  Contact Us  |  Help  |  A-Z Topic List
 
 
Bureau of Justice Statistics (BJS)
Home  |  Publications & Products

Bureau of Justice Statistics Customer Feedback Survey

By Marianne W. Zawitz, Chief, Publications and Electronic Dissemination

August, 2004

Contents: Background | Survey strategy | Findings | Conclusions | Methodology

Tom Hester, Chief, Publication Development and Verification; Priscilla Middleton, Information Specialist; and the NCJRS staff made significant contributions to these surveys and the report.

The Bureau of Justice Statistics (BJS), the official source of crime and justice statistics in the United States, disseminates the data it collects to Federal, State, and local policy makers, criminal justice practitioners, the media, educators, researchers, students, and the general public. BJS conducts surveys on criminal victimization and the operation of the U.S. justice system, which consists of over 50,000 different agencies.

BJS periodically surveys its regular customers; surveys were conducted in 1994, 1997 and 2003. With the advent of the Internet, the earlier surveys focused on the interest and ability of customers to move to electronic dissemination methods. Now that access to the Internet and familiarity with associated technologies are no longer issues for most BJS customers, we have turned to finding out what our customers do, whether they use statistics to perform their functions, and what information resources they use.

BJS publishes all materials in electronic formats; the release is on the Internet and, if warranted, publications are subsequently printed. Since the website was launched in 1996, usage has increased steadily. The BJS website hosts as many as 18,000 visitors a day. Over a third of our visitors are referred by search engines. On the other hand, over 20% are not referred by other sites but get to our site by bookmarks or direct entry of the URL. Around 18% of our website visitors are repeat visitors. We know the Internet has expanded our audience, and key audiences also access our information online.

As our website has matured, our paper distribution has declined, dwarfed by web activity. BJS continues to distribute a fairly large number of short paper publications to specific groups of customers through bulk mailings, a low cost method of paper distribution. Our distribution strategy has been to concentrate this bulk distribution to those customers who have registered with the National Criminal Justice Reference Service (NCJRS), a clearinghouse service sponsored by a number of Federal agencies with an interest in crime and justice.

Approximately 40,000 customers were registered when the study was conducted. Almost half of the registered users work in a justice agency. Less than 1% were affiliated with the media.



Organizational affiliation

Percent of registered users

Criminal justice agency 45.1%
Educational association/university 16.0%
Community organization 13.1%
Social services 4.6%
Private sector 4.0%
Health services 2.5%
Research/statistics organization 2.5%
Executive branch 1.4%
Professional association/society 1.1%
Media organization 0.8%
Legislative branch 0.6%
Indian tribal community 0.5%
None 7.8%
  100.0%

Most of the registered users were either administrators/managers or staff. At the time of the surveys, the registered user list had the following composition:

Position Percent of registered users
Administrator/ manager 41.1%
Staff 29.1%
Educator 10.0%
Researcher 7.2%
Student 6.2%
Elected/appointed official 2.8%
Volunteer 1.6%
General public 0.5%
Other and no response 1.4%
Total 100.0%

Registered users select topics of interest (law enforcement, violence, etc.) as well as type of information they are interested in (statistics, research, etc.) Forty percent of registered users indicated an interest in statistics. Customers who register are told they will automatically be sent documents from time to time, and they are subscribed to an E-mail newsletter announcing new publications.

In addition to the registered users, NCJRS maintains lists of targeted mail recipients, such as elected and appointed officials, association memberships, and related rosters. BJS has used these lists in their dissemination program but to a lesser degree than the registered user list focusing on Federal officials including the Congress, agency officials, and professional associations.

Many people on the registered and targeted lists are both users of our data and respondents to our surveys.

up arrow To the top

Survey strategy

Even though the number of web users dwarfs our paper distributions, we decided not to survey web users since such surveys suffer from extremely low response rates and other problems which affect the ability to infer results. We decided to survey paper recipients since we have much more information about them and know that they represent key audiences. In addition, we were able to conduct a sample survey of registered users so we could empirically assess quality of the results.

Past surveys had focused on registered users but not on user behavior with relation to specific reports from BJS. Therefore, we decided to tie the current effort to specific documents, selecting recent documents in topical areas to ensure coverage of audiences for subject matter such as law enforcement, courts, corrections and victimization. These areas of interest correspond to the primary audiences for BJS reports.

We assumed that members of each group were very different in terms of what they did and how they retrieved information. The following documents were selected for the survey:

    Registration selections
Title Length
in pages
Topical interests Statistics
interest
Local Police Departments 2000 33   Law enforcement  
Intimate Partner Violence, 1993-2001 2   Victims, Family Violence  

Prison and Jail Inmates
   at Midyear 2002

12   Corrections X
Juvenile Felony Defendants
    in Criminal Courts,1998
12   Courts, Juvenile Justice X

Since the registered user lists include phone numbers for most users, we decided to conduct a phone survey of registered users. The detailed information concerning the characteristics of the registered users also allowed us to do a sample survey of 300 registered users for each publication. For targeted users, we had only mailing addresses, so we conducted a mail survey. Two weeks after each report was sent by the bulk mail, a sample of registered users were called and a mail survey went to all of the targeted users except for "certainty units" like members of Congress. The following table summarizes the universe and responses for each report:

  Registered users Targeted users   
Reports  Sent the report Successful interview Sent the report Returned survey Total respondents
Local Police Departments,
    2000
17,391 300 2,800 352 652
Intimate Partner Violence,
    1993-2001
19,019 300 3,591 389 689
Prison and Jail Inmates at
    Midyear 2002
24,355 300 3,242 587 887
Juvenile Felony Defendants
    in Criminal Courts
8,988 300 6,010 258 558
Total 69,753 1,200 15,643 1,586 2,786

The registered customer profile drawn from the NCJRS Customer Relationship Management System data resembles the profile of the sampled users who completed the phone survey. The only exceptions were that smaller proportions of elected officials and students responded to the phone survey than were in the original list. We anticipated these proportions would be lower due to the difficulty in contacting elected officials directly by phone and in contacting students in the summer when school is out of session. Additional detail on the methodology is available at the end of this paper.

We also fielded a survey of people who ordered these specific publications. These mail surveys were sent after the order was fulfilled. Because the number of orders was small and the response rate was low, the responses from this survey are not included here.

We analyzed the results by publication and by type of recipient. Unless otherwise noted, responses between the registered user recipients of the various reports did not differ significantly. Since the survey of the targeted mail recipients had a very low response rate, the findings presented are not representative of the entire group.

up arrow To the top

Survey findings

What did customers do with the documents they received?

Most customers scanned the documents they received. Of the registered users, few read the document thoroughly. Over one-fourth of registered users did not receive the document or were not sure that they received it. A higher proportion of targeted mail recipients than registered users had read the document thoroughly or had scanned it. This result is probably due to the fact that the targeted mail survey was a voluntary survey so motivated recipients were more likely to respond.

  Registered users
Targeted mail recipients
Read it thoroughly 6% 25%
Scanned it 47% 59%
Did not read it 19% 9%
Not sure 18% 2%
Never received it 10% 5%
  • After their initial review, what did they do with the report?

    Until this survey, BJS had little information on whether the documents they sent to customers were being used or thrown away. Clearly, if users were throwing our documents away, the continued practice of sending unsolicited materials would need to be reconsidered. The survey showed that most respondents kept the document for reference and few threw it away. All of these documents contained statistical content which may be why they were kept for reference. Materials intended to notify users of publications, events, and other opportunities that do not contain reference content may not be treated the same way.

    We were also interested in the secondary distribution of our materials to others who were not the initial recipient. Of the registered users, almost a third gave the documents to others or circulated them, while half of the targeted users shared the documents. In either case, sharing the documents expands the number of people who get direct access and increases the outreach from a single mailing.
  Percent of respondents who acknowledged receipt
Registered users Targeted mail recipients
Gave it to someone else 9% 11%
Circulated it to others 20% 39%
Kept it for reference 66% 43%
Threw it away 5% 7%
  • If they threw it away, why?

    Of registered users who threw the document away, 60% said they were not interested in the topic. Some indicated that they threw it away because they could use the online version if needed later. Few said they never use statistics or had no space for the document. The majority of the registered users are selected to receive a document by their expressed topical preferences and an interest in statistics. Perhaps their lack of interest in the topic reflected a mismatch; the topics used in registration are very broad and the documents represented a narrower subtopic.
  Percent of respondents who threw document away
Reason for throwing it away Registered users Targeted mail recipients
Already read the online version 3% 0%
Not interested in this topic 60% 25%
Never use statistics 10% 21%
No space for it 10% 21%
Will use online version if needed again 18% 34%
  • Did they know about it before they received it?

    Slightly over a third of the registered users who received the document reported knowing about the document before they received it. Among targeted users, less than a quarter knew about it in advance of receipt. Prison and Jail Inmates at Midyear had the highest percentage of registered users with prior knowledge. This is probably because this is an annual publication they may have received before. In addition, some may be contributors to the surveys the report is based on.

  Percent who indicated knowledge of the publication before receiving it
  Registered
users
 Targeted
 mail
  recipients
Total 36% 23%
Prison and Jail Inmates at Midyear 43% 27%
Juvenile Felony Defendants 37% 16%
Local Police Departments 34% 28%
Intimate Partner Victims 28% 22%

Further probing in the survey revealed that of registered users who knew about the report before receiving it, a third knew about it through email notification. Of targeted users, a fifth knew about it by email notification. This represents 9% of the all registered users who were surveyed and 5% of the targeted mail recipients who responded. Browsing the Internet was the second most frequently given method of finding out about a document for registered users. Few reported that they learned about the report by other means, which was to be expected since most other means had not been actively pursued as marketing techniques. Only Prison and Jail Inmates at Midyear 2002 was released by a news release, and there was no direct attempt to use article placement or advertising in other professional publications.

NCJRS conducts regular notification of registered users through an electronic biweekly news letter, JUSTINFO. In addition, BJS operates a ListServe which immediately notifies interested subscribers of all statistical releases from three agencies in the Department of Justice. BJS announces their publications and releases in both. Therefore, the small number who reported that they knew of the report through email notices was disappointing. It is possible that they did not read the notification or that they did not remember the announcement since the electronic announcements occur as soon after the publication is available on the Internet, several months before print copies are available.

  Percent of
all respondents
How they knew it was available Registered
users
Targeted mail recipients
E-mail notification 34% 21%  
Browsing the Internet 27% 12%  
Some other way 25% 30%  
From professional publication 6% 21%  
From a colleague 5% 10%  
News report 3% 6%  
  • Would they have ordered this document if they had not received it?

    Over half of the registered users who received the document indicated that the document they were sent was something that they would have ordered, if they had not received it. For targeted mail respondents, less than one third would have ordered it. Registered users who received Juvenile Felony Defendants were most likely to indicate that they would order the document.

  Percent who would have ordered the document
Registered
users
Targeted mail recipients
Total 54% 31%
Local Police Departments 52% 37%
Intimate Partner Victims 54% 36%
Prison and Jail Inmates at Midyear 49% 26%
Juvenile Felony Defendants 60% 24%

The results for the registered users show that they represent a defined market for these documents. The results from the targeted mail respondents present a less clear picture since they were more likely to read the document thoroughly and were more likely to share it.

These surveys were conducted during a time when citizens could order most documents from NCJRS at no cost. Since that time, limits have been placed on the number of documents that can be ordered for free and on which documents are available for free. Therefore, the willingness of the respondents to order documents may change when faced with fees.

  • Do they mind receiving unsolicited mail from us?

    Both groups had positive feelings about being sent unsolicited materials. Few were negative about this practice. The support voiced by targeted mail recipients may be skewed due to the voluntary nature of the response. However, the registered users' positive answers show us that we can continue this approach without concern that we are intruding on our customers.
Feelings on being sent
unsolicited materials:
Registered
users
Targeted mail
recipients
  Very positive 61% 56%
  Somewhat positive 30% 26%
  Neutral 7% 15%
  Somewhat negative 2% 2%
  Very negative 1% 1%
    100% 100%

What do our customers do and do they use statistics?

  • What activities do they perform?

    A major focus of the survey was to ask what activities customers perform. We know they are employed in criminal justice agencies, educational institutions, and other government and private organizations. However, we were uncertain about the kinds of tasks they undertake and how they use statistics. Registered users most frequently responded that they write reports, give presentations, and plan. We have anecdotal evidence that they are often involved in writing proposals; however, slightly over half said they engage in this activity, far fewer than other activities listed.

    Fewer targeted users indicated performing most activities. Slightly more than half of the respondents indicated that they give presentations, plan, write reports, and prepare programs. We suspect that the differences between registered users' and targeted mail recipients' involvement in the activities is due to the different methods of data collection, phone survey vs. paper survey.
    Activities Registered Users   Targeted Mail Recipients  
    Writes Reports 83.9 % 53.0 %
    Gives presentations 82.8   57.1  
    Plans 78.1   54.8  
    Trains 76.3   46.8  
    Researches 67.7   43.9  
    Prepares programs 66.3   52.0  
    Conducts Conferences 61.9   25.8  
    Teaches 60.4   43.9  
    Writes proposals 52.8   40.5  
    Drafts legislation 32.2   19.9  
    Media 29.8   18.3  
    Does school work 25.1   6.8  
    Other activities 2.8   19.5  

    The small number of other activities listed is an indication that the list presented was fairly exhaustive, particularly for the registered users.
  • Do they use statistics in performing these activities?

    Ninety-two percent of the registered users indicated that they used statistics to complete the activities reported. Of targeted mail recipients, 83% said they used statistics. Since many registered users who received our documents had indicated an interest in statistics at registration, a high level of interest was expected. This high level of use by both groups suggests that our customers value statistics.

    We also asked them if they would use the report that we sent them in the activities reported. Overall, 72% of registered users and 86% of the targeted mail recipients said that they would use the report we sent them. Registered users who had received Local Police Departments and Intimate Partner Victims were slightly more likely than the recipients of Prison and Jail Inmates at Midyear and Juvenile Felony Defendants to say that they would use the documents. Targeted mail recipients were more likely than registered users to say they would use each of the documents, but this may have been a function of self-selection in responding to the mail survey.

  Would use the report they received to perform their frequent activities
Report Registered
users
Targeted
mail
 
Total 72%   86%  
Local Police Departments 80%   89%  
Intimate Partner Victims 75%   81%  
Prison and Jail Inmates at Midyear 63%   89%  
Juvenile Felony Defendants 69%   85%  
  • Where do they seek the information they need?

    As with the other multiple response categories, registered users were more likely to respond to this question. Almost 95% of the registered users said that they looked for information on the Internet, the most frequent behavior listed. Almost as many registered users would consult other materials, while 80% would ask a colleague for information. For targeted mail users, 82% reported that they would consult other materials and 78% would look for information on the Internet. Over three-quarters of the registered users would contact NCJRS and slightly fewer would directly contact the source. For targeted mail users more than half would directly contact the source of the information. They were much less likely to contact NCJRS than registered users (40% compared to 77%). For both groups, the least likely option for seeking information was going to the library.
Seeks information from: Registered
users
Targeted mail recipients
Looks for information on the Internet 94.7 % 77.6 %
Consults other materials 94.0   82.3  
Asks a colleague for information 80.0   44.5  
Contacts NCJRS for information 76.8   40.4  
Directly contacts source 72.2   52.5  
Goes to the library for information 43.1   21.2  
Note: Respondents could select more than one place.

We were also interested in what type of information our customers usually use. For registered users, 95% used the Internet followed by other government sources (92%) and other BJS reports (90%). In general, registered users were more likely than targeted mail recipients to use each type of information source. The most frequently cited sources for targeted mail recipients were journals and the Internet. The least likely source for both groups was magazines, although almost two-thirds of the registered users listed that as a possible source.

  Percent who use source
Information sources Registered
users
Targeted mail
The Internet 95.1 % 75.3 %
Other government sources 91.7   68.3  
Other BJS reports 90.3   64.3  
Journals 86.5   76.4  
Newspapers 73.1   44.9  
Own research 71.4   56.7  
Magazines 64.0   42.7  
Note: Respondents could select more than one source.

up arrow To the top

Conclusions

In general, the results of the surveys were very encouraging to BJS. The customers we are directly reaching use statistics and use BJS reports. The current approach of sending bulk mailings to registered users with additional augmentation from targeted lists appears to be successful. That most recipients keep or share the documents confirms the efficacy of this approach. Because there were few differences between the registered user audiences of distinct BJS products, we conclude that we are serving those audiences about equally well and that we have few obvious gaps to fill.

Email notification appears to fall below our expectations. We need more research on the efficacy of using other types of notification including the mailing of documents like catalogs and postcards.

This survey does not tell us if we are reaching most of our intended audiences. We need to understand coverage of target audiences in registered users. We may want to convert appropriate candidates among the targeted users into registered users. In addition, we need to find other ways to assess our outreach to those audiences that are not covered well by NCJRS such as the media, secondary school students and teachers, and the general public.

Clearly the findings have implications for how our statistics are packaged. Customer attitudes did not differ by publication even though the length of the publications and topics were diverse. Most users engage in activities that involve locating a few numbers rather knowing everything about a subject. A recent study supported by BJS showed that simply transferring publications to the web fails because users approach content on the web differently than content on paper.[1] We need further research on the best methods of delivering our content.

The surveys show that the Internet is most likely place to seek information. Therefore, we need to continue to support customers in their search to discover answers online. Traditionally, we have relied on intermediaries such a librarians and the NCJRS information center to convey cautions on proper use of statistics. The use of unmediated sources like the Internet has implications for proper use and understanding of statistics. Our customers also rely on other sources such as journals and, to a lesser extent, newspapers. We need to assess how these outlets can be leveraged to alert our customers to available information.

Traditionally, we focused considerable effort into getting our materials in libraries. The customers who responded to our surveys were less likely to use libraries when seeking information. We need to examine our dissemination to libraries and determine which audiences are best served through libraries.

up arrow To the top

Methodology

The survey was conducted during the summer of 2003. The NCJRS staff conducted the phone surveys about two weeks after each publication was sent by mail. Registered users with phone numbers in the NCJRS database were randomly contacted. The lists for each publication were cross checked so that no registered user could be contacted more than once. NCJRS staff members were instructed to select another user if the phone number was not working, if they encountered a busy signal or an answering machine, or if the user was not available. The results of the phone survey were as follows:

Sent report Called Calls that
did not get
through
Refused Incomplete
Survey
Completed
Survey
Local Police Departments
    2000
17,391 2,962 2,656 60 5   300
Inmate Partner Violence
    1993-2001
19,019 2,407 2,075 66 5   300
Prison and Jail Inmates
    at Midyear 2002
24,355 2,056 1,760 73 3   300
Juvenile Felony Defendants
    in Criminal Courts
8,988 2,424 2,050 75 4   300

A mail survey was sent to targeted mail users about two weeks after the mailing. Some targeted mail users were excluded as distribution was to these groups was not considered optional. The excluded groups were Congress, Sourcebook contributors, Statistical Analysis Centers, State Uniform Crime Reports Programs, BJS VIPs (interested persons who request all publications), and certain Federal Libraries. The targeted mailing response is as follows:

  Number targeted mail recipients Number of responses Targeted response rate
Local Police Departments 2000 2,800 352 13%
Intimate Partner Victims 1993-2001 3,591 389 11%
Prison and Jail Inmates at Midyear 2002 3,242 587 18%
Juvenile Felony Defendants in Criminal Courts 6,010 258 4%


[1] Carol A. Hert, Ph.D., "Positioning Sourcebook of Criminal Justice Statistics for the 21st Century," presentation to BJS, July 8, 2004.



Back to Top