A Follow-Up Conversation on Evaluation and Assessment

In July 2009, MPI's National Center on Immigrant Integration Policy issued a report on the assessment and evaluation methods used to build and assess language access services in social services agencies. In this conversation MPI Policy Analyst Laureen Laglagaron, the report's author, catches up with two of the administrators she interviewed for the original report to see how their agencies continue to monitor and assess their ability to provide services for limited English proficient individuals in this economic climate.

Jason Reed, Washington State Department of Social and Health Services: My name is Jason Reed and I am the limited English proficiency (LEP) Program Manager for the Economic Services Administration within the Washington State Department of Social and Health Services (DSHS). At DSHS, I develop and implement language access policies and procedures, write and oversee the interpreter/translation service contracts for the department, and train staff on the department's language access program.

Serafin "Jun" Colmenares, Hawaii Department of Labor and Industrial Relations Office of Language Access: My name is Serafin "Jun" Colmenares and I am Executive Director of the Office of Language Access (OLA) within the Department of Labor and Industrial Relations in the State of Hawaii. My office provides oversight, central coordination, and technical assistance to state agencies as they implement language access requirements. We also review and monitor each state agency's language access plan and investigate and resolve language access complaints.

Laureen Laglagaron, MPI: Jun, when we last spoke, you were in the midst of monitoring 17 state agencies and their compliance with Hawaii's language access law. Do you have results from that evaluation?

Colmenares: Yes. In November 2008, we conducted unannounced ocular inspections (site visits), administered a written questionnaire, and scheduled face-to-face meetings and site visits for 16 out of 17 state agencies. Based on these assessment tools and with the input of the participating agencies, each was given a final rating to reflect their compliance with Hawaii's language access law. The highest possible score was 52 and the lowest possible score was 0 (for failure to respond or participate in the monitoring process). The average score for state agencies was 32 and most agencies were found to be in partial compliance with the law.

The results show that most of the agencies are trying with varying degrees of effort, but that they can do better. Common shortfalls include unfamiliarity or lack of dissemination of their language access plan, lack of interpretation and translation resources, untrained bilingual staff, inability to identify or translate vital documents, data collection and reporting problems, lack of multilingual posters, lack of outreach or nonidentification of LEP communities, and failure to identify public contact positions.

Laglagaron: Washington State employs a different model for monitoring the state's DSHS field offices. Instead of one central office tasked with monitoring all state agencies, the Quality Assurance Office (QAO), reviews how each DSHS field office complies with a variety of regulations, including state and federal language access laws. Jason, when was DSHS's last review?

Reed: The most recent review was in November 2009. There are 65 field offices that are each reviewed independently. The larger offices are reviewed every 12 to 18 months and the Quality Assurance Office performs an overall assessment of management for each office. This includes assessing how cases are processed and reviewing federal requirements. The assessment also looks at services to LEP clients.

Laglagaron: Who performs the evaluation and assessment?

Reed: The Quality Assurance Office, a division of DSHS, which has approximately 8-10 headquarters staff and 45 field staff, administers our review. Typically one QAO staff person travels to the field office to conduct the review and gather case record information onsite. I supply information regarding how many cases that office handles and the different languages.

Colmenares: In Hawaii, our monitoring tool was administered by OLA staff - that includes myself, two research analysts, and our legal analyst.

Laglagaron: Who receives or acts on the results of your evaluation or assessment?

Reed: After the review the QAO issues a management evaluation and the field office is required to submit a corrective action plan, if deficiencies are found during the review. The plan responds to any issues encountered during the review process. There is generally no follow-up action or visits from the QAO. Further communication is usually carried out through written correspondence. Even though there are no formal consequences (such as loss of funding) to motivate corrective action, I have never experienced a situation where the field office failed to embark on the measures deemed necessary for improvement following a review.

Colmenares: After the monitoring process, we sent the results to the agencies. The results included any shortcomings we found and recommendations of how to improve. Only one agency failed to respond to our initiative. We also reported the results to the governor and the legislature as part of our annual report. We plan to follow up with the agencies to improve their language access services and will hopefully do another monitoring later this year. We have not yet received a response from the governor's office or the legislature.

Laglagaron: What are some typical problems in the provision of language access services that were identified as a result of your assessment and has your office taken any action to address them?

Colmenares: One specific area for improvement we identified was the dependence on bilingual or multilingual staff who appeared to lack appropriate training. While some agencies use telephonic services such as a language line to communicate with LEP clients, we noticed that many agencies primarily use staff who happen to be bilingual as interpreters and translators. After discovering the common use of bilingual staff as interpreters, we conducted trainings for bilingual staff as well as the frontline staff with experts from the University of Hawaii. We addressed how to provide services to LEP clients and included ethics for interpreters as well as cultural competency in the trainings. The two trainings reached almost 300 people.

Reed: The assessments have been helpful. In our field offices we have a system called the automatic client eligibility system that generates 200 different types of letters for clients. Every client is given a primary language code to indicate the written language in which they should receive correspondence. Letters are generated automatically in any of nine supported languages: Cambodian, Chinese, Korean, Laotian, Spanish, Russian, Somali, Vietnamese, and English. If a client does not read any of the supported languages, letters are generated in English and manually translated to the indicated language.

Through the quality assurance reviews we identified problems with language coding. By doing a fairly cursory review I am able to identify clients that appear to have been coded incorrectly. We discovered that one problem in the system itself was that workers could inadvertently change the language code. There was no prompt to ask if the worker wanted to change the code, so it went overlooked. We were able to ask for that to be corrected in the system.

Also, we have found that problems identified through management evaluations at one office are often a challenge for other offices. Therefore, I can anticipate those problems and address them during various training sessions that I provide year-round.

Laglagaron: Jun, since this is your first time administering the monitoring tool, are there some things that you would do differently in future evaluations?

Colmenares: Our monitoring initiative consisted of three components: (1) unannounced visits; (2) a questionnaire; and (3) face-to-face meetings. We plan to perform the next evaluation using a very similar structure, but we are considering using LEP individuals for the unannounced visits. We plan to reach out to community organizations and ask for their assistance with that effort. We will then help train LEP individuals to conduct the assessment. I will notify the language access coordinators at each agency that we plan to do this as part of the evaluation but they will not know the timing of the visit. We would like to use the top five languages - Ilocano, Tagalog, Japanese, Chinese (Mandarin and Cantonese), and Korean - in the unannounced visits.

Also, last year during the period of evaluation, we were conducting another survey of language access needs in the state. There were complaints and confusion about multiple questionnaires, one on language access needs and another evaluating compliance. Next time, we will not conduct simultaneous surveys.

Finally, we would like to have the support of all departments prior to conducting the evaluation. It would be ideal to have the governor issue a directive to all departments requiring them to respond.

Laglagaron: Were there any other unexpected challenges or lessons learned through the evaluation process?

Reed: I think the challenge right now is monitoring and having the resources to do it. While we would like to review more frequently, and a lot of what I do is a much more informal method of monitoring. That is the main challenge here in our agency. Especially now with budget constraints due to the recession, we don't have the staff and resources to go out as often as necessary and we know that problems exist. On the other hand, we do a good job of training our staff in these service areas. In our core training there is a short section on language access, which, in combination with what I do, helps ensure that effective and accurate services are provided to our clients.

Laglagaron: Speaking of the economic climate, how has the recession affected your programs?

Colmenares: Effective November 13, 2009, all four OLA staff were laid off and their positions were abolished as part of state cost-cutting measures. I am now the only full-time staff person. Our funding for operational expenses was also reduced. Our estimated budget, not including personnel, is about $200,000.

Fortunately, we have the help of several interns from the University of Hawaii.
We plan to conduct the next evaluation in the latter part of this year using interns for a large portion of the monitoring. At the same time, we are trying to make use of the IT staff within the department to allow some of the monitoring to be performed through electronic means.

Given our limited resources, a number of initiatives that were under consideration cannot be implemented at this time. We would have liked to offer more trainings, especially on neighboring islands, and construct a multilingual website. Also, the month of August is language access month and we originally had planned four to five events. Now we only have one event scheduled. We are, however, still planning to pursue a third language access conference. 

Reed: Washington State has faced similar budget cuts. DSHS administers a professional language certification exam that used to be available at six test sites. Budget cuts have reduced the number of language certification testing sites down to two.

In addition to budget woes, we have had to contend with problems of competing priorities as DSHS' service delivery model changes with the implementation of call centers. Since the agency's priority is moving towards the new service delivery system, adding languages to our automatic client eligibility system is no longer a high priority, even though it would pay for itself.

For example, we have approximately 100 language code values in our automatic client eligibility system and if we do not have a code for a particular language, we code it as "OT" for "other" and note the language. If a client's language is OT, any letter or notice to that client is generated in English and manually translated into the indicated language. Arabic is the next language we have scheduled to be added to the system. Although there is complexity to adding Arabic to our system, the reality is that we would recoup the original cost in a matter of months. In December 2009, we generated 613 letters in Arabic at a minimum cost of $12/letter for manual translation. That's over $7,000 whereas the cost is about $20,000 to program each additional language into our system. Clearly, it is much more cost effective to program the system to support a language than to continually translate letters manually. Given the economy, we anticipate that showing this potential cost savings in language access will likely attract more attention and resources in the future.

Laglagaron: Thank you very much for taking the time to reflect on the information you gained through your service evaluation processes. Many other language access program managers are looking for ways to monitor and improve the quality of their services and your efforts provide some very valuable lessons for how they might proceed.