But the industry has a checkered history. The AP found that at least 30 servicers have been accused in lawsuits of harassing borrowers, imposing illegal fees and charging for unnecessary insurance policies. More recently, the companies also have been criticized for not helping homeowners quickly enough delays that lead to more fees for homeowners and profits for servicers.

The biggest players in the servicing industry Bank of America, Wells Fargo & Co., JPMorgan Chase & Co. and Citigroup Inc. all face litigation, some of which has led to settlements with homeowners. All will receive federal money to modify loans.

But the industry's smaller players, which specialize in servicing riskier subprime loans and loans already in default, face harsher accusations that they systematically abused borrowers.

"The irony is, in essence, the government is paying servicers to do their job, which is to do loan modifications where appropriate," said Kurt Eggert, a law professor at Chapman University in Orange, Calif. "And that's not a part of their job they were ever especially good at."

The government says it has no choice but to partner with the servicers because they are the only link between borrowers and the investors who indirectly own their mortgages through securities. The companies acknowledge there have been abuses in their industry but argue many cases hinge on technicalities. They say borrowers facing foreclosure often sue out of desperation, trying to slow down the foreclosure process with frivolous allegations.