In a Ziplock bag combine cilantro, olive oil, lime juice, zest, garlic, chili powder, cumin, salt and pepper. Shape to combine. Add the chicken thighs and shape to coat them well with the marinade. Refrigerate for at least 2 hours to overnight.
Lightly grease the grates of an outdoor grill. Heat the grill to medium-high heat. Remove the chicken thighs from the marinade and place onto the grill. Cook turning at least once for 4-5 minutes per side or until fully cooked (the internal temperature has reached 165 degrees F.).