"We have segments in our population that are honestly fearful about having retail in town," Livingston Mayor Shawn Klein said.
According to Klein, the state has not created regulations yet when it comes to legal marijuana and cannabis and his township is concerned.
"Some safety and security measures around the licenses," Klein said. "We don't really know if we're going to be able to, if someone's violating the rules, whether the town is going to be able to take action against them."
In North Caldwell, all six council members voted unanimously against opening cannabis businesses in town.
"We don't have a lot of area for this type of manufacturing, etc," Mayor Joseph Alessi said. "It's a bedroom community. We have several schools and two churches."
Many residents across the state are concerned cannabis businesses might be located too close to churches, schools, playgrounds, and residential neighborhoods.
Municipalities planning on opening cannabis businesses include Newark, Atlantic City, Jersey City, and Hoboken.
"It can be open as soon as this fall or early next year," said Hoboken Mayor Ravi Bhalla. "We also have our finger on the pulse of the community and residents are receptive."
Jersey City Mayor Steven Fulop said "At the end of the day, there was a public question in the November election and voters in New Jersey overwhelmingly said they wanted this."
Even if municipalities opt-out now, they still have the option of changing their minds and opting in at any point.