But the Padres on Tuesday asked Joyner to not make the trip to Los Angeles, where the team began a three-game series against the Dodgers.
Towers was not available for comment on Tuesday.
"It was just apparent to me I wasn't being included in everything I think I should have been included in, and there was no way I could help without knowing it. And so I was frustrated and decided that I didn't care for it," Joyner said.
Joyner, who replaced Merv Rettenmund midway through last season, said he had been thinking of stepping down recently, a point manager Bud Black confirmed. Black said that he was hopeful to the end that Joyner would change his mind.
"I couldn't talk him out of it," Black said. "It was Wally's decision. He felt like it was something he had to do.
"I think Wally is a fine hitting coach. Wally coming in like he did last year in the middle of a season after not really being a coach at any level, I thought that he did a great job of picking up Merv. It's not easy coming in like he did."
The Padres have struggled offensively this season, which is a contributing factor as to why they went into Tuesday's game with 95 losses and ranked last in the National League in runs (618) and on-base percentage (.318).
Several players, such as third baseman Kevin Kouzmanoff, took well to Joyner's teachings.
"I got to know Wally, his philosophy, his teaching and I learned a lot about hitting," said Kouzmanoff, who has already set career highs in home runs (22), doubles (31) and RBIs (82) in his second full Major League season.
"The tough thing in this business is you get to know people and all of a sudden ... things change. I was sad when I got the notice. I had no clue it was going to happen."
But it's clear that this had been weighing on Joyner's mind in recent weeks. He finally decided to make it official on Monday, against the wishes of some in the organization, like Black and many of the Padres.
Joyner, who said he would like to coach again, either as a hitting coach or as a roving instructor, said he was hoping to see his hitters one last time this week in Los Angeles, though he was able to communicate to them through text messages.
"I truly love the players, I enjoy them very much. ... Believe it or not, we had some fun times regardless of our record," he said. "I think some of the players have had incredible years, they have had improvement from last year to this year and that was fun to be a part of. We faltered in other areas that I hope these guys understand and they will improve on it."