Sellers are no longer able to send negative feedback for buyers, it's one of those things that sellers hate, but buyers love because they feel safer to speak their mind without receiving bad feedback in retaliation.
It's really why I suggest that you have to be comfortable with your expectations, the seller is in essence a sitting duck these days!
Seller feedback should only be considered when making a purchasing decision, it shouldn't enter into it when deciding your own feedback to leave in response to the completed transaction.
Bottom line, if you feel that your expectations are just, then base your final decision on them, and on them alone in my opinion.
As you say, with that sort of feedback I would be surprised if you couldn't get something worked out between you both to keep everyone satisfied, and you are definitely doing the right thing in trying to resolve your concerns without jumping straight in and nailing him with negative feedback.
Just tread carefully, again I go back to ensuring your expectations are just, because it can be a fine line in these types of negotiations between what is just between both parties, and what is feedback extortion!
Keep a log of all of your communication to protect yourself from that sort of claim being made against you by the seller.