There is no "town side" - there is not even a town!
When you get off the Adirondack, walk the few feet to the road crossing, then walk the road downhill to the ferry dock. Total distance - less than 200 yards!
Yes, the ferry also carries just passengers.
Be aware that the ferry stops operating, and the Adirondack discontinues stopping in Port Kent, for the season in September!