The only reason US-Canadian border crossings are such a mess is that the institutional culture of our border agencies went haywire from 9/11 paranoia in the early 2000s. I think we in the US started it with the "Western Hemisphere Travel Initiative" that started requiring passports or enhanced ID for land crossings. Then I believe the Canadians just got annoyed at the perceived insult of our requiring them to have passports that they returned the favor.
I did numerous crossings of the border in the 1980s and 1990s, and the Canadians were always laid back and pleasant. Then in 2005, I crossed at Windsor with only a driver's license and got chewed out by the Canadian border guard. (He did let me through, though.) Then our family crossed at Coburn Gore in 2008, and they held us up for half an hour while I was interrogated about whether or not I had ever been arrested. They let us through in the end, though, as I, indeed, have never been arrested. Finally, in 2015 we crossed at Derby Line, and while we had no problem from the Canadian border guard, he was so brusque that I think he was taking lessons from the US CBP. My wife and I did so some foot crossings at Niagara Falls in 2010, and had no problems, though.
American border guards have always been a little brusque, but they could be nice if they wanted to be. Way back in the 80s, I crossed at Coburn Gore with my then South American girlfriend whose paperwork was, unknown to us, not 100% correct. (She had permission to stay in the US through the end of the summer, but her original visa to enter the US had long expired.) They held us until they could call the woman who was the regional immigration specialist. There was some delay, because she was up at Jackman helping process passengers on VIA's Atlantic, which ran through Maine at the time. In the end, she told them to let her enter the country. The customs guys may have made us get out of our car and wait in the customs house, and held up our trip, but they were pleasant and helpful, and certainly didn't make me feel like I was some sort of Coyote smuggling in young women for who knows what sort of purpose. On the other hand, back in 2015 when we crossed back to the US at Coburn Gore, the CBP guy was dressed all tactical like he was on a SWAT team, and nearly pointed his gun at me when I misunderstood him and opened my car door when he wanted me to stay in the car. Pretty rude treatment of a citizen who helps pay your salary.
I'm not sure that there's any actual security or trade issue or political disagreement that's so pressing that we need the level of security at the Northern border that we have. It's basically inertia that's keeping them from returning to the way things were before 9/11. That, and there's no real domestic push on either side to get the political leaders of both sides to sit down and make the whole process more rational.