It's not even colleagues right? It's more like do you remember the faces of all the people that ever entered the office building you work in, you don't.
In Deep Space 9 there's an episode called "Trials and Tribble-ations" where a number of the crew from DS9 go back in time and find themselves in the TOS episode "Trouble with Tribbles". The producers literally overlayed the DS9 cast onto TOS footage, and shot some new clips in the same setting, so canonically the DS9 crew was present for the events of the TOS episode (... at least as canonically as you can get when time travel is involved).
Warf and O'Brien were two of the DS9 crew involved in that episode, and they also happen to be on the Enterprise when Scotty is found trapped in a pattern buffer.
Wouldn't Scotty assume the Klingon in Tribbles and Tribulations was Worf's grandfather?
If anything his reaction should be that Worf looks exactly like his grandfather.