A scene association algorithm is proposed to automate the process of endoscopic surveillance. The algorithm constructs a neighborhood system for visually resemblant frames. An experimental evaluation of the algorithm has been conducted on a set of in vivo endoscopic videos.