The fiber isomorphisms $F_{p}\cong\mathbb{T}_{v}$ depend on the value of the phase of $s$. If we go around a loop $s\mapsto se^{2\pi i}$, we won’t come back to the original diffeomorphism $H_{s}^{\mathrm{sm}}\to W^{\epsilon}$. Rather, it will be a composition with a generalized Dehn twist, namely, the diffeomorphism $W^{\epsilon}\to W^{\epsilon}$ which is the fiber wise shift by a section of $W^{\epsilon}\to\Sigma\backslash N(D)$ (the tori are abelian groups). Such a section was explicitly written down in [Zha00].