You have to get the panning speed right but the camera software my struggle with panning such a wide angle lens.
Panning with wide angle lenses to create panoramas isn’t usually recommended, the Samyang 12mm has a fair bit of optical distortion and this is difficult for Panoramic modes to stitch properly, Given its massively wide field of view anyway I’m surprised you are finding the need to pan. You are also likely to end up with lots of foreground and sky with such a WA lens.
I would recommend a minimum panning focal length of 23mm to get a decent view and oftem 35mm+ gives better results…
I would also recommend taking several shots and stitching them together afterwards in something like lightroom, just overlap each shot by about 1/3. You can also pan with the camera in portrait mode giving you a greater vertical resolution.
8 shot pano with XF23mm in Portrait Mode, X-T1 (processed in lightroom)
160816 Salford Quays - 8 Shot Pano sm by David Yeoman, on Flickr